[openssl-users] libssl.so.1.0.0 -> Java1.6 net.ssl gives: dh key too small:s3_clnt.c:3617:

[Jakob Bohm]

On 25/08/2016 16:21, Matthias Apitz wrote:
> Hello,
>
> We have a C written OpenSSL application which talks to a server written
> in Java1.6. The client side (i.e. OpenSSL) rejects connecting with the
> error:
>
> 25.08.2016-10:58:06 Error - SSL_connect() returned:<-1> - connection failed
> 25.08.2016-10:58:06 SSL_get_error() returned SSL_ERROR_SSL, ERR_print_errors_fp():
> 4087322300:error:14082174:SSL routines:ssl3_check_cert_and_algorithm:dh key too small:s3_clnt.c:3617:
>
> I read in Don Google that is due to a stronger check in OpenSSL since
> somewhere in September 2015. The problem is of course with the old Java 1.6
> server and does not show up when we talk to a newer version of our
> server runninng on Java1.8. It works also with 1.6 when I use on the C
> side some older shared lib libssl.so.1.0.0 from Januar 2015, i.e. it
> seems exactly the bug as described in
> https://groups.google.com/forum/#!topic/ganeti/ds0TwfroS8A :
>
> The used keystore is generated with the Java keytool. It does not help
> generate the keystore with Java1.8 keytool and use this in the Java1.6
> server.
>
> Is there some workaround?
>
According to the information published by Qualsys, Java 1.6
supports DH group parameters (primes) up to 1024 bit.

OpenSSL and other modern clients rejects DH group parameters
smaller than 768 or 1024 bits.

So configuring the Java 1.6 server to use a (brand new, not
shared with other sites) 1024 bit prime in its DH group
parameters should be the best available compromise.

Upgrading to a current version of Java would allow using a
more secure prime size such as 2048 bits or more, but could
involve incompatibilities in other parts of Java.

I don't know how to tell the SSL/TLS code in Java 1.6 to use
specific DH group parameters, or to generate new such
parameters on a regular basis (it's a somewhat slow task, so
should be done in background while incoming connections use
the previous group parameters stored in an on-disk cache of
some kind).

Enjoy

Jakob
-- 
Jakob Bohm, CIO, Partner, WiseMo A/S.  https://www.wisemo.com
Transformervej 29, 2860 Søborg, Denmark.  Direct +45 31 13 16 10
This public discussion message is non-binding and may contain errors.
WiseMo - Remote Service Management for PCs, Phones and Embedded