Program works with older libssl, but not with newer

[Jan Just Keijser]

Hi,

On 31/03/20 19:42, Viktor Dukhovni wrote:
> On Tue, Mar 31, 2020 at 04:51:32PM +0200, Christoph Pleger wrote:
>
>>>> I have here a self-written server program and the corresponding
>>>> self-written client program. These run well together with libssl 1.1.0l,
>>>> but with libssl 1.1.1d, the same programs give errors SSL_ERROR_SYSCALL
>>>> in SSL_read(), no matter if I recompile the programs and then run them,
>>>> or just replace libssl with the newer version.
>>> OpenSSL 1.1.1 supports TLS 1.3, which OpenSSL 1.1.0 did not.
>>>
>>>> So, I want to ask if there are any known incompabilities in the libssl
>>>> versions that require me to change the code of the programs, or if there
>>>> is
>>>> any known bug in libssl1.1.1d that may cause the mentioned errors.
>>> Use of TLS 1.3 changes the communication patterns of the TLS protocol in
>>> some non-trivial ways, and, if your application were fragile, it might
>>> have gotten by with TLS 1.2, but the latent bugs could show up with TLS
>>> 1.3.
>> Now, I replaced TLS_server_method() and TLS_client_method() with
>> TLSv1_2_server_method() and TLSv1_2_client_method() respectively, and the same
>> error occurs.
> Well, in that case, you need to provide more detail.  Does the handshake
> complete?  If not, at what stage does it fail?
>
> A PCAP file may be needed.  And you need to explain what operation
> fails with SSL_ERROR_SYSCALL, and do an "strace" or equivalent to
> understand what the relevant socket read calls returned.
>
on a related note: I am experiencing similar problems with my ppp 
EAP-TLS patch ; I now that EAP-TLS + TLSv1.3 is experimental but I do 
notice that the entire handshake seems to differ for TLSv1.3 versus 
TLSv1.2 ; as a workaround I am adding
   SSL_CTX_set_max_proto_version(ctx, TLS1_2_VERSION);
to my code to avoid a TLSv1.3 handshake. I am not sure yet why and where 
it is failing, but it seems the client is not sending its certificate 
chain to the server.  I am positive it is a programming error on my side 
but I will say that this problem is particularly hard to track down.


JM2CW,

JJK / Jan Just Keijser