Openssl seems to inspects application data?

[Kreissl, Jochen]

Okay, I added a err_clear_all before calling ssl_read and now it seems to work.
I am quite baffled.
Is there anywhere in the docs I can read up on this (when to reset the err queue)?
________________________________
From: Kreissl, Jochen <Jochen.Kreissl at vector.com>
Sent: Wednesday, March 27, 2024 7:41:44 AM
To: Neil Horman <nhorman at openssl.org>
Cc: openssl-users at openssl.org <openssl-users at openssl.org>
Subject: Re: Openssl seems to inspects application data?


I am getting it on ssl_read.



Still debugging. Right now, it seems that our custom BIO is called three times.

- First ssl reads 5 bytes (header).

- Second: we fetch some 1600-ish bytes. Still not enough for the entire record (the chain is roughly 7 kb long)

- Last: our BIO is running out of data (network packages not available yet) and returns a 0 and sets the BIO flag to retry_send.  We have this retry_send behavior in other places too and it works (e.g. during handshake with the certificate message).



But in this instance, somewhere in the internal of ssl_read, an error occurs following the return code 0.

We get a -1 from ssl_read and then call SSL_get_error – which gives us a fatal SSL_ERROR_SSL.

We then call ERR_get_error and get the beforementioned, weird error code.

I’m still trying to find the exact spot where the internals of ssl_read fail.



________________________________

From: Neil Horman <nhorman at openssl.org>
Sent: Tuesday, March 26, 2024 6:55:25 PM
To: Kreissl, Jochen <Jochen.Kreissl at vector.com>
Cc: openssl-users at openssl.org <openssl-users at openssl.org>
Subject: Re: Openssl seems to inspects application data?



Sie erhalten nicht oft eine E-Mail von nhorman at openssl.org. Erfahren Sie, warum dies wichtig ist<https://aka.ms/LearnAboutSenderIdentification>

What library call are you getting that error in response to?  If you believe that this is coming from some attempt to interpret application data (which you are correct, it shouldn't be, unless the application auth protocol is somehow getting aliased as a tls control message of some sort), then I would, after the handshake, clear the error stack, and check it after a call from SSL_read returns.



On Tue, Mar 26, 2024 at 1:38 PM Kreissl, Jochen <Jochen.Kreissl at vector.com<mailto:Jochen.Kreissl at vector.com>> wrote:

Hi,



I am using openssl (3.2) in an application.

Handshake works just fine but I get a very weird behavior when I receive a big certificate chain inside application data (TLS 1.3 but NOT using Post-Handshake Auth, this is some level-7 auth protocol on top of tls).

The openssl error I get is error:0308010C:digital envelope routines::unsupported

Which … seems to indicate that openssl is trying (and failing) to interpret the certificate chain…?



I really don’t understand what is going on.

I thought openssl would treat any application data sent using SSL_write following a completed handshake would be opaque for openssl – because why would it look inside and try to parse something?



Does anyone have an explanation or have encountered something similar?





Regards

Jochen


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mta.openssl.org/pipermail/openssl-users/attachments/20240327/fc2a5f95/attachment-0001.htm>