Openssl seems to inspects application data?

[Viktor Dukhovni]

On Tue, Mar 26, 2024 at 01:55:25PM -0400, Neil Horman wrote:

> What library call are you getting that error in response to?  If you
> believe that this is coming from some attempt to interpret application data
> (which you are correct, it shouldn't be, unless the application auth
> protocol is somehow getting aliased as a tls control message of some sort),
> then I would, after the handshake, clear the error stack, and check it
> after a call from SSL_read returns.

I strongly support Neil's conjecture that is likely a "left-over"
element on the error stack, rather than an error condition in
SSL_read(3).  We can be absolutely certain that SSL_read(3) is not, in
fact, interpreting application data.

Also, errors on the stack are generally augemented with additional
detail text that was not reported.  It would be betst to include the
detailed reason in the error logging.

The error in question (EVR_LIB_EVP, ERR_R_UNSUPPORTED) appears be
largely confined to the low-level parameter parsing code:

    https://github.com/openssl/openssl/blob/1967539e212c17139dc810096da987c8100b1ba2/crypto/evp/ctrl_params_translate.c#L413-L414
    https://github.com/openssl/openssl/blob/1967539e212c17139dc810096da987c8100b1ba2/crypto/evp/ctrl_params_translate.c#L479-L482
    https://github.com/openssl/openssl/blob/1967539e212c17139dc810096da987c8100b1ba2/crypto/evp/ctrl_params_translate.c#L643-L647
    https://github.com/openssl/openssl/blob/1967539e212c17139dc810096da987c8100b1ba2/crypto/evp/ctrl_params_translate.c#L689-L693

All in the same default_fixup_args() function.  Call to this would not
normally be expected in the middle of application data reads, as all the
relevant ciphers, ... should already be in hand.  But the application
code parsing the user-level auth messages could well end up pushing such
errors onto the application error stack, which should be reset prior to
each SSL_read() to avoid logging spurious diagnostic messages.

-- 
    Viktor.