libcrypto failure on Openssh

[Michael Wojcik]

> From: openssl-users <openssl-users-bounces at openssl.org> On Behalf Of
> Michael Richardson
> Sent: Monday, 27 February, 2023 09:00
> 
> Hareesh Das Ulleri <hareesh.ulleri at ovt.com> wrote:
>     > When I tried to connect via OpenSSH it fails because of libcrypto
>     > error. In debug it found, it fails when an OpenSSH unprivileged child
>     > task calls a EVP_CipherInit function in OpenSSL and this calls my
>     > provider, which tries to open my device file which eventually fails.
> 
>     > My provider handles the file open operations in
>     > OSSL_FUNC_CIPHER_NEWCTX, Not sure this is the right approach for
>     > openssl provider or not !
> 
>     > Anyone knows or tried this scenario before ? What is the right approach
>     > of OpenSSL provider functions to handle file access in this scenario?
> 
> Could the open file have been marked to close upon fork()?
> (I don't know of a way to do that, but there are lots of new fangled
> permissions in Linux)

Or just standard UNIX close-on-exec (FD_CLOEXEC), depending on what OP means by "unprivileged child task".

> More likely, the OpenSSH child is closing all unneeded file descriptors, and
> you need to find a way to avoid having yours closed.  Or you need to open
> the FD again.

Yes. OP wrote "tries to open my device file", but from the problem description it seems more accurate to say that the "child task" is inheriting an EVP_CIPHER_CTX and passing that to EVP_CipherInit, and the descriptor opened by the provider under EVP_CIPHER_CTX_new has been closed. Reopening the device file as necessary seems like the better option.

In fact, it's a good idea to check that the descriptor still refers to your device anyway, e.g. by using fstat and checking the device and inode numbers match values you memoized when you opened it. It's not safe for libraries to assume descriptors they have allocated remain open unless that's part of the API contract (and it isn't here, because OpenSSL providers are transparent to callers), and because UNIX requires reusing the lowest available descriptor, it may well have been closed and reopened as something else.

-- 
Michael Wojcik