Troubles using Openssl ENGINE

[Gael GUEGAN]

Hello all,

I am currently having some trouble using an openssl engine with nginx.

I was having no problems using it for loading private key through my engine.
However after adding new capabilities about symmetric encryption (AES) to the engine, nginx is trying to use my engine instead of the default openssl implementation at some point.

And so the handshake is failing, trying to use the symmetric encryption of my engine that I don't want him to use, here a debug log :

2019/04/09 09:34:37 [debug] 9414#0: epoll timer: 59601
2019/04/09 09:34:37 [debug] 9414#0: epoll: fd:3 ev:0001 d:B6973109
2019/04/09 09:34:37 [debug] 9414#0: *3 SSL handshake handler: 0
Init Cipher Key ... (Debug Log from the engine code)
Cleaning up ... (Debug Log from the engine code)
2019/04/09 09:34:37 [debug] 9414#0: *3 SSL_do_handshake: -1
2019/04/09 09:34:37 [debug] 9414#0: *3 SSL_get_error: 1
2019/04/09 09:34:37 [crit] 9414#0: *3 SSL_do_handshake() failed (SSL: error:8009D064:tpm2-tss-engine:tpm2_cipher_init_key:Failed to read TPM2 data) while SSL handshaking, client: 192.168.13
2019/04/09 09:34:37 [debug] 9414#0: *3 close http connection: 3
2019/04/09 09:34:37 [debug] 9414#0: *3 event timer del: 3: 24375741
2019/04/09 09:34:37 [debug] 9414#0: *3 reusable connection: 0

My idea was to disable the symmetric functionality of the engine. And I have attempted to modify the file ngx_event_openssl.c by calling the function ENGINE_unregister_ciphers(...) or ENGINE_set_default(engine, ENGINE_METHOD_PKEY_METHS) or configuring the openssl.cnf with only RSA algo.
I have succeeded to do it in a small c code of mine, but in nginx it is like some function are resetting my configuration like SSL_CTX_new().

Is someone has an idea on how to resolve my problems ? I would highly appreciate some help.

Other information :
~$ sudo /usr/sbin/nginx -V
nginx version: nginx/1.12.1
built with OpenSSL 1.1.0h  27 Mar 2018
TLS SNI support enabled
configure arguments: --crossbuild=Linux:arm --with-endian=big --with-int=4 --with-long=4 --with-long-long=8 --with-ptr-size=4 --with-sig-atomic-t=4 --with-size-t=4 --with-off-t=4 --with-time-t=4 --with-sg

Here a link to the engine : https://github.com/tpm2-software/tpm2-tss-engine




Gael GUEGAN

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mta.openssl.org/pipermail/openssl-users/attachments/20190410/ab148aec/attachment-0001.html>