How to not use a configured engine?

[Lynch, Andrew]

Hi,

I support two bespoke engines which have been in use with OpenSSL 1.0.2.  Due to a new requirement for RSA-PSS keys we are in the process of migrating to 1.1.1a.  Implementing various minor API changes was no big deal and the engines still work as expected.  However the behaviour of openssl has changed when trying to _not_ use the engine.

There are several environments, each with OPENSSL_CONF pointing to a configuration file with an engines section for one of the two engines.  This specifies a number of PRE commands to configure the engine.  Mostly requests are processed with "openssl req" or "openssl x509" using HSM keys with -keyform ENGINE and the appropriate -engine option, however there are also some use cases with local file-based keys in the same environment.

Take the simplified example of creating a signature with "openssl dgst -sha256 -sign mykey_priv.pem -out foo.sig foo.txt"

There is no -engine option given, but the active configuration file does include an engines section.

Using OpenSSL 1.0.2 the signature is created using the given key.  Debug output from the engine shows that its init, finish and destroy functions have been called, but the sign operation does not go through the engine.  (Although ENGINE_set_RSA must also have been called?)

The same command line with OpenSSL 1.1.1a fails in the engine's rsa_sign method because of some missing setup that happens in the load_privkey method (which has not been called as no -engine or -keyform ENGINE option were provided).

Is this an intended change of behaviour?  If yes, how can I prevent 1.1.1a from using the engine's RSA method without having to change the configuration file?  Our current workaround is to repoint OPENSSL_CONF to a duplicate of the file in which the line "engines = engine_section" has been commented out.  Then the engine is not referenced at all.  As the configuration files contain a large number of other settings managing two almost identical copies is not desirable.


The implementation of our engines closely follows those included in the source distribution (e.g. e_capi.c).  bind_helper calls bind_enginename which registers all the functions and methods via ENGINE_set_init_function etc. including ENGINE_set_RSA and ENGINE_set_EC.

It appears that OpenSSL calls the bind_helper for every engine that appears in the configuration file.  If this includes two engines then always both init functions are called.  With an -engine option on the command line only the specified engine's method is used for the actual operation.  With no -engine option OpenSSL 1.0.2 uses its internal software method whereas OpenSSL 1.1.1a tries to use whatever engine happens to have been registered first (appears first in [engine_section]).

Assuming our engines' init function is always called, where is the right place to do any stuff that should only happen if that particular engine is actually set via the -engine option?

Regards,
Andrew.