CNG engine on GitHub

[Matt Caswell]

On 02/07/2021 16:18, Reinier Torenbeek wrote:
> It is not clear to me what you mean with "the OpenSSL engine interface 
> does allow using EVP_PKEY_METHOD callbacks instead of rsa_priv_dec 
> etc.". Can you elaborate (here or on the GitHub issue)?

You can hook the RSA calls at different abstraction levels. You can 
provide a custom RSA_METHOD in an enigne, which means calls to the 
various RSA_*() functions go via the custom RSA_METHOD.

However these RSA_*() functions are considered the "low level" 
functions, and it is preferred that applications use the "high level" 
EVP API instead (in fact the "low level" functions are all deprecated in 
3.0). RSA PSS padding is *only* available via the EVP API.

Algorithm specific EVP functionality is implemented via an 
EVP_PKEY_METHOD. The built-in RSA EVP_PKEY_METHOD mostly just calls the 
low level RSA_*() functions in the right places. However it implements 
PSS padding directly. Since padding has already been added via the 
RSA_PKEY_METHOD, when it actually calls the lower level RSA_*() 
functions it does so with RSA_NO_PADDING. So this means that the 
RSA_METHOD has no opportunity to influence the PSS padding.

However, an alternative is to implement a custom EVP_PKEY_METHOD. By 
doing this you get the opportunity to hook the PSS padding.

Not sure I explained that too well. I hope it makes sense.

Matt