[openssl-users] RSA_method_set_sign

[Dr. Stephen Henson]

On Sat, Jan 14, 2017, Melvyn Sopacua wrote:

> Hello all,
> 
> Some background: I'd like to have a workstation that uses OpenSSL 1.1 
> instead of a lower version. For that I'm porting various pieces of 
> software and quickly discovered that I was repeating myself. In addition 
> this teaches me more about the OpenSSL library, which I consider a great 
> benefit.
> This resulted in me working on a forwards-compatibility library, using 
> the OpenSSL Wiki as a guide and the KDE QCA library as a testbed. Work 
> in progress can be seen at [1] and [2].
> 
> However, I believe I've now hit a brick wall:
> Various functions in the realm RSA_method_set_* allow us to set 
> callbacks for RSA operations. However, I see no way to implement these, 
> since various (all?) X509 structures are now opaque. In addition, the 
> default RSA_sign implementation calls the rsa_sign callback in the 
> provided RSA structure, so we'll create an infinite loop if we wrap it 
> like this:
> 
> RSA_method_set_sign(meth, my_rsa_sign);
> int my_rsa_sign(...) {
> 	RSA_sign(...);
> 	store_state_on_our_object();
> }
> 
> This is caused by the code in [3].
> That file also shows the problem: OpenSSL itself has access to X509_SIG 
> (and friends) internals as demonstrated in encode_pkcs1(). But, I don't 
> see any way to setup the same context(s) from outside OpenSSL. There's 
> no X509_*_set_<anything> to setup the algorithm and parameters.
> 
> Am I missing something or is it simply no longer possible to implement 
> these callbacks properly?
> 

Can you give a pointer to the part that is causing problems?

The rsa_sign interface is used where the only interface available is passed
the digest algorithm and the raw digest and it performs its own formatting
using DigestInfo etc.

If you don't want to do that then the rsa_priv_enc method is more appropriate:
it gets passed the block to encrypt (sign) and all the DigestInfo formatting
is handled by OpenSSL itself.

If you really need to it should be possible to set up or examine an X509_SIG
structure using the available APIs. For example to retieve its fields you use
X509_SIG_get0 and to set them X509_SIG_getm.

The contained X509_ALGOR can be set up using X509_ALGOR_set0 and examined with
X509_ALGOR_get0.

Steve.
--
Dr Stephen N. Henson. OpenSSL project core developer.
Commercial tech support now available see: http://www.openssl.org