[openssl-users] RSA_method_set_sign

[Melvyn Sopacua]

Hello all,

Some background: I'd like to have a workstation that uses OpenSSL 1.1 
instead of a lower version. For that I'm porting various pieces of 
software and quickly discovered that I was repeating myself. In addition 
this teaches me more about the OpenSSL library, which I consider a great 
benefit.
This resulted in me working on a forwards-compatibility library, using 
the OpenSSL Wiki as a guide and the KDE QCA library as a testbed. Work 
in progress can be seen at [1] and [2].

However, I believe I've now hit a brick wall:
Various functions in the realm RSA_method_set_* allow us to set 
callbacks for RSA operations. However, I see no way to implement these, 
since various (all?) X509 structures are now opaque. In addition, the 
default RSA_sign implementation calls the rsa_sign callback in the 
provided RSA structure, so we'll create an infinite loop if we wrap it 
like this:

RSA_method_set_sign(meth, my_rsa_sign);
int my_rsa_sign(...) {
	RSA_sign(...);
	store_state_on_our_object();
}

This is caused by the code in [3].
That file also shows the problem: OpenSSL itself has access to X509_SIG 
(and friends) internals as demonstrated in encode_pkcs1(). But, I don't 
see any way to setup the same context(s) from outside OpenSSL. There's 
no X509_*_set_<anything> to setup the algorithm and parameters.

Am I missing something or is it simply no longer possible to implement 
these callbacks properly?


[1] https://github.com/melvyn-sopacua/qca/tree/openssl11-compat
[2] https://github.com/melvyn-sopacua/openssl-fwcompat
[3] 
<https://github.com/openssl/openssl/blob/master/crypto/rsa/rsa_sign.c#L77>
-- 
Melvyn Sopacua