[openssl-users] Understanding RSA_sign and type argument

[Ignacio Alamo Corsino]

Hello everyone,


i am having some issues understanding the RSA_sign function:

RSA_sign(int type, const unsigned char *m, unsigned int m_len, unsigned char *sigret, unsigned int *siglen, RSA *rsa);


As far as I know, the signing is a four step process:

- Calculate hash with digest algorithm <type> (given as argument to this function -> m)

- Encapsulate hash in a DigestInfo structure (X509_SIG)

- Structure padding (in RSA_private_encrypt)

- Private key operation on this padded structure (in RSA_private_encrypt)


Is that correct?


So, during the TLS handshake, the RSA_sign function is called in the CertificateVerify step.

For my tests, everytime this function is called, the hashing type is SHA512 even though I specify to use a SHA256 hash.

These are the commands that I use to test TLS:

#openssl s_server -accept 443 -cert cert.pem -key key.pem  -Verify 1 -msg -debug -cipher eNULL:aRSA:!SHA512:SHA256 -serverpref
#openssl s_client -connect localhost:443 -cert client_cert.pem   -key client.key -state -cipher eNULL:aRSA:!SHA512:SHA256

How can I force TLS to use a SHA256 digest for DH?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mta.openssl.org/pipermail/openssl-users/attachments/20170609/a474ab09/attachment.html>