[openssl-users] Error in X509_check_private_key when using pkcs11 engine (OpenSSL 1.0.2j)

[Anton Gerasimov]

Greetings,

I'm trying to make OpenSSL (v. 1.0.2j) get client certificate/private
key from a virtual PKCS#11 device, namely SoftHSM. I've imported a
certificate and private key into SoftHSM, configured openssl as
following and run

    openssl s_client -engine pkcs11 -connect mydomain.com:443 -cert
'/path/to/client.pem' -certform PEM -key
'pkcs11:serial=41e0a668eb25afbc;id=%02' -keyform engine

(I've made it take the certificate from a file just to narrow down field
of possible error causes, it gives similar results reading certifivate
from the device as well).

openssl then asks for a PIN and after receiving it returns "error
setting private key" and the following error:

139798174321496:error:0B080075:x509 certificate
routines:X509_check_private_key:unknown key type:x509_cmp.c:368:

After some more debugging I've learned that X509_check_private_key()
fails because EVP_PKEY_cmp() fails which in turn fails because
eckey_pub_cmp() fails (yes, I use an EC key) which fails because "pb",
return value of EC_KEY_get0_public_key() on the second parameter is
NULL. It also looks like both compared keys have 'engine' set to NULL.

I don't quite confident with what's going on here, so I would be
grateful for any help from someone who is closer familiar with OpenSSL
codebase.

Thanks,
Anton Gerasimov