[openssl-dev] EC based certificates not supported in CMS - why?

[Dr. Stephen Henson]

On Thu, Apr 09, 2015, Pawe?? Ka??mierczak wrote:

> I am affraid EC certs do not work in CMS openSSL 1.0.2. I just wrote a
> simple test procedure:
> 
> void cmsTest()
> {
>   //this RSA works
>   //auto certFileBio = BIO_new_file("c:\\a\\simplersa_noPem.cer", "rb");
>   //auto prvKeyFileBio = BIO_new_file("c:\\a\\simplersa_pkey.openssl",
> "rb");
> 
>   //this EC not
>   auto certFileBio = BIO_new_file("c:\\a\\advancedbr256r1_noPem.cer", "rb");
>   auto prvKeyFileBio = BIO_new_file("c:\\a\\advancedbr256r1_pkey.pkcs8",
> "rb");
> 
>   auto evpPkey = d2i_PrivateKey_bio(prvKeyFileBio, 0);
>   auto cert = d2i_X509_bio(certFileBio, 0);
>   stack_st_X509* certStack = sk_X509_new_null();
>   sk_X509_push(certStack, cert);
>   X509_STORE* store = X509_STORE_new();
>   X509_STORE_add_cert(store, cert);
> 
>   //sign
>   auto inFileBio = BIO_new_file("c:\\tmp\\0_inContent.txt", "rb");
>   CMS_ContentInfo *cms = CMS_sign(cert, evpPkey, 0, inFileBio, 0);
>   auto cmsOutFileBio = BIO_new_file("c:\\tmp\\1_signedCms.txt", "wb");
>   auto res = PEM_write_bio_CMS_stream(cmsOutFileBio, cms, 0, 0);
>   BIO_free(inFileBio);
>   BIO_free(cmsOutFileBio);
> 
>   //encrypt
>   inFileBio = BIO_new_file("c:\\tmp\\1_signedCms.txt", "rb");
>   cms = CMS_encrypt(certStack, inFileBio, EVP_aes_128_cbc(), 0);
>   auto ecnryptedCmsOutFileBio =
> BIO_new_file("c:\\tmp\\2_encryptedSignedCmsOut.txt", "wb");
>   res = PEM_write_bio_CMS_stream(ecnryptedCmsOutFileBio, cms, 0, 0);
>   BIO_free(inFileBio);
>   BIO_free(ecnryptedCmsOutFileBio);
> 
>   //decrypt
>   inFileBio = BIO_new_file("c:\\tmp\\2_encryptedSignedCmsOut.txt", "rb");
>   cms = PEM_read_bio_CMS(inFileBio, 0, 0, 0);
>   auto decryptedCmsOutFileBio =
> BIO_new_file("c:\\tmp\\3_decryptedSignedCmsOut.txt", "wb");
>   res = CMS_decrypt(cms, evpPkey, cert, 0, decryptedCmsOutFileBio, 0); //
> ERROR HERE **************************************************************
>   BIO_free(decryptedCmsOutFileBio);
>   BIO_free(inFileBio);
> 
>   //verify/read content CMS
>   inFileBio = BIO_new_file("c:\\tmp\\3_decryptedSignedCmsOut.txt", "rb");
>   cms = PEM_read_bio_CMS(inFileBio, 0, 0, 0);
>   auto decodedCmsOutFileBio = BIO_new_file("c:\\tmp\\4_inContext.txt",
> "wb");
>   res = CMS_verify(cms, certStack, store, 0, decodedCmsOutFileBio, 0);
>   auto signers = CMS_get0_signers(cms);
>   BIO_free(inFileBio);
>   BIO_free(decodedCmsOutFileBio);
> 
>   //deinit
>   EVP_PKEY_free(evpPkey);
>   sk_X509_free(certStack);
>   X509_STORE_free(store);
>   BIO_free(certFileBio);
>   BIO_free(prvKeyFileBio);
> }
> 
> and it works perfectly if RSA certificate is used but It fails during
> decrypt if I use the brainpool based certificates.
> The error occurs in cms_env.c, cms_env_asn1_ctrl function
> 
> int cms_env_asn1_ctrl(CMS_RecipientInfo *ri, int cmd)
> {
>     EVP_PKEY *pkey;
>     int i;
>     if (ri->type == CMS_RECIPINFO_TRANS)
>         pkey = ri->d.ktri->pkey;
>     else if (ri->type == CMS_RECIPINFO_AGREE) {
>         EVP_PKEY_CTX *pctx = ri->d.kari->pctx;
>         if (!pctx)
>             return 0;
>         pkey = EVP_PKEY_CTX_get0_pkey(pctx);
>         if (!pkey)
>             return 0;
>     } else
>         return 0;
>     if (!pkey->ameth || !pkey->ameth->pkey_ctrl)
>         return 1;
>     i = pkey->ameth->pkey_ctrl(pkey, ASN1_PKEY_CTRL_CMS_ENVELOPE, cmd, ri);
> // this returns 0 *********************
>     if (i == -2) {
>         CMSerr(CMS_F_CMS_ENV_ASN1_CTRL,
>                CMS_R_NOT_SUPPORTED_FOR_THIS_KEY_TYPE);
>         return 0;
>     }
>     if (i <= 0) {
>         CMSerr(CMS_F_CMS_ENV_ASN1_CTRL, CMS_R_CTRL_FAILURE);
>         return 0;
>     }
>     return 1;
> }
> 
> the i = pkey->ameth->pkey_ctrl call returns 0 and error
> CMSerr(CMS_F_CMS_ENV_ASN1_CTRL, CMS_R_CTRL_FAILURE is set.
> 

The standard OpenSSL CMS tests in 1.0.2 include an ECDH test but using P-256.

Does the cms utility do the same?

Hmm... that might be something related to the use of brainPool: I'll check.

Steve.
--
Dr Stephen N. Henson. OpenSSL project core developer.
Commercial tech support now available see: http://www.openssl.org