[openssl-users] Unable to decrypt CMS object encrypted with EC prime256v1 certificate

[Dr. Stephen Henson]

On Fri, Jul 01, 2016, Stephan M?hlstrasser wrote:

> Hi,
> 
> we are testing OpenSSL interoperability with a third-party
> application, and we cannot decrypt a CMS object that is encrypted by
> the third-party application with a prime256v1 elliptic-curve
> certificate.
> 
> I have attached the following files:
> 
> demo_signer_ec_secp256r1.cms.der: DER-encoded CMS object
> demo_signer_ec_secp256r1.cert.pem: recipient certificate
> demo_signer_ec_secp256r1.pkey.pem. recipient private key (no password)
> 
> I try to decrypt the CMS object with the following command using
> OpenSSL 1.0.2:
> 
> $ openssl version
> OpenSSL 1.0.2h  3 May 2016
> $ openssl cms -decrypt -in demo_signer_ec_secp256r1.cms.der  -inform
> DER -recip  demo_signer_ec_secp256r1.cert.pem  -inkey
> demo_signer_ec_secp256r1.pkey.pem
> Error decrypting CMS using private key
> 
> When I use OpenSSL 1.1.0 beta from today's HEAD of the master
> branch, I see an additional error message:
> 
> $ openssl version
> OpenSSL 1.1.0-pre6-dev  xx XXX xxxx
> $ openssl cms -decrypt -in demo_signer_ec_secp256r1.cms.der  -inform
> DER -recip  demo_signer_ec_secp256r1.cert.pem  -inkey
> demo_signer_ec_secp256r1.pkey.pem
> Error decrypting CMS using private key
> 140735294530304:error:0D06E0A4:asn1 encoding
> routines:asn1_do_adb:unsupported any defined by
> type:crypto/asn1/tasn_utl.c:238:
> 

The final error is bogus: fixed in current master.

> Is the CMS object broken, or is this a problem in OpenSSL?
> 

Well the OpenSSL version does interop OK with the Bouncy Castle version of
ECDH and CMS. I've checked through your test message and the problem is that
the AES unwrapping algorithm checks fail meaning it can't proceed any further.
That could be down to a CMS problem, an ECDH issue or a problem with the wrap
algorithm either in the version you are testing or OpenSSL.

Is it possible to get any debugging information from the other version you are
using: for example the content encryption key it is expecting or the ECDH
shared secret?

Have you tried generating an message with OpenSSL and decrypting it with the
other version?

Steve.
--
Dr Stephen N. Henson. OpenSSL project core developer.
Commercial tech support now available see: http://www.openssl.org