[openssl-dev] [openssl.org #4429] Cannot decrypt RC4-encrypted CMS object

[Viktor Dukhovni]

On Tue, Mar 15, 2016 at 06:33:32AM +0000, Viktor Dukhovni wrote:

> This is completely untested, may not even compile!  Enjoy.

It does seem to work, so one key remaining questions is whether it
is interoperable:

    $ ./util/shlib_wrap.sh ./apps/openssl req -config apps/openssl.cnf -new -x509 -newkey rsa:2048 -keyout key.pem -nodes -out cert.pem -days 100 -subj "/CN=RC4 CMS Test"
    <copious output>
    $ ./util/shlib_wrap.sh ./apps/openssl x509 -in cert.pem -noout -serial
    serial=ACD5DEDE758B9AA6
    $ echo sesame > data.txt
    $ ./util/shlib_wrap.sh ./apps/openssl cms -rc4 -encrypt -binary -in data.txt -out data.txt.cms -outform DER cert.pem
    $ ./util/shlib_wrap.sh ./apps/openssl cms -decrypt -in data.txt.cms -inform DER -out data2.txt -inkey key.pem -recip cert.pem
    $ diff -u data.txt data2.txt
    <no output>
    $ openssl asn1parse -inform DER -in data.txt.cms
	0:d=0  hl=4 l= 380 cons: SEQUENCE
	4:d=1  hl=2 l=   9 prim: OBJECT            :pkcs7-envelopedData
       15:d=1  hl=4 l= 365 cons: cont [ 0 ]
       19:d=2  hl=4 l= 361 cons: SEQUENCE
       23:d=3  hl=2 l=   1 prim: INTEGER           :00
       26:d=3  hl=4 l= 320 cons: SET
       30:d=4  hl=4 l= 316 cons: SEQUENCE
       34:d=5  hl=2 l=   1 prim: INTEGER           :00
       37:d=5  hl=2 l=  36 cons: SEQUENCE
       39:d=6  hl=2 l=  23 cons: SEQUENCE
       41:d=7  hl=2 l=  21 cons: SET
       43:d=8  hl=2 l=  19 cons: SEQUENCE
       45:d=9  hl=2 l=   3 prim: OBJECT            :commonName
       50:d=9  hl=2 l=  12 prim: UTF8STRING        :RC4 CMS Test
       64:d=6  hl=2 l=   9 prim: INTEGER           :ACD5DEDE758B9AA6
       75:d=5  hl=2 l=  13 cons: SEQUENCE
       77:d=6  hl=2 l=   9 prim: OBJECT            :rsaEncryption
       88:d=6  hl=2 l=   0 prim: NULL
       90:d=5  hl=4 l= 256 prim: OCTET STRING      [HEX DUMP]:70BD8B31ACD24F8184A54AF52446D10898DC09E4636456B8E14B3073701CAD5226C0AA03C0AD45B7056DB0A10F01487DC4DE0D35FDE7291875D665DEBB76049C6D660C885A011949A051874DF0CCEA181F9D60BC6BB8BD989B69900E917CCE170F60A34DC77A0EEFB935E13578F3AC9703AE02D972F853DBB3302BEB28F1F8E54964E7528E9E24EEA6950535EF2D1027C31CCAEB1FAB8F454ADBEB1DB9FD2A0F61F276498E64931483FDD40E90DD956BF991C3524C9EDA70211A256BEEFED941474B26ED7A4516873A12240C505813B6BD6EDFE6ED367FEAC86AEC2602A8E1C0C5ACE9C2745FA1B6702F1550FD1ECE322CD7F165DA621E984F1186CA981829AE
      350:d=3  hl=2 l=  32 cons: SEQUENCE
      352:d=4  hl=2 l=   9 prim: OBJECT            :pkcs7-data
      363:d=4  hl=2 l=  10 cons: SEQUENCE
      365:d=5  hl=2 l=   8 prim: OBJECT            :rc4
      375:d=4  hl=2 l=   7 prim: cont [ 0 ]
    $ tail -c8 data.txt.cms | od -tx1
    0000000   07  c3  e2  69  a0  ab  3b  ec
    0000010

That said, stream ciphers with unsigned CMS are especially unsafe.
Since the payload has no MAC or padding of any kind, it is trivial
to XOR any desired mask into the received plaintext:

    $ < data.txt.cms perl -e '
	($a, $b) = map { unpack("Q", "0$_\n") } qw(sesame unsafe);
	$/ = undef; $cms = <STDIN>;
	substr($cms, -8) = pack("Q", unpack("Q", substr($cms, -8)) ^ $a ^ $b);
	print $cms' > data.txt.cms2
    $ ./util/shlib_wrap.sh ./apps/openssl cms -decrypt -in data.txt.cms2 -inform DER -out data3.txt -inkey key.pem -recip cert.pem
    $ cat data3.txt
    unsafe

In the above example, a ciphertext-only transformation changes
'sesame' to 'unsafe'.  That, plus RC4's biases, make it unwise in
this context.  At the very least the CMS message MUST be signed,
and the first 256 bytes should not contain sensitive and yet
frequently transmitted content.

Don't let your children play with RC4 in CMS.

Of course, unsigned CMS payloads are also vulnerable to silent
corruption even with block ciphers in CBC mode, XOR of a mask into
a ciphertext block randomizes the plaintext of that block, but
makes a predictable change in the plaintext of the next block.

So, don't expect data integrity from unsigned CMS.

-- 
	Viktor.