[openssl-dev] [openssl.org #4651] [BUG] malloc_failure in ASN1_D2I_READ_BIO with large smime encoded file

[Brian Morton via RT]

Attempting to decrypt/decode a large smime encoded file created with
openssl fails regardless of the amount of OS memory available.

OpenSSL version 1.0.2d
Ubuntu 15.10 Linux 4.2.0-41 x86_64

Create keypair:

openssl req -x509 -nodes -newkey rsa:2048 \
    -keyout mysqldump-secure.priv.pem \
    -out mysqldump-secure.pub.pem

To create such a file on Linux:

dd if=/dev/urandom of=sample.txt bs=2G count=1

Encrypt/encode

openssl smime -encrypt -binary -text -aes256 -in sample.txt -out
sample.txt.enc -outform DER mysqldump-secure.pub.pem

Now decrypt/decode:

openssl smime -decrypt -binary -inkey mysqldump-secure.priv.pem -inform DEM
-in sample.txt.enc -out sample.txt.restored
*Error reading S/MIME message*
*140081024444064:error:07069041:memory buffer
routines:BUF_MEM_grow_clean:malloc failure:buffer.c:159:*
*140081024444064:error:0D06B041:asn1 encoding
routines:ASN1_D2I_READ_BIO:malloc failure:a_d2i_fp.c:255:*

Note that this problem occurs even on a system with 30GB+ of memory free.

Glad to take a stab at a patch if someone can point me in the right
direction as to the internal limitation.

Thank you,

Brian Morton

-- 
Ticket here: http://rt.openssl.org/Ticket/Display.html?id=4651
Please log in as guest with password guest if prompted