[openssl-users] openssl cms -decrypt failing due to malloc(3) failure

[Christian Böhme]

Hello all,

Assume that we have

$ uname -srvmpio
Linux 4.4.0-109-generic #132-Ubuntu SMP Tue Jan 9 19:52:39 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

$ openssl version
OpenSSL 1.0.2g  1 Mar 2016

$ printenv SHELL
/bin/bash

$ ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 63575
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 63575
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

$ dd if=/dev/zero of=plaintext.in bs=1024 count=$((1024 * 1024 * 2))
2097152+0 records in
2097152+0 records out
2147483648 bytes (2.1 GB, 2.0 GiB) copied, 7.7645 s, 277 MB/s

$ echo -n 'uno dos tres cuatro' | openssl cms -encrypt -aes-256-cbc -pwri_password fd:0 -in plaintext.in -binary -out ciphertext.der -outform DER

$ ls -lAF
total 4054804
-rw-rw-r-- 1 ubuntu ubuntu 2004623580 Jul 28 20:09 ciphertext.der
-rw-rw-r-- 1 ubuntu ubuntu 2147483648 Jul 28 19:55 plaintext.in

then we get

$ openssl asn1parse -in ciphertext.der -inform DER -i
140507999028888:error:07064041:memory buffer routines:BUF_MEM_grow:malloc failure:buffer.c:113:

or

$ echo -n 'uno dos tres cuatro' | openssl cms -decrypt -pwri_password fd:0 -in ciphertext.der -inform DER -out plaintext.out
Error reading S/MIME message
139871963694744:error:07069041:memory buffer routines:BUF_MEM_grow_clean:malloc failure:buffer.c:150:
139871963694744:error:0D06B041:asn1 encoding routines:ASN1_D2I_READ_BIO:malloc failure:a_d2i_fp.c:239:

It would appear that both commands fail due to them being unable to
allocate more memory to slurp the rest of the input file's contents into.
Is this intentional behaviour?

Both commands work when the plaintext file is half the size, i.e. 1 GiB, BTW.


Thanks,
Christian

-- 
*Christian Böhme*

Developer System Integration

CLOUD&HEAT

*CLOUD & HEAT Technologies GmbH*
Königsbrücker Str. 96 (Halle 15) | 01099 Dresden
Tel: +49 351 479 3670 - 100
Fax: +49 351 479 3670 - 110
E-Mail: christian.boehme at cloudandheat.com <mailto:christian.boehme at cloudandheat.com>
Web: https://www.cloudandheat.com <https://www.cloudandheat.com>

Handelsregister: Amtsgericht Dresden
Registernummer: HRB 30549
USt.-Ident.-Nr.: DE281093504
Geschäftsführer: Nicolas Röhrs


-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 533 bytes
Desc: OpenPGP digital signature
URL: <http://mta.openssl.org/pipermail/openssl-users/attachments/20180728/15070a79/attachment.sig>