[openssl-users] openssl cms -decrypt failing due to malloc(3) failure

Tue Jul 31 16:16:06 UTC 2018

On 30.07.2018 20:12, Michael Wojcik wrote:

>> From: openssl-users [mailto:openssl-users-bounces at openssl.org] On Behalf Of Jordan Brown
>> Sent: Monday, July 30, 2018 10:46
[…]
> FWIW, SUS Issue 5 defines RLIMIT_AS as applying to both malloc and mmap, but RLIMIT_DATA as
> applying only to malloc. (That is, mmap'd pages do not count against the data limit.)

mmap() , by defninition, populates the process' virtual address space, and if that
is limited in size, artificially or not, then the call is going to fail, eventually.

>> If you're a 32-bit process, then malloc'ing or mmap'ing a 2GB object will be difficult at best.
> 
> Agreed. And I'm not endorsing the mmap approach for this problem anyway - I'd use a streaming
> approach, so I'm not limited by address space.

Let's look at the following message that was produced by symmetrically encrypting
757 plaintext octets using the Camellia cipher in CBC mode with a 256 bit key derived
from a passphrase:

$ cat ciphertext.pem | openssl asn1parse -i -inform PEM
    0:d=0  hl=4 l= 978 cons: SEQUENCE
    4:d=1  hl=2 l=   9 prim:  OBJECT            :pkcs7-envelopedData
   15:d=1  hl=4 l= 963 cons:  cont [ 0 ]
   19:d=2  hl=4 l= 959 cons:   SEQUENCE
   23:d=3  hl=2 l=   1 prim:    INTEGER           :03
   26:d=3  hl=3 l= 133 cons:    SET
   29:d=4  hl=3 l= 130 cons:     cont [ 3 ]
   32:d=5  hl=2 l=   1 prim:      INTEGER           :00
   35:d=5  hl=2 l=  27 cons:      cont [ 0 ]
   37:d=6  hl=2 l=   9 prim:       OBJECT            :PBKDF2
   48:d=6  hl=2 l=  14 cons:       SEQUENCE
   50:d=7  hl=2 l=   8 prim:        OCTET STRING      [HEX DUMP]:948BAC4CEDB23DE2
   60:d=7  hl=2 l=   2 prim:        INTEGER           :0800
   64:d=5  hl=2 l=  46 cons:      SEQUENCE
   66:d=6  hl=2 l=  11 prim:       OBJECT            :id-alg-PWRI-KEK
   79:d=6  hl=2 l=  31 cons:       SEQUENCE
   81:d=7  hl=2 l=  11 prim:        OBJECT            :camellia-256-cbc
   94:d=7  hl=2 l=  16 prim:        OCTET STRING      [HEX DUMP]:D7A2F88C99A1881C1B8B6AA9E2BDD002
  112:d=5  hl=2 l=  48 prim:      OCTET STRING      [HEX DUMP]:445771F0EA6BAA64CAF35BFC2DA546845C…
  162:d=3  hl=4 l= 816 cons:    SEQUENCE
  166:d=4  hl=2 l=   9 prim:     OBJECT            :pkcs7-data
  177:d=4  hl=2 l=  31 cons:     SEQUENCE
  179:d=5  hl=2 l=  11 prim:      OBJECT            :camellia-256-cbc
  192:d=5  hl=2 l=  16 prim:      OCTET STRING      [HEX DUMP]:4F8DAFF8EE165FD78C35A644735CD082
  210:d=4  hl=4 l= 768 prim:     cont [ 0 ]

This structure, if held in a regular file, can be processed quite non-linearly,
and without  mmap()'ing  its entire contents.  The only parts that are going to
grow as the plaintext grows are the ciphertext (index 192 above) and, to a lesser
extend, the ones that depend on key sizes in the  recipientInfos  and the length
components.  Once the overall structure of the message is understood, sequential
processing of the ciphertext should pose no problem, whichever way implemented.

The pure streaming approach may be appropriate for file descriptors that are not
seekable like sockets, pipes, tty ends etcpp., whereas with egular files, random
access schemes using either  pread(v)(2)  or  lseek(2)  in combination with
read(v)(2)  can be employed.  Portability is certainly an issue, but isn't
this what the  configure  scripts are for to figure out?

I also do not quite get why CMS should not lend itself to "large" data, given the
above.  It would seem that the whole point of the definite TLV structures is to be
able to learn the type and size requirements of the data to come in the stream
/before/ processing it, allowing the "processor" to take appropriate measures,
and not necessarily in RAM alone.

Thanks,
Christian

-- 
*Christian Böhme*

Developer System Integration

CLOUD&HEAT

*CLOUD & HEAT Technologies GmbH*
Königsbrücker Str. 96 (Halle 15) | 01099 Dresden
Tel: +49 351 479 3670 - 100
Fax: +49 351 479 3670 - 110
E-Mail: christian.boehme at cloudandheat.com <mailto:christian.boehme at cloudandheat.com>
Web: https://www.cloudandheat.com <https://www.cloudandheat.com>

Handelsregister: Amtsgericht Dresden
Registernummer: HRB 30549
USt.-Ident.-Nr.: DE281093504
Geschäftsführer: Nicolas Röhrs

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 533 bytes
Desc: OpenPGP digital signature
URL: <http://mta.openssl.org/pipermail/openssl-users/attachments/20180731/66adfb4e/attachment.sig>