[openssl-users] openssl cms -decrypt failing due to malloc(3) failure

[Michael Wojcik]

> From: openssl-users [mailto:openssl-users-bounces at openssl.org] On Behalf
> Of Christian Böhme
> Sent: Tuesday, July 31, 2018 10:16
>
> On 30.07.2018 20:12, Michael Wojcik wrote:
>
> > FWIW, SUS Issue 5 defines RLIMIT_AS as applying to both malloc and mmap, but RLIMIT_DATA as
> > applying only to malloc. (That is, mmap'd pages do not count against the data limit.)
>
> mmap() , by defninition, populates the process' virtual address space, and if  that
> is limited in size, artificially or not, then the call is going to fail, eventually.

That's irrelevant to the statement you quoted, which was about the SUS process-limit mechanism (setrusage et al.), not the process address space.

> > Agreed. And I'm not endorsing the mmap approach for this problem anyway - I'd use a streaming
> > approach, so I'm not limited by address space.
>
> This structure, if held in a regular file, can be processed quite non-linearly,
> and without  mmap()'ing  its entire contents.

Indeed. I still don't see any compelling reason to mmap it at all.

> The pure streaming approach may be appropriate for file descriptors that are not
> seekable like sockets, pipes, tty ends etcpp., whereas with egular files, random
> access schemes using either  pread(v)(2)  or  lseek(2)  in combination with
> read(v)(2)  can be employed.

Or regular files could also be processed sequentially. What's the advantage of making seekable sources a special case?

In any case, the OpenSSL apps are a convenience and a set of samples. You can always write your own version of the cms app.

--
Michael Wojcik
Distinguished Engineer, Micro Focus