[openssl-users] openssl cms -decrypt failing due to malloc(3) failure

[Christian Böhme]

Hello,

On 01.08.2018 14:49, Michael Wojcik wrote:
>> On 30.07.2018 20:12, Michael Wojcik wrote:
>>
>>> FWIW, SUS Issue 5 defines RLIMIT_AS as applying to both malloc and mmap, but RLIMIT_DATA as
>>> applying only to malloc. (That is, mmap'd pages do not count against the data limit.)
>>
>> mmap() , by defninition, populates the process' virtual address space, and if  that
>> is limited in size, artificially or not, then the call is going to fail, eventually.
> 
> That's irrelevant to the statement you quoted, which was about the SUS process-limit mechanism
> (setrusage et al.), not the process address space.

I may have skipped a few steps and made too broad a statement, so here are the details
and corrections.

malloc(3)  is a pure C library call, whereas  mmap(2)  is a proper system call on those,
POSIX-conforming systems I have seen, anyway.

To a userland process, there is nothing more fundamental than the system call interface.
If it requires resources to do its work, it needs to go through this interface and ask
"the system" first.  The kernel as an implementation of "the system", where process-level
resource control actually happens, does not care about what userland code such as the
C library does unless that actually decides to call the system.

Except for SAS OSes, processes generally live in their own virtual address space, and
that is where "the process's data segment" is located that  RLIMIT_DATA  refers to.
s/brk(2)  are system calls to manage the end of the process's data segment AKA program
break, while  mmap() , as much broader a concept, does not have this limited view on the
process's address space and is therefore also more powerful.

Now, with Linux since 4.7 for example,  RLIMIT_DATA  also controls  mmap() .  What's more,
glibc  malloc()  on Linux is actually implemented in terms of  mmap()  these days, although
usage of  s/brk()  isn't phased out completely (apparently, the runtime linker still likes
those).  Since  s/brk()  and  mmap()  are the only means available to a Linux userland
process to manage those parts of its address space that it is designed to manage,
controlling a process's  RLIMIT_DATA  value ultimately controls all of its ability
to manage its address space, data segment or otherwise.

>> The pure streaming approach may be appropriate for file descriptors that are not
>> seekable like sockets, pipes, tty ends etcpp., whereas with egular files, random
>> access schemes using either  pread(v)(2)  or  lseek(2)  in combination with
>> read(v)(2)  can be employed.
> 
> Or regular files could also be processed sequentially. What's the advantage of making
> seekable sources a special case?

Making sure that the message under investigation is consistent w.r.t. the standards
according to which it was supposedly constructed, without having to consume resources
that aren't strictly necessary to make such a decision.  In other words, it's about
system performance /and/ making sure to get the logic right at the same time.

There is simply no need to slurp in the whole file just to get at its end (if so
required), when mechanisms for random access are readily available.  Please remember
our messages can get fairly large.  System performance and tight integration are
equally important to us as the security aspects.

> In any case, the OpenSSL apps are a convenience and a set of samples.

My original impression was that those tools represented some kind of reference
implementation of the libraries.  Clearly, I was wrong ;-)


Thanks,
Christian

-- 
*Christian Böhme*

Developer System Integration

CLOUD&HEAT

*CLOUD & HEAT Technologies GmbH*
Königsbrücker Str. 96 (Halle 15) | 01099 Dresden
Tel: +49 351 479 3670 - 100
Fax: +49 351 479 3670 - 110
E-Mail: christian.boehme at cloudandheat.com <mailto:christian.boehme at cloudandheat.com>
Web: https://www.cloudandheat.com <https://www.cloudandheat.com>

Handelsregister: Amtsgericht Dresden
Registernummer: HRB 30549
USt.-Ident.-Nr.: DE281093504
Geschäftsführer: Nicolas Röhrs


-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 533 bytes
Desc: OpenPGP digital signature
URL: <http://mta.openssl.org/pipermail/openssl-users/attachments/20180802/60827e52/attachment-0001.sig>