[openssl-dev] [openssl.org #4657] Bug - SSL Library Error - BUF_MEM_grow:malloc failure

[Nomalatha Aerampu via RT]

Hi,

Our product (32-bit process) uses OpenSSL third-party libraries for EAP protocols. During the debugging of a customer issue in PEAP protocol, we got to understand that SSL_Accept has returned failure.

STATE_HANDSHAKE SSL_ERROR_SSL error retrun code [1] and peak error translate code [39] and err_get_reason [1048] // Error code, Error cases(Reason)
8900:error:14094418:SSL routines:SSL3_READ_BYTES:tlsv1 alert unknown ca:.\ssl\s3_pkt.c:1235:SSL alert number 48 // Error Queue
SSL_ERROR_SSL Handshake failure

On further analyzing, we found that handshake failure is due to the "BUF_MEM_grow : malloc failure" in the server after the 'n' number of authentications.
The memory buffer gets completely allocated and handshake is failed when we tried to allocate the memory.

STATE_HANDSHAKE SSL_ERROR_SSL error retrun code [1] and peak error translate code [2] and err_get_reason [65]
7820:error:07064041:memory buffer routines:BUF_MEM_grow:malloc failure:.\crypto\buffer\buffer.c:122:
SSL_ERROR_SSL Handshake failure

Total memory that is installed on the server: 4GB
Process memory size when the issue occurs: 360 ~ 370 MB

Operating System Version - Windows Server 2008
OpenSSL version - 0.9.7e, 1.0.0s

Please let us know if there are known issues related to the same or let us know if you require further information for your debugging.

Regards,
Nomalatha A

-- 
Ticket here: http://rt.openssl.org/Ticket/Display.html?id=4657
Please log in as guest with password guest if prompted