SSL error (78c0100): malloc failure while implementing tls 1.3

[Ramaiah, Ravichandran Bagalur]

Hi All,

I'm trying to implement tls 1.3 support in my application. But I'm facing malloc failure error.

Could you please help me understand why this error is happening? How to solve this issue?


*Set TLSv1.3 Cipher list TLS_AES_128_GCM_SHA256 ret 1
*SipCmOpenSSLNew: TLS, mutual auth, tlsSipAuthRequired = FALSE
*SSL_OP_DONT_INSERT_EMPTY_FRAGMENTS for ssl is NOT set.
*SSL handshake started undefined:before SSL initialization 240
*SSL_accept:before SSL initialization
*SSL_accept:before SSL initialization
*SSL_accept:SSLv3/TLS read client hello
*SSL_accept:SSLv3/TLS write server hello
*SSL_accept:SSLv3/TLS write change cipher spec
*SSL_accept:TLSv1.3 early data
*SSL_accept:error in TLSv1.3 early data
*SipCmAcceptSocket, socketId 121, us 10.34.164.185, peer <unspec> protocol 8
*SSL_accept:TLSv1.3 early data
*SSL_accept:SSLv3/TLS read client hello
*SSL_accept:SSLv3/TLS write server hello
*SSL_accept:TLSv1.3 write encrypted extensions
*SSL_accept:SSLv3/TLS write certificate request
*SSL_accept:SSLv3/TLS write certificate
*SSL_accept:TLSv1.3 write server certificate verify
*SSL_accept:SSLv3/TLS write finished
*SSL_accept:TLSv1.3 early data
*SSL_accept:error in TLSv1.3 early data
*SSL_accept:TLSv1.3 early data
*SSL_accept:SSLv3/TLS read client certificate
*SSL_accept:SSLv3/TLS read certificate verify
*SSL_accept:SSLv3/TLS read finished
*SSL handshake done undefined:SSLv3/TLS write session ticket  240
*New session created on sigport 2
*SSL_accept:SSLv3/TLS write session ticket
*SSL_SESSION_free ref
 *Session deleted on 2
*SSL3 alert write:fatal:internal error
*SSL_accept:error in error
*SSL error (78c0100): malloc failure
*ERROR on SSL_read err=1 flag=0
*Initiating SSL shutdown





I generated client and server certificates using below commands. And I used TLS_AES_128_GCM_SHA256 cipher.

CA Certificate:

openssl_rbbn ecparam -name prime256v1 -genkey -noout -out ca.key

openssl_rbbn req -new -x509 -sha256 -key ca.key -out ca.crt

openssl_rbbn x509 -in ca.crt -inform PEM -out pk-ca.crt.der -outform DER


Server Certificate:

openssl_rbbn ecparam -name prime256v1 -genkey -noout -out server.key

openssl_rbbn req -new -sha256 -key server.key -out server.csr

openssl_rbbn x509 -req -in server.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out server.crt -days 1000 -sha256



Client Certificate:

openssl_rbbn ecparam -name prime256v1 -genkey -noout -out client1.key

openssl_rbbn req -new -sha256 -key client1.key -out client1.csr

openssl_rbbn x509 -req -in client1.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out client1.crt -days 1000 -sha256

Regards,
Ravi


Notice: This e-mail together with any attachments may contain information of Ribbon Communications Inc. and its Affiliates that is confidential and/or proprietary for the sole use of the intended recipient. Any review, disclosure, reliance or distribution by others or forwarding without express permission is strictly prohibited. If you are not the intended recipient, please notify the sender immediately and then delete all copies, including any attachments.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mta.openssl.org/pipermail/openssl-users/attachments/20220616/55e73413/attachment-0001.htm>