[openssl-dev] [openssl.org #4286] Debug in OpenSSL

Tiantian Liu via RT rt at openssl.org
Tue Feb 2 16:58:07 UTC 2016


Hi All,
Good morning.

I reported a OpenSSL function (PEM_read_RSAPrivateKey) crash yesterday.
Honestly, I doubt it's the issue of OpenSSL. After all, it has being used years.

I am suspecting maybe something, happened at the background on our customer's server,  which caused OpenSSL crash.
I think I should give you more information about how that function is used in our application. My code is:

RSA * createRSAWithFilename(char * filename,char* diag, int public)
{
    FILE * fp = fopen(filename,"rb");

    if(fp == NULL)
    {
        if(diag) SerialWriteTestLine_string_Time("Unable to open file:", filename, diag);
        return NULL;
    }

    RSA *rsa= RSA_new() ;
    if(diag) SerialWriteTestLine_string_Time("FILE open on:", filename, diag);

    if(public >0)
    {
        rsa = PEM_read_RSA_PUBKEY(fp, &rsa, NULL, NULL);
    }
    else
    {
        rsa = PEM_read_RSAPrivateKey(fp, &rsa, NULL, NULL);     <- CRASH HERE!
    }

    if(diag) SerialWriteTestLine_Time("after PEM_read_RSAPrivateKey/PEM_read_RSA_PUBKEY", diag);

    fclose(fp);

.........................................................
The code above is being used by our customer.  They have 2 or 3 times crash every day.
There are only 2 parameters passed to the rsa = PEM_read_RSAPrivateKey(fp, &rsa, NULL, NULL),
I found the code does not validate the value of rsa (if RSA_new successfully returned or not). But when I assigned NULL to rsa before calling PEM_read_RSAPrivateKey, it didn't crash.
But the first parameter, handle fp is not the cause of crash either. Because I also wrote another test program which keep opening and closing and overwriting the file, again it didn't crash.

So from your OpenSSL developer's perspective, what may cause the crash of PEM_read_RSAPrivateKey? For me, I can only control the parameters passed to it.
I know there are only 2 kinds of value returned by RSA_new(). Valid address upon success and NULL for failure. I am wondering does it possibly return a not NULL value but illegal memory address to rsa, which may cause the crash of PEM_read_RSAPrivateKey?

This is way I asked you guys about how can I step into the OpenSSL functions.
Thanks,
Tyler

From: Tiantian (Tyler) Liu
Sent: February-01-16 5:00 PM
To: 'rt at openssl.org'
Subject: Debug in OpenSSL

Hi, ALL,

I am software developer who is struggling with encryption and decryption issues in my application.

Our customer complained our application crashed at the point where OpenSSL method,  PEM_read_RSAPrivateKey, being called.

While I can't duplicate the crash in my machine. So I want to enable debug in OpenSSL and core dumping on their machine, then I can get the core dump file upon the crash on customer's side. And I can use GDB to debug the core dump to see what happened in side the so-called PEM_read_RSAPrivateKey.

Today, I re-compiled my OpenSSL (version openssl-1.0.1p). However, when I set the breakpoint at PEM_read_RSAPrivateKey, my GDB can't step into that function, just bypassed directly.
My machine is 32-bit RedHat Enterprise 5. What I did in configure and installation:

#./Configure -g debug-linux-elf -prefix=/usr shared
# make
# make install

All the new generated libs were installed under /usr/lib

I use GDB command to check my setup. It looks like my GDB can recognize all the OpenSSL source code and loaded OpenSSL shared library symbols. I post the part of information from GDB:
(gdb) info sharedlibrary
>From        To          Syms Read   Shared Object Library
0x00561a30  0x005c6364  Yes         /usr/lib/libkrb5.so.3
0x0064f590  0x00666e94  Yes         /usr/lib/libk5crypto.so.3
0x002407c0  0x004446c4  Yes         /usr/lib/libptcoresdk.so.2
0x0070a7f0  0x0070af84  Yes         /lib/libcom_err.so.2
0x008c55d0  0x00940594  Yes         /usr/lib/libstdc++.so.6
0x005e86b0  0x00631eb4  Yes         /usr/lib/libssl.so.1.0.0
0x00a73f00  0x00b81704  Yes         /usr/lib/libcrypto.so.1.0.0
0x004f7a50  0x004f8a64  Yes         /lib/libdl.so.2
0x004ff210  0x00509e34  Yes         /lib/i686/nosegneg/libpthread.so.0
0x00722bd0  0x0081a7d0  Yes         /lib/i686/nosegneg/libc.so.6
0x00513430  0x00517794  Yes         /usr/lib/libkrb5support.so.0
0x0053f0d0  0x0054a064  Yes         /lib/libresolv.so.2
0x0085a670  0x00861ea4  Yes         /lib/libgcc_s.so.1
0x00675410  0x00690654  Yes         /lib/i686/nosegneg/libm.so.6
0x00a1c7f0  0x00a3172f  Yes         /lib/ld-linux.so.2

And I also ran command:
(gdb) info source
.........................................
pem_pkey.c, /home/tyler28/openssl-1.0.1p/crypto/pem/pem_pkey.c, pem_pk8.c, /home/tyler28/openssl-1.0.1p/crypto/pem/pem_pk8.c,
pem_oth.c, /home/tyler28/openssl-1.0.1p/crypto/pem/pem_oth.c, pem_xaux.c, /home/tyler28/openssl-1.0.1p/crypto/pem/pem_xaux.c,
pem_x509.c, /home/tyler28/openssl-1.0.1p/crypto/pem/pem_x509.c, pem_err.c, /home/tyler28/openssl-1.0.1p/crypto/pem/pem_err.c,
pem_all.c, /home/tyler28/openssl-1.0.1p/crypto/pem/pem_all.c, pem_lib.c, /home/tyler28/openssl-1.0.1p/crypto/pem/pem_lib.c,
pem_info.c, /home/tyler28/openssl-1.0.1p/crypto/pem/pem_info.c, pem_seal.c, /home/tyler28/openssl-1.0.1p/crypto/pem/pem_seal.c,
pem_sign.c, /home/tyler28/openssl-1.0.1p/crypto/pem/pem_sign.c, asn_moid.c, /home/tyler28/openssl-1.0.1p/crypto/asn1/asn_moid.c,
...............................................

Then during debug, my GDB showed:
(gdb) break PEM_read_RSAPrivateKey
Breakpoint 2 at 0xb373fd: file pem_all.c, line 184.
(gdb) c
Continuing.
[Switching to Thread 14957456 (LWP 8796)]

Breakpoint 1, createRSAWithFilename (filename=0x82ef65a "out/private.pem", diag=0xe3ebdc "/MerchantConnectMulti/log/262.dg",
    public=0) at ../multi_client/source_Host_C_Code/ssl_open.c:1385
1385        FILE * fp = fopen(filename,"rb");
(gdb) n
1387        if(fp == NULL)
(gdb) n
1393        RSA *rsa= RSA_new() ;
(gdb) n
1394        if(diag) SerialWriteTestLine_string_Time("FILE open on:", filename, diag);
(gdb) n
1395        if(diag) SerialWriteTestLine_Time("after RSA_new", diag);
(gdb) n
1398        if (rsa == NULL) {
(gdb) n
1408        if(public >0)
(gdb) n
1415            rsa = PEM_read_RSAPrivateKey(fp, &rsa,NULL, NULL);
(gdb) s                               <<<<<<<<---------- GDB bypassed, I can't step into the function!
1419        if(diag) SerialWriteTestLine_Time("after PEM_read_RSAPrivateKey/PEM_read_RSA_PUBKEY", diag);

Beside that function, I found I can't step into any OpenSSL standard function either. For example, I can't step into the RSA_new too.
Based on the message I offered above, could you help me to figure out what mistakes I did? Could you help me?
In another word, I just want to step into the OpenSSL standard library functions. How can I do that?

I am eagerly waiting for your response and help, thank you in advance.

Thanks,
Tyler






More information about the openssl-dev mailing list