PKCS7_decrypt vs RSA OAEP padding

[Michal Moravec]

Thank you for the tip.

I replaced the Debian 1.1.1d with 1.1.1.k built from source using ./config -d
Using file on the libcrypto.so.1.1 library now shows "with debug_info, not stripped".
ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, BuildID[sha1]=29204b7f7a988f750cdc94e14be6ad9526c563ef, with debug_info, not stripped

Also I recompiled the libscep library with my changes to include the debug symbols as well and installed it into the lib directory.
I rebuilt the library cache using ldconfig and rebooted the OS just to be sure.

There is no difference in the trace output. Maybe because this is because it is reported by the kernel in the kern.log not the code which made it happen?

However I managed to crash one of the libscep tests in very similiar way. I am not sure if its 100% the same problem but it could be.
Here is a trace obtained via coredumpctl:

  PID: 3450 (test_message)
           UID: 0 (root)
           GID: 0 (root)
        Signal: 11 (SEGV)
     Timestamp: Thu 2021-04-15 15:40:28 CEST (4min 59s ago)
  Command Line: /root/libscep/build/tests/test_message
    Executable: /root/libscep/build/tests/test_message
 Control Group: /user.slice/user-998.slice/session-3.scope
          Unit: session-3.scope
         Slice: user-998.slice
       Session: 3
     Owner UID: 998 (testuser)
       Boot ID: 5867d606efa040f4acb4cf059c5349a2
    Machine ID: 5678312713094967b594b42360c03298
      Hostname: scepdev
       Storage: /var/lib/systemd/coredump/core.test_message.0.5867d606efa040f4acb4cf059c5349a2.3450.1618494028000000.
       Message: Process 3450 (test_message) of user 0 dumped core.

                Stack trace of thread 3450:
                #0  0x00007efd1ec4cd70 CMS_get0_type (libcrypto.so.1.1)
                #1  0x00007efd1ec51a45 CMS_decrypt (libcrypto.so.1.1)
                #2  0x00007efd1f1ef195 n/a (/root/libscep/build/src/libscep.so)
                #3  0x00007efd1f1f26d7 n/a (/root/libscep/build/src/libscep.so)
                #4  0x00007efd1f1f1c09 n/a (/root/libscep/build/src/libscep.so)
                #5  0x0000555c6c96ab31 n/a (/root/libscep/build/tests/test_message)
                #6  0x0000555c6c96fcc6 n/a (/root/libscep/build/tests/test_message)

I

MM


Try linking libcrypto.so.1.1 with debug symbols included (not
stripped).  This should make the error message point to the
function, maybe even show the call stack.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mta.openssl.org/pipermail/openssl-users/attachments/20210415/077c4864/attachment.html>