Can't link a static library with custom OpenSSL rsa engine

[Scott Neugroschl]

You need to put the static library at the END of your link command.  A static library is searched when it is encountered in the link stream, and only the items needed will be used from it.

Because you have it first, there are no undefined symbols, and no items will be used from it.

From: openssl-users <openssl-users-bounces at openssl.org> On Behalf Of Shariful Alam
Sent: Tuesday, November 17, 2020 12:40 PM
To: openssl-users at openssl.org
Subject: Can't link a static library with custom OpenSSL rsa engine

Hello,
I have a custom rsa engine. It builds and works fine. Later, I have added a static library with my custom engine code. My code compiles. However, when I try to load the custom engine it shows invalid engine "rsa-engine-new".  The full error is given below,
x at x:~/Downloads/x/x/x/rsa_engine$ openssl rsautl -decrypt -inkey private.pem -in msg.enc -engine rsa-engine-new
invalid engine "rsa-engine-new"
140112744122112:error:25066067:DSO support routines:dlfcn_load:could not load the shared library:crypto/dso/dso_dlfcn.c:119:filename(/opt/openssl/lib/engines-1.1/rsa-engine-new.so<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2frsa-engine-new.so&c=E,1,8MFfQUaoLlnS6gmd_niyiDduXeMUdluPk-jrMBw_BPH34rsf0PEA0Dv7zwy4i-COOOGl03UyV1TkoVRWZ9Zl4pTzjJkd5QnxI-mpuWBw&typo=1>): /opt/openssl/lib/engines-1.1/rsa-engine-new.so<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2frsa-engine-new.so&c=E,1,yZ0E8kBO-xC-AjU32zEiKHCGybnn-JJF_YPSr2ZBgAWL6DyoG4877idj9NO4H7DX2BC9f_0dGIfayxivSa9caNM_LtUEctaHHgYmaO44&typo=1>: undefined symbol: dune_init
140112744122112:error:25070067:DSO support routines:DSO_load:could not load the shared library:crypto/dso/dso_lib.c:162:
140112744122112:error:260B6084:engine routines:dynamic_load:dso not found:crypto/engine/eng_dyn.c:414:
140112744122112:error:2606A074:engine routines:ENGINE_by_id:no such engine:crypto/engine/eng_list.c:334:id=rsa-engine-new
140112744122112:error:25066067:DSO support routines:dlfcn_load:could not load the shared library:crypto/dso/dso_dlfcn.c:119:filename(librsa-engine-new.so<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2flibrsa-engine-new.so&c=E,1,7A5v40ih1yKJkQ1MQXgdsgIcz0QX7BfZjbTVW525GIwdanaAcrro9rDKrM97clgfEHiWtSoHQWELls4jdhFGZeZQNznXqtpt-okohpCJ07u9v2OqEOEtp3iXN74,&typo=1>): librsa-engine-new.so<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2flibrsa-engine-new.so&c=E,1,LIYVHMYYZmkN3YvxC0loOZ2qEyA6ce6grFXm8tUPGpxI3XpP2-0P9DV4TbhF2veGWNUwcuxthn0W7zI-IUcXMzrRyjB72JONredHeuwRuHBUig577YKKqMkC&typo=1>: cannot open shared object file: No such file or directory
140112744122112:error:25070067:DSO support routines:DSO_load:could not load the shared library:crypto/dso/dso_lib.c:162:
140112744122112:error:260B6084:engine routines:dynamic_load:dso not found:crypto/engine/eng_dyn.c:414:

Now the error doesn't say much about the cause of invalid engine. However my guess is it is from the  "undefined symbol: dune_init". "dune_init" is from the static library. Therefire I believe my linking is not working. I use the following Makefile to compile the engine,

  1.  rsa-engine: rsa/rsa.c rsa/bignum.c rsa/aes.c rsa/x509parse.c rsa/pem.c
  2.  gcc -fPIC -o rsa/rsa.o -c rsa/rsa.c
  3.  gcc -fPIC -o rsa/bignum.o -c rsa/bignum.c
  4.  gcc -fPIC -o rsa/aes.o -c rsa/aes.c
  5.  gcc -fPIC -o rsa/x509parse.o -c rsa/x509parse.c
  6.  gcc -fPIC -o rsa/pem.o -c rsa/pem.c
  7.  gcc -fPIC -c rsa-engine.c
  8.  gcc -shared -o librsa_engine.so<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2flibrsa_engine.so&c=E,1,N_9eth6NdpIazbe3_j8EdohDb6pIn-PO43kU7bxbgdGLXKBOUC5HXR5F48otNDxySKG93X0q5krjPfBvVFAldMrnv5856Oo-uoSY-xJ5Y7m_y_q4wxvu&typo=1> libdune/libdune.a -lcrypto rsa-engine.o rsa/rsa.o rsa/bignum.o rsa/aes.o rsa/x509parse.o rsa/pem.o
  9.  mv librsa_engine.so<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2flibrsa_engine.so&c=E,1,7geFCnxRhWfkcdK7jWRVB2N_LnXYGRX8r9JpSaukZwQmWiqf1H0s3yIjrW4LtoVYe9dCjUvDEqdamKO-lwVSZzRpmHQu8fH1fNdzMqEvmA,,&typo=1> rsa-engine-new.so<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2frsa-engine-new.so&c=E,1,Tyo5JxmmfA2ttXY9vqeJ1tLVs76qTcaZOjPKAudqDhbW2Ikkwauuegat_ao4q7fXq6RwR2FJBQA1P3KfMhqYUvSqljdyh7C-l-lV5ZhPnN_h-yMXotVWISMInOU,&typo=1>
  10. sudo cp rsa-engine-new.so<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2frsa-engine-new.so&c=E,1,Jyat_hKejDMyyX3dPO3dLs3pVai3U4qQiwdSuOgV9AvIbM2go7xZTGSscwik_t7z6yc4NB2uyLHuiWvLg6yLISk50r6zerXdbtcVWZqcJXBI_TWbmb_B7Imb7zY,&typo=1> /opt/openssl/lib/engines-1.1/
  11. clean:
  12. rm -f *.o rsa/*.o *.so rsa-engine
So, can anyone please if my guess is correct or not? If my guess is correct, how can I fix my Makefile?

N.B: Static library

  *   libdune/libdune.a is in the same directory with the main rsa-engine.c
  *   libdune/libdune.a is compiled with -fPIC flag

Thanks,
Shariful
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mta.openssl.org/pipermail/openssl-users/attachments/20201117/5ec728d3/attachment-0001.html>