Can't link a static library with custom OpenSSL rsa engine

[Shariful Alam]

Hello,
I have a custom rsa engine. It builds and works fine. Later, I have added a
static library with my custom engine code. My code compiles. However, when
I try to load the custom engine it shows *invalid engine "rsa-engine-new".
*The full error is given below,

x at x:~/Downloads/x/x/x/rsa_engine$ openssl rsautl -decrypt -inkey
private.pem -in msg.enc -engine rsa-engine-new
invalid engine "rsa-engine-new"
140112744122112:error:25066067:DSO support routines:dlfcn_load:could not
load the shared
library:crypto/dso/dso_dlfcn.c:119:filename(/opt/openssl/lib/engines-1.1/rsa-engine-new.so):
/opt/openssl/lib/engines-1.1/rsa-engine-new.so: undefined symbol: dune_init
140112744122112:error:25070067:DSO support routines:DSO_load:could not load
the shared library:crypto/dso/dso_lib.c:162:
140112744122112:error:260B6084:engine routines:dynamic_load:dso not
found:crypto/engine/eng_dyn.c:414:
140112744122112:error:2606A074:engine routines:ENGINE_by_id:no such
engine:crypto/engine/eng_list.c:334:id=rsa-engine-new
140112744122112:error:25066067:DSO support routines:dlfcn_load:could not
load the shared
library:crypto/dso/dso_dlfcn.c:119:filename(librsa-engine-new.so):
librsa-engine-new.so: cannot open shared object file: No such file or
directory
140112744122112:error:25070067:DSO support routines:DSO_load:could not load
the shared library:crypto/dso/dso_lib.c:162:
140112744122112:error:260B6084:engine routines:dynamic_load:dso not
found:crypto/engine/eng_dyn.c:414:


Now the error doesn't say much about the cause of invalid engine. However
my guess is it is from the  "*undefined symbol: dune_init*". "dune_init" is
from the static library. Therefire I believe my linking is not working. I
use the following Makefile to compile the engine,


   1. rsa-engine: rsa/rsa.c rsa/bignum.c rsa/aes.c rsa/x509parse.c rsa/pem.c
   2. gcc -fPIC -o rsa/rsa.o -c rsa/rsa.c
   3. gcc -fPIC -o rsa/bignum.o -c rsa/bignum.c
   4. gcc -fPIC -o rsa/aes.o -c rsa/aes.c
   5. gcc -fPIC -o rsa/x509parse.o -c rsa/x509parse.c
   6. gcc -fPIC -o rsa/pem.o -c rsa/pem.c
   7. gcc -fPIC -c rsa-engine.c
   8. gcc -shared -o librsa_engine.so libdune/libdune.a -lcrypto
   rsa-engine.o rsa/rsa.o rsa/bignum.o rsa/aes.o rsa/x509parse.o rsa/pem.o
   9. mv librsa_engine.so rsa-engine-new.so
   10. sudo cp rsa-engine-new.so /opt/openssl/lib/engines-1.1/
   11. clean:
   12. rm -f *.o rsa/*.o *.so rsa-engine

So, can anyone please if my guess is correct or not? If my guess is
correct, how can I fix my Makefile?

N.B: Static library

   - libdune/libdune.a is in the same directory with the main rsa-engine.c
   - libdune/libdune.a is compiled with -fPIC flag


Thanks,
Shariful
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mta.openssl.org/pipermail/openssl-users/attachments/20201117/bc653d8b/attachment.html>