[openssl-users] gpgsm/openssl cms detached signatures verification fails

[Andreas Fenkart]

Hi list,
I'm experimenting with openssl/gnupg interoperability. It seems the
least common multiple is cms. I didn't find a way to produce output
with opengpg to work with 'openssl dgst' directly. Am I wrong, is
there a simpler way?

generate gpgsm keyring: https://lists.gt.net/gnupg/devel/53489
$ gpgsm --generate-key > x.pem; CN=Joe, O=Corp, C=unv
$ gpgsm --import x.pem
$ gpgsm -a --output pub.pem --export 0x3E3AB34C

GNUPG -> OPENSSL:

it works when using non-detached signatures
$ gpgsm -as -o somefile.sig somefile
$ sed -i 's/SIGNED MESSAGE/CMS/' somefile.sig
$ openssl cms -verify -in somefile.sig -inform PEM -certfile pub.pem -noverify
Verification successful

But fails when using detached signatures:
$ gpgsm -asb -o somefile.sig somefile
$ sed -i 's/SIGNED MESSAGE/CMS/' somefile.sig
$ openssl cms -verify -in somefile.sig -inform PEM -content somefile
-certfile pub.pem -noverify

OPENSSL -> GNUPG:

create key:
$ openssl req -x509 -newkey rsa:2048 -nodes -keyout mycert.key.pem \
    -out mycert.cert.pem -subj "/O=SWUpdate /CN=target"
$ gpgsm --import mycert.cert.pem

again works with non-detached signature:
$ openssl cms -sign -in somefile -out somefile.sig -nodetach \
  -signer mycert.cert.pem -inkey mycert.key.pem -outform PEM -nosmimecap
$ gpgsm --disable-crl-checks --verify somefile.sig
gpgsm: Good signature from "/CN=target/O= SWUpdate"

and fails with detached signatures:
$ openssl cms -sign -in somefile -out somefile.sig \
  -signer mycert.cert.pem -inkey mycert.key.pem -outform PEM -nosmimecap

$ gpgsm --disable-crl-checks --verify somefile.sig somefile
gpgsm: Signature made 2017-05-15 14:44:36 using certificate ID 0x5F7C51D8
gpgsm: invalid signature: message digest attribute does not match computed one

COMPARING ASN1 output:

I parsed the signatures using asn1parse, then compared the outputs:
$ openssl asn1parse -in somefile.sig

  998:d=7  hl=2 l=   9 prim: OBJECT            :messageDigest
   1009:d=7  hl=2 l=  34 cons: SET
   1011:d=8  hl=2 l=  32 prim: OCTET STRING      [HEX
DUMP]:CC83AEAE49B66CCADA3DFFEB87E27AF53D99437F5E24485D31F4F11BF092FA6D
   1045:d=5  hl=2 l=  13 cons: SEQUENCE
   1047:d=6  hl=2 l=   9 prim: OBJECT            :rsaEncryption
   1058:d=6  hl=2 l=   0 prim: NULL
   1060:d=5  hl=4 l= 256 prim: OCTET STRING      [HEX
DUMP]:7440A9007CACDD5484076661B57181AE64249506FF631CFF1EE5B0D74CBD431D9AB57533E44BE6B56F2D18DF52ED9B24C6DD040B6E8F9B3A6F1E3

The :messageDigest hash codes are the same for detached/non-detached
but the :rsaEncryption differ (salt, timestamp?) The messageDigest
does not match the output from sha256sum

Repeating the some for the gpgsm genarated signatures, I can see that
the messageDigest matches directly the output of sha256sum.

So openssl cms/gpgsm compute the sha256sum differently in the detached case.
Is there a hidden flag to make either tool behave like the other?

/Andi