PKCS#7/CMS verify reports bad signature

[Matt Caswell]

On 01/04/2019 14:46, Steffen wrote:
> Hello,
> 
> I am struggling with using OpenSSL 1.1.1 to verify a PKCS #7/CMS structure.
> Verification succeeds when I use OpenSSL 1.0.2, but 1.1.0 and 1.1.1 fails with
> "bad signature". I initially had this problem when using the OpenSSL library but
> I see that the problem also applies to the OpenSSL CLI.

Could be this (from CHANGES):

  *) Fixed a text canonicalisation bug in CMS

     Where a CMS detached signature is used with text content the text goes
     through a canonicalisation process first prior to signing or verifying a
     signature. This process strips trailing space at the end of lines, converts
     line terminators to CRLF and removes additional trailing line terminators
     at the end of a file. A bug in the canonicalisation process meant that
     some characters, such as form-feed, were incorrectly treated as whitespace
     and removed. This is contrary to the specification (RFC5485). This fix
     could mean that detached text data signed with an earlier version of
     OpenSSL 1.1.0 may fail to verify using the fixed version, or text data
     signed with a fixed OpenSSL may fail to verify with an earlier version of
     OpenSSL 1.1.0. A workaround is to only verify the canonicalised text data
     and use the "-binary" flag (for the "cms" command line application) or set
     the SMIME_BINARY/PKCS7_BINARY/CMS_BINARY flags (if using CMS_verify()).
     [Matt Caswell]

Matt

> 
> I am at loss and need some help with this issue. Please see the commands I used
> below. Thank you for any assistance you can provide!
> 
> Notes:
> 
>   * "-noverify" was used because the certificates expired.
>   * Verification succeeds when specifying "-nosigs".
>   * "openssl cms -verify [...]" behaves the same way.
>   * Since the files I am working with (test.der and test-data.bin) are part of a
>     private project, I am not ready to share these in public.
>   * I do not know exactly how the message structure was created but I guess
>     either with some OpenSSL 1.0.2, Java with or without BouncyCastle.
> 
> Commands used:
> 
> # Environment: macOS 10.14.3 / Homebrew
>  
> $ /usr/local/opt/openssl/bin/openssl version
> OpenSSL 1.0.2r  26 Feb 2019
>  
> $ /usr/local/opt/openssl/bin/openssl smime -verify -inform der -in test.der
> -content test-data.bin -noverify
> Verification successful
>  
> $ /usr/local/opt/openssl\@1.1/bin/openssl version
> OpenSSL 1.1.1b  26 Feb 2019
>  
> $ /usr/local/opt/openssl\@1.1/bin/openssl smime -verify -inform der -in test.der
> -content test-data.bin -noverify
> Verification failure
> 4563408320:error:04091068:rsa routines:int_rsa_verify:bad
> signature:crypto/rsa/rsa_sign.c:220:
> 4563408320:error:21071069:PKCS7 routines:PKCS7_signatureVerify:signature
> failure:crypto/pkcs7/pk7_doit.c:1037:
> 4563408320:error:21075069:PKCS7 routines:PKCS7_verify:signature
> failure:crypto/pkcs7/pk7_smime.c:353: