[openssl-users] Correct the check of RSA_FLAG_SIGN_VER

[Richard Levitte]

In message <BN6PR07MB3553B5B834436F685450D434F6A50 at BN6PR07MB3553.namprd07.prod.outlook.com> on Tue, 3 Apr 2018 22:55:06 +0000, "Eichenberger, John" <John.Eichenberger at Honeywell.com> said:

John.Eichenberger> Honeywell Internal
John.Eichenberger> 
John.Eichenberger> I think I have an answer for why this commit needed to be reverted:
John.Eichenberger> 
John.Eichenberger> Author: Dr. Stephen Henson <steve at openssl.org>
John.Eichenberger> Author date: 2 years ago (12/20/2015 10:18:43 AM)
John.Eichenberger> Commit date: 2 years ago (12/20/2015 11:27:03 AM)
John.Eichenberger> Commit hash: 6656ba7152dfe4bba865e327dd362ea08544aa80
John.Eichenberger> Children: 1c7de36f62
John.Eichenberger> Parent(s): 17592f323a
John.Eichenberger> 
John.Eichenberger> Don't check RSA_FLAG_SIGN_VER.
John.Eichenberger> 
John.Eichenberger> Reviewed-by: Richard Levitte <levitte at openssl.org>
John.Eichenberger> 
John.Eichenberger> The change made in that commit was to simply remove
John.Eichenberger> the attempt to check for the RSA_FLAG_SIGN_VER flag.
John.Eichenberger> But that’s not what is wrong with this code that
John.Eichenberger> required changing. The change should be to add
John.Eichenberger> “meth” prior to flags:

Well, not quite, actually.  We can easily study the code prior to this
change by looking at the 1.0.2 source.

1. in RSA_new_method(), which is used to create new instances of the
   RSA structure, there's this line:

    ret->flags = ret->meth->flags & ~RSA_FLAG_NON_FIPS_ALLOW;

   So that makes the check of rsa->flags valid, no need to go via
   rsa->meth

2. In rsa.h (crypto/rsa/rsa.h in 1.0.2), you'll find this comment in
   the middle of the definition of rsa_meth_st

    /*
     * New sign and verify functions: some libraries don't allow arbitrary
     * data to be signed/verified: this allows them to be used. Note: for
     * this to work the RSA_public_decrypt() and RSA_private_encrypt() should
     * *NOT* be used RSA_sign(), RSA_verify() should be used instead. Note:
     * for backwards compatibility this functionality is only enabled if the
     * RSA_FLAG_SIGN_VER option is set in 'flags'.
     */

   Do note the note on backward compatibility...  you see, there were
   versions of OpenSSL where the fields 'rsa_sign' and 'rsa_verify'
   didn't exist (they appeared in OpenSSL 0.9.5), so for the sake of
   allowing older applications to work with the newer OpenSSL without
   recompilation, we required all new RSA method implementations to
   use that flag to have the 'rsa_sign' and 'rsa_verify' functions
   used.  Without that flag, those functions were assumed not to
   exist, that the RSA method structure was pre-0.9.5.  However, this
   was somewhere in 2000.

   Fast forward to 2015, when we were starting to make certain types
   opaque, and someone noticed this flag still hanging around, and we
   figured that 0.9.5 is long gone, and 1.0.1 was a year away from its
   end of life, and we figured that the reason to have this flag at
   all was a matter of years long past, it was time to simply drop its
   use.  It had grown to become irrelevant a long time ago.

Cheers,
Richard

-- 
Richard Levitte         levitte at openssl.org
OpenSSL Project         http://www.openssl.org/~levitte/