[openssl-users] RSA_PKCS1_OAEP_PADDING

[Dr. Stephen Henson]

On Thu, May 11, 2017, RudyAC wrote:

> Hello,
> 
> I have the requirement to encrypt e-mails using RSA-OAEP padding. I use the
> library openssl-1.0.2k and encrypt with CMS container. The following
> function describes my method. My problem is that I'm not sure if this method
> really uses the RSA-OAEP padding.
> 
> bool
> smime_encrypt_cms(const std::string& infile, const std::string& outfile)
> {
>     bool                bResult = false;
>     const char*         inmode = "r";
>     const char*         outmode = "w";
>     const EVP_CIPHER*   cipher = NULL;
> 
> 
>     STACK_OF(X509)*     encerts = NULL;
>     BIO*                in = NULL;
>     BIO*                out = NULL;
>     BIO*                bio_err = NULL;
>     int                 flags = 0;
> 
> 	X509 *recip;
> 	int i = 0;
> 	unsigned char *oaep_label = NULL;
> 	int oaep_label_l = 0;
> 	int nflags = CMS_PARTIAL | CMS_KEY_PARAM;
> 	CMS_ContentInfo* cms = CMS_encrypt(NULL, NULL, cipher, nflags);
> 	EVP_PKEY_CTX* wrap_ctx = NULL;
> 
>     KWlog ( EV_D_APPL_14 , "smime_encrypt_cms () started" );
> 
>     cipher = get_cipher();
>     SMTPD_RAND_load_file ( NULL , bio_err , 0 );
> 
>     encerts = sk_X509_new_null();
> 
>     FOR_CONST_IT(EmailAndCertList, itRecip, _m_recipCertsList)
>     {
>         SMIME_key_list recip_encerts = (*itRecip)->smime_enc();
> 
>         FOR_CONST_IT(SMIME_key_list, iter, recip_encerts)
>         {
>             sk_X509_push( encerts, (*iter).dup_cert());
>         }
>     }
> 
> 
>     if ( ! ( in = BIO_new_file ( infile.c_str() , inmode ))) {
>         KWlog_appl ( EV_E_APPL_INFO , "Can't open input file %s",
> infile.c_str() );
>         _error_messages.push_back("Internal Error");
>         goto exit;
>     }
> 
>     if ( ! ( out = BIO_new_file ( outfile.c_str() , outmode ))) {
>         KWlog_appl ( EV_E_APPL_INFO , "Can't open output file %s",
> outfile.c_str() );
>         _error_messages.push_back("Internal Error");
>         goto exit;
>     }
> 
>     for (i = 0; i < sk_X509_num(encerts); i++) {
> 
>     	CMS_RecipientInfo* r_info;
> 
> 		recip = sk_X509_value(encerts, i);
> 		r_info = CMS_add1_recipient_cert(cms, recip, nflags);
> 		if (!r_info) {
> 				KWlog_appl(EV_E_APPL_INFO,
> 						"smime_encrypt_cms(): Error while adding recipient certs to CMS info
> structure");
> 				return false;
> 		}
> 		wrap_ctx = CMS_RecipientInfo_get0_pkey_ctx(r_info);
> 		KWlog ( EV_D_APPL_14 , "smime_encrypt_cms () Set OAEP Padding");
> 		EVP_PKEY_CTX_set_rsa_padding(wrap_ctx, RSA_PKCS1_OAEP_PADDING);
> 		EVP_PKEY_CTX_set_rsa_oaep_md(wrap_ctx, EVP_sha256());
> 		EVP_PKEY_CTX_set_rsa_mgf1_md(wrap_ctx, EVP_sha256());
> 		EVP_PKEY_CTX_set0_rsa_oaep_label(wrap_ctx, oaep_label, oaep_label_l);
> 	}
> 
>        CMS_final(cms, in, NULL, nflags);
> 
>     /* encrypt content */
>     cms = CMS_encrypt(encerts, in, cipher, flags);
> 
> 
>     if( ! cms ) {
>         KWlog ( EV_E_APPL_INFO , "Error creating CMS structure");
>         KWlog_SSL ;
>         _error_messages.push_back("Internal Error");
>         goto exit;
>     }
> 
>     flags |= SMIME_OLDMIME;
> 
>     /* Write out S/MIME message */
>     if (!SMIME_write_CMS(out, cms, in, flags))
>     	goto exit;
> 
>     bResult = true;
> 
>  exit:
>     SMTPD_RAND_write_file (NULL, bio_err);
>     sk_X509_pop_free(encerts, X509_free);
>     if (cms)
>     	CMS_ContentInfo_free(cms);
>     BIO_free(in);
>     BIO_free_all(out);
> 
>     KWlog ( EV_D_APPL_14 , "smime_encrypt_cms () finished" );
>     return ( bResult );
> }
> 
> When using this function to encrypt an e-mail Thunderbird can decrypt the
> message. But is RSA-OAEP padding really used or is the default padding still
> used? How can I check this?
> 
> For comments I would be very grateful
> 

You can try printing out all the fields of the message with:

	openssl cms -cmsout -noout -print

Near the top you should see: 

        keyEncryptionAlgorithm: 
          algorithm: rsaesOaep (1.2.840.113549.1.1.7)

while the default padding give:

        keyEncryptionAlgorithm: 
          algorithm: rsaEncryption (1.2.840.113549.1.1.1)

Steve.
--
Dr Stephen N. Henson. OpenSSL project core developer.
Commercial tech support now available see: http://www.openssl.org