OpenSSL 1.0.2k EVP_PKEY decryption failing

[Viken Kondakji]

Hello,

I am new to openssl.

My objective is to encrypt data using RSA public key, with SHA-256 as hash function and MGF1 as mask generating function.
To do that, I am using the EVP_PKEY API.
I seem to be able to encrypt, but when I try to decrypt, I fail, with
                                                    EVP_PKEY_decrypt error:040A1079:rsa routines:RSA_padding_check_PKCS1_OAEP_mgf1:oaep decoding error

My environ:
$ openssl version -a
OpenSSL 1.0.2k-fips  26 Jan 2017
built on: reproducible build, date unspecified
platform: linux-x86_64
options:  bn(64,64) md2(int) rc4(16x,int) des(idx,cisc,16,int) idea(int) blowfish(idx)
compiler: gcc -I. -I.. -I../include  -fPIC -DOPENSSL_PIC -DZLIB -DOPENSSL_THREADS -D_REENTRANT -DDSO_DLFCN -DHAVE_DLFCN_H -DKRB5_MIT -m64 -DL_ENDIAN -Wall -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches   -m64 -mtune=generic -Wa,--noexecstack -DPURIFY -DOPENSSL_IA32_SSE2 -DOPENSSL_BN_ASM_MONT -DOPE
NSSL_BN_ASM_MONT5 -DOPENSSL_BN_ASM_GF2m -DRC4_ASM -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DMD5_ASM -DAES_ASM -DVPAES_ASM -DBSAES_ASM -DWHIRLPOOL_ASM -DGHASH_ASM -DECP_NISTZ256_ASM
OPENSSLDIR: "/etc/pki/tls"
engines:  dynamic

Code snippet is below.
 
My issue is, after encryption (assuming what I'm doing is correct), how to I go about decrypting using the EVP_PKEY API and the associated RSA private key?
Can someone guide me please?

Thanks
Viken Kondakji



/* =============================Code Used============================================ */
int decrypte(unsigned char*, size_t);

void getError(std::array<char,120>& error) {
  ERR_load_crypto_strings();
  ERR_error_string_n(ERR_get_error(), error.data(), error.size());
 }

int padding = RSA_PKCS1_OAEP_PADDING;

EVP_PKEY * createEvpKeyFromFile(const char* theFile, bool publicKey) {
  std::array<char, 120> error;

  BIO *bioFile = BIO_new(BIO_s_file());
  if (!BIO_read_filename(bioFile, theFile))
  {
    BIO_free(bioFile);
    getError(error);
    throw std::runtime_error(std::string("Failed to read the file ") +
        theFile + " " + std::string(error.data()));
  }

  EVP_PKEY *evpKey= nullptr;
  evpKey = publicKey ? PEM_read_bio_PUBKEY(bioFile, NULL, NULL, NULL)
                     : PEM_read_bio_PrivateKey(bioFile, NULL, NULL, NULL);
  BIO_free(bioFile);
  if(!evpKey)
  {
    getError(error);
    throw std::runtime_error(std::string("Failed to create EVP_KEY from the file ")
        + theFile + ' ' + std::string(error.data()));
  }
  return evpKey;
}

int main(int argc, char* argv[])
{
  unsigned char plainText[] = "Hello World! Nice to meet you. How are you?";

  EVP_PKEY * evpKey = createEvpKeyFromFile("public2048bitsKey.pem", 1);
  
  std::array<char, 120> error;

  EVP_PKEY_CTX* ctx = EVP_PKEY_CTX_new(evpKey, NULL);
  if (nullptr == ctx)
  {
    getError(error);
    throw std::runtime_error(std::string("Failed in EVP_PKEY_CTX_new ") + std::string(error.data()));
  }

  if (EVP_PKEY_encrypt_init(ctx) <= 0)
  {
    getError(error);
    throw std::runtime_error(std::string("EVP_PKEY_encrypt_init ") + std::string(error.data()));
  }

  if (EVP_PKEY_CTX_set_rsa_padding(ctx, RSA_PKCS1_OAEP_PADDING) <= 0)
  {
    getError(error);
    throw std::runtime_error(std::string("EVP_PKEY_CTX_set_rsa_padding ") + std::string(error.data()));
  }

  if (EVP_PKEY_CTX_set_rsa_oaep_md(ctx, EVP_sha256()) <=0 )
  {
    getError(error);
    throw std::runtime_error(std::string("EVP_PKEY_CTX_set_rsa_oaep_md ") + std::string(error.data()));
  }

  if (EVP_PKEY_CTX_set_rsa_mgf1_md(ctx, EVP_sha256()) <=0 )
  {
    getError(error);
    throw std::runtime_error(std::string("EVP_PKEY_CTX_set_rsa_mgf1_md ") + std::string(error.data()));
  }

  // Determine encryption buffer length
  size_t encryptedLen {};
  if (EVP_PKEY_encrypt(ctx, NULL, &encryptedLen, plainText, sizeof(plainText)) <= 0)
  {
    getError(error);
    throw std::runtime_error(std::string("Determine encryption length ") + std::string(error.data()));
  }
  std::cout << "bytes needed to encrypt = " << encryptedLen << '\n';
  unsigned char* encrypted = (unsigned char*) OPENSSL_malloc(encryptedLen);
  if (!encrypted)
  {
    throw std::runtime_error("OPENSSL_malloc failure");
  }

  if (EVP_PKEY_encrypt(ctx, encrypted, &encryptedLen, plainText, sizeof(plainText)) <= 0)
  {
    getError(error);
    throw std::runtime_error(std::string("EVP_PKEY_encrypt failed ") + std::string(error.data()));
  }

  int base64LenNeeded = std::ceil(encryptedLen/3.0) * 4;
  std::cout << "base64LenNeeded = " << base64LenNeeded << '\n';

  unsigned char base64Encoded[8196];
  size_t base64EncodedLength = EVP_EncodeBlock(base64Encoded, (unsigned char *)encrypted, encryptedLen);
  std::cout << "base64EncodedLength = " << base64EncodedLength << '\n';
  std::cout << "base64Encoded\n" << base64Encoded << '\n';

  OPENSSL_free(encrypted);

  EVP_PKEY_CTX_free(ctx);

/* encryption seems to be fine. Let's check our work and attempt to decrypt... */

//When I call the below, I gey  EVP_PKEY_decrypt error:040A1079:rsa routines:RSA_padding_check_PKCS1_OAEP_mgf1:oaep decoding error
  decrypte(encrypted, encryptedLen);               
  return 0;
}

int decrypte(unsigned char* encrypted, size_t encryptedLen) {
  EVP_PKEY * evpKey = createEvpKeyFromFile("private2048bitsKey.pem", 0);

  std::array<char, 120> error;

  EVP_PKEY_CTX* ctx = EVP_PKEY_CTX_new(evpKey, NULL);
  if (nullptr == ctx)
  {
    getError(error);
    throw std::runtime_error(std::string("Failed in EVP_PKEY_CTX_new ") + std::string(error.data()));
  }

  // initializes a public key algorithm context
  if (EVP_PKEY_decrypt_init(ctx) <= 0)
  {
    getError(error);
    throw std::runtime_error(std::string("EVP_PKEY_decrypt_init ") + std::string(error.data()));
  }

  if (EVP_PKEY_CTX_set_rsa_padding(ctx, RSA_PKCS1_OAEP_PADDING) <= 0)
  {
    getError(error);
    throw std::runtime_error(std::string("EVP_PKEY_CTX_set_rsa_padding ") + std::string(error.data()));
  }

  if (EVP_PKEY_CTX_set_rsa_oaep_md(ctx, EVP_sha256()) <=0 )
  {
    getError(error);
    throw std::runtime_error(std::string("EVP_PKEY_CTX_set_rsa_oaep_md ") + std::string(error.data()));
  }

  if (EVP_PKEY_CTX_set_rsa_mgf1_md(ctx, EVP_sha256()) <=0 )
  {
    getError(error);
    throw std::runtime_error(std::string("EVP_PKEY_CTX_set_rsa_mgf1_md ") + std::string(error.data()));
  }

  // determine buffer length
  size_t outLen {};
  size_t privateKeyLen = strlen((const char*)privateKey);
  if (EVP_PKEY_decrypt(ctx, NULL, &outLen, privateKey, privateKeyLen) <= 0)
  {
    getError(error);
    throw std::runtime_error(std::string("determine buffer length in decrypt ") + std::string(error.data()));
  }
  std::cout << "outLen = " << outLen << '\n';
  unsigned char* out = (unsigned char*)OPENSSL_malloc(outLen);
  if(!out)
  {
    throw std::runtime_error(std::string("malloc failed in decrypt ") + std::string(error.data()));
  }

  if (EVP_PKEY_decrypt(ctx, out, &outLen, encrypted, encryptedLen) <= 0 )  /* I fail in this call */
  {
    getError(error);
    std::ostringstream os;
    os << "at line #" << __LINE__ << "...EVP_PKEY_decrypt " << error.data();
    throw std::runtime_error(os.str());
  }

  return 0;
}

****Disclaimer****This message, including all attachments, is private and confidential, may contain proprietary or privileged information and material and is intended solely for use by the named addressee(s). If you receive this transmission in error, please immediately notify the sender and destroy this message in its entirety, whether in electronic or hard copy format. Any unauthorized use (and reliance thereon), copying, disclosure, retention, or distribution of this transmission or the material herein is forbidden. We reserve the right to retain, monitor, intercept and archive electronic communications. This message does not constitute an offer or solicitation with respect to the purchase or sale of any security. It should not be construed to contain any recommendation regarding any security or strategy unless expressly stated therein. Any reference to the terms of executed transactions should be treated as preliminary only and subject to formal written confirmation. Any views expressed are those of the individual sender, except where the message states otherwise and the sender is authorized to state them to be the views of any such entity. This message is provided on an “as is” basis. It contains material that is owned by Instinet Incorporated, its subsidiaries or its or their licensors, and may not, in whole or in part, be (i) copied, photocopied or duplicated in any form, by any means, or (ii) redistributed, posted, published, excerpted, or quoted without Instinet Incorporated's prior written consent. No confidentiality or privilege is waived or lost by any mistransmission of this message. Instinet, LLC (member SIPC) and Instinet Canada Limited, Member – Canadian Investor Protection Fund are subsidiaries of Instinet Incorporated that are locally registered or otherwise authorized to provide securities brokerage products and services. Please refer to the following links for additional disclosures and disclaimers that apply to this message. In the United States: http://instinet.com/docs/legal/le_disclaimers.html. In Canada: http://www.instinet.com/docs/legal/le_ca_disclosures.html. Canadian residents: You are receiving this electronic communication because of your existing relationship with Instinet Canada Limited or an authorized affiliate. Canadian residents who wish to unsubscribe from commercial electronic messages: please e-mail iclcompliance at instinet.com. Please note that you will continue to receive non-commercial electronic messages, such as account statements, invoices, client communications, and other similar factual electronic communications.
For recipients of Wolfe research, please see https://www.nomuranow.com/research/globalresearchportal/pages/disclosures/disclosures.aspx. for important research-related disclosures.