Signing using EVP_PKEY_encrypt when using pkcs11 engine

[Martin Townsend]

On Mon, Jun 3, 2019 at 4:35 PM Martin Townsend <mtownsend1973 at gmail.com> wrote:
>
> Hi,
>
> I'm trying to modify the evm/ima utility so that it can use a HSM to
> perform signing.  I've setup SoftHSM and used this to create a
> certificate with an RSA public key pair.  The evmctl code creates the
> hash and then calls a function to perform the sign operation which
> ends up calling
> len = RSA_private_encrypt(size + asn1->size, buf, hdr->sig,
>   key, RSA_PKCS1_PADDING);
>
> My idea was to keep the hash calculation as is, and replace the
> RSA_private_encrypt with code that uses the private key in the HSM to
> encrypt the hash buffer that has been calculated.
>
> My initialisation looks like this
>     /* Load the configuration using OPENSSL_CONF environment variable */
>     OPENSSL_config(NULL);
>     /* Try and load PKCS11 engine */
>     const char* s = getenv("OPENSSL_CONF");
>     printf("Trying to load pkcs#11 engine\n");
>     printf("OPENSSL_CONF=%s\n", s);
>     pkcs_engine = ENGINE_by_id("pkcs11");
>     if (!pkcs_engine) {
>         printf("PKCS#11 engine not found, not using HSM\n");
>     } else {
>         int rv = ENGINE_init(pkcs_engine);
>
>         if (!rv) {
>             fprintf(stderr, "PKCS#11 could not be initialised\n");
>             ENGINE_free(pkcs_engine);
>             pkcs_engine = NULL;
>         }
>
>         ENGINE_set_default(pkcs_engine, ENGINE_METHOD_ALL);
>     }
>
>     OpenSSL_add_all_algorithms();
>     OpenSSL_add_all_digests();
>     ERR_load_crypto_strings();
>
> and then I load the private key with
>
>     key = ENGINE_load_private_key(pkcs_engine, keyid, UI_OpenSSL(), NULL);
>     if (!key) {
>         log_err("%s: Failed to load private key with id: %s\n", keyid,
> __func__);
>         ERR_print_errors_fp(stderr);
>     }
>
>
> and then use the following to perform the encryption
>
>         /* Create context */
>         hsm_key_ctx = EVP_PKEY_CTX_new(hsm_key, NULL);
>         if (hsm_key_ctx == NULL) {
>             log_err("sign_hash_v2: failed to create context\n");
>             ERR_print_errors_fp(stderr);
>             return -1;
>         }
>         rv = EVP_PKEY_encrypt_init(hsm_key_ctx);
>         if (rv <= 0) {
>             log_err("sign_hash_v2: failed to init encrypt (rv=%d\n", rv);
>             ERR_print_errors_fp(stderr);
>             EVP_PKEY_CTX_free(hsm_key_ctx);
>             return -1;
>         }
>         /* TODO: What padding??? RSA_PKCS1_PSS_PADDING?? */
>         rv = EVP_PKEY_CTX_set_rsa_padding(hsm_key_ctx, RSA_PKCS1_PADDING);
>         if (rv <= 0) {
>             log_err("sign_hash_v2: failed to set RSA_PKCS1_PADDING
> (rv=%d\n", rv);
>             ERR_print_errors_fp(stderr);
>             EVP_PKEY_CTX_free(hsm_key_ctx);
>             return -1;
>         }
>         if (rv <= 0) {
>             log_err("sign_hash_v2: failed to set RSA_PKCS1_PADDING
> (rv=%d\n", rv);
>             ERR_print_errors_fp(stderr);
>             EVP_PKEY_CTX_free(hsm_key_ctx);
>             return -1;
>         }
>
>         /* Create signature */
>         outlen = 0;
>         rv = EVP_PKEY_encrypt(hsm_key_ctx, NULL, &outlen, buf, size +
> asn1->size);
>         if ((rv <= 0) || (outlen == 0)) {
>             log_err("sign_hash_v2: failed to learn needed output buf
> len (rv=%d)\n", rv);
>             ERR_print_errors_fp(stderr);
>             EVP_PKEY_CTX_free(hsm_key_ctx);
>             return -1;
>         }
>         /*outlen = 256;*/
>         log_info("EVP_PKEY_encrypt: outlen: %lu\n", outlen);
>         rv = EVP_PKEY_encrypt(hsm_key_ctx, hdr->sig, &outlen, buf,
> size + asn1->size);
>         if (rv <= 0) {
>             log_err("sign_hash_v2: EVP_PKEY_encrypt() failed (rv=%d)\n", rv);
>             ERR_print_errors_fp(stderr);
>             EVP_PKEY_CTX_free(hsm_key_ctx);
>             return -1;
>         }
>
> But I find that when I create a signature for a particular file it's
> always different with each invocation of the utility. If I try and
> verify it using the associated public key I get:
>
> RSA_public_decrypt() failed: -1
> errno: No data available (61)
> error:0407006A:rsa routines:RSA_padding_check_PKCS1_type_1:block type is not 01
> error:04067072:rsa routines:RSA_EAY_PUBLIC_DECRYPT:padding check failed
>
> Is there something I am missing in my code above?  I tried setting the
> Engine in EVP_PKEY_CTX_new but get:
> sign_hash_v2: failed to create context
> 140174165591744:error:260C0065:engine
> routines:ENGINE_get_pkey_meth:unimplemented public key
> method:tb_pkmeth.c:128:
> 140174165591744:error:0609D09C:digital envelope
> routines:INT_CTX_NEW:unsupported algorithm:pmeth_lib.c:166:
> errno: Invalid argument (22)
>
> Any help appreciated,
> Martin.

I switched over to using the Cryptoki API of SoftHSMv2 and encryption
using the private key isn't allowed (I get
CKR_KEY_FUNCTION_NOT_PERMITTED error as I'm guessing the CKA_ENCRYPT
flag isn't set) which makes sense as you don't usually encrypt with a
private key but this means that I can't do what I wanted with evmctl
where the hash has already been created and just needs encrypting.  I
will take a look modifying evmctl to use the Signing function of
SoftHSM via C_Sign.