How to distinguish between encrypted/unencrypted key in a PEM file

[Viktor Dukhovni]

> On 26 Jan 2022, at 9:14 am, Bartlomiej <bartek at conclusive.pl> wrote:
> 
> I have a PEM file on the device which can contain an encrypted/non-encrypted private key. When it's encrypted, it's using PBES/PBKDF. The file is accessed from a C++ application which uses the OpenSSL library. If the key is encrypted, then it should be PKCS#8, but checking it is PKCS#8 by using e.g. `PEM_read_PKCS8` is not enough to confirm it is actually encrypted, since an unencrypted key can also be stored as PKCS#8. Is there a way to check whether the key is encrypted or not using OpenSSL APIs?

If nobody else can suggest anything better, and without an exhaustive
check for higher-level alternatives, I can suggest the low-level type-
agnostic PEM_read_bio(3) that reads a PEM header and data, leaving it
up to you to interpret the data as you want, based on the PEM header.

For example:

  https://github.com/vdukhovni/postfix/blob/master/postfix/src/tls/tls_dane.c#L1189-L1219

In the case of PKCS8, you'd be looking for:

  openssl/pem.h:# define PEM_STRING_PKCS8 "ENCRYPTED PRIVATE KEY"

as opposed to one of:

  openssl/pem.h:# define PEM_STRING_EVP_PKEY     "ANY PRIVATE KEY"
  openssl/pem.h:# define PEM_STRING_RSA          "RSA PRIVATE KEY"
  openssl/pem.h:# define PEM_STRING_DSA          "DSA PRIVATE KEY"
  openssl/pem.h:# define PEM_STRING_PKCS8INF     "PRIVATE KEY"
  openssl/pem.h:# define PEM_STRING_ECPRIVATEKEY "EC PRIVATE KEY"

-- 
	Viktor.