[openssl-dev] [openssl.org #3887] PATCH: rsautl and intelligent retry for Public Key parse after Traditional/Subject Public Key Info parse fails

[noloader@gmail.com via RT]

On Sun, May 31, 2015 at 12:27 PM, Richard Levitte via RT <rt at openssl.org> wrote:
> Nice idea, I'm however thinking that much of the trying different formats could
> be moved to load_key / load_pubkey, all that would be needed is a keyformat
> denoting "try anything". -1, perhaps?
>

I like the idea, and I was entertaining it. The goal here ways to
improve rsautl, so I deferred.

There are some potential issues with moving the logic into load_key
and load_pubkey. The first issue is the change in behavior utility
wide. That is, it will affect most (all?) utilities. I think its a
good idea, but others may not.

The second issue (related to the first), is the rsa utility has tried
to address these issue already. So there's a few other
switches/options for the rsa utility. For example, RSAPublicKey_in and
RSAPublicKey_out. This may or may not be an issue.

The third issue (related to the first), is that other similar
functions, like load_certificate, may not get the change. This may or
may not matter, and may or may not be an issue.

The fourth issue is the cracker. Effectively, apps{c|h} only
recognizes PEM, PEMRSA, ASN1, and ASN1RSA (see str2fmt, which appears
to be removed in 1.1.0). So OpenSSL would need a way to identify other
encodings, types and algorithms. For example, a PEM encoded DSA
SubjectPublicKeyInfo or PrivateKeyInfo (traditional key) versus a
ANS.1/DER encoded DSA public key or private key.

**********

I think there's little difference between load_key and load_pubkey.
Fold them together, and call the function that remains load_key and
return the EVP_PKEY.

load_key should take a encoding hint (PEM vs ASN.1, etc), a key type
hint (SPKI/Traditional versus Key-Only, etc), a key pair hint (public
versus private), and a key algorithm hint (RSA, DSA, etc).

I would envision it to be similar to:

    int encoding_hint = -1, key_type_hint = -1, key_pair_hint = -1,
key_alg_hint = -1;
    key_hints(…, &encoding_hint, &key_type_hint, &key_pair_hint, &key_alg_hint);

That's going to lead to an ugly message cracker, but I don't know how
to avoid it.

**********

There is a key_file_format function in apps.c, but its not really
useful. I think this is the keystone to making things work here. It
should peek at the key, and return the tuple {key encoding, key type,
key pair, key algorithm} hints. Its similar to str2fmt, but it works
directly with the key specified by the user rather than command line
arguments.

One of the things I give the user credit for is that they know the
key-file they want to use. They may not know the encoding, format or
type - but they know the filename. So something to peek into the file
and then make intelligent decisions would probably be very helpful to
the user.

I understand key_file_format won't work with stdin. So maybe the
course of action is to peek from a BIO. If its a file BIO, then
everything just works. If its from stdin, then wrap it in a memory bio
so everything works as expected.

**********

Jeff