[openssl-users] Differences in openssl 0.9.8 and 1.0.1x for private pem key file

[Dave Thompson]

> From: openssl-users On Behalf Of Jaya Nageswar
> Sent: Tuesday, December 30, 2014 02:36

> ... the output [is] different between openssl 0.9.8 and 1.0.1x versions as the following methods 
> are being used in the code flow for the method PEM_write_bio_PrivateKey.
> 1.0.1x - PEM_write_bio_PKCS8PrivateKey
> 0.9.8 - PEM_ASN1_write_bio((i2d_of_void *)i2d_PrivateKey,...)

Yes. To be complete, it's 0.9.8anything versus 1.0.0anything OR 1.0.1anything.

> 1. As I mentioned earlier, We have a sample application where we try to read 
> a  sample pem key file, create an EVP_PKEY indirectly using PEM_read_bio_PrivateKey 
> and try to create pem key files encrypted using different ciphers like (RC2, RC4    etc.) 
> using the method PEM_write_bio_PrivateKey. I am getting a different output in 1.0.1x 
> while using the cipher RC2 by using the method PEM_write_bio_PrivateKey.That is 
> understandable as we use PKCS8 in 1.0.1x. However if I try to use the cipher RC4 
> for encyrption,PEM_write_bio_PKCS8PrivateKey is failing.Is there a known issue or a bug for RC4.  

I don't see anything in RT (the bug tracker) but yes privatekey encryption doesn't work 
for RC4, apparently because it's a stream cipher with no IV. The symptoms vary:

- writing PKCS8 encrypted gives an error, in either DER or PEM (PKCS8 is encrypted 
in the DER, the PEM just base64's it). In 1.0.0+ PEM_write_PrivateKey maps to 
PEM_write_PKCS8PrivateKey and therefore gets this.

- writing traditional RSA/etc encrypted PEM (which encrypts at the PEM level) 
writes a file and returns success, but that file can't be decrypted because it has no IV.
In 0.9.8 PEM_write_PrivateKey maps to PEM_write_{RSA/etc}PrivateKey and gets this.

- for completeness remember there is no traditional encrypted DER format.

> 2. Also Can I use the method PEM_ASN1_write_bio((i2d_of_void *)i2d_PrivateKey,...) 
> in 1.0.1x instead of the method PEM_write_bio_PrivateKey if I want to have the same output similar to 0.9.8.

It looks like you can, but it's not documented that I can see and looks a bit fragile.

The long-documented way that works on all versions (so far!) is to call the correct 
per-algorithm routine PEM_write_{RSA,DSA,EC}PrivateKey .