[openssl-project] To use or not use the iconv API, and to use or not use other libraries

[Richard Levitte]

In message <333784C8-4870-4DDB-A892-13D552724222 at dukhovni.org> on Tue, 12 Jun 2018 16:02:16 -0400, Viktor Dukhovni <openssl-users at dukhovni.org> said:

openssl-users> 
openssl-users> 
openssl-users> > On Jun 12, 2018, at 3:39 PM, Richard Levitte <levitte at openssl.org> wrote:
openssl-users> > 
openssl-users> >> The flags I'd like to see are:
openssl-users> >> 
openssl-users> >>   -latin1:  Passphrase is a stream of octets, each of which is a single unicode
openssl-users> >>             character in the range 0-255.
openssl-users> > 
openssl-users> > I would prefer to call it -binary or something like that...  it
openssl-users> > certainly comes down to the same thing in practice, and should
openssl-users> > translate exactly to the pre-1.1.0 behaviour.
openssl-users> 
openssl-users> I won't quibble over the name.
openssl-users> 
openssl-users> > 
openssl-users> >>   -utf8:    Passphrase is already utf-8 encoded
openssl-users> >> 
openssl-users> >>   -ascii:   Passphrase must be ASCII, reject inadvertent 8-bit input.
openssl-users> > 
openssl-users> > ... and if none of these are given?
openssl-users> 
openssl-users> Not sure.  We could opt for "-binary" by default, which is backwards
openssl-users> compatible, but it produces non-standard outputs, which is a disfavour
openssl-users> to new users.  We could go with "-ascii" as a default, forcing failure
openssl-users> for non-ascii passwords without an explicit indication of encoding.
openssl-users> The second seems more appealing to me.

Same here.

openssl-users> >> And as available:
openssl-users> >> 
openssl-users> >>   -toutf8:   Convert passphrase from the input encoding to UTF-8.
openssl-users> >> 	     Either using the locale-specific encoding, or yet
openssl-users> >>              another flag:
openssl-users> >> 
openssl-users> >>   -encoding: A platform-specific name for the input encoding understood
openssl-users> >>              by the system's encoding conversion library (iconv on Unix).
openssl-users> > 
openssl-users> > If the availability of -toutf8 depends on the presumed presence of
openssl-users> > iconv(), then we can assume that nl_langinfo() is present as well.
openssl-users> > That renders -encoding unnecessary, unless you want to use it to
openssl-users> > override the locale-specific encoding.
openssl-users> 
openssl-users> The purpose is specifically to override the encoding when it is wrong
openssl-users> for some reason.  The iconv library takes the empty string as the
openssl-users> locale-specific encoding, so we should not need nl_langinfo(), unless
openssl-users> that's known to produce better results.

Some implementations of the iconv library take the empty string as
the locale-specific encoding, but that is in no way universal, and
isn't specified in the standard:

http://pubs.opengroup.org/onlinepubs/009695399/functions/iconv_open.html

Using nl_langinfo() to get the locale-specific encoding will, as far
as I know, always get you what you expect.

Cheers,
Richard

-- 
Richard Levitte         levitte at openssl.org
OpenSSL Project         http://www.openssl.org/~levitte/