TLS handshake fails ("SSL_accept:error in error") for server->server connection (smtp submit dovecot->postfix) if /etc/pki/tls/openssl.cnf "Options=" includes 'ServerPreference' ?

[Viktor Dukhovni]

On Thu, Sep 24, 2020 at 06:43:05PM -0700, PGNet Dev wrote:

> Been awhile since I 'de-noised' a comms dump; I'll dust off my notes, & work on getting a useful/relevant PCAP file ...

    # tcpdump -s0 -w /some/file tcp port 12345
    <start delivery that will fail, hit ^C once that happens>
    ^C

    # tcpdump -r /some/file 'tcp[13] & 0x12 == 2'
    <prints all connection initiating SYN packets, note the client's source port>

    # tcpdump -r /some/file -s0 -w session.pcap tcp port <theport>

    # tcpdump -r session.pcap 
    <should contain one short session from 3-way SYN to 3-way FIN and/or RST>

See also:

    https://www.spinics.net/lists/openssl-users/msg05623.html

for notes on using "tshark" to extract the detailed protocol
diagnostics.  These can be somewhat disappointing with TLS 1.3,
because privacy...(most of the handshake is encrypted).

> Version is
> 
>  openssl version
>   OpenSSL 1.1.1g FIPS  21 Apr 2020

Looking at the upstream OpenSSL source, the first thing to note is that
the prefer chacha setting has no effect at all, unless you also have
server preference set (which you can do with Postfix settings, rather
than globally in the config file).

Secondly, the effect of "prefer chacha" is to just synthesize a
transient ordered list of server cipher preferences that moves any
cha-cha ciphers to the top, cipher selection then continues as usual.

So your reported symptoms re protocol version mismatch look rather
perplexing, don't know what the client is doing, and whether the client
is even linked with OpenSSL?  Is dovecot using OpenSSL or GnuTLS?

-- 
    Viktor.