TLS handshake fails ("SSL_accept:error in error") for server->server connection (smtp submit dovecot->postfix) if /etc/pki/tls/openssl.cnf "Options=" includes 'ServerPreference' ?

[Viktor Dukhovni]

On Thu, Sep 24, 2020 at 09:26:26PM -0700, PGNet Dev wrote:

> > It is surprising that the client sent "QUIT<CRLF>" only .14 seconds after SYN,
> > since if it expected to do SMTP STARTLS, it would typically wait for the
> > server greeting for more than a fraction of a second.
> 
> So, iiuc, that's a dovecot faux pas?

No, it is a misconfiguration on your part.  Dovecot, as configured, is
expecting to talk to a standard SUBMIT service (i.e. SMTP + STARTTLS),
but you've configured the server port for TLS wrapper-mode (port 465
SMTP inside implicit TLS).

You have to either configure Dovecot to submit to port 587 (or similar)
that does not do TLS wrapper-mode (implicit TLS).  Or configure it
to use implicit TLS.

I must lodge a complaint on wasting my time here, you intimated that
just changing openssl.cnf makes the difference.  But that is clearly not
the case, because you're testing different server endpoints, with port
60465 for the "working" case, and "465" for the non-working case.

It seems likely that you don't have TLS wrapper mode on port 60465.

-- 
    Viktor.