[openssl-users] Troubleshooting failed TLS negotiation

[Graham Allan]

I'm moving samba service between a couple of FreeBSD systems (9.3 to 
10.2), and I'm stuck on getting samba on the new machine to connect to 
our openldap server over ssl - frustrating since I've been running 
samba+ldap for 15 years or so; feel sure I'm missing something basic!

The smbd-to-ldap connection works fine with no encryption, but I get 
errors when using either TLS to port 389 ("Failed to issue the StartTLS 
instruction: Connect error"), or for SSL to 636 I get:

failed to bind to server ldaps://ldap-fqdn with dn="cn=admin,dc=..." 
Error: Can't contact LDAP server
error:0D0C5006:asn1 encoding routines:ASN1_item_verify:EVP lib

I'm pretty certain it's not a certificate or CA validation issue. All my 
other ldap clients on that server are working as expected, including a 
simple "ldapsearch -ZZ"; and openssl s_client is happy connecting to the 
ldaps port. I tested different settings in openldap's ldap.conf, eg 
using TLS_CACERTDIR vs TLS_CACERT and different values of TLS_REQCERT; 
all seem to work equally well for ldapsearch (and equally badly for smbd).

Capturing the packet exchange between smbd and slapd, I'm seeing the 
(smbd) client sends a "decrypt error" (TLS alert code 51) to the ldap 
server after receiving the certificate, while the working "ldapsearch 
-ZZ" moves on to client key exchange etc.

The biggest difference I can think of between the working and 
non-working systems is the openssl version (FreeBSD 10.2 uses 1.0.1p 
while 9.3 uses 0.9.8zd - the ldap server is using the latter). However 
that doesn't explain all my other 10.x ldap/ssl clients working 
successfully...

It sounds a bit like this posting from couple of years ago (which I 
unfortunately couldn't see any resolution to):
http://comments.gmane.org/gmane.comp.encryption.openssl.user/49142

I'm not sure where to try looking next for the cause, would welcome any 
suggestions...

Thanks, Graham