Public-key based authentication of clients

[Jeremy Friesner]

Hi All,

I have a simple C++ client/server application, in which the clients use OpenSSL's PSK (Pre-Shared-Key) mechanism to log in to the server via TLS -- i.e. the client GUI prompts the user to enter a username and password, and when the client connects, it calls SSL_set_psk_client_callback(), and the server calls SSL_set_psk_server_callback(), and OpenSSL does its thing, and if the client's username and password matches the ones the server is expecting, the TLS connection is allowed, otherwise it is rejected.

That all works great; however, what I'd like to do now (mostly just to see if it can be done) is add an alternative, keypair-based mechanism, similar to what github (and I imagine many other web services) provide.

I imagine it would work like this: The user runs the openssl app to create a private/public keypair, and (by some external mechanism) gives the public key-file to the server, and the private key-file to the client. Then, when the client program connects to the server, it has to prove to the server (via cryptographic math) that it has the private-key file in its possession, and the server uses the public-key to verify that proof, before deciding whether to let the client continue or not.

This seems like it should be pretty straightforward, but I haven't been able to find any clear documentation or examples of how to implement it using the OpenSSL library.  Is it possible to implement this behavior using OpenSSL, and if so, how might I go about it?

Thanks,
Jeremy