Fwd: Proper API usage with DTLS over custom net transport

[Павел Балашов]

Hello,

I would be really grateful if someone can point me in the right direction
with proper OpenSSL API usage in the following scenario.
I have a custom network transport - ICE (essentially UDP socket, as a part
of typical WebRTC stack) and I need to implement a DTLS connection over it.
So given some hints from the internet I assume that using BIO_s_mem is a
tool of choice in this situation.
Suppose SSL_CTX, SSL, BIOs are created and ready.
Overall my current understanding of API usage is somewhere like this (very
broad strokes, kind of pseudocode):

if (client)  SSL_set_connect_state();
if (server) SSL_set_accept_state();
while (true)
{
    // On initial iteration this would make OpenSSL put some data for us to
read from out_BIO to send it over
    // if we are client, and don't put anything if we are server
    // On following iterations pretty much the same behaviour remains.
    SSL_do_handshake();

    if (out_BIO_has_data) send_data_from_out_BIO_to_net();

    // So at some loop iteration on line above we've sent everything
OpenSSL wanted us to send and
    // OpenSSL tells us handshake is done, so we quit the loop
    if (SSL_is_init_finished()) break;

    // We've sent some flight of DTLS handshake, but there could be need to
retransmit
    // we ask for retransmit timeout
    timeout = DTLSv1_get_timeout();

    // "select"-like machinery
    wait_for_timeout_or_socket_data();

    if (select_resulted_in_timeout)
    {
        // I assume this is going to generate DTLS-protocol-wise correct
retransmission flight data in out_BIO
        DTLSv1_handle_timeout();
        continue;
    }

    // socket_data that is going to be put in in_BIO is about to be
processed by OpenSSL internals in next SSL_do_handshake() on next loop iter
above
    if (select_resulted_in_socket_data) put_socket_data_to_in_BIO();
}

// So here I assume as client we've got last flight from server and ready
to send data
// And if we are a server we've sent out last flight, but we don't know
whether it has got through
// So if everything more or less correct up to this point we can expect
multiplexed traffic on ICE "socket": SRTP, SRTCP, and DTLS
// DeMultiplexing is not a problem (some RFCs cover this)
while (true)
{
    sock_data = read_socket_data();
    if (is_rtp(sock_data)) ...;
    if (is_rtcp(sock_data)) ...;
    if (is_dtls(sock_data)) ????
}

So now the questions:
(1) If we receive some dtls data at the line above with '????' what should
we do in terms of OpenSSL API calls ?  I assume this dtls data could be a
client's retransmission due to server's last flight was lost or this could
be client receiving server's last flight duplicated (theoretically
could happen as long as lower layer protocol is UDP) or this could be
DTLS-encrypted real-app data or this could be DTLS-renegotiation, this also
could be a DTLS shutdown alert and anything else DTSL-related. What is the
supposed way of inferring and reacting to those different events with API ?
(2) Is the whole usage of OpenSSL even right for this scenario - maybe the
structure and sequence of API calls should be rearranged somehow ?
(3) There is an option to pass custom info_callback
with SSL_CTX_set_info_callback(). Would there be a proper usage of
this kind of callback in this scenario ?

Any other input, links to any kind of relevant supplemental material is
really appreciated.

Thanks a lot for reading, very special thanks to authors and maintainers
for all the hard work on this project.
---
Regards,
Pavel.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mta.openssl.org/pipermail/openssl-users/attachments/20221020/381f04a9/attachment.htm>