[openssl-dev] sizeof (HMAC_CTX) changes with update, breaks binary compatibility

[Dan McDonald]

I noticed that a new field was added to HMAC_CTX in the 1.0.2a->b or 1.0.1m->n update:

typedef struct hmac_ctx_st {
   const EVP_MD *md;
   EVP_MD_CTX md_ctx;
   EVP_MD_CTX i_ctx;
   EVP_MD_CTX o_ctx;
   unsigned int key_length;
   unsigned char key[HMAC_MAX_MD_CBLOCK];
+ int key_init;
} HMAC_CTX;

This breaks binary compatibility.  I found this out the hard way during an attempt to update OmniOS's OpenSSL to 1.0.2b ('014, bloody) or 1.0.1n (006, 012).  Observe our use of HMAC_CTX in illumos (which OmniOS is a distribution of):

struct Mac {
        char            *name;
        int             enabled;
        u_int           mac_len;
        u_char          *key;
        u_int           key_len;
        int             type;
        const EVP_MD    *evp_md;
        HMAC_CTX        evp_ctx;
};
struct Comp {
        int     type;
        int     enabled;
        char    *name;
};
struct Newkeys {
        Enc     enc;
        Mac     mac;
        Comp    comp; /* XXX KEBE SAYS THIS GETS CLOBBERED!!! */
};

You can see the code here:

	http://src.illumos.org/source/xref/illumos-gate/usr/src/cmd/ssh/include/kex.h#100

What is supposed to happen in this situation?  I was under the impression that letter releases don't break binary compatibility.  The SSH in illumos breaks because of this, but it appears OpenSSH has worked around such a situation.

Clues are welcome.

Thanks,
Dan McDonald -- OmniOS Engineering