CAPI engine seems to break server validation

[Brett Stahlman]

On Mon, Oct 26, 2020 at 9:35 AM Jakob Bohm <jb-openssl at wisemo.com> wrote:

> On 2020-10-24 16:09, Brett Stahlman wrote:
> > Jakob,
> > I don't really understand why the engine *needs* to do PSS. Neither of
> > the badssl certificates seem to use it for signatures. (I'm assuming the
> > fact that a cert was signed with RSA-PSS would show up in the Windows
> > certificate viewer...) If you could give a short summary of the problem
> > as you understand it, perhaps it would help me narrow in on a
> > workaround. I'd be happy with even an ugly patch at this point. Given
> > that server verification works fine with a ca-bundle file, I wonder
> > whether it would be possible to have the capi engine handle only the
> > client authentication. As you understand it, would the problem breaking
> > server verification also preclude client authentication with the capi
> > engine?
> >
>
>  From the content of your mails, I inferred that whatever you tried to
> do caused OpenSSL to attempt to generate PSS signatures, but failing to
> pass that job to the CAPI engine.  I was commenting on how that might be
> made to work.
>
> The failure was occurring during server cert verification. Which was
surprising because I'd provided a ca-bundle.crt file to be used for server
authentication, and CAPI doesn't seem to get involved directly in server
authentication anyways. But apparently, the presence of the CAPI engine was
preventing openssl from validating the server cert the way it normally
would. The error occurred in do_sigver_init (crypto/evp/m_sigver.c) at the
`ERR_raise` below:

```
        /*
         * If we couldn't find a keymgmt at all try legacy.
         * TODO(3.0): Once all legacy algorithms (SM2, HMAC etc) have
provider
         * based implementations this fallback shouldn't be necessary.
Either
         * we have an ENGINE based implementation (in which case we should
have
         * already fallen back in the test above here), or we don't have the
         * provider based implementation loaded (in which case this is an
         * application config error)
         */
        if (locpctx->keymgmt == NULL)
            goto legacy;
        ERR_raise(ERR_LIB_EVP, EVP_R_INITIALIZATION_ERROR);
        goto err;
```
I was able to get past the error simply by commenting the conditional
several lines higher in the function:
//if (evp_pkey_ctx_is_legacy(locpctx))
    goto legacy;

For some reason, the call to evp_pkey_export_to_provider() fails when the
CAPI engine is in use, but forcing the jump to legacy bypasses the test
altogether and permits me to use the (working) legacy logic for server cert
validation. I need to debug further to understand why
evp_pkey_export_to_provider() is failing, but the simple change shown above
allows two-way authentication to work with the client.badssl.com test site:
client auth uses a cert in my Personal "MY" store; server auth uses the CA
certs in ca-bundle.cer.

Any light anyone can shed on the history of the legacy fallback logic would
be greatly appreciated...

Thanks,
Brett S.



> > On Fri, Oct 23, 2020 at 11:34 AM Jakob Bohm via openssl-users
> > <openssl-users at openssl.org <mailto:openssl-users at openssl.org>> wrote:
> >
> >     On 2020-10-23 15:45, Matt Caswell wrote:
> >      >
> >      > On 23/10/2020 14:10, Brett Stahlman wrote:
> >      >> It seems that the CAPI engine is breaking the server
> >     verification somehow.
> >      >> Note that the only reason I'm using the ca-bundle.crt is that I
> >     couldn't
> >      >> figure out how to get CAPI to load the Windows "ROOT" certificate
> >      >> store, which contains the requisite CA certs. Ideally, server
> >      >> authentication would use the CA certs in the Windows "ROOT"
> >     store, and
> >      >> client authentication would use the certs in the Windows "MY"
> >     store, but
> >      >> CAPI doesn't appear to be loading either one.
> >      > This is probably the following issue:
> >      >
> >      > https://github.com/openssl/openssl/issues/8872
> >      >
> >      > Matt
> >     Looking at the brutal wontfixing of that bug, maybe reconsider if the
> >     existing engine interface can do PSS by simply having the CAPI/CAPIng
> >     engine export the generic PKEY type for PSS-capable RSA keys.  Also,
> >     maybe use a compatible stronger CAPI "provider" (their engines) to do
> >     stronger hashes etc.
> >
> >
>
>
> Enjoy
>
> Jakob
> --
> Jakob Bohm, CIO, Partner, WiseMo A/S.  https://www.wisemo.com
> Transformervej 29, 2860 Søborg, Denmark.  Direct +45 31 13 16 10
> This public discussion message is non-binding and may contain errors.
> WiseMo - Remote Service Management for PCs, Phones and Embedded
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mta.openssl.org/pipermail/openssl-users/attachments/20201026/8e427536/attachment.html>