[openssl-dev] openssl.org #4615 Cache utility behaving strange with X509_LOOKUP_add_dir

Tue Jul 19 20:00:15 UTC 2016

Guys,

I will make things more simpler based on all the explanations that you have given.

A $hash.r0 CRL file has been loaded in the internal store since it was unique (new name) as suggested by you. Currently, this file was present on disk under the CRL directory.
Now, this CRL file has been purged from the CRL directory (now the directory is empty with no CRL files).
As suggested by you, I got a new refreshed CRL file (which has a new list of certificates that have been revoked) from the same issuer. I save this file on disk under the CRL directory with extension same as before i.e $hash.r0. Problem persists again, that openssl does not do a lookup for this newly placed CRL file but only refers to the old CRL file of the issuer from its cache as you have mentioned below:
“Once that file is loaded, it's not been loaded from disk again (as implied by the 'caches'), also not when it changes.”

How to make this new CRL file count which is just a replacement of the old CRL file belonging to the same issuer. We cannot every time reload the store so that the cache refreshes for every new (refreshed/replaced) CRL file.

So, another question linked to this is: Is there any way to clear the store cache every time before the validation begins (so that it may look for disk files and load them since they will always be unique)? Is there any API which does that?

From: openssl-dev [mailto:openssl-dev-bounces at openssl.org] On Behalf Of Patel, Anirudh (Anirudh)
Sent: Tuesday, July 19, 2016 2:42 PM
To: openssl-dev at openssl.org
Subject: Re: [openssl-dev] openssl.org #4615 Cache utility behaving strange with X509_LOOKUP_add_dir

Fine and thanks for all the explanation.

First let me give my scenario again and then I will come to Mischa’s point.

I got your point Richard but, my scenario is slightly different. What you are considering is that both $hash{sub_ca}.r0 and $hash{sub_ca}.r1 are already present in the CRL directory and has been loaded up. There might be issues with this as you mentioned below.

My case is slightly different:

1. At time T1, CRL file $hash{sub_ca}.r0 is present in the CRL directory. Store is loaded up with it and maybe as you said if, no other object with the same hash is already there this will be put in the internal store.

2. Now I get an incoming client connection and for which this CRL file is looked up and everything is fine.

3. At time T2, I removed this CRL file $hash{sub_ca}.r0 from the CRL directory (but remember it is in the internal store/cache).

4. Now I get the same incoming connection from the client as above for which openssl is currently doing a lookup and gives no error as the corresponding CRL file $hash{sub_ca}.r0 was there in the internal

    store/cache. I was expecting an error since I removed the CRL file but I did not and that is how I understood that openssl is keeping the file in the cache.

5. Above client has now been revoked by sub_ca which generated a new CRL.

6. At time T3, I placed the new CRL file as $hash{sub_ca}.r1 under the CRL directory and it is the only one since I have deleted the old one $hash{sub_ca}.r0 from the directory.

7. Again the same client tries to connect and passes away which now becomes a security concern. As per the statement mentioned on the man page, the lookup should have found and considered the new CRL (.r1)

   and not the one in the cache since its sequence number is old.

Now, from what you have just said let me ask you this:
I have .r0 file which has been loaded up in the internal store since, that issuer was not already present. Right, this is what you are saying: ”if the file is new or if that issuer is not already in the store it, it will be loaded”.
Suppose, I remove this file from the disk (.r0) and place a replacement of it as you said (this replaced CRL file has been recently published which has a list of new certificates that have been revoked). I place this file under the CRL directory as .r0 again, but openssl will now again not lookup for this file as the same issuer name file was already present in its internal store/cache. This is a replaced CRL file of an existing file CRL file but how to make it count. We need to start and stop the server again for loading up the store and refreshing the cache. This seems to be lot of work and inefficient.

Hope I have been able to made my point.

From: openssl-dev [mailto:openssl-dev-bounces at openssl.org] On Behalf Of Mischa Salle
Sent: Tuesday, July 19, 2016 2:28 PM
To: openssl-dev at openssl.org<mailto:openssl-dev at openssl.org>
Subject: Re: [openssl-dev] openssl.org #4615 Cache utility behaving strange with X509_LOOKUP_add_dir

Hi,
the point is that they are loaded when a new file appears, meaning, a file with a name not yet present. Once that file is loaded, it's not been loaded from disk again (as implied by the 'caches'), also not when it changes.
The .r1 files are rarely used (only for rekeying of the CA etc., as far as I remember), so normally you would get only replacements of existing files, meaning OpenSSL will get then from its cache.
Note that I don't like this behaviour, but I think it is as intended.
Mischa

On Tue, Jul 19, 2016 at 10:24 AM, Patel, Anirudh (Anirudh) <anirudhp at avaya.com<mailto:anirudhp at avaya.com>> wrote:
Man page for X509_LOOKUP_hash_dir states something different though:

X509_LOOKUP_hash_dir is a more advanced method, which loads certificates and CRLs on demand, and caches them in memory once they are loaded. As of OpenSSL 1.0.0, it also checks for newer CRLs upon each lookup, so that newer CRLs are as soon as they appear in the directory

When checking for new CRLs once one CRL for given hash value is loaded, hash_dir lookup method checks only for certificates with sequence number greater than that of the already cached CRL.

a)      If you look at my scenario, firstly I did not had any CRL files under the CRL directory which has been loaded in the store when my server starts. For the incoming chain as ID/Sub CA/Root CA verify_callback is invoked which gives 3 errors X509_V_ERR_UNABLE_TO_GET_CRL one for each certificate and then follows with further validation.

b)      Then, when I placed the respective CRL files under the directory and I get the same incoming connection (chain: ID/Sub CA/Root CA) openssl verify_callback stops complaining about CRL files not found for the certs. Please note that I did not stop/start the server to load the store with CRL directory again. As stated in the man page, openssl rightly did a lookup and found new CRL files during the handshake.

c)       The problem is when a new CRL file for one the above issuer is placed under the CRL directory (with an incremented sequence number .rN) openssl is not looking at the newer CRL file but only considering the ones in the cache.

Let me know if the manual page description meant something different.

Thanks.

From: openssl-dev [mailto:openssl-dev-bounces at openssl.org<mailto:openssl-dev-bounces at openssl.org>] On Behalf Of Mischa Salle
Sent: Tuesday, July 19, 2016 1:27 PM

To: openssl-dev at openssl.org<mailto:openssl-dev at openssl.org>
Subject: Re: [openssl-dev] openssl.org<https://urldefense.proofpoint.com/v2/url?u=http-3A__openssl.org&d=CwMFaQ&c=BFpWQw8bsuKpl1SgiZH64Q&r=r_yFHjnA3pyorIMQVU-vjyndTmY6-rsuMCBf8EzS6oU&m=NM3FtBB4wlVT4uq5ccJIq8GhBUYhH_xiQVBkJ1QVX_s&s=7-gGMTBKo65sNA3NAdCAT48WVK9-kiJa_d54QmGxVmM&e=> #4615 Cache utility behaving strange with X509_LOOKUP_add_dir

Hi Anirudh,
this is as far as I know a very old issue (at least since 2002 or so). Basically a server needs to restart periodically in order to pick up changed CRLs. There are some workarounds, like forcibly reloading all the CRLs periodically, even those already in the store.
Mischa Salle

On Tue, Jul 19, 2016 at 9:32 AM, Patel, Anirudh (Anirudh) <anirudhp at avaya.com<mailto:anirudhp at avaya.com>> wrote:
It is not re-checking the files (new CRL for the same issuer) in the CRL directory
IssuerHash_YYYY.r0 (old crl for sub-ca)
IssuerHash_YYYY.r1 (new crl for sub-ca) ---> not looked up for an incoming client connection
IssuerXXXX.r0 (old crl for root ca)

I have mentioned the complete scenario in the ticket#4615

-----Original Message-----
From: openssl-dev [mailto:openssl-dev-bounces at openssl.org<mailto:openssl-dev-bounces at openssl.org>] On Behalf Of Salz, Rich
Sent: Tuesday, July 19, 2016 12:55 PM
To: openssl-dev at openssl.org<mailto:openssl-dev at openssl.org>
Subject: Re: [openssl-dev] openssl.org<https://urldefense.proofpoint.com/v2/url?u=http-3A__openssl.org&d=CwMFaQ&c=BFpWQw8bsuKpl1SgiZH64Q&r=r_yFHjnA3pyorIMQVU-vjyndTmY6-rsuMCBf8EzS6oU&m=LGoV-7-LnJ190GCUHmne799_dEEFi6KVc51xhuhtJ10&s=ultotvw_l8QsxQ1Zf_5rbTjOo-JG_iZbExIaV7qPLOs&e=> #4615 Cache utility behaving strange with X509_LOOKUP_add_dir

> I have earlier raised an issue on how openssl is not looking up for newer CRLs in each lookup. The only CRL files it is taking into consideration are the ones present in the cache.

> Could you please provide some inputs on this as I am blocked on the implementation front.

You mean it's not fetching CRL's over the network?  Or re-checking the files?

--
openssl-dev mailing list
To unsubscribe: https://urldefense.proofpoint.com/v2/url?u=https-3A__mta.openssl.org_mailman_listinfo_openssl-2Ddev&d=CwIF-g&c=BFpWQw8bsuKpl1SgiZH64Q&r=r_yFHjnA3pyorIMQVU-vjyndTmY6-rsuMCBf8EzS6oU&m=aetYwxnSuG9CLQakXoaWRTkyEyx2DzRAan4VyAwUF44&s=V6DU-ZDPxeXtjMHdOVafHx4u7EzISeITtikifV3D7gs&e=
--
openssl-dev mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev<https://urldefense.proofpoint.com/v2/url?u=https-3A__mta.openssl.org_mailman_listinfo_openssl-2Ddev&d=CwMFaQ&c=BFpWQw8bsuKpl1SgiZH64Q&r=r_yFHjnA3pyorIMQVU-vjyndTmY6-rsuMCBf8EzS6oU&m=LGoV-7-LnJ190GCUHmne799_dEEFi6KVc51xhuhtJ10&s=F3JmB5AFiuX4fyQGRse7p8_yKB86uPYSSBlCWfApGfM&e=>

--
openssl-dev mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev<https://urldefense.proofpoint.com/v2/url?u=https-3A__mta.openssl.org_mailman_listinfo_openssl-2Ddev&d=CwMFaQ&c=BFpWQw8bsuKpl1SgiZH64Q&r=r_yFHjnA3pyorIMQVU-vjyndTmY6-rsuMCBf8EzS6oU&m=NM3FtBB4wlVT4uq5ccJIq8GhBUYhH_xiQVBkJ1QVX_s&s=gu_2BoTnFPbjE2U9cAYiFYofV66JRKrPpbhxdz3Nbmg&e=>

--
Van Boshuizenstraat 443
1082 AR Amsterdam
The Netherlands
Tel. (+31/0)20-4043782
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mta.openssl.org/pipermail/openssl-dev/attachments/20160719/35e09f4a/attachment-0001.html>