OpenSSF Security Metrics Initiative

Tue May 4 14:27:50 UTC 2021

Hi Nicola!
Thanks for reaching out! (And for understanding that the metric dashboard is alpha quality, and on a good day, an approximation of the health of a project.

So yes, many of the items don’t reflect reality.

For OpenSSL, the Scorecard data is gathered by this project (which I’m just a consumer of):
ossf/scorecard: Security Scorecards - Security health metrics for Open Source (github.com)<https://github.com/ossf/scorecard>

For example, the SECURITY.md, it appears<https://github.com/ossf/scorecard/blob/main/checks/security_policy.go> that project is just looking for a file with that name in either the root for the .github folder. Similarly for the others, but I will go through OpenSSL today to make sure it’s at least *technically* working correctly.

As we move this project forward, we want to be able to pull out metrics on non-GitHub projects as well. I’ll also add a note to the dashboard page itself about the content being alpha quality, etc.

But to the larger going forward point, yes, on behalf of the working group, we would greatly appreciate additional insight and thoughts into how we can make this useful and reflect reality – the intent isn’t to make *any* project look “bad”, especially due to our implementation.

Thanks again!

Mike

From: Nicola Tuveri<mailto:nic.tuv at gmail.com>
Sent: Tuesday, May 4, 2021 3:12 AM
To: OpenSSL Project<mailto:openssl-project at openssl.org>; otc at openssl.org<mailto:otc at openssl.org>
Cc: Michael Scovetta<mailto:Michael.Scovetta at microsoft.com>
Subject: OpenSSF Security Metrics Initiative

Hi,

I wanted to point out to the OMC and to openssl-project a new initiative from the [Open Source Security Foundation](www.openssf.org<https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.openssf.org%2F&data=04%7C01%7Cmichael.scovetta%40microsoft.com%7Caa7009fae37b409b8be808d90ee461ae%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637557199518547959%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=IZnS4v%2Fis0bWmwKiMoqUvP5w%2FCxDYTnf5LkGnYIrdbE%3D&reserved=0>): the Security Metrics Initiative.

A more detailed description is available at <https://openssf.org/blog/2021/05/03/introducing-the-security-metrics-project/<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fopenssf.org%2Fblog%2F2021%2F05%2F03%2Fintroducing-the-security-metrics-project%2F&data=04%7C01%7Cmichael.scovetta%40microsoft.com%7Caa7009fae37b409b8be808d90ee461ae%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637557199518557915%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=2NhTSK5Wq0wrbl2dOYEXEuDP4XhNNrPjYvhx7iTb624%3D&reserved=0>>.
It should be remarked that the <metrics.openssf.org<https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Fmetrics.openssf.org%2F&data=04%7C01%7Cmichael.scovetta%40microsoft.com%7Caa7009fae37b409b8be808d90ee461ae%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637557199518557915%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=Vv3u%2FW1306qjsMY0CMgaknp7%2BGRw2rKGLZlVPa9g2M8%3D&reserved=0>> service is to be considered alpha, and that changes in the API, in data sources might occur at this stage, and that there might be inaccuracies in the reported data.

Here is a direct link to what the initiative reports for the OpenSSL project:
<https://metrics.openssf.org/grafana/d/default/metric-dashboard?orgId=1&var-PackageURL=pkg%3Agithub/openssl/openssl<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmetrics.openssf.org%2Fgrafana%2Fd%2Fdefault%2Fmetric-dashboard%3ForgId%3D1%26var-PackageURL%3Dpkg%253Agithub%2Fopenssl%2Fopenssl&data=04%7C01%7Cmichael.scovetta%40microsoft.com%7Caa7009fae37b409b8be808d90ee461ae%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637557199518567873%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=FHJmwJsN8fDLMX2Em4tXQsIwaTQPlOum8esW7Tie%2FeQ%3D&reserved=0>>.

In particular it seems we score quite low on the OpenSSF Scorecard (30.8% as I am writing this mail) and, also for the data coming from the OpenSSF Best Practices Badge Program, it looks like the project has many negative marks.

It should also be noted that the description field in the project information for `github:/openssl/openssl` reports:

> This is a historical badge entry for the OpenSSL project before the Heartbleed vulnerability was reported, circa February 2014. Please note that the OpenSSL project's status has changed substantially since then. For the current state of OpenSSL, see the current OpenSSL badge entry. [...]

So maybe it is not too alarming that many of the negative marks are coming from unexpected entries: e.g. it seems at the moment it reports we don't have/use static/dynamic analysis, we don't have vulnerability reporting, code review, CI Tests or Pull Requests.

Nonetheless given this tool might soon be used to pick among alternatives when making critical infrastructure design choices, or affect funding decisions or resource planning, it might be a good thing for the OMC to get proactive and reach out to straighten the record for current OpenSSL releases, to offer suggestions on alternative metrics to be considered, on redefining criteria for existing metrics, and possibly incorporate feedback from the Security Metrics initiative to adapt plans regarding future roadmap for OpenSSL.

I finish reporting in this email the last paragraph from the Security Metrics Initiative announcement, as it might be of interest for all subscribers to openssl-project:

> Your [feedback](https://github.com/ossf/Project-Security-Metrics/issues<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fossf%2FProject-Security-Metrics%2Fissues&data=04%7C01%7Cmichael.scovetta%40microsoft.com%7Caa7009fae37b409b8be808d90ee461ae%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637557199518567873%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=1YYtuiuTGuWh2msSmCfMxIgwUHXpjRloT3VXvCOUxZE%3D&reserved=0>) is most welcome, and if you're interested in learning more or joining this effort, please reach out to [Michael Scovetta](mailto://michael.scovetta@microsoft.com<mailto:michael.scovetta@microsoft.com>) or join us at our next [working group](https://github.com/ossf/wg-identifying-security-threats<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fossf%2Fwg-identifying-security-threats&data=04%7C01%7Cmichael.scovetta%40microsoft.com%7Caa7009fae37b409b8be808d90ee461ae%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637557199518577822%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=pbDTWeCEOv0hgAN8arvQp2kbhzmAdW%2BEVwn3iMa%2FDRk%3D&reserved=0>) meeting.

Best regards,

Nicola Tuveri

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mta.openssl.org/pipermail/openssl-project/attachments/20210504/d428ad17/attachment-0001.html>