BUG: Scorecard can't analyze files which are export-ignore'd in a .gitattributes file

[diogoteles08]

Describe the bug
The Scorecards check for Security Policy did not find any Security Policy file on Laravel project, but it does have a SECURITY.md file inside .github folder.

Reproduction steps
Steps to reproduce the behavior:

1. Run Scorecards CLI for the repo https://github.com/laravel/framework, or access the already evaluated result here
2. Note that for Security-Policy check it's got a null punctuation with reason "security policy file not detected"
3. Note that the repository actually have a security policy. It's located on .github/SECURITY.md, which is a location considered by GitHub itself, as you can check here.

Expected behavior
Scorecards should detect the SECURITY.md file and use its content to evaluate the punctuation for Security-Policy check.

[spencerschrock]

So the tarball downloaded by hitting https://api.github.com/repos/laravel/framework/tarball/ doesn't contain the .github folder at all, which is why scorecard isn't picking it up. The scorecard repo downloaded via the equivalent link does include the .github folder for example

I'm not familiar with the tarball endpoint, and the GitHub docs don't seem to explain the behavior.

/tmp$ tar -tvf laravel-framework-v9.41.0-25-gdefd920.tar.gz | grep .github
/tmp$

[laurentsimon]

That's an interesting finding. Other checks look for files under .github workflow (Token-Permissions, Dangerous-Workflows, Pinned-Dependencies). Does this affect the other checks too for this repository?

[gabibguti]

@laurentsimon Yes. Scorecards gives a 10 for Token-Permissions, when they do not set it as Scorecards expects, and also a 10 for Pinned-Dependencies, when they do not hash-pin the dependencies and for some not even tag-pin. So, if the TAR comes "wrong" from GH API, these checks that look at the repo's source code will be mistaken.

[spencerschrock]

This is a configuration issue. By excluding the .github folder in their .gitattributes file, it's unavailable for analysis by scorecard
https://github.com/laravel/framework/blob/926cf9686c28ea6424990e2bd36dd607695eb104/.gitattributes#L9

[azeemshaikh38]

Great find @spencerschrock! Maybe #1709 could help here?

[diogoteles08]

Great find indeed! Would help if I close this issue and create a different one focusing on this main cause?

[azeemshaikh38]

Let's keep this open. This bug has all the context needed for someone in the future.

[spencerschrock]

The above go-git issue would help, but part of the reason it was never done was speed (primarily targeted at our weekly analysis of public repos). We we may want a compatibility mode to the binary/action which does a clone with go-git.

[JBludau]

Hmm how is the state on this? We would like to avoid exporting our workflows.

[spencerschrock]

Hmm how is the state on this? We would like to avoid exporting our workflows.

I've been playing around today with a compatibility mode mentioned above. Testing it on kokkos/kokkos shows a 10/10 for Dangerous-Workflow