BUG: List GitHub commit crashing for pytorch/pytorch

[gabibguti]

Describe the bug
The query that lists GitHub commits is crashing for pytorch/pytorch and possibly affecting the results of multiple checks that depend on this information, such as CI-Tests. CI-Tests is currently resulting in "no pull request found", but manually analysing there are PRs.

Reproduction steps
Steps to reproduce the behavior:

1. Run Scorecard v4.12.0 for pytorch/pytorch
2. View the following error

Error: RunScorecard: internal error: ListCommits:error during graphqlHandler.setup: internal error: githubv4.Query: non-200 OK status code: 502 Bad Gateway body: "{\n   \"data\": null,\n   \"errors\":[\n      {\n         \"message\":\"Something went wrong while executing your query. This may be the result of a timeout, or it could be a GitHub bug. Please include `CC1C:6B70:ABEBD8:15D971F:64FB2BE3` when reporting this issue.\"\n      }\n   ]\n}\n"
2023/09/08 14:13:04 error during command execution: RunScorecard: internal error: ListCommits:error during graphqlHandler.setup: internal error: githubv4.Query: non-200 OK status code: 502 Bad Gateway body: "{\n   \"data\": null,\n   \"errors\":[\n      {\n         \"message\":\"Something went wrong while executing your query. This may be the result of a timeout, or it could be a GitHub bug. Please include `CC1C:6B70:ABEBD8:15D971F:64FB2BE3` when reporting this issue.\"\n      }\n   ]\n}\n"

Expected behavior
I expected Scorecard to detect the commits.

Additional context

[raghavkaul]

I'm noticing the same issue intermittently - sometimes I get results, sometimes not. When I set --commit-depth lower, e.g. to 5, I still get errors but less frequently. Re-running scorecard might fix the issue in the Scorecard viewer.

The GraphQL handler calls GitHub's GraphQL endpoint here:

scorecard/clients/githubrepo/graphql.go

Line 204 in 846fb19

if err := handler.client.Query(handler.ctx, handler.data, vars); err != nil {

In GraphQL, handler.RateLimit.Cost usually returns 1, so I doubt we're exceeding a hardcoded limit (and the 502 confirms this). According to https://github.com/orgs/community/discussions/24844, all API calls (including GraphQL) hit a 10-second timeout, which can happen when the data returned is too large. We could try reducing the commit depth required to page (e.g. from 99 -> 20?) in:

scorecard/clients/githubrepo/graphql.go

Line 198 in 846fb19

if handler.commitDepth > 99 {

But I don't think commit depth alone is the culprit, we could also try playing with the parameters here:

scorecard/clients/githubrepo/graphql.go

Lines 30 to 37 in 846fb19

	const (
	pullRequestsToAnalyze = 1
	checksToAnalyze = 30
	issuesToAnalyze = 30
	issueCommentsToAnalyze = 30
	reviewsToAnalyze = 30
	labelsToAnalyze = 30
	)

[gabibguti]

Oh, right! Thanks for the info, Raghav! As you said, it's probably because the data returned is too large in this case. I believe playing with the pullRequestsToAnalyze and other parameters may be the right direction here, instead of commit depth.