CI logs / artifacts analyzing tool

[Totktonada]

In short:

• Implement as an external crawler (see reasoning below).
• A service that holds collected logs / artifacts and shows generated HTML reports.
• Weekly HTML reports to track progress, statistics timeline and graphs.
• Ability to regenerate old HTML reports, when a new matcher is added (or existing one is changed).
• The service should allow to download all collected logs and artifacts at once (for local experiments).
• Support are script language for problem matchers.
• Easy to experiment locally with a new matcher and propose it via GitHub pull request.
• Collect all jobs, not only ones on release branches, to have great amount of statistics.

Reasoning behind external crawler

Here we compare two approaches:

1. Send completed (successful or failed) job id from the job itself.
2. Gather past jobs information externally.

The first approach is simpler, it is the main its advance.

The second approach has the following advances:

• Able to track GitHub Actions and self-hosted runners infrastructure fails.
• In case of an error in the crawler or outage of the service, we can fix it and collect past jobs.

Matchers API thoughts

Input: log and artifact files in argv.

Output: simple key-value pairs.

type: flaky test
issue: gh-xxxx
summary: blah blah
(and so on)

[ligurio]

I believe ELK should match to your requirements

[Totktonada]

The initial implementation is there: https://github.com/tarantool/multivac

It has the following features at the moment:

• Fetch and update jobs metainfo and logs.
• Generate last seen test fails report over the whole logs collection (there is also filtering per branch). The report can be in the CSV or the HTML format.
• (There is also spent time calculator, but it is the side feature, out of context here.)

It is possible to add the script into cron, setup nginx and generate daily / weekly reports about overall situation with tests.

There are some known problems and ideas about a new functionality:

• sensors/test_status: QA Notice leads to a false fail detection #1
• Show a table with all failed tests, filtered by dashboard's variables #2
• last_seen: no link to locally stored logs #3
• last_seen: split fails by OS / machine architecture? #4

But I think that the problem is generally solved from the scripting side.

The next action is to setup a machine and provide HTTP access to regularly updated reports. @kyukhin volunteered for this.

[Totktonada]

Now the service can be found here: https://shame.tarantool.dev/crawler/

[NickVolynkin]

Pretty much of this issue is done already in multivac/InfluxDB/Graphana toolchain. I will move this ticket to tarantool/multivac and close it.