TeamCity: archive debug logs if there are over 1000 of them

[SarahFrench]

Description

This PR addresses issues like in this build:

The plugin used for uploading artifacts to S3 has a limit of 1000 artifacts per build. After contacting JetBrains support they suggested archiving files as a solution. The downside to this is that this would make it harder to open specific artifacts in your web browser, instead you'd need to download the entire zip/tar file, open it and then find the logs file you want.

The compromise discussed in the team chat was to archive logs only if a build exceeds the limit. This PR introduces that behaviour.

Testing

I used this PR to create a test project in TeamCity for manual testing of the bash in the new build step. This PR has some commits from when I forced builds to create >1000 or <1000 debug*.txt files and then see how the new build step reacted.

When there are >=1000 artifacts present

This commit 7b966a7 shows how I forced builds to look like they had >1000 debug logs.

Here's a build in TeamCity showing the outcome:

In step 1 >1000 debug files are made

In step 2

21:33:51 Step 2/2: Tasks after running nightly tests: archive artifacts(debug logs) if there are >=1000 before S3 upload (Command Line)
21:33:51   Starting: /opt/teamcity-agent/temp/agentTmp/custom_script8203854418285142512
21:33:51   in directory: /opt/teamcity-agent/work/d9c38c33567f381b
21:33:51   Post-test step - archive artifacts(debug logs) if there are >=1000 before S3 upload
21:33:51   Found >=1000 debug artifacts; archiving before upload to S3
21:33:51   Listing files matching the artifact rule value /opt/teamcity-agent/work/d9c38c33567f381b/debug*
21:33:51   debug-google-1-TerraformProviders_SarahTesting1000artifactFix_GOOGLE_NIGHTLYTESTS_GOOGLE_PACKAGE_ACCESSAPPROVAL-archive.tar.gz
21:33:51   Finished
21:33:51   Process exited with code 0

The result is the artifacts tab contains only that .tar.gz file:

When there are <1000 artifacts present

This build shows the opposite scenario, where I made the builds instead create only 10 debug files (bfbca51)

Step 2 now looks like this:

22:10:50 Step 2/2: Tasks after running nightly tests: archive artifacts(debug logs) if there are >=1000 before S3 upload (Command Line)
22:10:50   Starting: /opt/teamcity-agent/temp/agentTmp/custom_script15021977902611658096
22:10:50   in directory: /opt/teamcity-agent/work/d9c38c33567f381b
22:10:50   Post-test step - archive artifacts(debug logs) if there are >=1000 before S3 upload
22:10:50   Found <1000 debug artifacts; we won't archive them before upload to S3
22:10:50 

And the artifacts tab shows the 10 files unchanged:

Release Note Template for Downstream PRs (will be copied)

See Write release notes for guidance.

[modular-magician]

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR generated some diffs in downstreams - here they are.

google provider: Diff ( 5 files changed, 63 insertions(+), 3 deletions(-))

[modular-magician]

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR generated some diffs in downstreams - here they are.

google provider: Diff ( 5 files changed, 65 insertions(+), 4 deletions(-))

[modular-magician]

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR generated some diffs in downstreams - here they are.

google provider: Diff ( 5 files changed, 96 insertions(+), 10 deletions(-))

[modular-magician]

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR generated some diffs in downstreams - here they are.

google provider: Diff ( 6 files changed, 126 insertions(+), 40 deletions(-))

[modular-magician]

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR generated some diffs in downstreams - here they are.

google provider: Diff ( 6 files changed, 127 insertions(+), 41 deletions(-))

[modular-magician]

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR generated some diffs in downstreams - here they are.

google provider: Diff ( 6 files changed, 127 insertions(+), 41 deletions(-))

[modular-magician]

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR generated some diffs in downstreams - here they are.

google provider: Diff ( 5 files changed, 66 insertions(+), 5 deletions(-))

[SarahFrench]

Hi @shuyama1 - this PR addresses recent issues with artifacts for the Compute builds in TeamCity. The PR description links to some builds where I show the new build step in action.

Something annoying I noticed is that the TeamCity UI makes it looks like you can open archived files in the UI:

However if you click to expand it causes a 500 error (you can try here):

The error mentions "java.util.zip.ZipException: Not in GZIP format" but I think the file format is fine? I'm not sure if this is something I can resolve myself.

The file is still ok to be downloaded and opened locally though.

[shuyama1]

Thank you for the fix!