Test dataset downloadability on a schedule

[pmeier]

Currently we rely on user feedback to detect failing dataset downloads. In #2610 I've included a downloadability test by asserting a head request to every URL used in the dataset is successful. We should expand these test for all downloadable datasets.

That being said, I don't think we should run this as part of every PR or push, but rather on a schedule (for example daily). In that case we need some way to inform us about failed tests.

cc @seemethere @pmeier

[fmassa]

@seemethere how do you think would be a good way for a scheduled CI job to signal us that a test stopped working (due to an external server going down / stopping to work)? Sending an e-mail is one option, but is there a better one?

[seemethere]

We could also possibly have it just a send a message in a slack channel as well.

That way community members as well as maintainers can have a way to see the signal in a more public space.

Another good middle ground is to have a github action that posts a comment to a specific issue every time a dataset is not able to be downloaded

[pmeier]

I'm currently looking into probot to do this. @seemethere Do you know if CircleCI can fire a webhook if it fails?

[seemethere]

It's entirely possible, but we haven't had much success trying to create an alerting system through CircleCI

[pmeier]

A simple solution could be this https://github.com/JasonEtco/create-an-issue. If we test the download as part of a GitHub Actions workflow, this could simply create an issue from a given template if the workflow fails.

@seemethere @fmassa Is GitHub Actions a alternative to CircleCI for this?

[pmeier]

I've created a proof of concept repo. With this minimal setup it will create an issue like this https://github.com/pmeier/test-issue-on-fail/issues/9 every time the workflow fails. I think that might already be enough.

[pmeier]

Re-opening this, since #2665 only laid the ground work, but not the actual testing on a schedule.

[fmassa]

I've created a proof of concept repo. With this minimal setup it will create an issue like this pmeier/test-issue-on-fail#9 every time the workflow fails. I think that might already be enough.

I think this looks pretty good! One thing to check is to see if the bot creates a repeated issue every day if the CI is not fixed the day it fails.

[pmeier]

Unfortunately, right now it will. I have more time at the end of September / early October to build a proper bot. Until then, we have stick to close it manually. That being said, with the retry functionality and wait time between the individual requests, I don't think this will fail often. Actual dead links or broken downloads are quite rare.

Should I send a PR for this?

[fmassa]

Hum, that would be a bit annoying to be spammed by a known-problem by the bot. But I suppose having an initial PR would be good to have

[pmeier]

Given that I'm responsible for the datasets, it will mostly spam me 😉