Add dependabot to GitHub repository

[kinow]

This is a small change with no associated Issue.

Maybe worth a try here before considering adding it to cylc-flow? It's a free service, used by other projects. It runs daily, and reports any dependency in our dependency range that can be updated.

I normally go through the main dependencies in our setup.py and look them up at PYPI. But dependabot will handle that.

Furthermore, it is probably already enabled in GitHub if our repository settings has not disabled the dependabot security checks (click settings / security I think, then look for dependabot security).

I enabled it for one of my repositories with Python. It executed almost immediately.

Each PR created by dependabot will run the CI pipeline too, confirming whether there was any build/test failures due to the new version. Developers can also check out the branch and try the code.

So I think it could be helpful to use this, instead of relying on us remembering to update the dependencies before a release?

Requirements check-list

• I have read CONTRIBUTING.md and added my name as a Code Contributor.
• Contains logically grouped changes (else tidy your branch by rebase).
• Does not contain off-topic changes (use other PRs for other changes).
• Already covered by existing tests.
• No change log entry required (why? e.g. invisible to users).
• No documentation update required.
• No dependency changes.

[kinow]

Accidentally merged this PR thinking I was looking at another repository, then reverted it. But the commit history didn't look OK with both PR's merged, so I dropped the commits and push-forced (confirming three times my master was identical to the one on GitHub).

Re-opening the PR now, and then closing this tab so I don't accidentally merge it again.

Sorry!!! ☕

[kinow]

Updated a PR to one of our conda forge repositories, and realized that with dependabot PR's we would have to remember to update Conda Forge recipe before we release that. In Cylc Flow I think we also have to update the environment.yml file (not updated automatically by dependabot)

[hjoliver]

Seems like a good thing to do.

[oliver-sanders]

Would be great to do this, I had a look at doing it for cylc-flow a little while back. The barrier I hit was that Dependabot doesn't like setup.py files (because they are program logic making it hard to parse out dependencies). I think they implemented some logic to scrape deps out of setup.py files (can't find the blog post) but noted that it won't work in all cases and may miss deps.

They now don't mention setup.py support on their website - https://dependabot.com/python/

I think we need to move our deps from setup.py to requirements.txt for Dependabot to be able to do its job properly.

[kinow]

Would be great to do this, I had a look at doing it for cylc-flow a little while back. The barrier I hit was that Dependabot doesn't like setup.py files (because they are program logic making it hard to parse out dependencies). I think they implemented some logic to scrape deps out of setup.py files (can't find the blog post) but noted that it won't work in all cases and may miss deps.

I think it may depend on the setup.py. The project I enabled it uses a setup.py https://github.com/kinow/protobuf-uml-diagram/blob/d955b0b528947e4a4e8ee57145302846fba9b97d/setup.py#L29-L31

dependabot found a couple more updates last night too kinow/protobuf-uml-diagram#22, so far it appears to be working, but my setup.py is much simpler than cylc-flow's. cylc-uiserver is a bit simpler.

They now don't mention setup.py support on their website - https://dependabot.com/python/

Their website appears to be have more marketing material, and information about what it does without going into detail :-( there's more in their github repo: https://github.com/dependabot/dependabot-core/blob/564ca68bfcdaa20df925c165d20834868a987d67/python/lib/dependabot/python.rb#L18-L28

If you search the issues, there are reports from cases where setup.py didn't work. I think it should work for cylc-uiserver, but not sure about cylc-flow.

I think we need to move our deps from setup.py to requirements.txt for Dependabot to be able to do its job properly.

I think either way should work. It's probably easier for dependabot to parse a requirements.txt.

[oliver-sanders]

Managed to dig it out of my browser history, it's a note from GitHub

https://docs.github.com/en/github/visualizing-repository-data-with-graphs/about-the-dependency-graph#supported-package-ecosystems

Note: If you list your Python dependencies within a setup.py file, we may not be able to parse and list every dependency in your project.

However, I also took a quick look at dependabot's Python parsing and it looks ok:

https://github.com/dependabot/dependabot-core/blob/f5593cbaf7e333a8d0b445a280a73876ad494f3c/python/helpers/lib/parser.py#L136

It does actually run the setup.py file so it should be able to handle the way we do things. I don't know if it looks at the extra deps, will see if I can find out...

...yes

https://github.com/dependabot/dependabot-core/blob/f5593cbaf7e333a8d0b445a280a73876ad494f3c/python/helpers/lib/parser.py#L88-L91

Very nice, great, we can copy this to all our Python projects 👍.

[oliver-sanders]

MyPy failures are completely unrelated.