Coveralls promoted to green, expand on what coverage reporting *is*.

[samcunliffe]

• Solves Codecov should go amber, and expand on the descriptions in the table. #254

[dstansby]

I disagree about moving codecov to amber - although it can be a bit flaky sometimes, unless we can recommend a better tool I think it should stay green.

For reference our guidance on amber is:

🟠 We don’t discourage using this, but it may duplicate functionality of a green tool.

[paddyroddy]

I disagree about moving codecov to amber - although it can be a bit flaky sometimes, unless we can recommend a better tool I think it should stay green.

Without being a nag, we discussed various points in person yesterday. Would have been good if you were able to join these, consider it already "decided".

🫐

[dstansby]

And FWIW I think codecov should be green above coveralls, because you can use codecov with private repos for free (see their pricing), but can't use coveralls with private repos for free (see their pricing)

[dstansby]

I disagree about moving codecov to amber - although it can be a bit flaky sometimes, unless we can recommend a better tool I think it should stay green.

Without being a nag, we discussed various points in person yesterday. Would have been good if you were able to join these, consider it already "decided".

This is totally not fair - stuff shouldn't be discussed in person, that discussion not documented, and others not able to challenge or question it afterwards just because they weren't in the room.

[samcunliffe]

Why did we decide it should go 🟠 ? Was it just because slow and a little flakey?

Perhaps it should stay green because it's widely used (by us).

In answer to a question (maybe mine) from yesterday...
We generate coverage reports in the template, but we don't upload anywhere.

[samcunliffe]

And FWIW I think codecov should be green above coveralls, because you can use codecov with private repos for free (see their pricing), but can't use coveralls with private repos for free (see their pricing)

This would also convince me. Not all research software projects are open.

[paddyroddy]

Why did we decide it should go 🟠 ? Was it just because slow and a little flakey?

I think it was because we were claiming we can only have 🟢 and the tools were considered as good as each other. Personally, I've had fewer issues with coveralls over codecov - in terms of flaky-ness.

[paddyroddy]

This is totally not fair - stuff shouldn't be discussed in person, that discussion not documented, and others not able to challenge or question it afterwards just because they weren't in the room.

I see your point, but we wouldn't achieve much if literally everything discussed had to be documented. Unless we had someone taking minutes.

[dstansby]

I see your point, but we wouldn't achieve much if literally everything discussed had to be documented. Unless we had someone taking minutes.

I think in the interests of being an open and collaborative project reasons for decisions should at least be written down somewhere.

[dstansby]

I think it was because we were claiming we can only have 🟢 and the tools were considered as good as each other. Personally, I've had fewer issues with coveralls over codecov - in terms of flaky-ness.

I haven't used coveralls in ages, so could believe it's less flaky.

Another point in favour of codecov is that it's more widely used by big OSS scientific Python projects (AFAIK)?

[samcunliffe]

I'm convinced to put these comments into the page. I've not got a strong feeling about 🟢 or 🟠 .

[matt-graham]

I'm convinced to put these comments into the page. I've not got a strong feeling about 🟢 or 🟠 .

I also have no opinion on 🟢 versus 🟠 but I strongly agree with @samcunliffe's suggestion to put the details discussed here comparing Codecov / Coveralls in the table / on the page, particularly @dstansby points about being able to use Codecov with private repositories, and that Codecov appears to be more widely used by major Scientific Python packages (from a quick scan of repositories, all of NumPy, Matplotlib and pandas appear to use Codecov).

[matt-graham]

This is totally not fair - stuff shouldn't be discussed in person, that discussion not documented, and others not able to challenge or question it afterwards just because they weren't in the room.

I think in the interests of being an open and collaborative project reasons for decisions should at least be written down somewhere.

I also think @dstansby points here are very valid and given we've been explicitly discussing how we might encourage getting outside contributors, making sure we have processes which make sure everyone can equally contribute to discussions is important.

[paddyroddy]

I also think @dstansby points here are very valid and given we've been explicitly discussing how we might encourage getting outside contributors, making sure we have processes which make sure everyone can equally contribute to discussions is important.

I agree, but it is very easy to say without actually doing anything. The only way for that to actually happen is someone to spend the whole time taking minutes, or recording the whole thing. I would not volunteer for either of those roles.

[matt-graham]

I agree, but it is very easy to say without actually doing anything. The only way for that to actually happen is someone to spend the whole time taking minutes, or recording the whole thing. I would not volunteer for either of those roles.

I don't think we need minutes, we just need to make sure if we have an off-GitHub discussion (be it in-person or on Slack) and this leads to an issue being raised around some decision point about a change to be made we record the key points of the discussion and why the decision was made.

[paddyroddy]

I don't think we need minutes, we just need to make sure if we have an off-GitHub discussion (be it in-person or on Slack) and this leads to an issue being raised around some decision point about a change to be made we record the key points of the discussion and why the decision was made.

Yes, but to remember those points...

[matt-graham]

Yes, but to remember those points...

If we are recording the points in the initial issue comment as we create an issue I don't think there should be a problem remembering them.

For example if we're creating issues in a discussion / planning session like we had in the first part of yesterday morning, I think the extra overhead of having someone (probably distinct from the person leading the planning) note down any key points discussed would be minimal. And this would also have the additional benefit of making it easier to understand why decisions were made previously if we subsequently review and decided to change.

[paddyroddy]

If we are recording the points in the initial issue comment as we create an issue I don't think there should be a problem remembering them.

Sounds like a plan.

[matt-graham]

Sorry realise this went quite off-topic but I though it was a useful discussion to have. I'll create a new issue to remind us to somehow agree on and document a process around this.

[dstansby]

Yeah, to be clear not looking for full discussion notes or anything, just a brief summary of why a decision has been made so it's clear to anyone who wasn't part of the discussion. For this PR, a summary in the body of #254 were what I was expecting.

[samcunliffe]

Still not a strong opinion, but I think two 🟠 is probably an accurate representation of the situation. We don't have a consensus about which is better. And we don't have either implemented in our template.

[dstansby]

In that case I'd say have two 🟢 ? I think it's better to have at least one green per category, otherwise we run the risk of people not choosing anything if there's just 🟠 ?

[dstansby]

But also, as I've said above codecov 1) can be used for private repos 2) is more widely used across scipy ecosystem, and I still think that's enough to edge out coveralls unless anyone has any other arguments for coveralls above codecov?

[samcunliffe]

But also, as I've said above codecov 1) can be used for private repos 2) is more widely used across scipy ecosystem, and I still think that's enough to edge out coveralls unless anyone has any other arguments for coveralls above codecov?

Should we go all in and add it to the template's CI? (+ a badge, obvs.)

[paddyroddy]

But also, as I've said above codecov 1) can be used for private repos 2) is more widely used across scipy ecosystem, and I still think that's enough to edge out coveralls unless anyone has any other arguments for coveralls above codecov?

Flakiness, from my own experience

[paddyroddy]

Should we go all in and add it to the template's CI? (+ a badge, obvs.)

That would be good, but it's another thing they would need to set up #205

[dstansby]

Flakiness, from my own experience

Ah fair enough, sorry for forgetting/not reading properly. I'm still pro both as green? Won't block this PR if others think both amber though

[dstansby]

I'd be happy to merge this if the amber > green change is made

[samcunliffe]

I've updated the PR title and commit message to reflect our decision.

[paddyroddy]

https://hynek.me/articles/ditch-codecov-python

[samcunliffe]

For those who can access it, we did a very lightweight solution over in INFORMus.
https://github.com/inform-us/INFORMus/blob/dev/.github/workflows/api.yml#L49-L75

And it's working well so far.

Using

• https://github.com/irongut/CodeCoverageSummary (which can fail CI if coverage is too low)
• https://github.com/marocchino/sticky-pull-request-comment (for posting bot comments 🤖 )