Update reference tests to the latest stable version?

[gdalle]

People only test locally on the latest stable version, and as a result some PRs can introduce test failures (like #441). This would be easier to avoid if the reference tests were kept up to date with the latest stable version, so 1.9.4 at the time of writing.

On the other hand, it means that once we re-generate the fixtures with a new version, they are basically useless. Indeed, by definition, they will agree with the output of the code.
@oxinabox any idea how to resolve this conundrum?

[gdalle]

More generally, this raises the question of what reference tests are for

[oxinabox]

We actually need to keep testing on 1.7 even if we update the fixtures,
because 1.8 changes how IO works in tests,
so we can't run the interactive tests on 1.8

though maybe it would just be ok to stop testing them?

[oxinabox]

In general the notion of reference tests is that you look at them before merging after regenerating, and confirm that they actually are correct

[gdalle]

In general the notion of reference tests is that you look at them before merging after regenerating, and confirm that they actually are correct

So if we work on the assumption that the code is correct whenever reference tests pass, we could use that correct code to re-generate new reference tests at each Julia release?

[oxinabox]

Yes, because we assume that new julia releases are not breaking our stuff.

[gdalle]

Okay. I'll try a big PR that does only that, re-generate all the fixtures with 1.9.4

[gdalle]

Solved by #453