Run tests on Windows using CircleCI

[eurunuela]

Detailed Description

Now that we have CircleCI running and is not giving us any problems, we should write some tests on Windows. This would make our testing suite only depend on CircleCI (i.e., we would not need Azure anymore). This way, we avoid having two different setups and two different places to check the logs.

Possible Implementation

I would only run the integration tests on Windows to avoid having too many testing jobs on CircleCI. We currently have:

• 4 testing jobs,
• the job to create the environment,
• the "build the docs" job,
• the style check job,
• the merge coverage job.

So, that's 8 jobs we currently have. Adding another 4 (2 integration + 2 unit test) jobs would be a bit redundant imo. However, I'd like to know what you guys think about this @tsalo @vinferrer @smoia .

The steps to implement this can be found on the CircleCI documentation page.

[vinferrer]

I think this is great to only depend on one platform, however i don't understand the problems with the second part. Are you suggesting to have less test on windows, like only integration tests?

[eurunuela]

Yes, that's my suggestion. I don't think unit tests would be necessary on Windows. If something doesn't work on Windows, I think the integration tests would find it.

[vinferrer]

I completely agree with you

[smoia]

I agree on the fact that we probably don't need to run all the tests in Windows since we run them already in Linux, but we do need to be sure that all the code is run at least once in Windows, especially for those commands that might be affected by the different OS (so far, paths, but maybe something else too). I'm not sure that integrations are the best candidate for that, while maybe one integration test, the one that runs the most code, plus the unit tests that cover the rest, might be better.
However, as long as all the code is run at least once, it's fine by me!

[vinferrer]

Question. Can we have a different config.yml for the windows tests?

[eurunuela]

I believe it must be the same.

[vinferrer]

well, then I would propose to have to jobs: One that runs only one integration test and another that runs the unit tests

[eurunuela]

What do you mean?

[vinferrer]

Two windows jobs. This way we ensure our code runs in windows properly

[eurunuela]

Sure. I think those would be sufficient but I think @smoia thinks otherwise.

[vinferrer]

really,

maybe one integration test, the one that runs the most code, plus the unit tests that cover the rest, might be better.
However, as long as all the code is run at least once, it's fine by me!

I think that's exactly what he says here