Document our philosophy / best practices about test suites

[kytrinyx]

I couldn't find a document that talks about how we think about the exercise test suites.
There's a little bit in here: https://github.com/exercism/docs/blob/master/contributing-to-language-tracks/improving-consistency-across-tracks.md but I think it would be great to have a top-level philosophy document that other documentation could refer to.

@parkerl @jtigger @IanWhitney @petertseng @kotp @ErikSchierboom you all immediately come to mind as maintainers who care about this, and have a good understanding of this.

Would you suggest some bullet points of what should be included in such a doc?

---

TODO

• determine bullet points of what should be included in the doc
• write a draft / submit PR of doc
• (when merged) pare down the boilerplate for new tracks (https://github.com/exercism/request-new-language-track) and link to this document from the CHECKLIST and the TRAVIS files

[petertseng]

• Follow x-common's canonical-data.json if it's available
• If it's not available, use http://x.exercism.io/problems/hello-world (replace hello-world with the exercise in question) or http://exercism.io/languages/vimscript/todo (replace vimscript with your track) to find other implementing tracks.
• Test generator if you think it will make translation from JSON -> your track easier, not required of course.
• Explain why we skip tests.
  • We should summarise past discussions.
    • Should we really have skipped tests in the assignments? exercism#856
    • @Ignore all but the first test case? kotlin#25
    • Why do we skip tests? python#105
    • Skip all but the first test. python#171
  • As an alternative that achieves the same ends, consider a test runner that runs the tests but stops at the first failure.

[kotp]

This is excellent @petertseng set to start from.

[kytrinyx]

Also perhaps:

• consider the user experience (what do the failure messages / error messages look like, are they communicative?)
• consider the test cases: try to avoid a test that introduces multiple requirements at once, try to avoid tests that test for the same thing, try to make the order of the tests increase the requirements gradually

[petertseng]

That's a great point about user experience. It reminded me of a point:

• Can we make it as easy as possible to run the tests?
  • A command that is short and easy to remember (such as make test) will probably have the advantage over one that has a bunch of flags.
  • A bit of previous discussion in All tracks that have discussed whether to use a build system or not discussions#117, but unsure how much of it is considered relevant

[kytrinyx]

Can we make it as easy as possible to run the tests?

Yeah, that's a good point. We should optimize for ease of running the tests, while still balancing against idiomatic use of the language's tools.

[ErikSchierboom]

The only thing I would add is:

• Try to make the test code as simple as can be.

I think it is very useful if people can actually read the tests, which means that they should be as simple as possible.

Furthermore, perhaps "Follow x-common's canonical-data.json if it's available" should be specified in more detail. Something like:

• Use the test description as specified in the canonical data
• Use the test order as specified in the canonical data

[kytrinyx]

"Follow x-common's canonical-data.json if it's available" should be specified in more detail

I think we should mention that these are what we think are good defaults, but if you have a good reason (language idioms, etc) then don't hesitate to deviate from them.

[parkerl]

Dropping this here for reference from https://github.com/exercism/request-new-language-track/edit/master/TRAVIS.

When implementing an exercise test suite, we want to provide a good user experience for the people writing a solution to the exercise. People should not be confused or overwhelmed.

In most Exercism language tracks, we simulate Test-Driven Development (TDD) by implementing the tests in order of increasing complexity. We try to ensure that each test either

- helps triangulate a solution to be more generic, or
- requires new functionality incrementally.

Many test frameworks will randomize the order of the tests when running them. This is an excellent practice, which helps ensure that subsequent tests are not dependent on side effects from earlier tests. However, in order to simulate TDD we want tests to run *in the order that they are defined*, and we want them to *fail fast*, that is to say, as soon as the test suite encounters a failure, we want the execution to stop. This ensures that the person implementing the solution sees only one error or failure message at a time, unless they make a change which causes prior tests to fail.

This is the same experience that they would get if they were implementing each new test themselves.

Most testing frameworks do not have the necessary configuration options to get this behavior directly, but they often do have a way of marking tests as _skipped_ or _pending_. The mechanism for this will vary from language to language and from test framework to test framework.

Whatever the mechanism—functions, methods, annotations, directives, commenting out tests, or some other approach—these are changes made directly to the test file. The person solving the exercise will need to edit the test file in order to "activate" each subsequent test.

Any tests that are marked as skipped will not be verified by the track test suite unless special care is taken.

Additionally, in some programming languages, the name of the file containing the solution is hard-coded in the test suite, and the example solution is not named in the way that we expect people to name their files.

We will need to temporarily (and programmatically) edit the exercise test suites to ensure that all of their tests are active. We may also need to rename the example solution file(s) in order for the exercise test suite to run against it.

[jtigger]

As part of this work, as discussed in exercism/generic-track#12 please add a link from https://github.com/exercism/request-new-language-track/blob/master/TRAVIS to the document you create here.