Hoist CI build requirements to top of issue template #12

jtigger · 2017-03-23T13:50:31Z

Feels like a lot of unilateral decisions in this PR; I hope I'm understanding the purpose and audience.

suggesting that we make explicit all the "requirements" of setting up CI at the very top of the file (the what).
elaborate the background on each "requirement" with similarly named sections (the why).
keep the excellent examples at the tail of the file (the how).

Also, there is background for writing high quality exercise test suites. Suggesting we extract these to a separate file. In fact, I believe this would go in the new docs repo in some section giving background on writing test suites.

- extract background for writing high quality exercise test suites to a separate file.

kytrinyx · 2017-03-23T14:35:45Z

In fact, I believe this would go in the new docs repo in some section giving background on writing test suites.

Yes, I think that is right. I think that probably goes in contributing-to-language-tracks, but maintaining-a-track would also be correct.

kytrinyx

This is great. I love the direction.

kytrinyx · 2017-03-23T14:36:26Z

TRAVIS

+
+- for each exercise in the track,
+  - run that exercise's test suite (aka the "exercise test suite");
+  - use the exercise's solution as the code under test.


"sample solution" or "example solution"?

👍 the tooling is looking for an "example" solution, that one.

kytrinyx · 2017-03-23T14:37:07Z

TRAVIS

+- for each exercise in the track,
+  - run that exercise's test suite (aka the "exercise test suite");
+  - use the exercise's solution as the code under test.
+- ensure all test cases are enabled;


Should ensuring all test cases are enabled be first in the list?

Noice. In fact, I think it would best serve as a sub-bullet of "run that exercise's test suite".

kytrinyx · 2017-03-23T14:38:00Z

TRAVIS

+  - use the exercise's solution as the code under test.
+- ensure all test cases are enabled;
+- be executed on Travis CI;
+- be easy to run on a contributor/maintainer's local machine (i.e. a build script with any needed instructions).



Ah, yes. Instructions. Let's specify that this needs to be documented in the README of the track.

There is a placeholder for this as the last bullet under "Submitting a Pull Request" in the TRACK_README template.

@parkerl would that bullet contain the instructions to run the script? Or would it reference those instructions? (on some tracks, there might be a little more setup surrounding the script execution... ...cough... Java ...cough... for example ...wheeez...).

kytrinyx · 2017-03-23T14:41:54Z

TRAVIS


-Many test frameworks will randomize the order of the tests when running them. This is an excellent practice, which helps ensure that subsequent tests are not dependent on side effects from earlier tests. However, in order to simulate TDD we want tests to run *in the order that they are defined*, and we want them to *fail fast*, that is to say, as soon as the test suite encounters a failure, we want the execution to stop. This ensures that the person implementing the solution sees only one error or failure message at a time, unless they make a change which causes prior tests to fail.
+As described in [Contributing to Exercism: Writing exercise test suites](writing_tests.md), exercise test suites are written such that all but the first test case are skipped/pending/ignored.  For the purpose of the track test suite, all test cases should be enabled (simulating an Exercism user having completed the exercise).


s/Writing/writing/

s/writing_tests.md/writing-tests.md

exercise test suites are written such that all but the first test case are skipped/pending/ignored.

s/test suites/test suites in most language tracks/. Some language tracks are able to implement this without skipping, pending, ignoring (e.g. Go has a t.Fatal() that will stop on the first error or failure, whereas t.Error() will report the error and move on. I think also some language tracks have decided not to fail fast. That might be a decision that we need to consider when we start looking at the curriculum from a high level and consider the user experience.

I really like the simulating an Exercism user having completed the exercise bit.

kytrinyx · 2017-03-23T14:43:10Z

TRAVIS


-Whatever the mechanism—functions, methods, annotations, directives, commenting out tests, or some other approach—these are changes made directly to the test file. The person solving the exercise will need to edit the test file in order to "activate" each subsequent test.
+It's important that if we rewrite files in any way during a test run, that these changes do not accidentally get checked in to the git repository.


s/git/Git/ (Git is a proper name)

kytrinyx · 2017-03-23T14:43:32Z

TRAVIS


-Any tests that are marked as skipped will not be verified by the track test suite unless special care is taken.
+Therefore, many language tracks write the track test suite in such a way that it _copies_ the exercise to a temporary location outside of the git repository before editing or rewriting the exercise files during a test run.


s/git/Git

kytrinyx · 2017-03-23T14:44:57Z

TRAVIS


-Additionally, in some programming languages, the name of the file containing the solution is hard-coded in the test suite, and the example solution is not named in the way that we expect people to name their files.
+_Configuring Travis CI_


I now realize that we sometimes have AppVeyor (Microsoft stack) and Circle CI (Apple stack). Travis CI only runs on Linux.

kytrinyx · 2017-03-23T14:45:28Z

TRAVIS


-**Avoiding accidental git check-ins**


And I clearly forgot that Git is a proper name. Shame on me.

kytrinyx · 2017-03-23T14:46:45Z

writing_tests.md

@@ -0,0 +1,16 @@
+_(in fact, this would be part of the new "docs" repo, not a file in this repo.)_


Part of this was stolen by "someone" and placed in the TRACK_README. I agree a part of it should also go into docs but some of this about randomization is relevant here.

Yeah, how about a one-liner that links to the docs?

... Also, I move to hunt down that "someone" and bring him to justice! ;) 🍻

(clears his throat to appear serious) So, which parts do you feel should be in-lined, @parkerl ? As I'm suggesting in this PR, we're really nudging pretty hard to go read that section in the central guide. I'm of the mind that these characteristics (including how some tooling randomizes tests while others don't) are really part and parcel to this general discussion about test suites.

parkerl

Nice work @jtigger

parkerl · 2017-03-27T03:50:16Z

TRAVIS

+  - use the exercise's solution as the code under test.
+- ensure all test cases are enabled;
+- be executed on Travis CI;
+- be easy to run on a contributor/maintainer's local machine (i.e. a build script with any needed instructions).



There is a placeholder for this as the last bullet under "Submitting a Pull Request" in the TRACK_README template.

parkerl · 2017-03-27T03:51:24Z

TRAVIS


-Many test frameworks will randomize the order of the tests when running them. This is an excellent practice, which helps ensure that subsequent tests are not dependent on side effects from earlier tests. However, in order to simulate TDD we want tests to run *in the order that they are defined*, and we want them to *fail fast*, that is to say, as soon as the test suite encounters a failure, we want the execution to stop. This ensures that the person implementing the solution sees only one error or failure message at a time, unless they make a change which causes prior tests to fail.
+As described in [Contributing to Exercism: Writing exercise test suites](writing_tests.md), exercise test suites are written such that all but the first test case are skipped/pending/ignored.  For the purpose of the track test suite, all test cases should be enabled (simulating an Exercism user having completed the exercise).


parkerl · 2017-03-27T03:52:43Z

TRAVIS


-**Avoiding accidental git check-ins**


parkerl · 2017-03-27T03:56:03Z

writing_tests.md

@@ -0,0 +1,16 @@
+_(in fact, this would be part of the new "docs" repo, not a file in this repo.)_


Part of this was stolen by "someone" and placed in the TRACK_README. I agree a part of it should also go into docs but some of this about randomization is relevant here.

jtigger · 2017-03-28T14:10:58Z

Okay, I'm really diggin' the copy editing! Thank you for the thorough review, folks.

I believe I addressed the points brought up. A little discussion still about landing spots for a few bits.

kytrinyx · 2017-03-29T20:03:43Z

A little discussion still about landing spots for a few bits.

Would you rephrase? I can't figure out how to parse this. (Sorry! I am traveling, brain might not actually be working right) 🌷

jtigger · 2017-04-04T15:04:34Z

Sorry for the delay, @kytrinyx. In fact, I missed your suggestion to put in a one-liner and link to the docs. That's a perfect solution, here.

That reference would be to the page requested by exercism/docs#8. I'm gonna update that issue to request the reference be made as part of that work.

parkerl · 2017-05-02T03:54:53Z

@jtigger I'm lost. Where are we with this one?

kytrinyx · 2019-02-26T00:15:52Z

I ended up removing the issue template, as it contained too many assumptions about the state of the track. I'm going to go ahead and close this issue.

Hoist CI build requirements to top of issue template

1f2d116

- extract background for writing high quality exercise test suites to a separate file.

kytrinyx requested changes Mar 23, 2017

View reviewed changes

parkerl reviewed Mar 27, 2017

View reviewed changes

Nips and tucks.

e3127a4

jtigger mentioned this pull request Apr 4, 2017

Document our philosophy / best practices about test suites exercism/legacy-docs#8

Closed

3 tasks

kytrinyx closed this Feb 26, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Hoist CI build requirements to top of issue template #12

Hoist CI build requirements to top of issue template #12

jtigger commented Mar 23, 2017 •

edited

Loading

kytrinyx commented Mar 23, 2017

kytrinyx left a comment

kytrinyx Mar 23, 2017

jtigger Mar 28, 2017

kytrinyx Mar 23, 2017

jtigger Mar 28, 2017

kytrinyx Mar 23, 2017

parkerl Mar 27, 2017

jtigger Mar 28, 2017

kytrinyx Mar 23, 2017

parkerl Mar 27, 2017

kytrinyx Mar 23, 2017

kytrinyx Mar 23, 2017

kytrinyx Mar 23, 2017

kytrinyx Mar 23, 2017

parkerl Mar 27, 2017

kytrinyx Mar 23, 2017

parkerl Mar 27, 2017

kytrinyx Mar 27, 2017

jtigger Mar 28, 2017

jtigger Mar 28, 2017

parkerl left a comment

parkerl Mar 27, 2017

parkerl Mar 27, 2017

parkerl Mar 27, 2017

parkerl Mar 27, 2017

jtigger commented Mar 28, 2017

kytrinyx commented Mar 29, 2017

jtigger commented Apr 4, 2017

parkerl commented May 2, 2017

kytrinyx commented Feb 26, 2019


		Many test frameworks will randomize the order of the tests when running them. This is an excellent practice, which helps ensure that subsequent tests are not dependent on side effects from earlier tests. However, in order to simulate TDD we want tests to run in the order that they are defined, and we want them to fail fast, that is to say, as soon as the test suite encounters a failure, we want the execution to stop. This ensures that the person implementing the solution sees only one error or failure message at a time, unless they make a change which causes prior tests to fail.
		As described in [Contributing to Exercism: Writing exercise test suites](writing_tests.md), exercise test suites are written such that all but the first test case are skipped/pending/ignored. For the purpose of the track test suite, all test cases should be enabled (simulating an Exercism user having completed the exercise).


		Whatever the mechanism—functions, methods, annotations, directives, commenting out tests, or some other approach—these are changes made directly to the test file. The person solving the exercise will need to edit the test file in order to "activate" each subsequent test.
		It's important that if we rewrite files in any way during a test run, that these changes do not accidentally get checked in to the git repository.


		Any tests that are marked as skipped will not be verified by the track test suite unless special care is taken.
		Therefore, many language tracks write the track test suite in such a way that it _copies_ the exercise to a temporary location outside of the git repository before editing or rewriting the exercise files during a test run.


		Additionally, in some programming languages, the name of the file containing the solution is hard-coded in the test suite, and the example solution is not named in the way that we expect people to name their files.
		_Configuring Travis CI_

		@@ -0,0 +1,16 @@
		_(in fact, this would be part of the new "docs" repo, not a file in this repo.)_

Hoist CI build requirements to top of issue template #12

Hoist CI build requirements to top of issue template #12

Conversation

jtigger commented Mar 23, 2017 • edited Loading

kytrinyx commented Mar 23, 2017

kytrinyx left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

parkerl left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jtigger commented Mar 28, 2017

kytrinyx commented Mar 29, 2017

jtigger commented Apr 4, 2017

parkerl commented May 2, 2017

kytrinyx commented Feb 26, 2019

jtigger commented Mar 23, 2017 •

edited

Loading