Skip to content

Commit

Permalink
Expanding greatly in response to code review.
Browse files Browse the repository at this point in the history
Some typo fixes.

Much more linking.

A few more configuration hooks get mentions.

Features have a pattern description before the literal example.

Examples are more completely narrated.

Added a whole section about script-vs-sequence (because the one tiny
paranthetical about that wasn't even close to enough).
  • Loading branch information
warpfork committed Nov 27, 2022
1 parent 7978cc9 commit 9c5618d
Show file tree
Hide file tree
Showing 2 changed files with 81 additions and 19 deletions.
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -308,7 +308,7 @@ There's a bunch more features too, like ways to check exit codes,
ways to check stdout and stderr streams separately,
ways to populate fixture filesystems (using hunks like `mytest/fs/dirname/filename.ext`), etc.

See that package, and its additional README, for more details!
See that package, and its additional [README](./testexec/README.md), for more details!


Changelog
Expand Down
98 changes: 80 additions & 18 deletions testexec/README.md
Original file line number Diff line number Diff line change
@@ -1,12 +1,12 @@
the testexec extension to testmark
==================================

See the README in the parent directory for a broader introduction to testmark as a format.
See the [README](../README.md) in the parent directory for a broader introduction to testmark as a format.

"testexec" is a convention used within the testmark format to describe
tests for executing processes.

Use the [well-known hunk names](well-known-hunk-names) to write your fixtures,
Use the [well-known hunk names](#well-known-hunk-names) to write your fixtures,
then connect it to `go test` with the connecting the code you can see in the [example code](#examples).
It supports [autopatching and fixture generation](#autopatching),
and the [exec hooks are configurable](#configurable-exec-hooks).
Expand All @@ -22,9 +22,8 @@ Firstly, either of these two forms can be used to specify the commands to test:
- "`script`" -- will feed the entire hunk of text as a script to a shell (by default, bash).
- "`sequence`" -- will run each line as a command as its own executable (parsing args by simply whitespace split).

("sequence" mode is meant to be a little less complicated to implement, and more portable, but is also a bit less flexible.
"script" mode can do whatever the shell does -- so, handle quoting, redirections, etc -- the only challenge is then you have to know what your shell does!)
i
(See [Script vs Sequence](#script-vs-sequence) for more on why there's two options.)

Second, there's the hunks that contain output assertions:

- "`output`" -- if present, will cause the commands to be given a unified stdout and stderr buffer, and it will be checked against this data when done.
Expand All @@ -47,10 +46,14 @@ Input streams (a.k.a. "stdin") can also be specified:

Beyond this, you can also specify fixture filesystems:

- "`fs/somedir/thefile.ext`" -- (and names of that general pattern: anything starting with "fs/")
- "`fs/*`" -- everything under here will be placed in a (temporary!) working directory during the run.
- "`fs/somedir/thefile.ext`" -- for example, causes "somedir" to be created, and places "thefile.ext" inside it.

And last of all, sequences of causally related tests can be created.
Any time a testexec script or sequence has siblings named "`then-*`",
that creates sub-tests.

And last of all, sequences of causally related tests can be created:
given a series of hunks with names like the following:
For example, given a series of hunks with names like the following:

```
test-one/script
Expand All @@ -59,20 +62,30 @@ test-one/then-test-two/then-test-three/script
test-one/then-test-two-b/script
```

In a scenario like the above, the nested "then-*" tests will inherit the working directory from their parent --
In this scenario above, a total of four tests will occur.
Each of the tests that are related by the nested "`then-*`" hunk naming will inherit the working directory from their parent --
or more specifically, a *copy* of it.
So: the script for `then-test-two` receives whatever the filesystem state left behind by `test-one` was;
the `then-test-three` script receives whatever filesystem state left behind by `then-test-two` was;
and `then-test-two-b` receives the filesystem as it was *after `test-one`* (! _not_ what was left by `then-test-three`).
(This lets you do one-time setup work, or build narratives -- while also still keeping tests isolated.)
Tests that *aren't* related don't receive copies of the working directories!
(This means you can make diverging tests of various filesystem state evolution stories easily.)

So: in the scenario above:
- the script for `then-test-two` receives whatever filesystem state was left behind by the evalution of `test-one`;
- the script for `then-test-three` receives whatever filesystem state was left behind by `then-test-two`;
- and the script for `then-test-two-b` receives the filesystem as it was *after `test-one`*.
- Note! _not_ what was left by `then-test-three`! The relationships are only explicit :) and the order of the testmark hunks doesn't matter.

The "then-*" system, combined with filesystem forking,
lets you do one-time setup work, or build narratives -- while still keeping tests cleanly isolated and very explicit.


Examples
--------

See the `selfexercise.md` file in this directory for fully worked examples.
See the [`selfexercise.md`](./selfexercise.md) file in this directory for fully worked examples.
(Note: if you're reading this on Github: You may also want to look at the raw mode of it!
You won't see the testmark annotations in the rendered markdown on Github!)

See the `testexec_test.go` file for the complete wiring to execute the examples --
See the [`testexec_test.go`](./testexec_test.go) file for an example of the complete wiring to execute the examples --
including the autopatcher support.


Expand All @@ -98,14 +111,63 @@ Note that regen mode will only update hunks that already exist; it won't add the
(E.g., if you only have a `stdout` hunk, regen won't _add_ a `stderr` hunk.)


Configurable Exec Hooks
-----------------------
Script vs Sequence
------------------

The testexec convention covers two different kinds of instruction specification hunk:
"script" and "sequence".

Each mode is similar, and the content of each looks roughly like a shell script, but they differ slightly.

"sequence" mode is defined as just splitting words based on whitespace, and processing args that way.
It's meant to be a little less complicated to implement, and is a bit more portable (it's not invoking a shell!),
but of course it's also a bit less flexible.

"script" mode is defined as doing whatever a shell does -- so, it should handle quoting, support redirections, etc.
In exchange, the challenge is then you have to know what your shell does!
(And internally, the default implementation is literally executing the host shell -- so there's some portability considerations, there.)

In this library, each of these modes can be set up to do something different, by use of different callbacks when you're setting up the code --
see the next section about [configurable exec hooks](#configurable-exec-hooks).

In practice: we often find that the "sequence" mode's callback is easy to implement
and connect to the program's "main" method...
which makes it very easy to write tests that exercise a CLI-style application!
(The alternative would likely be doing a separate "go install" phase and invoking the application again as a subprocess,
but this is considerably more complicated to implement and maintain.)
But we still also find people usually reaching for "script" mode when testing how one program interacts with others,
because "sequence" mode is incapable of supporting piping programs together, etc.
Ultimately, both approaches have some attractions :)


Configuration Hooks
-------------------

### Configurable Exec Hooks

The default behaviors for this testexec implementation are:

- the exec mode does literally the system exec calls (e.g. for a single process).
- the sequence mode does literally the system exec calls (e.g. for a single process).
- the script mode will invoke bash and feed the script to it.

This is customizable in the Go code.
See the `testexec.Tester` struct, and its fields `ExecFn` and `ScriptFn`
for the two callbacks which can replace the default execution behaviors.

### Bring your own assertion library

... if you want.

You can configure your choice of test equality checking function in the `testexec.Tester` struct by setting the `AssertFn` field.

(The authors of this package happen to like `frankban/quicktest`, for example -- but we didn't want to force that choice on you!)

The default function does some very simple string comparisons,
and has zero dependencies,
but does not provide rich diff output or any other nice-to-have features.

### Filtrations

Sometimes you want to test an application that is mostly predictable, but perhaps includes some unpredictable outputs, like timestamps for example.

You can set the `FilterFn` field in the `testexec.Tester` struct to apply some normalizing transforms to the output streams before doing the comparisons.

0 comments on commit 9c5618d

Please sign in to comment.