From 7978cc95fe0c12407cddf6ea0f31bbb9f2b26b02 Mon Sep 17 00:00:00 2001 From: Eric Myhre Date: Sun, 27 Nov 2022 00:30:16 +0100 Subject: [PATCH 1/2] Add a readme for the testexec extension and its conventions. --- README.md | 2 +- testexec/README.md | 111 +++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 112 insertions(+), 1 deletion(-) create mode 100644 testexec/README.md diff --git a/README.md b/README.md index 5bf1c37..c9a6101 100644 --- a/README.md +++ b/README.md @@ -307,8 +307,8 @@ and a hunk called `mytest/output` and put the expected output in it, and you're There's a bunch more features too, like ways to check exit codes, ways to check stdout and stderr streams separately, ways to populate fixture filesystems (using hunks like `mytest/fs/dirname/filename.ext`), etc. -See that package for more details! +See that package, and its additional README, for more details! Changelog diff --git a/testexec/README.md b/testexec/README.md new file mode 100644 index 0000000..b771ace --- /dev/null +++ b/testexec/README.md @@ -0,0 +1,111 @@ +the testexec extension to testmark +================================== + +See the README in the parent directory for a broader introduction to testmark as a format. + +"testexec" is a convention used within the testmark format to describe +tests for executing processes. + +Use the [well-known hunk names](well-known-hunk-names) to write your fixtures, +then connect it to `go test` with the connecting the code you can see in the [example code](#examples). +It supports [autopatching and fixture generation](#autopatching), +and the [exec hooks are configurable](#configurable-exec-hooks). + + +Well-known hunk names +--------------------- + +The "testexec" extension is as simple as a series of well-known hunk names: + +Firstly, either of these two forms can be used to specify the commands to test: + +- "`script`" -- will feed the entire hunk of text as a script to a shell (by default, bash). +- "`sequence`" -- will run each line as a command as its own executable (parsing args by simply whitespace split). + +("sequence" mode is meant to be a little less complicated to implement, and more portable, but is also a bit less flexible. +"script" mode can do whatever the shell does -- so, handle quoting, redirections, etc -- the only challenge is then you have to know what your shell does!) +i +Second, there's the hunks that contain output assertions: + +- "`output`" -- if present, will cause the commands to be given a unified stdout and stderr buffer, and it will be checked against this data when done. +- "`stdout`" -- if present, works like "output", but only collects stdout. Cannot be combined with "output". +- "`stderr`" -- ditto "stdout", but for (you guessed it) stderr. +- "`exitcode`" -- if present, should contains a base-10 number for the expected exit code. If not present, an exitcode of 0 will be expected. + +For hunks not present, nothing will be checked. +(In other words, if you have a `stdout` hunk, but not a `stderr` hunk, whatever the stderr output is, it will be ignored by the test.) + +Note that not every testexec dataset has to contain _any_ of "output", "stdout", "stderr", or "exitcode": +containing none of them means the commands in the sequence will all be run, +and the exitcode is expected to be zero from each, +and nothing about their stdout nor stderr will be checked. +(If you want to mandate that their output is empty, you must create a blank data hunk to say so explicitly.) + +Input streams (a.k.a. "stdin") can also be specified: + +- "`input`" -- if present, will be fed to the stdin stream of the execution. + +Beyond this, you can also specify fixture filesystems: + +- "`fs/somedir/thefile.ext`" -- (and names of that general pattern: anything starting with "fs/") + +And last of all, sequences of causally related tests can be created: +given a series of hunks with names like the following: + +``` +test-one/script +test-one/then-test-two/script +test-one/then-test-two/then-test-three/script +test-one/then-test-two-b/script +``` + +In a scenario like the above, the nested "then-*" tests will inherit the working directory from their parent -- +or more specifically, a *copy* of it. +So: the script for `then-test-two` receives whatever the filesystem state left behind by `test-one` was; +the `then-test-three` script receives whatever filesystem state left behind by `then-test-two` was; +and `then-test-two-b` receives the filesystem as it was *after `test-one`* (! _not_ what was left by `then-test-three`). +(This lets you do one-time setup work, or build narratives -- while also still keeping tests isolated.) + + +Examples +-------- + +See the `selfexercise.md` file in this directory for fully worked examples. + +See the `testexec_test.go` file for the complete wiring to execute the examples -- +including the autopatcher support. + + +Autopatching +------------ + +**Yes** -- this testexec implementation supports automatic fixture regeneration. + +There's a tiny amount of setup required: +you do have to create a PatchAccumulator, +assign it to the `Tester{}.Patches` field, +and then tell the PatchAccumulator where to write any updates to. +(This is the norm for use of the patching system.) +(PRs welcome for further automating this.) +Once that setup is done: + +Just write your 'script' hunk, +and whichever of the 'output', 'stdout', 'stderr', or 'exitcode' hunks that you want to track, +and run your tests like `go test ./... -testmark.regen`. +The document will be patched with the actual output inserted into those hunks. + +Note that regen mode will only update hunks that already exist; it won't add them. +(E.g., if you only have a `stdout` hunk, regen won't _add_ a `stderr` hunk.) + + +Configurable Exec Hooks +----------------------- + +The default behaviors for this testexec implementation are: + +- the exec mode does literally the system exec calls (e.g. for a single process). +- the script mode will invoke bash and feed the script to it. + +This is customizable in the Go code. +See the `testexec.Tester` struct, and its fields `ExecFn` and `ScriptFn` +for the two callbacks which can replace the default execution behaviors. From 3c24d09dbeec7051902b64d8e6836032576ae2f2 Mon Sep 17 00:00:00 2001 From: Eric Myhre Date: Sun, 27 Nov 2022 02:08:04 +0100 Subject: [PATCH 2/2] Expanding greatly in response to code review. Some typo fixes. Much more linking. A few more configuration hooks get mentions. Features have a pattern description before the literal example. Examples are more completely narrated. Added a whole section about script-vs-sequence (because the one tiny paranthetical about that wasn't even close to enough). --- README.md | 2 +- testexec/README.md | 100 +++++++++++++++++++++++++++++++++++++-------- 2 files changed, 83 insertions(+), 19 deletions(-) diff --git a/README.md b/README.md index c9a6101..5cb9187 100644 --- a/README.md +++ b/README.md @@ -308,7 +308,7 @@ There's a bunch more features too, like ways to check exit codes, ways to check stdout and stderr streams separately, ways to populate fixture filesystems (using hunks like `mytest/fs/dirname/filename.ext`), etc. -See that package, and its additional README, for more details! +See that package, and its additional [README](./testexec/README.md), for more details! Changelog diff --git a/testexec/README.md b/testexec/README.md index b771ace..5a96961 100644 --- a/testexec/README.md +++ b/testexec/README.md @@ -1,12 +1,12 @@ the testexec extension to testmark ================================== -See the README in the parent directory for a broader introduction to testmark as a format. +See the [README](../README.md) in the parent directory for a broader introduction to testmark as a format. "testexec" is a convention used within the testmark format to describe tests for executing processes. -Use the [well-known hunk names](well-known-hunk-names) to write your fixtures, +Use the [well-known hunk names](#well-known-hunk-names) to write your fixtures, then connect it to `go test` with the connecting the code you can see in the [example code](#examples). It supports [autopatching and fixture generation](#autopatching), and the [exec hooks are configurable](#configurable-exec-hooks). @@ -22,9 +22,8 @@ Firstly, either of these two forms can be used to specify the commands to test: - "`script`" -- will feed the entire hunk of text as a script to a shell (by default, bash). - "`sequence`" -- will run each line as a command as its own executable (parsing args by simply whitespace split). -("sequence" mode is meant to be a little less complicated to implement, and more portable, but is also a bit less flexible. -"script" mode can do whatever the shell does -- so, handle quoting, redirections, etc -- the only challenge is then you have to know what your shell does!) -i +(See [Script vs Sequence](#script-vs-sequence) for more on why there's two options.) + Second, there's the hunks that contain output assertions: - "`output`" -- if present, will cause the commands to be given a unified stdout and stderr buffer, and it will be checked against this data when done. @@ -47,10 +46,14 @@ Input streams (a.k.a. "stdin") can also be specified: Beyond this, you can also specify fixture filesystems: -- "`fs/somedir/thefile.ext`" -- (and names of that general pattern: anything starting with "fs/") +- "`fs/*`" -- everything under here will be placed in a (temporary!) working directory during the run. +- "`fs/somedir/thefile.ext`" -- for example, causes "somedir" to be created, and places "thefile.ext" inside it. + +And last of all, sequences of causally related tests can be created. +Any time a testexec script or sequence has siblings named "`then-*`", +that creates sub-tests. -And last of all, sequences of causally related tests can be created: -given a series of hunks with names like the following: +For example, given a series of hunks with names like the following: ``` test-one/script @@ -59,20 +62,32 @@ test-one/then-test-two/then-test-three/script test-one/then-test-two-b/script ``` -In a scenario like the above, the nested "then-*" tests will inherit the working directory from their parent -- +In the scenario above, a total of four tests will occur. +Each of the tests that are related by the nested "`then-*`" hunk naming will inherit the working directory from their parent -- or more specifically, a *copy* of it. -So: the script for `then-test-two` receives whatever the filesystem state left behind by `test-one` was; -the `then-test-three` script receives whatever filesystem state left behind by `then-test-two` was; -and `then-test-two-b` receives the filesystem as it was *after `test-one`* (! _not_ what was left by `then-test-three`). -(This lets you do one-time setup work, or build narratives -- while also still keeping tests isolated.) +Tests that *aren't* related don't receive copies of the working directories! +(This means you can make diverging tests of various filesystem state evolution stories easily.) + +Walking through the scenario above step by step: + +- the script for `test-one` executes first; +- the script for `then-test-two` receives whatever filesystem state was left behind by the evalution of `test-one`; +- the script for `then-test-three` receives whatever filesystem state was left behind by `then-test-two`; +- and the script for `then-test-two-b` receives the filesystem as it was *after `test-one`*. + - Note! _not_ what was left by `then-test-three`! The relationships are only explicit :) and the order of the testmark hunks doesn't matter. + +The "`then-*`" system (and its helpful support for filesystem forking), +lets you do one-time setup work, or build narratives -- while still keeping tests cleanly isolated and very explicit. Examples -------- -See the `selfexercise.md` file in this directory for fully worked examples. +See the [`selfexercise.md`](./selfexercise.md) file in this directory for fully worked examples. +(Note: if you're reading this on Github: You may also want to look at the raw mode of it! +You won't see the testmark annotations in the rendered markdown on Github!) -See the `testexec_test.go` file for the complete wiring to execute the examples -- +See the [`testexec_test.go`](./testexec_test.go) file for an example of the complete wiring to execute the examples -- including the autopatcher support. @@ -98,14 +113,63 @@ Note that regen mode will only update hunks that already exist; it won't add the (E.g., if you only have a `stdout` hunk, regen won't _add_ a `stderr` hunk.) -Configurable Exec Hooks ------------------------ +Script vs Sequence +------------------ + +The testexec convention covers two different kinds of instruction specification hunk: +"script" and "sequence". + +Each mode is similar, and the content of each looks roughly like a shell script, but they differ slightly. + +"sequence" mode is defined as just splitting words based on whitespace, and processing args that way. +It's meant to be a little less complicated to implement, and is a bit more portable (it's not invoking a shell!), +but of course it's also a bit less flexible. + +"script" mode is defined as doing whatever a shell does -- so, it should handle quoting, support redirections, etc. +In exchange, the challenge is then you have to know what your shell does! +(And internally, the default implementation is literally executing the host shell -- so there's some portability considerations, there.) + +In this library, each of these modes can be set up to do something different, by use of different callbacks when you're setting up the code -- +see the next section about [configurable exec hooks](#configurable-exec-hooks). + +In practice: we often find that the "sequence" mode's callback is easy to implement +and connect to the program's "main" method... +which makes it very easy to write tests that exercise a CLI-style application! +(The alternative would likely be doing a separate "go install" phase and invoking the application again as a subprocess, +but this is considerably more complicated to implement and maintain.) +But we still also find people usually reaching for "script" mode when testing how one program interacts with others, +because "sequence" mode is incapable of supporting piping programs together, etc. +Ultimately, both approaches have some attractions :) + + +Configuration Hooks +------------------- + +### Configurable Exec Hooks The default behaviors for this testexec implementation are: -- the exec mode does literally the system exec calls (e.g. for a single process). +- the sequence mode does literally the system exec calls (e.g. for a single process). - the script mode will invoke bash and feed the script to it. This is customizable in the Go code. See the `testexec.Tester` struct, and its fields `ExecFn` and `ScriptFn` for the two callbacks which can replace the default execution behaviors. + +### Bring your own assertion library + +... if you want. + +You can configure your choice of test equality checking function in the `testexec.Tester` struct by setting the `AssertFn` field. + +(The authors of this package happen to like `frankban/quicktest`, for example -- but we didn't want to force that choice on you!) + +The default function does some very simple string comparisons, +and has zero dependencies, +but does not provide rich diff output or any other nice-to-have features. + +### Filtrations + +Sometimes you want to test an application that is mostly predictable, but perhaps includes some unpredictable outputs, like timestamps for example. + +You can set the `FilterFn` field in the `testexec.Tester` struct to apply some normalizing transforms to the output streams before doing the comparisons.