txm 
Purpose
Command-line program that verifies the correctness of code examples in a given
Markdown file. It parses the file for code blocks preceded by
HTML comments containing !test annotations, and checks that given
inputs to the given program result in given outputs.
Features
- Language-agnostic. Runs tests with any shell command.
- TAP format output.
- Process-level parallelism.
- Clear diagnostics when tests fail.
- Users retain full choice of formatting.
Non-features
- No compilation step.
- No language-specific features.
- No annotations visible in the rendered document.
example
README.md:
# console.log
The [console.log][1] function in [Node.js][2] stringifies the given arguments
and writes them to `stdout`, followed by a newline. For example:
<!-- !test program node -->
<!-- !test in simple example -->
console.log('a')
console.log(42)
console.log([1, 2, 3])
The output is:
<!-- !test out simple example -->
a
42
[ 1, 2, 3 ]
[1]: https://nodejs.org/api/console.html#console_console_log_data_args
[2]: https://nodejs.org/Run:
$ txm README.mdOutput:
TAP version 13 1..1 ok 1 simple example # 1/1 passed # OK
Examples of other use-cases:
Testing C code with gcc
Any sequence of shell commands is a valid !test program, so you can e.g. cat
the test input into a file, then compile and run it:
<!-- !test program
cat > /tmp/program.c
gcc /tmp/program.c -o /tmp/test-program && /tmp/test-program -->
Here is a simple example C program that computes the answer to life, the
universe, and everything:
<!-- !test in printf -->
#include <stdio.h>
int main () {
printf("%d\n", 6 * 7);
}
<!-- !test out printf -->
42TAP version 13 1..1 ok 1 printf # 1/1 passed # OK
In practice you might want to invoke mktemp in the !test program to avoid
multiple parallel tests overwrting each other's files. Or pass --jobs 1 to
run tests serially.
Replacing package imports with local file imports
(For example, require('module-name') → require('./index.js'))
In languages with package managers, users will likely be using your library by
importing it using its package name (e.g. require('module-name'). However,
it makes sense to actually run your tests such that they use your local
implementation (e.g. ./index.js, or whatever is listed as the main file in
package.json).
So here's a markdown file with a test program specified that loads the name of
the main file out of ./package.json, and replaces the first require(...)
call with that:
<!-- !test program
# First read stdin into a temporary file
TEMP_FILE="$(mktemp --suffix=js)"
cat > "$TEMP_FILE"
# Read the package name and main file from package.json
PACKAGE_NAME=$(node -e "console.log(require('./package.json').name)")
LOCAL_MAIN_FILE=$(node -e "console.log(require('./package.json').main)")
# Run a version of the input code where requires for the package name are
# replaced with the local file path
cat "$TEMP_FILE" \
| sed -e "s#require('$PACKAGE_NAME')#require('./$LOCAL_MAIN_FILE')#" \
| node
-->
Did you know you can also use `txm` as a module to use it programmatically?
<!-- !test in use library -->
const parseAndRunTests = require('txm')
parseAndRunTests(`
# Markdown heading!
<!-- !test program node -->
<!-- !test check print -->
require('assert')(true)
`)
It produces output onto console:
<!-- !test out use library -->
TAP version 13
1..1
ok 1 print
# 1/1 passed
# OKTAP version 13 1..1 ok 1 use library # 1/1 passed # OK
Redirecting stderr→stdout, to test both in the same
block
Prepending 2>&1 to a shell command redirects stderr
to stdout. This can be handy if you don't want to write separate !test out
and !test err blocks.
<!-- !test program 2>&1 node -->
<!-- !test in print to both stdout and stderr -->
console.error("This goes to stderr!")
console.log("This goes to stdout!")
<!-- !test out print to both stdout and stderr -->
This goes to stderr!
This goes to stdout!TAP version 13 1..1 ok 1 print to both stdout and stderr # 1/1 passed # OK
Testing a program that exits non-zero
txm assumes that if the test program exits non-zero, it must have been
unintentional. You can put || true after the program command to make the
shell swallow the exit code and pretend to txm that it was 0 and everything
is fine.
<!-- !test program node || true -->
<!-- !test in don't fail -->
console.log("Hi before throw!")
throw new Error("AAAAAA!")
<!-- !test out don't fail -->
Hi before throw!TAP version 13 1..1 ok 1 don't fail # 1/1 passed # OK
Testing examples that call assert
If your example code calls assert or such (which throw an error and exit
nonzero when the assert fails), then you don't really need an output block,
because the example already documents its assumptions.
In such cases you can use use a !test check annotation. This simply runs the
code, and checks that the program exits with status 0, ignoring its output.
<!-- !test program node -->
<!-- !test check laws of mathematics -->
const assert = require('assert')
assert(1 + 1 == 2)
TAP version 13 1..1 ok 1 laws of mathematics # 1/1 passed # OK
install
Requires Node.js. Install with npm install -g txm.
use
txm [--jobs <n>] [filename]
-
filename: Input file (default: read fromstdin) -
--jobs: How many tests may run in parallel.(default:os.cpus().length)Results will always be shown in insertion order in the output, regardless of the order parallel tests complete.
-
--version -
--help
txm exits 0 if and only if all tests pass.
Coloured output is on if outputting to a terminal, and off otherwise. If you
want no colors ever, set the env variable NO_COLOR=1. If you want to output
colour codes even when not in a TTY, set FORCE_COLOR=1.
Annotations (inside HTML comments):
-
!test program <program>The
<program>is run as a shell command for each following matching input/output pair. It gets the input onstdin, and is expected to produce the output onstdout.To use the same program for each test, just declare it once.
The program sees these environment variables:
TXM_INDEX(1-based number of test)TXM_NAME(name of test)TXM_INDEX_FIRST,TXM_INDEX_LAST(indexes of first and last tests)TXM_INPUT_LANG(the language tag of the input/check markdown code block, if applicable)
-
!test in <name>/!test out <name>/!test err <name>The next code block is read as given input, or expected stdout or stderr.
These are matched by
<name>, and may be anywhere. -
!test check <name>The next code block is read as a check test. The previously-specified program gets this as input, but its output is ignored. The test passes if the program exits
0.Use this for code examples that check their own correctness, (e.g. by calling
assert), or if your test program is a linter.
Note that 2 consecutive hyphens (--) inside HTML comments are disallowed by
the HTML spec. For this reason, txm lets you escape
hyphens: #- is automatically replaced by -. If you need to write literally
#-, write ##- instead, and so on. # acts normally everywhere else.
screenshot
