Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

runtimetest: Granular TAP output #1

Closed
wants to merge 3 commits into from
Closed

runtimetest: Granular TAP output #1

wants to merge 3 commits into from

Conversation

wking
Copy link

@wking wking commented Jan 11, 2017 •
edited
Loading

Granular TAP output makes it easier to see what's being tested (and what's going wrong). You could feed this into any TAP harness you like, but I've chosen prove (it seems popular). You need a TAP harness looking for test failures, because runtimetest now returns zero when it successfully runs the tests (regardless of whether the tests all pass).

The OPTIND shift is out of getopts(1p).

I've dropped the “VALIDATING RUNTIME” header since prove displays the command it's running in its usual output (e.g. /usr/bin/runc .. ok).

Full example output:

/usr/bin/runc .. 
TAP version 13
ok 1 - # SKIP root.readonly falsy
ok 2 - hostname matches expected value
ok 3 - mount "/proc" found
ok 4 - mount "/dev" found
ok 5 - mount "/dev/pts" found
ok 6 - mount "/dev/shm" found
ok 7 - mount "/dev/mqueue" found
ok 8 - mount "/sys" found
ok 9 - expected capability CAP_CHOWN set
ok 10 - expected capability CAP_DAC_OVERRIDE set
ok 11 - unexpected capability CAP_DAC_READ_SEARCH not set
ok 12 - expected capability CAP_FOWNER set
ok 13 - expected capability CAP_FSETID set
ok 14 - expected capability CAP_KILL set
ok 15 - expected capability CAP_SETGID set
ok 16 - expected capability CAP_SETUID set
ok 17 - expected capability CAP_SETPCAP set
ok 18 - unexpected capability CAP_LINUX_IMMUTABLE not set
ok 19 - expected capability CAP_NET_BIND_SERVICE set
ok 20 - unexpected capability CAP_NET_BROADCAST not set
ok 21 - unexpected capability CAP_NET_ADMIN not set
ok 22 - expected capability CAP_NET_RAW set
ok 23 - unexpected capability CAP_IPC_LOCK not set
ok 24 - unexpected capability CAP_IPC_OWNER not set
ok 25 - unexpected capability CAP_SYS_MODULE not set
ok 26 - unexpected capability CAP_SYS_RAWIO not set
ok 27 - expected capability CAP_SYS_CHROOT set
ok 28 - unexpected capability CAP_SYS_PTRACE not set
ok 29 - unexpected capability CAP_SYS_PACCT not set
ok 30 - unexpected capability CAP_SYS_ADMIN not set
ok 31 - unexpected capability CAP_SYS_BOOT not set
ok 32 - unexpected capability CAP_SYS_NICE not set
ok 33 - unexpected capability CAP_SYS_RESOURCE not set
ok 34 - unexpected capability CAP_SYS_TIME not set
ok 35 - unexpected capability CAP_SYS_TTY_CONFIG not set
ok 36 - expected capability CAP_MKNOD set
ok 37 - unexpected capability CAP_LEASE not set
ok 38 - expected capability CAP_AUDIT_WRITE set
ok 39 - unexpected capability CAP_AUDIT_CONTROL not set
ok 40 - expected capability CAP_SETFCAP set
ok 41 - unexpected capability CAP_MAC_OVERRIDE not set
ok 42 - unexpected capability CAP_MAC_ADMIN not set
ok 43 - unexpected capability CAP_SYSLOG not set
ok 44 - unexpected capability CAP_WAKE_ALARM not set
ok 45 - unexpected capability CAP_BLOCK_SUSPEND not set
ok 46 - unexpected capability CAP_AUDIT_READ not set
ok 47 - lstat default symlink /dev/fd
ok 48 - default symlink /dev/fd is a symlink
ok 49 - default symlink /dev/fd has expected target
ok 50 - lstat default symlink /dev/stdin
ok 51 - default symlink /dev/stdin is a symlink
ok 52 - default symlink /dev/stdin has expected target
ok 53 - lstat default symlink /dev/stdout
ok 54 - default symlink /dev/stdout is a symlink
ok 55 - default symlink /dev/stdout has expected target
ok 56 - lstat default symlink /dev/stderr
ok 57 - default symlink /dev/stderr is a symlink
ok 58 - default symlink /dev/stderr has expected target
ok 59 - mount /proc has expected type
ok 60 - mount /sys has expected type
ok 61 - mount /dev/pts has expected type
ok 62 - mount /dev/shm has expected type
ok 63 - stat default device /dev/null
ok 64 - default device /dev/null is a device
ok 65 - stat default device /dev/zero
ok 66 - default device /dev/zero is a device
ok 67 - stat default device /dev/full
ok 68 - default device /dev/full is a device
ok 69 - stat default device /dev/random
ok 70 - default device /dev/random is a device
ok 71 - stat default device /dev/urandom
ok 72 - default device /dev/urandom is a device
ok 73 - stat default device /dev/tty
ok 74 - default device /dev/tty is a device
ok 75 - stat default device /dev/ptmx
ok 76 - default device /dev/ptmx is a device
ok 77 - # SKIP linux.devices (no devices configured)
ok 78 - has expected working directory
ok 79 - has expected environment variable PATH
ok 80 - has expected environment variable TERM
ok 81 - has expected user ID
ok 82 - has expected group ID
ok 83 - has expected number of process arguments
ok 84 - has expected process argument 0
ok 85 - has expected noNewPrivileges
ok 86 - # SKIP linux.resources.oomScoreAdj falsy
ok 87 - has expected soft RLIMIT_NOFILE
ok 88 - has expected hard RLIMIT_NOFILE
ok 89 - # SKIP linux.uidMappings checks (no mappings specified)
ok 90 - # SKIP linux.gidMappings checks (no mappings specified)
1..90
ok
All tests successful.
Files=1, Tests=90,  0 wallclock secs ( 0.02 usr  0.00 sys +  0.02 cusr  0.00 csys =  0.04 CPU)
Result: PASS

And failing (if I just run runtimetest in my host namespace):

$ oci-runtime-tool generate >config.json
$ ./runtimetest
TAP version 13
ok 1 - # SKIP root.readonly falsy
not ok 2 - hostname matches expected value
# expected mrsdalloway, actual myhost
ok 3 - mount "/proc" found
not ok 4 - expected mount "/dev" found
ok 5 - mount "/dev/pts" found
ok 6 - mount "/dev/shm" found
ok 7 - mount "/dev/mqueue" found
ok 8 - mount "/sys" found
not ok 9 - expected capability CAP_CHOWN set
not ok 10 - expected capability CAP_DAC_OVERRIDE set
ok 11 - unexpected capability CAP_DAC_READ_SEARCH not set
not ok 12 - expected capability CAP_FOWNER set
not ok 13 - expected capability CAP_FSETID set
not ok 14 - expected capability CAP_KILL set
not ok 15 - expected capability CAP_SETGID set
not ok 16 - expected capability CAP_SETUID set
not ok 17 - expected capability CAP_SETPCAP set
ok 18 - unexpected capability CAP_LINUX_IMMUTABLE not set
not ok 19 - expected capability CAP_NET_BIND_SERVICE set
ok 20 - unexpected capability CAP_NET_BROADCAST not set
ok 21 - unexpected capability CAP_NET_ADMIN not set
not ok 22 - expected capability CAP_NET_RAW set
ok 23 - unexpected capability CAP_IPC_LOCK not set
ok 24 - unexpected capability CAP_IPC_OWNER not set
ok 25 - unexpected capability CAP_SYS_MODULE not set
ok 26 - unexpected capability CAP_SYS_RAWIO not set
not ok 27 - expected capability CAP_SYS_CHROOT set
ok 28 - unexpected capability CAP_SYS_PTRACE not set
ok 29 - unexpected capability CAP_SYS_PACCT not set
ok 30 - unexpected capability CAP_SYS_ADMIN not set
ok 31 - unexpected capability CAP_SYS_BOOT not set
ok 32 - unexpected capability CAP_SYS_NICE not set
ok 33 - unexpected capability CAP_SYS_RESOURCE not set
ok 34 - unexpected capability CAP_SYS_TIME not set
ok 35 - unexpected capability CAP_SYS_TTY_CONFIG not set
not ok 36 - expected capability CAP_MKNOD set
ok 37 - unexpected capability CAP_LEASE not set
not ok 38 - expected capability CAP_AUDIT_WRITE set
ok 39 - unexpected capability CAP_AUDIT_CONTROL not set
not ok 40 - expected capability CAP_SETFCAP set
ok 41 - unexpected capability CAP_MAC_OVERRIDE not set
ok 42 - unexpected capability CAP_MAC_ADMIN not set
ok 43 - unexpected capability CAP_SYSLOG not set
ok 44 - unexpected capability CAP_WAKE_ALARM not set
ok 45 - unexpected capability CAP_BLOCK_SUSPEND not set
ok 46 - unexpected capability CAP_AUDIT_READ not set
ok 47 - lstat default symlink /dev/fd
ok 48 - default symlink /dev/fd is a symlink
ok 49 - default symlink /dev/fd has expected target
ok 50 - lstat default symlink /dev/stdin
ok 51 - default symlink /dev/stdin is a symlink
ok 52 - default symlink /dev/stdin has expected target
ok 53 - lstat default symlink /dev/stdout
ok 54 - default symlink /dev/stdout is a symlink
ok 55 - default symlink /dev/stdout has expected target
ok 56 - lstat default symlink /dev/stderr
ok 57 - default symlink /dev/stderr is a symlink
ok 58 - default symlink /dev/stderr has expected target
ok 59 - mount /dev/pts has expected type
ok 60 - mount /dev/shm has expected type
ok 61 - mount /proc has expected type
ok 62 - mount /sys has expected type
ok 63 - stat default device /dev/null
ok 64 - default device /dev/null is a device
ok 65 - stat default device /dev/zero
ok 66 - default device /dev/zero is a device
ok 67 - stat default device /dev/full
ok 68 - default device /dev/full is a device
ok 69 - stat default device /dev/random
ok 70 - default device /dev/random is a device
ok 71 - stat default device /dev/urandom
ok 72 - default device /dev/urandom is a device
ok 73 - stat default device /dev/tty
ok 74 - default device /dev/tty is a device
ok 75 - stat default device /dev/ptmx
ok 76 - default device /dev/ptmx is a device
ok 77 - # SKIP linux.devices (no devices configured)
not ok 78 - has expected working directory
# working directory expected: /, actual: /home/wking/.local/lib/go/src/github.com/opencontainers/runtime-tools
not ok 79 - has expected environment variable PATH
# environment variable PATH expected: /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin, actual: /home/wking/bin:/home/wking/.local/bin:/home/wking/.gem/ruby/2.1.0/bin:/home/wking/.gem/ruby/2.0.0/bin:/usr/local/bin:/usr/bin:/bin:/opt/bin:/usr/x86_64-pc-linux-gnu/gcc-bin/4.9.3:/usr/sbin:/sbin
ok 80 - has expected environment variable TERM
not ok 81 - has expected user ID
# user ID expected: 0, actual 1000
not ok 82 - has expected group ID
# group ID expected: 0, actual 1000
ok 83 - has expected number of process arguments
not ok 84 - has expected process argument 0
# expected process argument 0: sh, actual: init [3]
ok 85 - has expected noNewPrivileges
ok 86 - # SKIP linux.resources.oomScoreAdj falsy
ok 87 - has expected soft RLIMIT_NOFILE
not ok 88 - has expected hard RLIMIT_NOFILE
# hard RLIMIT_NOFILE expected: 1024, actual: 4096
ok 89 - # SKIP linux.uidMappings checks (no mappings specified)
ok 90 - # SKIP linux.gidMappings checks (no mappings specified)
1..90

@wking wking mentioned this pull request Jan 11, 2017
Merged
@wking
Copy link
Author

wking commented Jan 11, 2017

You could also pull the TAP harness outside of test_runtime.sh, and suggest folks run:

$ prove sudo test_runtime.sh

or similar. That way folks can easily swap in their own preferred harness. Let me know if you want me to reroll taking that approach.

@wking
Copy link
Author

wking commented Jan 13, 2017

Better skip support is on the way with mndrix/tap-go#6, so it's probably best to hold off here until that lands and I reroll this to take advantage of it.

@wking
Copy link
Author

wking commented Jan 14, 2017

I've pushed b462307 → f3c0d27 with .Skip handling and prove pulled out of test_runtime.sh.

wking added 3 commits January 16, 2017 20:50
@wking
Revert "godep: Add go-multierror dependency"
168230f 
This reverts commit 67daea2.

We won't need these with granular TAP handling.

Conflicts:
	Godeps/Godeps.json

Signed-off-by: W. Trevor King <[email protected]>
@wking
godep: Bump tap-go to pick up the Skip method
7d409aa 
I couldn't figure out how to do this with 'godep update', so I just
copied the files over and bumped the hash by hand.

Signed-off-by: W. Trevor King <[email protected]>
@wking
runtimetest: Make TAP output more granular
cf0d014 
Granular TAP output makes it easier to see what's being tested (and
what's going wrong).  You could feed this into any TAP harness you
like, but I've chosen prove (it seems popular).  You need a TAP
harness looking for test failures, because runtimetest now returns
zero when it successfully runs the tests (regardless of whether the
tests all pass).

With this commit, runtimetest will exit zero when it can successfully
*run the tests* (regardless of whether all the tests pass).  A TAP
harness like prove should be used to consume the output TAP to decide
if validation succeeded or not.  To avoid confusing the TAP consumer,
all of the non-TAP logging goes to stderr (and not stdout).

Signed-off-by: W. Trevor King <[email protected]>
@wking
Copy link
Author

wking commented Jan 17, 2017

Shifted to opencontainers#308.

@wking wking closed this Jan 17, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@wking