runtimetest: Make TAP output more granular #308
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
wking
force-pushed
the
granular-tap
branch
2 times, most recently
from
c30a87d to
459c384
Compare
|
I think that's too detailed... |
|
On Tue, Jan 17, 2017 at 12:22:00AM -0800, Ma Shimiao wrote:
I think that's too detailed...
Most users will not consume the raw TAP. They'll be looking at the
prove summary [1] (which is only a few lines). The detail is just for
the curious and for failures. For an example of the failure display,
here's what you get with a no-op runtime:
$ ./oci-runtime-tool generate >config.json
$ prove ./runtimetest
./runtimetest .. Failed 22/90 subtests
(less 5 skipped subtests: 63 okay)
Test Summary Report
-------------------
./runtimetest (Wstat: 0 Tests: 90 Failed: 22)
Failed tests: 2, 4, 9-10, 12-17, 19, 22, 27, 36, 38, 40
78-79, 81-82, 84, 88
Files=1, Tests=90, 0 wallclock secs ( 0.02 usr + 0.00 sys = 0.02 CPU)
Result: FAIL
Or showing the failed tests:
$ prove -f ./runtimetest
./runtimetest .. 1/?
not ok 2 - hostname matches expected value
not ok 4 - expected mount "/dev" found
not ok 9 - expected capability CAP_CHOWN set
not ok 10 - expected capability CAP_DAC_OVERRIDE set
not ok 12 - expected capability CAP_FOWNER set
not ok 13 - expected capability CAP_FSETID set
not ok 14 - expected capability CAP_KILL set
not ok 15 - expected capability CAP_SETGID set
not ok 16 - expected capability CAP_SETUID set
not ok 17 - expected capability CAP_SETPCAP set
not ok 19 - expected capability CAP_NET_BIND_SERVICE set
not ok 22 - expected capability CAP_NET_RAW set
not ok 27 - expected capability CAP_SYS_CHROOT set
not ok 36 - expected capability CAP_MKNOD set
not ok 38 - expected capability CAP_AUDIT_WRITE set
not ok 40 - expected capability CAP_SETFCAP set
not ok 78 - has expected working directory
not ok 79 - has expected environment variable PATH
not ok 81 - has expected user ID
not ok 82 - has expected group ID
not ok 84 - has expected process argument 0
not ok 88 - has expected hard RLIMIT_NOFILE
./runtimetest .. Failed 22/90 subtests
(less 5 skipped subtests: 63 okay)
Test Summary Report
-------------------
./runtimetest (Wstat: 0 Tests: 90 Failed: 22)
Failed tests: 2, 4, 9-10, 12-17, 19, 22, 27, 36, 38, 40
78-79, 81-82, 84, 88
Files=1, Tests=90, 0 wallclock secs ( 0.02 usr + 0.00 sys = 0.02 CPU)
Result: FAIL
Or showing the failures and comments:
$ prove -fo ./runtimetest
./runtimetest .. 1/?
not ok 2 - hostname matches expected value
# hostname expected: mrsdalloway, actual: my-hostname
not ok 4 - expected mount "/dev" found
not ok 9 - expected capability CAP_CHOWN set
not ok 10 - expected capability CAP_DAC_OVERRIDE set
not ok 12 - expected capability CAP_FOWNER set
not ok 13 - expected capability CAP_FSETID set
not ok 14 - expected capability CAP_KILL set
not ok 15 - expected capability CAP_SETGID set
not ok 16 - expected capability CAP_SETUID set
not ok 17 - expected capability CAP_SETPCAP set
not ok 19 - expected capability CAP_NET_BIND_SERVICE set
not ok 22 - expected capability CAP_NET_RAW set
not ok 27 - expected capability CAP_SYS_CHROOT set
not ok 36 - expected capability CAP_MKNOD set
not ok 38 - expected capability CAP_AUDIT_WRITE set
not ok 40 - expected capability CAP_SETFCAP set
not ok 78 - has expected working directory
# working directory expected: /, actual: /home/wking/src/runtime-tools
not ok 79 - has expected environment variable PATH
# environment variable PATH expected: /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin, actual: /home/wking/bin:/usr/local/bin:/usr/bin:/bin:/opt/bin:/usr/sbin:/sbin
not ok 81 - has expected user ID
# user ID expected: 0, actual: 1000
not ok 82 - has expected group ID
# group ID expected: 0, actual: 1000
not ok 84 - has expected process argument 0
# expected process argument 0: sh, actual: init [3]
not ok 88 - has expected hard RLIMIT_NOFILE
# hard RLIMIT_NOFILE expected: 1024, actual: 4096
./runtimetest .. Failed 22/90 subtests
(less 5 skipped subtests: 63 okay)
Test Summary Report
-------------------
./runtimetest (Wstat: 0 Tests: 90 Failed: 22)
Failed tests: 2, 4, 9-10, 12-17, 19, 22, 27, 36, 38, 40
78-79, 81-82, 84, 88
Files=1, Tests=90, 0 wallclock secs ( 0.02 usr + 0.00 sys = 0.02 CPU)
Result: FAIL
So you can easily tune the output to be as noisy as you like.
[1]: https://github.com/wking/ocitools-v2/blob/459c3848b94184d4831a00f4bff47b79592352b7/README.md#testing-oci-runtimes
|
|
My first reaction was that it was too granular but I see how it could be helpful. |
|
@wking can you rebase this? |
|
On Tue, Mar 07, 2017 at 09:36:49PM -0800, Ma Shimiao wrote:
@wking can you rebase this?
I expect this is going to be a pain in the neck to continually rebase
;). Can we at least get a nominal ok on the general idea from another
maintainer before I go through the trouble?
|
|
@liangchenye How do you think about this PR? |
|
If folks are considering v0.4.0 PRs in ascending-date order, I'd prefer they skip over this one and look at #439 first. That PR is current, and it's easier to rebase than this one. I'll try and get this rebased in the next week or so, but this PR will need another rebase if the older and also milestoned for v0.4.0 #93 lands first. |
|
#93 needs more work, I have put it into lower priority than this one. |
This was referenced Dec 13, 2017
|
@wking Others have been merged,I think it is time to update this pr. |
wking
added a commit
to wking/ocitools-v2
that referenced
this pull request
Jan 11, 2018
No semantic changes here, but the DRYer approach makes future pivots easier (e.g. opencontainers#308). Signed-off-by: W. Trevor King <[email protected]>
wking
added a commit
to wking/ocitools-v2
that referenced
this pull request
Jan 11, 2018
No semantic changes here, but the DRYer approach makes future pivots easier (e.g. opencontainers#308). Signed-off-by: W. Trevor King <[email protected]>
|
@wking Excuse me, how about this patch update? I'm ready release 0.4 version, I think this patch need to merge before the release. |
|
Sorry, I'm only ~ half way through, and I still need to make adjustments to handle |
|
Fine. |
|
@wking Can you update this pr? So we can put it in 0.6. |
|
Finally rebased onto master with 459c384 → 7c384c1. The diff is huge, and I only just got through it, so give it a careful read before merging. On the other hand, don't wait too long or we'll have another big rebase to go through ;). I'll go back through in the next few days and try to double check things as well. |
wking
force-pushed
the
granular-tap
branch
2 times, most recently
from
9ec0740 to
d7e4d91
Compare
alban
reviewed
Apr 6, 2018
cmd/runtimetest/main.go
Outdated
| if err == nil { | ||
| return true, nil | ||
| } else if err == io.EOF { | ||
| return false, nil |
There was a problem hiding this comment.
I find this surprising to return "non-readable" when the file might well be readable but just empty.
There was a problem hiding this comment.
I find this surprising to return "non-readable" when the file might well be readable but just empty.
Yeah, but I'm not sure how to detect readable-but-empty (and what I have here is what we've been using). Do you have ideas? The lack of support for that case makes this brittle, but I'm ok with it if we update our tests (e.g. here) to use non-empty test files. runtimetest is for internal testing only (not part of our public API), so a few cludges to work around the runtimetest limitation are unfortunate but survivable.
There was a problem hiding this comment.
Actually, our current tests use an empty file and treat an immediate EOF as success. So I am changing the logic. But the old logic doesn't make much sense to me. What if the runtime implemented linux.readonlyPaths by binding /dev/null over the listed paths? That would seem to satisfy the spec (which is pretty weak on this anyway), while still resulting in immediate EOFs.
There was a problem hiding this comment.
Maybe just add a comment to explain the situation:
} else if err == io.EOF {
// linux_readonly_paths.go only tests non-empty files. So if we find
// an empty file, the runtime did not apply the ReadonlyPaths correctly
return false, nil
I agree about updating the test you mention :)
There was a problem hiding this comment.
cmd/runtimetest/main.go
Outdated
| tmpfile.Close() | ||
| os.RemoveAll(filepath.Join(path, tmpfile.Name())) | ||
| if err != nil { |
There was a problem hiding this comment.
this branch cannot be taken because it was tested a few lines above.
alban
reviewed
Apr 6, 2018
| } | ||
| if spec.Linux.RootfsPropagation == "shared" { | ||
| return nil | ||
| c.harness.Ok(exposed, fmt.Sprintf("shared root propogation exposes %q", targetFile)) | ||
| } else { |
There was a problem hiding this comment.
I find the spec quite weak about this RootfsPropagation:
- a mount could be both shared and slave
rootfsPropagation=slaveis only meaningful if we say what is the master of that enslaved rootfs. Should it necessarily be wrt to a mount in the host namespace?
But for the purpose of this PR, your change is fine.
There was a problem hiding this comment.
Should it necessarily be wrt to a mount in the host namespace?
This is another case where we can work around runtimetest limitations by limiting the configs we create in validation/ to things runtimetest can check for ;).
But for the purpose of this PR, your change is fine.
Yeah, I'm fine making not growing this PR to fix everything ;). And if folks want me to spin off anything I am doing (like adjusting the readonly tests), let me know and I'll try to spin those out into their own PRs.
cmd/runtimetest/main.go
Outdated
| } else { | ||
| exists = true | ||
| } | ||
| rfcError, err := c.Ok(exists, specerror.DevicesAvailable, version, fmt.Sprintf("has a file at %s", description)) |
There was a problem hiding this comment.
specerror.DevicesAvailable -> condition
There was a problem hiding this comment.
ditto in the 6 other occurrences of specerror.DevicesAvailable below in this function
| }) | ||
| } | ||
|
|
||
| if description == "/dev/console (default device)" { |
There was a problem hiding this comment.
wouldn't it be better to test
if device.Path == "/dev/console" {
There was a problem hiding this comment.
wouldn't it be better to test
if device.Path == "/dev/console" {
I'm fine with that. My current check ensures that we do perform the rest of these checks if the config JSON explicitly included /dev/console in linux.devices. But we don't do that in validation/, and I don't expect us to start, so it probably doesn't matter either way. Do any maintainers have a preference?
cmd/runtimetest/main.go
Outdated
| continue | ||
| } | ||
|
|
||
| isSymlink := fi.Mode()&os.ModeSymlink == os.ModeSymlink |
There was a problem hiding this comment.
I find it more clear written this way:
isSymlink := fi.Mode()&os.ModeType == os.ModeSymlink
but that's equivalent in Golang because os.ModeType is a bitfield.
It confused me at first because in C, stat.st_mode is not a bitfield (man 7 inode):
S_IFLNK = S_IFREG | S_IFCHR
The following mask values are defined for the file type:
S_IFMT 0170000 bit mask for the file type bit field
S_IFSOCK 0140000 socket
S_IFLNK 0120000 symbolic link
S_IFREG 0100000 regular file
S_IFBLK 0060000 block device
S_IFDIR 0040000 directory
S_IFCHR 0020000 character device
S_IFIFO 0010000 FIFO
(Things I learnt today)
There was a problem hiding this comment.
I see it was already written like that before. Feel free to ignore my comment ;)
There was a problem hiding this comment.
I find it more clear written this way…
I like that argument. I've pushed the change as a new commit with 7ecd679, in case the maintainers want me to spin it out into a separate PR.
cmd/runtimetest/main.go
Outdated
|
|
||
| for i, path := range spec.Linux.ReadonlyPaths { | ||
| readable, err := testReadAccess(path) | ||
| if err != nil && !os.IsNotExist(err) { |
There was a problem hiding this comment.
Why ignoring the os.IsNotExist error?
There was a problem hiding this comment.
cmd/runtimetest/main.go
Outdated
| if err != nil && !os.IsNotExist(err) { | ||
| return err | ||
| } | ||
| if !writable { |
There was a problem hiding this comment.
why the if? The condition is already in the c.harness.Ok() below.
wking
force-pushed
the
granular-tap
branch
3 times, most recently
from
5e395e8 to
1e67d33
Compare
wking
added a commit
to wking/ocitools-v2
that referenced
this pull request
Apr 6, 2018
We've been using ModeSymlink as the mask since the check landed in da25004 (runtimetest: add linux default symbolic link validation, 2016-11-30, opencontainers#284). POSIX provides S_IS*(m) macros to portably interpret the mode type, but does not define values for each type [2]. Alban pointed out that st_mode is not a bitfield on Linux [1]. For example, Linux defines [3]: S_IFBLK 060000 S_IFDIR 040000 S_IFCHR 020000 So 'm&S_IFCHR == S_IFCHR', for example, would succeed for both character and block devices. Go translates the system values to a platform-agnostic bitfield [4], so the previous approach works on Go. But it may be confusing for people used to the native non-bitfield mode, so this commit moves us to an approach that does not rely on Go's using a bitfield. [1]: opencontainers#308 (comment) [2]: http://pubs.opengroup.org/onlinepubs/9699919799/basedefs/sys_stat.h.html [3]: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/include/uapi/linux/stat.h?h=v4.16#n9 [4]: https://github.com/golang/go/blob/b0d437f866eb8987cde7e6550cacd77876f36d4b/src/os/types.go#L45 Signed-off-by: W. Trevor King <[email protected]>
wking
added a commit
to wking/fileutils
that referenced
this pull request
Apr 6, 2018
POSIX provides S_IS*(m) macros to portably interpret the mode type, but does not define values for each type [2]. Alban pointed out that st_mode is not a bitfield on Linux [1]. For example, Linux defines [3]: S_IFBLK 060000 S_IFDIR 040000 S_IFCHR 020000 So 'm&S_IFCHR == S_IFCHR', for example, would succeed for both character and block devices. Go translates the system values to a platform-agnostic bitfield [4], so the previous approach works on Go. But it may be confusing for people used to the native non-bitfield mode, so this commit moves us to an approach that does not rely on Go's using a bitfield. [1]: opencontainers/runtime-tools#308 (comment) [2]: http://pubs.opengroup.org/onlinepubs/9699919799/basedefs/sys_stat.h.html [3]: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/include/uapi/linux/stat.h?h=v4.16#n9 [4]: https://github.com/golang/go/blob/b0d437f866eb8987cde7e6550cacd77876f36d4b/src/os/types.go#L45
wking
added a commit
to wking/fileutils
that referenced
this pull request
Apr 6, 2018
POSIX provides S_IS*(m) macros to portably interpret the mode type, but does not define values for each type [1]. Alban pointed out that st_mode is not a bitfield on Linux [2]. For example, Linux defines [3]: S_IFBLK 060000 S_IFDIR 040000 S_IFCHR 020000 So 'm&S_IFCHR == S_IFCHR', for example, would succeed for both character and block devices. Go translates the system values to a platform-agnostic bitfield [4], so the previous approach works on Go. But it may be confusing for people used to the native non-bitfield mode, so this commit moves us to an approach that does not rely on Go's using a bitfield. [1]: http://pubs.opengroup.org/onlinepubs/9699919799/basedefs/sys_stat.h.html [2]: opencontainers/runtime-tools#308 (comment) [3]: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/include/uapi/linux/stat.h?h=v4.16#n9 [4]: https://github.com/golang/go/blob/b0d437f866eb8987cde7e6550cacd77876f36d4b/src/os/types.go#L45
wking
added a commit
to wking/fileutils
that referenced
this pull request
Apr 6, 2018
POSIX provides S_IS*(m) macros to portably interpret the mode type, but does not define values for each type [1]. Alban pointed out that st_mode is not a bitfield on Linux [2]. For example, Linux defines [3]: S_IFBLK 060000 S_IFDIR 040000 S_IFCHR 020000 So 'm&S_IFCHR == S_IFCHR', for example, would succeed for both character and block devices. Go translates the system values to a platform-agnostic bitfield [4], so the previous approach works on Go. But it may be confusing for people used to the native non-bitfield mode, so this commit moves us to an approach that does not rely on Go's using a bitfield. I've also dropped the 07000 portion of the previous 07777 mask in favor of the cross-platform ModePerm mask. This avoids an internal magic number, and the sticky, suid, and sgid bits don't make sense for device nodes. And I'm using some contants from os instead of their syscall analogs. We can't drop the syscall dependency, because we're still using syscall to construct the Mknod arguments, but with this commit we're no longer using it to inspect the source file type. [1]: http://pubs.opengroup.org/onlinepubs/9699919799/basedefs/sys_stat.h.html [2]: opencontainers/runtime-tools#308 (comment) [3]: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/include/uapi/linux/stat.h?h=v4.16#n9 [4]: https://github.com/golang/go/blob/b0d437f866eb8987cde7e6550cacd77876f36d4b/src/os/types.go#L45
wking
added a commit
to wking/ocitools-v2
that referenced
this pull request
Apr 6, 2018
We've been using ModeSymlink as the mask since the check landed in da25004 (runtimetest: add linux default symbolic link validation, 2016-11-30, opencontainers#284). POSIX provides S_IS*(m) macros to portably interpret the mode type, but does not define values for each type [2]. Alban pointed out that st_mode is not a bitfield on Linux [1]. For example, Linux defines [3]: S_IFBLK 060000 S_IFDIR 040000 S_IFCHR 020000 So 'm&S_IFCHR == S_IFCHR', for example, would succeed for both character and block devices. Go translates the system values to a platform-agnostic bitfield [4], so the previous approach works on Go. But it may be confusing for people used to the native non-bitfield mode, so this commit moves us to an approach that does not rely on Go's using a bitfield. [1]: opencontainers#308 (comment) [2]: http://pubs.opengroup.org/onlinepubs/9699919799/basedefs/sys_stat.h.html [3]: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/include/uapi/linux/stat.h?h=v4.16#n9 [4]: https://github.com/golang/go/blob/b0d437f866eb8987cde7e6550cacd77876f36d4b/src/os/types.go#L45 Signed-off-by: W. Trevor King <[email protected]>
wking
added a commit
to wking/ocitools-v2
that referenced
this pull request
Apr 6, 2018
We've been using ModeSymlink as the mask since the check landed in da25004 (runtimetest: add linux default symbolic link validation, 2016-11-30, opencontainers#284). POSIX provides S_IS*(m) macros to portably interpret the mode type, but does not define values for each type [2]. Alban pointed out that st_mode is not a bitfield on Linux [1]. For example, Linux defines [3]: S_IFBLK 060000 S_IFDIR 040000 S_IFCHR 020000 So 'm&S_IFCHR == S_IFCHR', for example, would succeed for both character and block devices. Go translates the system values to a platform-agnostic bitfield [4], so the previous approach works on Go. But it may be confusing for people used to the native non-bitfield mode, so this commit moves us to an approach that does not rely on Go's using a bitfield. [1]: opencontainers#308 (comment) [2]: http://pubs.opengroup.org/onlinepubs/9699919799/basedefs/sys_stat.h.html [3]: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/include/uapi/linux/stat.h?h=v4.16#n9 [4]: https://github.com/golang/go/blob/b0d437f866eb8987cde7e6550cacd77876f36d4b/src/os/types.go#L45 Signed-off-by: W. Trevor King <[email protected]>
wking
added a commit
to wking/ocitools-v2
that referenced
this pull request
Apr 6, 2018
We've been using ModeSymlink as the mask since the check landed in da25004 (runtimetest: add linux default symbolic link validation, 2016-11-30, opencontainers#284). POSIX provides S_IS*(m) macros to portably interpret the mode type, but does not define values for each type [2]. Alban pointed out that st_mode is not a bitfield on Linux [1]. For example, Linux defines [3]: S_IFBLK 060000 S_IFDIR 040000 S_IFCHR 020000 So 'm&S_IFCHR == S_IFCHR', for example, would succeed for both character and block devices. Go translates the system values to a platform-agnostic bitfield [4], so the previous approach works on Go. But it may be confusing for people used to the native non-bitfield mode, so this commit moves us to an approach that does not rely on Go's using a bitfield. [1]: opencontainers#308 (comment) [2]: http://pubs.opengroup.org/onlinepubs/9699919799/basedefs/sys_stat.h.html [3]: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/include/uapi/linux/stat.h?h=v4.16#n9 [4]: https://github.com/golang/go/blob/b0d437f866eb8987cde7e6550cacd77876f36d4b/src/os/types.go#L45 Signed-off-by: W. Trevor King <[email protected]>
Granular TAP output makes it easier to see what's being tested (and what's going wrong). You'll want a TAP harness looking for test failures, because runtimetest now returns a lot of information, most of which is (hopefully ;) about passing tests. In order to accomplish this change, I've setup a complianceTester structure to make it easy to pass the TAP harness down into the helper functions. complianceTester also holds the target compliance level, so it can decide whether a spec violation is sufficiently serious to be a failed test violoation or is minor enough to be a warning. I've used skips for the warnings, since they seemed like the closest fit from the TAP spec. And I've made a number of other improvements while I was working through this, but haven't written them all up in this commit message. One change is the new support for testing file write access; previously we only tested directory write access. Its possible that some WriteFile errors should kill the test suite, but for now I'm counting all of them as successful "cannot write" determinations. Signed-off-by: W. Trevor King <[email protected]>
Previously we were only looking at directories. But the spec does not say these are directory-only options, so test files too. Put some content in the files, because runtimetest currently has no way to check the readability of empty files. Signed-off-by: W. Trevor King <[email protected]>
We've been using ModeSymlink as the mask since the check landed in da25004 (runtimetest: add linux default symbolic link validation, 2016-11-30, opencontainers#284). POSIX provides S_IS*(m) macros to portably interpret the mode type, but does not define values for each type [2]. Alban pointed out that st_mode is not a bitfield on Linux [1]. For example, Linux defines [3]: S_IFBLK 060000 S_IFDIR 040000 S_IFCHR 020000 So 'm&S_IFCHR == S_IFCHR', for example, would succeed for both character and block devices. Go translates the system values to a platform-agnostic bitfield [4], so the previous approach works on Go. But it may be confusing for people used to the native non-bitfield mode, so this commit moves us to an approach that does not rely on Go's using a bitfield. [1]: opencontainers#308 (comment) [2]: http://pubs.opengroup.org/onlinepubs/9699919799/basedefs/sys_stat.h.html [3]: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/include/uapi/linux/stat.h?h=v4.16#n9 [4]: https://github.com/golang/go/blob/b0d437f866eb8987cde7e6550cacd77876f36d4b/src/os/types.go#L45 Signed-off-by: W. Trevor King <[email protected]>
|
I've fixed a few typos that had snuck in with my recent rerolls, and Travis is happy again. Also, in case folks are interested, here's an example of how node-tap renders the $ tap ./runtimetest
./runtimetest ..................................... 199/275
not ok has expected working directory
actual: /…/src/github.com/opencontainers/runtime-tools
expected: /
not ok mounts[1] (/tmp) found
not ok mounts[2] (/dev) found in order
config:
destination: /dev
type: devtmpfs
source: devtmpfs
earlier:
config:
destination: /tmp
type: tmpfs
source: none
indexConfig: 0
indexSystem: 18
indexConfig: 2
indexSystem: 0
level: MUST
reference: 'https://github.com/opencontainers/runtime-spec/blob/v1.1.0/config.md#mounts'
not ok has expected user ID
actual: 1000
expected: 1
not ok has expected group ID
actual: 1000
expected: 1
not ok unexpected bounding capability CAP_CHOWN not set
not ok unexpected bounding capability CAP_DAC_OVERRIDE not set
not ok unexpected bounding capability CAP_DAC_READ_SEARCH not set
not ok unexpected bounding capability CAP_FOWNER not set
not ok unexpected bounding capability CAP_FSETID not set
not ok unexpected bounding capability CAP_KILL not set
not ok unexpected bounding capability CAP_SETGID not set
not ok unexpected bounding capability CAP_SETUID not set
not ok unexpected bounding capability CAP_SETPCAP not set
not ok unexpected bounding capability CAP_LINUX_IMMUTABLE not set
not ok unexpected bounding capability CAP_NET_BIND_SERVICE not set
not ok unexpected bounding capability CAP_NET_BROADCAST not set
not ok unexpected bounding capability CAP_NET_ADMIN not set
not ok unexpected bounding capability CAP_NET_RAW not set
not ok unexpected bounding capability CAP_IPC_LOCK not set
not ok unexpected bounding capability CAP_IPC_OWNER not set
not ok unexpected bounding capability CAP_SYS_MODULE not set
not ok unexpected bounding capability CAP_SYS_RAWIO not set
not ok unexpected bounding capability CAP_SYS_CHROOT not set
not ok unexpected bounding capability CAP_SYS_PTRACE not set
not ok unexpected bounding capability CAP_SYS_PACCT not set
not ok unexpected bounding capability CAP_SYS_ADMIN not set
not ok unexpected bounding capability CAP_SYS_BOOT not set
not ok unexpected bounding capability CAP_SYS_NICE not set
not ok unexpected bounding capability CAP_SYS_RESOURCE not set
not ok unexpected bounding capability CAP_SYS_TIME not set
not ok unexpected bounding capability CAP_SYS_TTY_CONFIG not set
not ok unexpected bounding capability CAP_MKNOD not set
not ok unexpected bounding capability CAP_LEASE not set
not ok unexpected bounding capability CAP_AUDIT_WRITE not set
not ok unexpected bounding capability CAP_AUDIT_CONTROL not set
not ok unexpected bounding capability CAP_SETFCAP not set
not ok unexpected bounding capability CAP_MAC_OVERRIDE not set
not ok unexpected bounding capability CAP_MAC_ADMIN not set
not ok unexpected bounding capability CAP_SYSLOG not set
not ok unexpected bounding capability CAP_WAKE_ALARM not set
not ok unexpected bounding capability CAP_BLOCK_SUSPEND not set
not ok unexpected bounding capability CAP_AUDIT_READ not set
not ok has expected process argument 0
actual: ./runtimetest
expected: sh
index: 0
Skipped: 32
root.readonly is false but the root filesystem is still not writable
hostname not set
/dev/null (default device) has unconfigured permissions
/dev/null (default device) has an unconfigured user ID
/dev/null (default device) has an unconfigured group ID
/dev/zero (default device) has unconfigured permissions
/dev/zero (default device) has an unconfigured user ID
/dev/zero (default device) has an unconfigured group ID
/dev/full (default device) has unconfigured permissions
/dev/full (default device) has an unconfigured user ID
/dev/full (default device) has an unconfigured group ID
/dev/random (default device) has unconfigured permissions
/dev/random (default device) has an unconfigured user ID
/dev/random (default device) has an unconfigured group ID
/dev/urandom (default device) has unconfigured permissions
/dev/urandom (default device) has an unconfigured user ID
/dev/urandom (default device) has an unconfigured group ID
/dev/tty (default device) has unconfigured permissions
/dev/tty (default device) has an unconfigured user ID
/dev/tty (default device) has an unconfigured group ID
/dev/ptmx (default device) has unconfigured permissions
/dev/ptmx (default device) has an unconfigured user ID
/dev/ptmx (default device) has an unconfigured group ID
linux.devices is not set
linux.maskedPaths not set
process.oomScoreAdj not set
linux.seccomp not set
linux.readonlyPaths not set
linux.rootfsPropagation not set
linux.sysctl not set
linux.uidMappings not set
linux.gidMappings not set
total ............................................. 199/275
199 passing (102.442ms)
32 pending
44 failing
|
I ran these examples again with: $ runc --version runc version 1.0.0-rc5+dev commit: cc4307ab6643668ce5abc6b524e1764a54c32550 spec: 1.0.0 $ uname -r 4.15.0 I don't know what that hugetlb permission error is about, but that's what I see ;). It could be an issue with my local system. Signed-off-by: W. Trevor King <[email protected]>
|
LGTM |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Granular TAP output makes it easier to see what's being tested (and what's going wrong). You could feed this into any TAP harness you like, but I've chosen prove (it seems popular). You need a TAP harness looking for test failures, because runtimetest now returns zero when it successfully runs the tests (regardless of whether the tests all pass). To avoid confusing the TAP consumer, all of the non-TAP logging goes to stderr (and not stdout).
With this change, the verbose output looks like: