test(shell): ensure that fields that aren't checked are empty #4900
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Important: The tests are supposed to fail.
They are supposed to fail on master too, and this PR only surfaces the issue. I'll probably open a PR in the future fixing the issue too, but first, this one should be merged.
What's the problem this PR addresses?
Most shell tests only check some of the result fields (
exitCode
,stdout
, andstderr
).Because of this, some bugs can slip through, such as #4892 (where the test passes because the error is a recoverable shell error, so the shell writes it to
stderr
and exits with exit code1
, but we don't check for the exit code).How did you fix it?
I made all tests use
expectResult
, which ensures that fields that aren't specified are empty (0
or empty string).Checklist