First time a test is skipped causes rest of tests in shell script to be skipped as well.

[edwardhartnett]

Describe the bug
We are using the keyword "skip" in our shell scripts to skip tests. This results in execution like this:

bash -x ./test_mpp_domains2.sh
+ . ../test_common.sh
++ set -e
++ TEST_PASS=0
++ TEST_FAIL=1
++ TEST_SKIP=77
++ TEST_ERROR=99
++ top_srcdir=/home/ed/f1
++ top_buildir=/home/ed/f1
++ mpi_launcher=mpirun
++ oversubscribe=
++ uname -s
+ '[' xLinux = xDarwin ']'
+ '[' x = xtrue ']'
+ echo '1: Test simple functionality'
1: Test simple functionality
+ run_test test_domains_simple 4
+ test x = xskip
+ test xmpirun '!=' x
+ npes='-n 4'
+ mpirun -n 4 ./test_domains_simple
NOTE from PE     0: MPP_DOMAINS_SET_STACK_SIZE: stack size set to    32768.
NOTE from PE     0: MPP_DOMAINS_SET_STACK_SIZE: stack size set to  4000000.
 pe, tile, ipos, jpos=           0           1           0           0  pearray(:,jpos)=           0           1  pearray(ipos,:)=           0           2
 pe, tile, ipos, jpos=           1           1           1           0  pearray(:,jpos)=           0           1  pearray(ipos,:)=           1           3
 pe, tile, ipos, jpos=           2           1           0           1  pearray(:,jpos)=           2           3  pearray(ipos,:)=           0           2
 pe, tile, ipos, jpos=           3           1           1           1  pearray(:,jpos)=           2           3  pearray(ipos,:)=           1           3
 MPP_DOMAINS_STACK high water mark=           0

Tabulating mpp_clock statistics across      4 PEs...

                                          tmin          tmax          tavg          tstd  tfrac grain pemin pemax
Total runtime                         0.000547      0.000548      0.000547      0.000000  0.999     0     0     3
 MPP_STACK high water mark=           0
+ run_test test_mpp_domains 2 skip
+ test xskip = xskip
+ return 77

Note that the first skipped test causes the shell script to return. That's because of this code in test_fms/test_common.sh:

# Set exit codes to control the failure modes.  These exit codes are found in
# The GNU Automake manual: https://www.gnu.org/software/automake/manual/html_node/Scripts_002dbased-Testsuites.html
# Note, any non-zero status, other than SKIP and ERROR is a FAIL
TEST_PASS=0
TEST_FAIL=1
TEST_SKIP=77
TEST_ERROR=99

run_test()
{
    # If the tests is known to fail exit
    if test "x${3}" = "xskip" ; then
        return $TEST_SKIP
    else
        # If there is no mpi launcher just ./{job_script}
        if test "x$mpi_launcher" != "x" ; then
            npes="-n ${2}"
        fi
    fi

    # Run test
    $mpi_launcher $oversubscribe $npes ./${1}
}

So you think you are skipping just one test, but you are skipping all the remaining tests in the shell script.

To Reproduce
Run any of the test scripts which skip tests, using bash -x.

Expected behavior
I expected the test to be skipped, but subsequent tests to run.

Given that the first skip causes all remaining tests in the shell script to be skipped as well, perhaps we want to comment out tests, instead of skipping them.

System Environment
I'm on a GNU/Linux system.

Additional context
Add any other context about the problem. If applicable, include where any files
that help describe, or reproduce the problem exist.