Framework: AT: No PRs merging due to potential communication issues and failing framework tests

[GrahamBenHarper]

Bug Report

@trilinos/framework

As reported in TRILINOSHD-201, several pull requests are experiencing framework issues. Specifically, there's the following communication issue in #10983:

NOTICE: The AutoTester has encountered an internal error (usually a Communications Timeout), testing will be restarted, previous tests may still be running but will be ignored by the AutoTester...

After digging deeper, it's also clear that there are framework tests failing related to #10972, #10983, and more. Some output from the failing test is included below:

XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

Advanced Test: TrilinosFrameworkTests_PullRequestLinuxDriver_UnitTests

Selected Test/CTest Properties:
  CATEGORIES = BASIC
  PROCESSORS = 1
  TIMEOUT    = DEFAULT

Creating new working directory '/scratch/trilinos/jenkins/ascic158/workspace/Trilinos_PR_python3/pull_request_test/commonTools/framework/TrilinosFrameworkTests_PullRequestLinuxDriver_UnitTests'

Running test commands: TEST_0

================================================================================

TEST_0

Running: "bash" "pr_python_test_driver.sh" "/projects/sems/install/rhel7-x86_64/sems/v2/utility/anaconda3/2020.07/gcc/7.3.0/7t6phgv/bin/python3" "pip3"

  Running in working directory "/scratch/trilinos/jenkins/ascic158/workspace/Trilinos_PR_python3/Trilinos/commonTools/framework/../../packages/framework/pr_tools"

--------------------------------------------------------------------------------


+----------------------------------------------------------+
|           P Y T H O N   T E S T   D R I V E R            |
|                                                          |
|                   2022-09-02 16:25:28                    |
+----------------------------------------------------------+


+----------------------------------------------------------+
|                P A C K A G E   S E T U P                 |
|                                                          |
|                   2022-09-02 16:25:28                    |
+----------------------------------------------------------+
--- python: /projects/sems/install/rhel7-x86_64/sems/v2/utility/anaconda3/2020.07/gcc/7.3.0/7t6phgv/bin/python3
--- pip   : pip3
PRDriver> pip executable: /projects/sems/install/rhel7-x86_64/sems/v2/utility/anaconda3/2020.07/gcc/7.3.0/7t6phgv/bin/python3 -m pip
PRDriver> 
PRDriver> /projects/sems/install/rhel7-x86_64/sems/v2/utility/anaconda3/2020.07/gcc/7.3.0/7t6phgv/bin/python3 -m pip install --user configparser
Requirement already satisfied: configparser in /home/trilinos/.local/lib/python3.8/site-packages (5.3.0)
/scratch/trilinos/jenkins/ascic158/workspace/Trilinos_PR_python3/Trilinos/packages/framework/pr_tools/common.bash: line 78: color: command not found
PRDriver> OK (3 s)


+----------------------------------------------------------+
|                E X E C U T E   T E S T S                 |
|                                                          |
|                   2022-09-02 16:25:31                    |
+----------------------------------------------------------+
--- /projects/sems/install/rhel7-x86_64/sems/v2/utility/anaconda3/2020.07/gcc/7.3.0/7t6phgv/bin/python3 -m pytest --cov=. --cov-report term-missing
ERROR: usage: __main__.py [options] [file_or_dir] [file_or_dir] [...]
__main__.py: error: unrecognized arguments: --cov=. --cov-report
  inifile: None
  rootdir: /scratch/trilinos/jenkins/ascic158/workspace/Trilinos_PR_python3/Trilinos/packages/framework/pr_tools


--- STATUS: 4

--------------------------------------------------------------------------------

TEST_0: Return code = 4
TEST_0: Pass criteria = Match any REGEX {=+ [0-9]+ passed in} [not matched]
TEST_0: Result = FAILED

================================================================================

OVERALL FINAL RESULT: TEST FAILED (TrilinosFrameworkTests_PullRequestLinuxDriver_UnitTests)

XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

[bartlettroscoe]

Take a look at this query and this query. These failures have nothing to do the particular PR being tested and every PR test iteration will fail until someone does something to fix this failing test TrilinosFrameworkTests_PullRequestLinuxDriver_UnitTests.

[jhux2]

Take a look at this query and this query. These failures have nothing to do the particular PR being tested and every PR test iteration will fail until someone does something to fix this failing test TrilinosFrameworkTests_PullRequestLinuxDriver_UnitTests.

@bartlettroscoe Clicking through these two links shows exactly the error message in the description.

[jmlapre]

They are failing because the color command can't be found. If you notice in the script (common.bash) it fails due to color not being found after it installs configparser. Then it stops loading the rest of the packages and that's why the coverage is failing (it hasn't installed pytest-cov due to the earlier failure).

[jhux2]

Might be fixed by #10994.

[jhux2]

This is fixed. Dashboard results before and after merge of #10994.