Add infrastructure for CUDA PR build #4217

jwillenbring · 2019-01-18T16:49:37Z

@trilinos/framework

Description

This PR adds/modifies files in preparation of adding a dev->master CUDA build and then a PR CUDA build.

Motivation and Context

We need to protect against breaking the CUDA build at PR time. This is in response to #2464.

How Has This Been Tested?

@ZUUL42 set up Jenkins infrastructure for this build also, and we have a large subset of Trilinos tests passing. We have disabled temporarily what is failing so that we can get a PR build in place that will protect what is passing while we work on the remaining issues.

trilinos-autotester · 2019-01-18T17:16:15Z

Status Flag 'Pre-Test Inspection' - Auto Inspected - Inspection Is Not Necessary for this Pull Request.

trilinos-autotester · 2019-01-18T17:16:58Z

Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects:

Pull Request Auto Testing STARTING (click to expand)

Build Information

Test Name: Trilinos_pullrequest_gcc_4.8.4

Build Num: 2268
Status: STARTED

Jenkins Parameters

Parameter Name	Value
COMPILER_MODULE	sems-gcc/4.8.4
JENKINS_BUILD_TYPE	Release
JENKINS_COMM_TYPE	MPI
JENKINS_DO_COMPLEX	OFF
JENKINS_JOB_TYPE	Experimental
MPI_MODULE	sems-openmpi/1.8.7
PULLREQUESTNUM	4217
TEST_REPO_ALIAS	TRILINOS
TRILINOS_SOURCE_BRANCH	ride_build_pr
TRILINOS_SOURCE_REPO	https://github.com/jwillenbring/Trilinos
TRILINOS_SOURCE_SHA	`1fbde49`
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	`882d842`

Build Information

Test Name: Trilinos_pullrequest_intel_17.0.1

Build Num: 2069
Status: STARTED

Jenkins Parameters

Parameter Name	Value
PULLREQUESTNUM	4217
TEST_REPO_ALIAS	TRILINOS
TRILINOS_SOURCE_BRANCH	ride_build_pr
TRILINOS_SOURCE_REPO	https://github.com/jwillenbring/Trilinos
TRILINOS_SOURCE_SHA	`1fbde49`
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	`882d842`

Build Information

Test Name: Trilinos_pullrequest_gcc_4.9.3_SERIAL

Build Num: 556
Status: STARTED

Jenkins Parameters

Parameter Name	Value
PULLREQUESTNUM	4217
TEST_REPO_ALIAS	TRILINOS
TRILINOS_SOURCE_BRANCH	ride_build_pr
TRILINOS_SOURCE_REPO	https://github.com/jwillenbring/Trilinos
TRILINOS_SOURCE_SHA	`1fbde49`
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	`882d842`

Build Information

Test Name: Trilinos_pullrequest_gcc_7.2.0

Build Num: 170
Status: STARTED

Jenkins Parameters

Parameter Name	Value
PULLREQUESTNUM	4217
TEST_REPO_ALIAS	TRILINOS
TRILINOS_SOURCE_BRANCH	ride_build_pr
TRILINOS_SOURCE_REPO	https://github.com/jwillenbring/Trilinos
TRILINOS_SOURCE_SHA	`1fbde49`
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	`882d842`

Using Repos:

Repo: TRILINOS (jwillenbring/Trilinos)

Pull Request Author: jwillenbring

ZUUL42

Looks good and the last few builds have been successful.

trilinos-autotester · 2019-01-18T21:32:06Z

Status Flag 'Pull Request AutoTester' - Jenkins Testing: all Jobs PASSED

Pull Request Auto Testing has PASSED (click to expand)

Build Information

Test Name: Trilinos_pullrequest_gcc_4.8.4

Build Num: 2268
Status: PASSED

Jenkins Parameters

Parameter Name	Value
COMPILER_MODULE	sems-gcc/4.8.4
JENKINS_BUILD_TYPE	Release
JENKINS_COMM_TYPE	MPI
JENKINS_DO_COMPLEX	OFF
JENKINS_JOB_TYPE	Experimental
MPI_MODULE	sems-openmpi/1.8.7
PULLREQUESTNUM	4217
TEST_REPO_ALIAS	TRILINOS
TRILINOS_SOURCE_BRANCH	ride_build_pr
TRILINOS_SOURCE_REPO	https://github.com/jwillenbring/Trilinos
TRILINOS_SOURCE_SHA	`1fbde49`
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	`882d842`

Build Information

Test Name: Trilinos_pullrequest_intel_17.0.1

Build Num: 2069
Status: PASSED

Jenkins Parameters

Parameter Name	Value
PULLREQUESTNUM	4217
TEST_REPO_ALIAS	TRILINOS
TRILINOS_SOURCE_BRANCH	ride_build_pr
TRILINOS_SOURCE_REPO	https://github.com/jwillenbring/Trilinos
TRILINOS_SOURCE_SHA	`1fbde49`
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	`882d842`

Build Information

Test Name: Trilinos_pullrequest_gcc_4.9.3_SERIAL

Build Num: 556
Status: PASSED

Jenkins Parameters

Parameter Name	Value
PULLREQUESTNUM	4217
TEST_REPO_ALIAS	TRILINOS
TRILINOS_SOURCE_BRANCH	ride_build_pr
TRILINOS_SOURCE_REPO	https://github.com/jwillenbring/Trilinos
TRILINOS_SOURCE_SHA	`1fbde49`
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	`882d842`

Build Information

Test Name: Trilinos_pullrequest_gcc_7.2.0

Build Num: 170
Status: PASSED

Jenkins Parameters

Parameter Name	Value
PULLREQUESTNUM	4217
TEST_REPO_ALIAS	TRILINOS
TRILINOS_SOURCE_BRANCH	ride_build_pr
TRILINOS_SOURCE_REPO	https://github.com/jwillenbring/Trilinos
TRILINOS_SOURCE_SHA	`1fbde49`
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	`882d842`

CDash Test Results for PR# 4217.

trilinos-autotester · 2019-01-18T21:32:23Z

Status Flag 'Pre-Merge Inspection' - SUCCESS: The last commit to this Pull Request has been INSPECTED AND APPROVED by [ ZUUL42 ]!

trilinos-autotester · 2019-01-18T21:32:30Z

Status Flag 'Pull Request AutoTester' - AutoMerge IS ENABLED, but the Label AT: AUTOMERGE is not set. Either set Label AT: AUTOMERGE or manually merge the PR...

trilinos-autotester · 2019-01-19T21:46:11Z

Status Flag 'Pull Request AutoTester' - AutoMerge IS ENABLED, but the Label AT: AUTOMERGE is not set. Either set Label AT: AUTOMERGE or manually merge the PR...

trilinos-autotester · 2019-01-20T21:46:15Z

Status Flag 'Pull Request AutoTester' - AutoMerge IS ENABLED, but the Label AT: AUTOMERGE is not set. Either set Label AT: AUTOMERGE or manually merge the PR...

william76 · 2019-01-21T18:27:29Z

cmake/std/PullRequestLinuxCudaDriver.sh

@@ -0,0 +1,23 @@
+#!/bin/bash -l
+
+if [ "${BSUB_CTEST_TIME_LIMIT}" == "" ] ; then


Is BSUB_CTEST_TIME_LIMIT a system defined environment variable or is this a new Jenkins parameter that's being introduced?
If this doesn't exist, can it break things?

@bartlettroscoe @ZUUL42 @william76

Is BSUB_CTEST_TIME_LIMIT a system defined environment variable or is this a new Jenkins parameter that's being introduced?
If this doesn't exist, can it break things?

I am not certain, but I think it is just an environment variable that can be defined by a user.
@bartlettroscoe may be able to confirm or correct this.
If it is not defined, it is set to a default value, so it shouldn't break things. The variable is specific to this build as far as current PR tests go (because ride uses bsub).

This was just an env var that allowed specialization in the individual driver scripts. This came from the file:

Trilinos/cmake/ctest/drivers/atdm/ride/local-driver.sh

which has:

#!/bin/bash -l if [ "${BSUB_CTEST_TIME_LIMIT}" == "" ] ; then export BSUB_CTEST_TIME_LIMIT=12:00 fi ... source $WORKSPACE/Trilinos/cmake/std/atdm/load-env.sh $JOB_NAME set -x bsub -x -Is -q $ATDM_CONFIG_QUEUE -n 16 -J $JOB_NAME -W $BSUB_CTEST_TIME_LIMIT \ $WORKSPACE/Trilinos/cmake/ctest/drivers/atdm/ctest-s-driver.sh

Any questions about this?

william76 · 2019-01-21T18:37:11Z

cmake/std/PullRequestLinuxDriver-Test.sh

-
-
+regex=".*(_cuda_).*"
+if [[ ! ${JOB_BASE_NAME:?} =~ ${regex} ]]; then


I checked this regex test in a script and it works... but I'm not sure we should make an assumption that if the job has _cuda_ in its name that it's on Ride. This might lead to bugs down the line if we create a CUDA job later on for a machine that isn't Ride... I'd rather see something like:

if job_name doesn't have CUDA: load non-cuda environment else if machine is Ride load git module for Ride else throw a meaningful error and notify DevOPs fi

@prwolfe @jwillenbring your thougs?

Ya, that would be better than just a warning note about only putting CUDA builds on Ride.
I can add the extra check for only running CUDA builds on Ride.

Agreed - Thanks @ZUUL42

Committed the additional check.

jwillenbring · 2019-01-21T19:54:26Z

As discussed at the @trilinos/framework meeting this morning, I fired off another PR build (not the CUDA build) to make sure we didn't break anything for other cases. The build came back clean, but we will wait for @ZUUL42 to make the other change and test again before merging.

trilinos-autotester · 2019-01-22T01:31:27Z

Status Flag 'Pre-Test Inspection' - Auto Inspected - Inspection Is Not Necessary for this Pull Request.

trilinos-autotester · 2019-01-22T01:32:10Z

Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects:

Pull Request Auto Testing STARTING (click to expand)

Build Information

Test Name: Trilinos_pullrequest_gcc_4.8.4

Build Num: 2283
Status: STARTED

Jenkins Parameters

Parameter Name	Value
COMPILER_MODULE	sems-gcc/4.8.4
JENKINS_BUILD_TYPE	Release
JENKINS_COMM_TYPE	MPI
JENKINS_DO_COMPLEX	OFF
JENKINS_JOB_TYPE	Experimental
MPI_MODULE	sems-openmpi/1.8.7
PULLREQUESTNUM	4217
TEST_REPO_ALIAS	TRILINOS
TRILINOS_SOURCE_BRANCH	ride_build_pr
TRILINOS_SOURCE_REPO	https://github.com/jwillenbring/Trilinos
TRILINOS_SOURCE_SHA	`d8cad58`
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	`882d842`

Build Information

Test Name: Trilinos_pullrequest_intel_17.0.1

Build Num: 2084
Status: STARTED

Jenkins Parameters

Parameter Name	Value
PULLREQUESTNUM	4217
TEST_REPO_ALIAS	TRILINOS
TRILINOS_SOURCE_BRANCH	ride_build_pr
TRILINOS_SOURCE_REPO	https://github.com/jwillenbring/Trilinos
TRILINOS_SOURCE_SHA	`d8cad58`
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	`882d842`

Build Information

Test Name: Trilinos_pullrequest_gcc_4.9.3_SERIAL

Build Num: 571
Status: STARTED

Jenkins Parameters

Parameter Name	Value
PULLREQUESTNUM	4217
TEST_REPO_ALIAS	TRILINOS
TRILINOS_SOURCE_BRANCH	ride_build_pr
TRILINOS_SOURCE_REPO	https://github.com/jwillenbring/Trilinos
TRILINOS_SOURCE_SHA	`d8cad58`
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	`882d842`

Build Information

Test Name: Trilinos_pullrequest_gcc_7.2.0

Build Num: 189
Status: STARTED

Jenkins Parameters

Parameter Name	Value
PULLREQUESTNUM	4217
TEST_REPO_ALIAS	TRILINOS
TRILINOS_SOURCE_BRANCH	ride_build_pr
TRILINOS_SOURCE_REPO	https://github.com/jwillenbring/Trilinos
TRILINOS_SOURCE_SHA	`d8cad58`
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	`882d842`

Using Repos:

Repo: TRILINOS (jwillenbring/Trilinos)

Pull Request Author: jwillenbring

trilinos-autotester · 2019-01-22T06:43:55Z

Status Flag 'Pull Request AutoTester' - Jenkins Testing: all Jobs PASSED

Pull Request Auto Testing has PASSED (click to expand)

Build Information

Test Name: Trilinos_pullrequest_gcc_4.8.4

Build Num: 2283
Status: PASSED

Jenkins Parameters

Parameter Name	Value
COMPILER_MODULE	sems-gcc/4.8.4
JENKINS_BUILD_TYPE	Release
JENKINS_COMM_TYPE	MPI
JENKINS_DO_COMPLEX	OFF
JENKINS_JOB_TYPE	Experimental
MPI_MODULE	sems-openmpi/1.8.7
PULLREQUESTNUM	4217
TEST_REPO_ALIAS	TRILINOS
TRILINOS_SOURCE_BRANCH	ride_build_pr
TRILINOS_SOURCE_REPO	https://github.com/jwillenbring/Trilinos
TRILINOS_SOURCE_SHA	`d8cad58`
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	`882d842`

Build Information

Test Name: Trilinos_pullrequest_intel_17.0.1

Build Num: 2084
Status: PASSED

Jenkins Parameters

Parameter Name	Value
PULLREQUESTNUM	4217
TEST_REPO_ALIAS	TRILINOS
TRILINOS_SOURCE_BRANCH	ride_build_pr
TRILINOS_SOURCE_REPO	https://github.com/jwillenbring/Trilinos
TRILINOS_SOURCE_SHA	`d8cad58`
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	`882d842`

Build Information

Test Name: Trilinos_pullrequest_gcc_4.9.3_SERIAL

Build Num: 571
Status: PASSED

Jenkins Parameters

Parameter Name	Value
PULLREQUESTNUM	4217
TEST_REPO_ALIAS	TRILINOS
TRILINOS_SOURCE_BRANCH	ride_build_pr
TRILINOS_SOURCE_REPO	https://github.com/jwillenbring/Trilinos
TRILINOS_SOURCE_SHA	`d8cad58`
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	`882d842`

Build Information

Test Name: Trilinos_pullrequest_gcc_7.2.0

Build Num: 189
Status: PASSED

Jenkins Parameters

Parameter Name	Value
PULLREQUESTNUM	4217
TEST_REPO_ALIAS	TRILINOS
TRILINOS_SOURCE_BRANCH	ride_build_pr
TRILINOS_SOURCE_REPO	https://github.com/jwillenbring/Trilinos
TRILINOS_SOURCE_SHA	`d8cad58`
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	`882d842`

CDash Test Results for PR# 4217.

trilinos-autotester · 2019-01-22T06:44:13Z

Status Flag 'Pre-Merge Inspection' - - This Pull Request Requires Inspection... The code must be inspected by a member of the Team before Testing/Merging
THE LAST COMMIT TO THIS PULL REQUEST HAS BEEN REVIEWED, BUT NOT ACCEPTED OR REQUIRES CHANGES

trilinos-autotester · 2019-01-22T06:44:19Z

All Jobs Finished; status = PASSED, However Inspection must be performed before merge can occur...

trilinos-autotester · 2019-01-23T06:46:12Z

All Jobs Finished; status = PASSED, However Inspection must be performed before merge can occur...

william76 · 2019-01-23T16:09:35Z

@ZUUL42 looks good to me.

trilinos-autotester · 2019-01-23T16:46:28Z

Status Flag 'Pre-Merge Inspection' - SUCCESS: The last commit to this Pull Request has been INSPECTED AND APPROVED by [ william76 ]!

trilinos-autotester · 2019-01-23T16:46:36Z

Status Flag 'Pull Request AutoTester' - AutoMerge IS ENABLED, but the Label AT: AUTOMERGE is not set. Either set Label AT: AUTOMERGE or manually merge the PR...

ZUUL42 and others added 19 commits January 18, 2019 09:21

Add infrastructure for testing cuda build

3a32119

Removing sems module init for cuda & fix workspace ref

7993516

Fix MPI and sems init calls for cuda

c476288

MPI_NAME -> MPI_VENDOR

04bfef8

Setting KOKKOS_ARCH

6daed62

Fixing CUDA configuration

aa161c9

Added BLAS and LAPACK options.

a5b2204

Explicitly disabling Scotch and ParMetis.

cb2edca

Adding Netcdf library configuration.

df416a1

Disabled SuperLU and boostlib

60b4424

Disabled SEACAS for now.

969e5a5

Disabled Moertel.

915ed4a

Disabling shared libs and Komplex

14c493a

Disabled debug to match ATDM build

b5aebc2

Set Tpetra_INST_SERIAL=ON to match atdm build.

1fc7808

Turning off Debug symbols.

c504c36

Turning off secondary tested code.

559a7e4

Disable a failing test.

5fefffa

Disabled remaining failed tests.

1fbde49

jwillenbring added Framework tasks Framework tasks (used internally by Framework team) PA: Framework Issues that fall under the Trilinos Framework Product Area labels Jan 18, 2019

jwillenbring requested review from william76 and ZUUL42 January 18, 2019 16:49

bartlettroscoe added the stage: in progress Work on the issue has started label Jan 18, 2019

ZUUL42 previously approved these changes Jan 18, 2019

View reviewed changes

william76 reviewed Jan 21, 2019

View reviewed changes

Added check for cuda build for if node is ride

26ce0f0

ZUUL42 dismissed their stale review via 26ce0f0 January 21, 2019 21:43

Added check for cuda build for if node is ride

d8cad58

william76 approved these changes Jan 23, 2019

View reviewed changes

jwillenbring merged commit 09facb2 into trilinos:develop Jan 23, 2019

bartlettroscoe removed the stage: in progress Work on the issue has started label Jan 23, 2019

kddevin mentioned this pull request Jan 29, 2019

Tpetra: unreliable test condition in ScopeGuard tests #3453

Closed

trilinos-autotester mentioned this pull request Dec 20, 2021

Testing on Geminga: Use new SEMS modules #10045

Merged

		@@ -0,0 +1,23 @@
		#!/bin/bash -l

		if [ "${BSUB_CTEST_TIME_LIMIT}" == "" ] ; then



		regex=".(_cuda_)."
		if [[ ! ${JOB_BASE_NAME:?} =~ ${regex} ]]; then

Add infrastructure for CUDA PR build #4217

Add infrastructure for CUDA PR build #4217

Conversation

jwillenbring commented Jan 18, 2019

Description

Motivation and Context

How Has This Been Tested?

trilinos-autotester commented Jan 18, 2019

trilinos-autotester commented Jan 18, 2019

Build Information

Test Name: Trilinos_pullrequest_gcc_4.8.4

Jenkins Parameters

Build Information

Test Name: Trilinos_pullrequest_intel_17.0.1

Jenkins Parameters

Build Information

Test Name: Trilinos_pullrequest_gcc_4.9.3_SERIAL

Jenkins Parameters

Build Information

Test Name: Trilinos_pullrequest_gcc_7.2.0

Jenkins Parameters

Using Repos:

ZUUL42 left a comment

Choose a reason for hiding this comment

trilinos-autotester commented Jan 18, 2019

Build Information

Test Name: Trilinos_pullrequest_gcc_4.8.4

Jenkins Parameters

Build Information

Test Name: Trilinos_pullrequest_intel_17.0.1

Jenkins Parameters

Build Information

Test Name: Trilinos_pullrequest_gcc_4.9.3_SERIAL

Jenkins Parameters

Build Information

Test Name: Trilinos_pullrequest_gcc_7.2.0

Jenkins Parameters

trilinos-autotester commented Jan 18, 2019

trilinos-autotester commented Jan 18, 2019

trilinos-autotester commented Jan 19, 2019

trilinos-autotester commented Jan 20, 2019

william76 Jan 21, 2019 • edited Loading

Choose a reason for hiding this comment

jwillenbring Jan 22, 2019

Choose a reason for hiding this comment

bartlettroscoe Jan 22, 2019

Choose a reason for hiding this comment

william76 Jan 21, 2019

Choose a reason for hiding this comment

ZUUL42 Jan 21, 2019 • edited Loading

Choose a reason for hiding this comment

prwolfe Jan 21, 2019

Choose a reason for hiding this comment

ZUUL42 Jan 21, 2019

Choose a reason for hiding this comment

jwillenbring commented Jan 21, 2019

trilinos-autotester commented Jan 22, 2019

trilinos-autotester commented Jan 22, 2019

Build Information

Test Name: Trilinos_pullrequest_gcc_4.8.4

Jenkins Parameters

Build Information

Test Name: Trilinos_pullrequest_intel_17.0.1

Jenkins Parameters

Build Information

Test Name: Trilinos_pullrequest_gcc_4.9.3_SERIAL

Jenkins Parameters

Build Information

Test Name: Trilinos_pullrequest_gcc_7.2.0

Jenkins Parameters

Using Repos:

trilinos-autotester commented Jan 22, 2019

Build Information

Test Name: Trilinos_pullrequest_gcc_4.8.4

Jenkins Parameters

Build Information

Test Name: Trilinos_pullrequest_intel_17.0.1

Jenkins Parameters

Build Information

Test Name: Trilinos_pullrequest_gcc_4.9.3_SERIAL

william76 Jan 21, 2019 •

edited

Loading

ZUUL42 Jan 21, 2019 •

edited

Loading