Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Tpetra: fix race condition in transfers with AbsMax for CUDA_AWARE_MPI #3999

Merged

Conversation

rppawlo
Copy link
Contributor

@rppawlo rppawlo commented Dec 5, 2018

#3968

Changes to each package are in separate commits.

Thanks to Mark Hoemmen for the tpetra fix!

While these are not necessary to fix trilinos#3968, this cleans up some old
and probably not so safe code. Mainly switching to using dual view
semantics instead of the get 1d and 2d arrays and adding fences.
@rppawlo rppawlo added pkg: Tpetra pkg: Panzer AT: AUTOMERGE Causes the PR autotester to automatically merge the PR branch once approvals are completed labels Dec 5, 2018
@rppawlo rppawlo self-assigned this Dec 5, 2018
@rppawlo rppawlo requested review from mhoemmen and etphipp December 5, 2018 15:23
@rppawlo rppawlo requested a review from a team as a code owner December 5, 2018 15:23
@bartlettroscoe bartlettroscoe added the stage: in progress Work on the issue has started label Dec 5, 2018
etphipp
etphipp previously approved these changes Dec 5, 2018
@rppawlo rppawlo removed the AT: AUTOMERGE Causes the PR autotester to automatically merge the PR branch once approvals are completed label Dec 5, 2018
@trilinos-autotester
Copy link
Contributor

Status Flag 'Pre-Test Inspection' - Auto Inspected - Inspection Is Not Necessary for this Pull Request.

mhoemmen
mhoemmen previously approved these changes Dec 5, 2018
Copy link
Contributor

@mhoemmen mhoemmen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @rppawlo ! I'll close #3996 and bring in its unit test later. See comments (things I'd like changed but not now).

@@ -805,7 +810,7 @@ DOFManager<LO,GO>::buildGlobalUnknowns_GUN(const Tpetra::MultiVector<GO,LO,GO,pa

// do a prefix sum
GO scanResult = 0;
Teuchos::scan<int, GO> (*getComm(), Teuchos::REDUCE_SUM, static_cast<size_t> (localsum), Teuchos::outArg (scanResult));
Teuchos::scan<int, GO> (*getComm(), Teuchos::REDUCE_SUM, static_cast<int> (localsum), Teuchos::outArg (scanResult));
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is only helpful for removing warnings if GO is signed. You really want to cast localsum to GO since that's the type of the scan. It will get cast from int to GO here anyway.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks! will change

@@ -930,7 +938,10 @@ DOFManager<LO,GO>::buildTaggedMultiVector(const ElementBlockAccess & ownedAccess

// temporary working vector to fill each row in tagged array
std::vector<int> working(overlap_mv->getNumVectors());
ArrayRCP<ArrayRCP<GO> > edittwoview = overlap_mv->get2dViewNonConst();
auto dual_view = overlap_mv->getDualView();
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would prefer that you not call getDualView. Please call the functions that get the Kokkos::View of the appropriate memory space directly. This is fine for now, but I do plan on getting rid of getDualView at some point, since it exposes an implementation detail that constrains Tpetra to have allocations on both sides at all times (it can't do lazy allocation).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok. Will fix that as well.

auto dual_view = overlap_mv->getDualView();
dual_view.sync_host();
PHX::Device::fence();
auto edittwoview_host = dual_view.view_host();
for (size_t b = 0; b < blockOrder_.size(); ++b) {
// there has to be a field pattern assocaited with the block
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"associated"

@@ -1304,7 +1322,10 @@ fillGIDsFromOverlappedMV(const ElementBlockAccess & access,
using Teuchos::ArrayRCP;

//To generate elementGIDs we need to go through all of the local elements.
ArrayRCP<ArrayRCP<const GO> > twoview = overlap_mv.get2dView();
auto dual_view = overlap_mv.getDualView();
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See above. Get Kokkos::View objects from the Tpetra::MultiVector directly; don't access the DualView.

@mhoemmen mhoemmen added the AT: AUTOMERGE Causes the PR autotester to automatically merge the PR branch once approvals are completed label Dec 5, 2018
@trilinos-autotester
Copy link
Contributor

Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects:

Pull Request Auto Testing STARTING (click to expand)

Build Information

Test Name: Trilinos_pullrequest_intel_17.0.1

  • Build Num: 1795
  • Status: STARTED

Jenkins Parameters

Parameter Name Value
PULLREQUESTNUM 3999
TEST_REPO_ALIAS TRILINOS
TRILINOS_SOURCE_BRANCH 3968-tpetra-cuda-aware-absmax
TRILINOS_SOURCE_REPO https://github.com/rppawlo/Trilinos
TRILINOS_SOURCE_SHA 3ff9338
TRILINOS_TARGET_BRANCH develop
TRILINOS_TARGET_REPO https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA 436a132

Build Information

Test Name: Trilinos_pullrequest_gcc_4.9.3

  • Build Num: 2358
  • Status: STARTED

Jenkins Parameters

Parameter Name Value
COMPILER_MODULE sems-gcc/4.9.3
JENKINS_BUILD_TYPE Release
JENKINS_COMM_TYPE MPI
JENKINS_DO_COMPLEX OFF
JENKINS_JOB_TYPE Experimental
MPI_MODULE sems-openmpi/1.8.7
PULLREQUESTNUM 3999
TEST_REPO_ALIAS TRILINOS
TRILINOS_SOURCE_BRANCH 3968-tpetra-cuda-aware-absmax
TRILINOS_SOURCE_REPO https://github.com/rppawlo/Trilinos
TRILINOS_SOURCE_SHA 3ff9338
TRILINOS_TARGET_BRANCH develop
TRILINOS_TARGET_REPO https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA 436a132

Build Information

Test Name: Trilinos_pullrequest_gcc_4.8.4

  • Build Num: 2018
  • Status: STARTED

Jenkins Parameters

Parameter Name Value
COMPILER_MODULE sems-gcc/4.8.4
JENKINS_BUILD_TYPE Release
JENKINS_COMM_TYPE MPI
JENKINS_DO_COMPLEX OFF
JENKINS_JOB_TYPE Experimental
MPI_MODULE sems-openmpi/1.8.7
PULLREQUESTNUM 3999
TEST_REPO_ALIAS TRILINOS
TRILINOS_SOURCE_BRANCH 3968-tpetra-cuda-aware-absmax
TRILINOS_SOURCE_REPO https://github.com/rppawlo/Trilinos
TRILINOS_SOURCE_SHA 3ff9338
TRILINOS_TARGET_BRANCH develop
TRILINOS_TARGET_REPO https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA 436a132

Build Information

Test Name: Trilinos_pullrequest_gcc_4.9.3_SERIAL

  • Build Num: 327
  • Status: STARTED

Jenkins Parameters

Parameter Name Value
PULLREQUESTNUM 3999
TEST_REPO_ALIAS TRILINOS
TRILINOS_SOURCE_BRANCH 3968-tpetra-cuda-aware-absmax
TRILINOS_SOURCE_REPO https://github.com/rppawlo/Trilinos
TRILINOS_SOURCE_SHA 3ff9338
TRILINOS_TARGET_BRANCH develop
TRILINOS_TARGET_REPO https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA 436a132

Using Repos:

Repo: TRILINOS (rppawlo/Trilinos)
  • Branch: 3968-tpetra-cuda-aware-absmax
  • SHA: 3ff9338
  • Mode: TEST_REPO

Pull Request Author: rppawlo

@rppawlo
Copy link
Contributor Author

rppawlo commented Dec 5, 2018

@mhoemmen - I'm going to remove the automerge for now since I need to push the changes you requested.

@rppawlo rppawlo added AT: WIP Causes the PR autotester to not test the PR. (Remove to allow testing to occur.) and removed AT: AUTOMERGE Causes the PR autotester to automatically merge the PR branch once approvals are completed labels Dec 5, 2018
@rppawlo rppawlo dismissed stale reviews from mhoemmen and etphipp via 5f91b56 December 5, 2018 19:14
@trilinos-autotester
Copy link
Contributor

Status Flag 'Pull Request AutoTester' - Error: Jenkins Jobs - A user has commited a change to the PR before testing completed. The original testing SHA = 3ff9338 Does not match the current commit SHA = 5f91b56. The Jenkins Jobs will be shutdown; Testing of this PR must occur again.

@rppawlo rppawlo added AT: AUTOMERGE Causes the PR autotester to automatically merge the PR branch once approvals are completed and removed AT: WIP Causes the PR autotester to not test the PR. (Remove to allow testing to occur.) labels Dec 5, 2018
@trilinos-autotester
Copy link
Contributor

Status Flag 'Pull Request AutoTester' - Jenkins Testing: 1 or more Jobs FAILED

Note: Testing will normally be attempted again in approx. 2 Hrs 30 Mins. If a change to the PR source branch occurs, the testing will be attempted again on next available autotester run.

Pull Request Auto Testing has FAILED (click to expand)

Build Information

Test Name: Trilinos_pullrequest_intel_17.0.1

  • Build Num: 1795
  • Status: ERROR

Jenkins Parameters

Parameter Name Value
PULLREQUESTNUM 3999
TEST_REPO_ALIAS TRILINOS
TRILINOS_SOURCE_BRANCH 3968-tpetra-cuda-aware-absmax
TRILINOS_SOURCE_REPO https://github.com/rppawlo/Trilinos
TRILINOS_SOURCE_SHA 3ff9338
TRILINOS_TARGET_BRANCH develop
TRILINOS_TARGET_REPO https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA 436a132

Build Information

Test Name: Trilinos_pullrequest_gcc_4.9.3

  • Build Num: 2358
  • Status: ERROR

Jenkins Parameters

Parameter Name Value
COMPILER_MODULE sems-gcc/4.9.3
JENKINS_BUILD_TYPE Release
JENKINS_COMM_TYPE MPI
JENKINS_DO_COMPLEX OFF
JENKINS_JOB_TYPE Experimental
MPI_MODULE sems-openmpi/1.8.7
PULLREQUESTNUM 3999
TEST_REPO_ALIAS TRILINOS
TRILINOS_SOURCE_BRANCH 3968-tpetra-cuda-aware-absmax
TRILINOS_SOURCE_REPO https://github.com/rppawlo/Trilinos
TRILINOS_SOURCE_SHA 3ff9338
TRILINOS_TARGET_BRANCH develop
TRILINOS_TARGET_REPO https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA 436a132

Build Information

Test Name: Trilinos_pullrequest_gcc_4.8.4

  • Build Num: 2018
  • Status: ERROR

Jenkins Parameters

Parameter Name Value
COMPILER_MODULE sems-gcc/4.8.4
JENKINS_BUILD_TYPE Release
JENKINS_COMM_TYPE MPI
JENKINS_DO_COMPLEX OFF
JENKINS_JOB_TYPE Experimental
MPI_MODULE sems-openmpi/1.8.7
PULLREQUESTNUM 3999
TEST_REPO_ALIAS TRILINOS
TRILINOS_SOURCE_BRANCH 3968-tpetra-cuda-aware-absmax
TRILINOS_SOURCE_REPO https://github.com/rppawlo/Trilinos
TRILINOS_SOURCE_SHA 3ff9338
TRILINOS_TARGET_BRANCH develop
TRILINOS_TARGET_REPO https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA 436a132

Build Information

Test Name: Trilinos_pullrequest_gcc_4.9.3_SERIAL

  • Build Num: 327
  • Status: ERROR

Jenkins Parameters

Parameter Name Value
PULLREQUESTNUM 3999
TEST_REPO_ALIAS TRILINOS
TRILINOS_SOURCE_BRANCH 3968-tpetra-cuda-aware-absmax
TRILINOS_SOURCE_REPO https://github.com/rppawlo/Trilinos
TRILINOS_SOURCE_SHA 3ff9338
TRILINOS_TARGET_BRANCH develop
TRILINOS_TARGET_REPO https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA 436a132
Console Output (last 100 lines) : Trilinos_pullrequest_intel_17.0.1 # 1795 (click to expand)


*** Generating set of Trilinos enables given modified packages from
*** git commit origin/develop to HEAD


TRILINOS_DIR=/scratch/trilinos/workspace/trilinos-folder/Trilinos_pullrequest_intel_17.0.1/Trilinos
TRILINOS_SCRIPTS_DIR=/scratch/trilinos/workspace/trilinos-folder/Trilinos_pullrequest_intel_17.0.1/Trilinos
TRIBITS_DIR=/scratch/trilinos/workspace/trilinos-folder/Trilinos_pullrequest_intel_17.0.1/Trilinos/cmake/tribits

A) Generate the Trilinos Packages definition and depencencies XML file

Wrote the file 'TrilinosPackageDependencies.xml'

B) Get the set of changed files

Current directory: /scratch/trilinos/workspace/trilinos-folder/Trilinos_pullrequest_intel_17.0.1/Trilinos

git diff --name-only origin/develop..HEAD > /scratch/trilinos/workspace/trilinos-folder/Trilinos_pullrequest_intel_17.0.1/changed-files.txt

Wrote file 'changed-files.txt'

Current directory: /scratch/trilinos/workspace/trilinos-folder/Trilinos_pullrequest_intel_17.0.1

C) Get the unfiltered list of changed Trilinos packages (including 'ALL_PACKAGES')

CHANGED_PACKAGES_FULL_LIST='PanzerAdaptersSTK,PanzerDofMgr,TpetraCore'

D) Filter list of changed packages to get only the PT packages

CHANGED_PACKAGES_PT_LIST='PanzerAdaptersSTK,PanzerDofMgr,TpetraCore'

E) Generate the *.cmake enables file

Wrote file 'packageEnables.cmake'

  • set +x
    Enabled packages:
    -- Setting Trilinos_ENABLE_PanzerAdaptersSTK = ON
    -- Setting Trilinos_ENABLE_PanzerDofMgr = ON
    -- Setting Trilinos_ENABLE_TpetraCore = ON
    Set CWD = /scratch/trilinos/workspace/trilinos-folder/Trilinos_pullrequest_intel_17.0.1/TFW_testing_single_configure_prototype
    Build name = PR-3999-test-Trilinos_pullrequest_intel_17.0.1-1795
    Cur dir = /scratch/trilinos/workspace/trilinos-folder/Trilinos_pullrequest_intel_17.0.1/TFW_testing_single_configure_prototype
    Source dir = /scratch/trilinos/workspace/trilinos-folder/Trilinos_pullrequest_intel_17.0.1/Trilinos
    Binary dir = /scratch/trilinos/workspace/trilinos-folder/Trilinos_pullrequest_intel_17.0.1/pull_request_test
    Parallel level = 18
    skip_by_parts_submit = OFF
    skip_single_submit = ON
    skip_update_step = ON
    skip_upload_config_files = OFF
    skip_clean_build_dir = OFF
    Subproject count = 53
    Dashboard model = Experimental
    Dashboard track = Pull Request
    Running configuration:
    /projects/sems/install/rhel6-x86_64/atdm/binary-install/cmake-3.11.1-Linux-x86_64/bin/cmake
    -C "/scratch/trilinos/workspace/trilinos-folder/Trilinos_pullrequest_intel_17.0.1/Trilinos/cmake/std/PullRequestLinuxIntelTestingSettings.cmake"
    -C "/scratch/trilinos/workspace/trilinos-folder/Trilinos_pullrequest_intel_17.0.1/packageEnables.cmake"
    -DTrilinos_ENABLE_TESTS:BOOL=ON
    -G "Ninja"
    /scratch/trilinos/workspace/trilinos-folder/Trilinos_pullrequest_intel_17.0.1/Trilinos
    CTEST_DROP_LOCATION = /cdash/submit.php?project=Trilinos
    CDash URL1 = https://testing-vm.sandia.gov/cdash/index.php?project=Trilinos&display=project&filtercount=3&showfilters=1&filtercombine=and&field1=site&compare1=61&value1=ascic158&field2=buildname&compare2=61&value2=PR-3999-test-Trilinos_pullrequest_intel_17.0.1-1795&field3=buildstamp&compare3=61&value3=20181205-1718-Pull Request
    CDash URL2 = https://testing-vm.sandia.gov/cdash/index.php?project=Trilinos&display=project&filtercount=2&showfilters=0&filtercombine=and&field1=buildname&compare1=61&value1=PR-3999-test-Trilinos_pullrequest_intel_17.0.1-1795&field2=buildstamp&compare2=61&value2=20181205-1718-Pull Request
    CDash URL3 = https://testing-vm.sandia.gov/cdash/index.php?project=Trilinos&filtercount=2&showfilters=0&filtercombine=and&field1=buildname&compare1=61&value1=PR-3999-test-Trilinos_pullrequest_intel_17.0.1-1795&field2=buildstamp&compare2=61&value2=20181205-1718-Pull Request
    Starting configure step.
    Each . represents 1024 bytes of output
    .................................................. Size: 50K
    .................................................. Size: 100K
    .................................................. Size: 150K
    .................................................. Size: 200K
    .................................................. Size: 250K
    ......................... Size of output: 274K
    configure submit error = 0
    Configure suceeded.
    Starting build step.
    Each symbol represents 1024 bytes of output.
    .................................................. Size: 49K
    .................................................. Size: 99K
    .................................................. Size: 149K
    .................................................. Size: 199K
    .................................................. Size: 250K
    .................................................. Size: 299K
    .................................................. Size: 349K
    .................................................. Size: 399K
    .................................................. Size: 449K
    .................................................. Size: 499K
    .................................................. Size: 549K
    .................................................. Size: 599K
    .................................................. Size: 649K
    .................................................. Size: 699K
    .................................................. Size: 749K
    .................................................. Size: 799K
    .................................................. Size: 849K
    .................................................. Size: 899K
    .................................................. Size: 949K
    ......................Build was aborted
    Aborted by James M Willenbring
    Archiving artifacts
    Finished: ABORTED
Console Output (last 100 lines) : Trilinos_pullrequest_gcc_4.9.3 # 2358 (click to expand)

***

TRILINOS_DIR=/scratch/trilinos/workspace/trilinos-folder/Trilinos_pullrequest_gcc_4.9.3/Trilinos
TRILINOS_SCRIPTS_DIR=/scratch/trilinos/workspace/trilinos-folder/Trilinos_pullrequest_gcc_4.9.3/Trilinos
TRIBITS_DIR=/scratch/trilinos/workspace/trilinos-folder/Trilinos_pullrequest_gcc_4.9.3/Trilinos/cmake/tribits

A) Generate the Trilinos Packages definition and depencencies XML file

Wrote the file 'TrilinosPackageDependencies.xml'

B) Get the set of changed files

Current directory: /scratch/trilinos/workspace/trilinos-folder/Trilinos_pullrequest_gcc_4.9.3/Trilinos

git diff --name-only origin/develop..HEAD > /scratch/trilinos/workspace/trilinos-folder/Trilinos_pullrequest_gcc_4.9.3/changed-files.txt

Wrote file 'changed-files.txt'

Current directory: /scratch/trilinos/workspace/trilinos-folder/Trilinos_pullrequest_gcc_4.9.3

C) Get the unfiltered list of changed Trilinos packages (including 'ALL_PACKAGES')

CHANGED_PACKAGES_FULL_LIST='PanzerAdaptersSTK,PanzerDofMgr,TpetraCore'

D) Filter list of changed packages to get only the PT packages

CHANGED_PACKAGES_PT_LIST='PanzerAdaptersSTK,PanzerDofMgr,TpetraCore'

E) Generate the *.cmake enables file

Wrote file 'packageEnables.cmake'

  • set +x
    Enabled packages:
    -- Setting Trilinos_ENABLE_PanzerAdaptersSTK = ON
    -- Setting Trilinos_ENABLE_PanzerDofMgr = ON
    -- Setting Trilinos_ENABLE_TpetraCore = ON
    Set CWD = /scratch/trilinos/workspace/trilinos-folder/Trilinos_pullrequest_gcc_4.9.3/TFW_testing_single_configure_prototype
    Build name = PR-3999-test-Trilinos_pullrequest_gcc_4.9.3-2358
    Cur dir = /scratch/trilinos/workspace/trilinos-folder/Trilinos_pullrequest_gcc_4.9.3/TFW_testing_single_configure_prototype
    Source dir = /scratch/trilinos/workspace/trilinos-folder/Trilinos_pullrequest_gcc_4.9.3/Trilinos
    Binary dir = /scratch/trilinos/workspace/trilinos-folder/Trilinos_pullrequest_gcc_4.9.3/pull_request_test
    Parallel level = 18
    skip_by_parts_submit = OFF
    skip_single_submit = ON
    skip_update_step = ON
    skip_upload_config_files = OFF
    skip_clean_build_dir = OFF
    Subproject count = 53
    Dashboard model = Experimental
    Dashboard track = Pull Request
    Running configuration:
    /projects/sems/install/rhel6-x86_64/atdm/binary-install/cmake-3.11.1-Linux-x86_64/bin/cmake
    -C "/scratch/trilinos/workspace/trilinos-folder/Trilinos_pullrequest_gcc_4.9.3/Trilinos/cmake/std/PullRequestLinuxGCC4.9.3TestingSettings.cmake"
    -C "/scratch/trilinos/workspace/trilinos-folder/Trilinos_pullrequest_gcc_4.9.3/packageEnables.cmake"
    -DTrilinos_ENABLE_TESTS:BOOL=ON
    -G "Ninja"
    /scratch/trilinos/workspace/trilinos-folder/Trilinos_pullrequest_gcc_4.9.3/Trilinos
    CTEST_DROP_LOCATION = /cdash/submit.php?project=Trilinos
    CDash URL1 = https://testing-vm.sandia.gov/cdash/index.php?project=Trilinos&display=project&filtercount=3&showfilters=1&filtercombine=and&field1=site&compare1=61&value1=ascic142&field2=buildname&compare2=61&value2=PR-3999-test-Trilinos_pullrequest_gcc_4.9.3-2358&field3=buildstamp&compare3=61&value3=20181205-1721-Pull Request
    CDash URL2 = https://testing-vm.sandia.gov/cdash/index.php?project=Trilinos&display=project&filtercount=2&showfilters=0&filtercombine=and&field1=buildname&compare1=61&value1=PR-3999-test-Trilinos_pullrequest_gcc_4.9.3-2358&field2=buildstamp&compare2=61&value2=20181205-1721-Pull Request
    CDash URL3 = https://testing-vm.sandia.gov/cdash/index.php?project=Trilinos&filtercount=2&showfilters=0&filtercombine=and&field1=buildname&compare1=61&value1=PR-3999-test-Trilinos_pullrequest_gcc_4.9.3-2358&field2=buildstamp&compare2=61&value2=20181205-1721-Pull Request
    Starting configure step.
    Each . represents 1024 bytes of output
    .................................................. Size: 50K
    .................................................. Size: 100K
    .................................................. Size: 150K
    .................................................. Size: 200K
    .................................................. Size: 250K
    ......................... Size of output: 274K
    configure submit error = 0
    Configure suceeded.
    Starting build step.
    Each symbol represents 1024 bytes of output.
    .................................................. Size: 49K
    .................................................. Size: 99K
    .................................................. Size: 149K
    .................................................. Size: 199K
    .................................................. Size: 249K
    .................................................. Size: 299K
    .................................................. Size: 349K
    .................................................. Size: 399K
    .................................................. Size: 449K
    .................................................. Size: 499K
    .................................................. Size: 549K
    .................................................. Size: 599K
    .................................................. Size: 649K
    .................................................. Size: 699K
    .................................................. Size: 749K
    .................................................. Size: 799K
    .................................................. Size: 849K
    .................................................. Size: 899K
    .................................................. Size: 950K
    .................................................. Size: 999K
    .................................................. Size: 1050K
    .................................................. Size: 1099K
    .................................................. Size: 1149K
    .................................................. Size: 1199K
    .................................................. Size: 1249K
    .................................................. Size: 1300K
    .................................................. Size: 1350K
    ..........
Console Output (last 100 lines) : Trilinos_pullrequest_gcc_4.8.4 # 2018 (click to expand)

***
*** Generating set of Trilinos enables given modified packages from
*** git commit origin/develop to HEAD
***

TRILINOS_DIR=/scratch/trilinos/workspace/trilinos-folder/Trilinos_pullrequest_gcc_4.8.4/Trilinos
TRILINOS_SCRIPTS_DIR=/scratch/trilinos/workspace/trilinos-folder/Trilinos_pullrequest_gcc_4.8.4/Trilinos
TRIBITS_DIR=/scratch/trilinos/workspace/trilinos-folder/Trilinos_pullrequest_gcc_4.8.4/Trilinos/cmake/tribits

A) Generate the Trilinos Packages definition and depencencies XML file

Wrote the file 'TrilinosPackageDependencies.xml'

B) Get the set of changed files

Current directory: /scratch/trilinos/workspace/trilinos-folder/Trilinos_pullrequest_gcc_4.8.4/Trilinos

git diff --name-only origin/develop..HEAD > /scratch/trilinos/workspace/trilinos-folder/Trilinos_pullrequest_gcc_4.8.4/changed-files.txt

Wrote file 'changed-files.txt'

Current directory: /scratch/trilinos/workspace/trilinos-folder/Trilinos_pullrequest_gcc_4.8.4

C) Get the unfiltered list of changed Trilinos packages (including 'ALL_PACKAGES')

CHANGED_PACKAGES_FULL_LIST='PanzerAdaptersSTK,PanzerDofMgr,TpetraCore'

D) Filter list of changed packages to get only the PT packages

CHANGED_PACKAGES_PT_LIST='PanzerAdaptersSTK,PanzerDofMgr,TpetraCore'

E) Generate the *.cmake enables file

Wrote file 'packageEnables.cmake'

  • set +x
    Enabled packages:
    -- Setting Trilinos_ENABLE_PanzerAdaptersSTK = ON
    -- Setting Trilinos_ENABLE_PanzerDofMgr = ON
    -- Setting Trilinos_ENABLE_TpetraCore = ON
    Set CWD = /scratch/trilinos/workspace/trilinos-folder/Trilinos_pullrequest_gcc_4.8.4/TFW_testing_single_configure_prototype
    Build name = PR-3999-test-Trilinos_pullrequest_gcc_4.8.4-2018
    Cur dir = /scratch/trilinos/workspace/trilinos-folder/Trilinos_pullrequest_gcc_4.8.4/TFW_testing_single_configure_prototype
    Source dir = /scratch/trilinos/workspace/trilinos-folder/Trilinos_pullrequest_gcc_4.8.4/Trilinos
    Binary dir = /scratch/trilinos/workspace/trilinos-folder/Trilinos_pullrequest_gcc_4.8.4/pull_request_test
    Parallel level = 18
    skip_by_parts_submit = OFF
    skip_single_submit = ON
    skip_update_step = ON
    skip_upload_config_files = OFF
    skip_clean_build_dir = OFF
    Subproject count = 53
    Dashboard model = Experimental
    Dashboard track = Pull Request
    Running configuration:
    /projects/sems/install/rhel6-x86_64/atdm/binary-install/cmake-3.11.1-Linux-x86_64/bin/cmake
    -C "/scratch/trilinos/workspace/trilinos-folder/Trilinos_pullrequest_gcc_4.8.4/Trilinos/cmake/std/PullRequestLinuxGCC4.8.4TestingSettings.cmake"
    -C "/scratch/trilinos/workspace/trilinos-folder/Trilinos_pullrequest_gcc_4.8.4/packageEnables.cmake"
    -DTrilinos_ENABLE_TESTS:BOOL=ON
    -G "Ninja"
    /scratch/trilinos/workspace/trilinos-folder/Trilinos_pullrequest_gcc_4.8.4/Trilinos
    CTEST_DROP_LOCATION = /cdash/submit.php?project=Trilinos
    CDash URL1 = https://testing-vm.sandia.gov/cdash/index.php?project=Trilinos&display=project&filtercount=3&showfilters=1&filtercombine=and&field1=site&compare1=61&value1=ascic142&field2=buildname&compare2=61&value2=PR-3999-test-Trilinos_pullrequest_gcc_4.8.4-2018&field3=buildstamp&compare3=61&value3=20181205-1725-Pull Request
    CDash URL2 = https://testing-vm.sandia.gov/cdash/index.php?project=Trilinos&display=project&filtercount=2&showfilters=0&filtercombine=and&field1=buildname&compare1=61&value1=PR-3999-test-Trilinos_pullrequest_gcc_4.8.4-2018&field2=buildstamp&compare2=61&value2=20181205-1725-Pull Request
    CDash URL3 = https://testing-vm.sandia.gov/cdash/index.php?project=Trilinos&filtercount=2&showfilters=0&filtercombine=and&field1=buildname&compare1=61&value1=PR-3999-test-Trilinos_pullrequest_gcc_4.8.4-2018&field2=buildstamp&compare2=61&value2=20181205-1725-Pull Request
    Starting configure step.
    Each . represents 1024 bytes of output
    .................................................. Size: 50K
    .................................................. Size: 100K
    .................................................. Size: 150K
    .................................................. Size: 200K
    .................................................. Size: 250K
    ........................... Size of output: 276K
    configure submit error = 0
    Configure suceeded.
    Starting build step.
    Each symbol represents 1024 bytes of output.
    .................................................. Size: 49K
    .................................................. Size: 99K
    .................................................. Size: 149K
    .................................................. Size: 199K
    .................................................. Size: 249K
    .................................................. Size: 299K
    .................................................. Size: 349K
    .................................................. Size: 399K
    .................................................. Size: 450K
    .................................................. Size: 499K
    .................................................. Size: 549K
    .................................................. Size: 599K
    .................................................. Size: 649K
    .................................................. Size: 699K
    .................................................. Size: 749K
    .................................................. Size: 799K
    .................................................. Size: 849K
    .................................................. Size: 899K
    .................................................. Size: 949K
    .................................................. Size: 999K
    .................................................. Size: 1049K
    .................................................. Size: 1099K
    .................................................. Size: 1149K
    .................................................. Size: 1199K
    ....................................
Console Output (last 100 lines) : Trilinos_pullrequest_gcc_4.9.3_SERIAL # 327 (click to expand)

A) Generate the Trilinos Packages definition and depencencies XML file

Wrote the file 'TrilinosPackageDependencies.xml'

B) Get the set of changed files

Current directory: /scratch/trilinos/workspace/trilinos-folder/Trilinos_pullrequest_gcc_4.9.3_SERIAL/Trilinos

git diff --name-only origin/develop..HEAD > /scratch/trilinos/workspace/trilinos-folder/Trilinos_pullrequest_gcc_4.9.3_SERIAL/changed-files.txt

Wrote file 'changed-files.txt'

Current directory: /scratch/trilinos/workspace/trilinos-folder/Trilinos_pullrequest_gcc_4.9.3_SERIAL

C) Get the unfiltered list of changed Trilinos packages (including 'ALL_PACKAGES')

CHANGED_PACKAGES_FULL_LIST='PanzerAdaptersSTK,PanzerDofMgr,TpetraCore'

D) Filter list of changed packages to get only the PT packages

CHANGED_PACKAGES_PT_LIST='PanzerAdaptersSTK,PanzerDofMgr,TpetraCore'

E) Generate the *.cmake enables file

Wrote file 'packageEnables.cmake'

  • set +x
    Enabled packages:
    -- Setting Trilinos_ENABLE_PanzerAdaptersSTK = ON
    -- Setting Trilinos_ENABLE_PanzerDofMgr = ON
    -- Setting Trilinos_ENABLE_TpetraCore = ON
    Set CWD = /scratch/trilinos/workspace/trilinos-folder/Trilinos_pullrequest_gcc_4.9.3_SERIAL/TFW_testing_single_configure_prototype
    Build name = PR-3999-test-Trilinos_pullrequest_gcc_4.9.3_SERIAL-327
    Cur dir = /scratch/trilinos/workspace/trilinos-folder/Trilinos_pullrequest_gcc_4.9.3_SERIAL/TFW_testing_single_configure_prototype
    Source dir = /scratch/trilinos/workspace/trilinos-folder/Trilinos_pullrequest_gcc_4.9.3_SERIAL/Trilinos
    Binary dir = /scratch/trilinos/workspace/trilinos-folder/Trilinos_pullrequest_gcc_4.9.3_SERIAL/pull_request_test
    Parallel level = 18
    skip_by_parts_submit = OFF
    skip_single_submit = ON
    skip_update_step = ON
    skip_upload_config_files = OFF
    skip_clean_build_dir = OFF
    Subproject count = 53
    Dashboard model = Experimental
    Dashboard track = Pull Request
    Running configuration:
    /projects/sems/install/rhel6-x86_64/atdm/binary-install/cmake-3.11.1-Linux-x86_64/bin/cmake
    -C "/scratch/trilinos/workspace/trilinos-folder/Trilinos_pullrequest_gcc_4.9.3_SERIAL/Trilinos/cmake/std/PullRequestLinuxGCC4.9.3TestingSettingsSERIAL.cmake"
    -C "/scratch/trilinos/workspace/trilinos-folder/Trilinos_pullrequest_gcc_4.9.3_SERIAL/packageEnables.cmake"
    -DTrilinos_ENABLE_TESTS:BOOL=ON
    -G "Ninja"
    /scratch/trilinos/workspace/trilinos-folder/Trilinos_pullrequest_gcc_4.9.3_SERIAL/Trilinos
    CTEST_DROP_LOCATION = /cdash/submit.php?project=Trilinos
    CDash URL1 = https://testing-vm.sandia.gov/cdash/index.php?project=Trilinos&display=project&filtercount=3&showfilters=1&filtercombine=and&field1=site&compare1=61&value1=ascic115&field2=buildname&compare2=61&value2=PR-3999-test-Trilinos_pullrequest_gcc_4.9.3_SERIAL-327&field3=buildstamp&compare3=61&value3=20181205-1734-Pull Request
    CDash URL2 = https://testing-vm.sandia.gov/cdash/index.php?project=Trilinos&display=project&filtercount=2&showfilters=0&filtercombine=and&field1=buildname&compare1=61&value1=PR-3999-test-Trilinos_pullrequest_gcc_4.9.3_SERIAL-327&field2=buildstamp&compare2=61&value2=20181205-1734-Pull Request
    CDash URL3 = https://testing-vm.sandia.gov/cdash/index.php?project=Trilinos&filtercount=2&showfilters=0&filtercombine=and&field1=buildname&compare1=61&value1=PR-3999-test-Trilinos_pullrequest_gcc_4.9.3_SERIAL-327&field2=buildstamp&compare2=61&value2=20181205-1734-Pull Request
    Starting configure step.
    Each . represents 1024 bytes of output
    .................................................. Size: 50K
    .................................................. Size: 100K
    .................................................. Size: 150K
    .................................................. Size: 200K
    .................................................. Size: 250K
    ............. Size of output: 262K
    configure submit error = 0
    Configure suceeded.
    Starting build step.
    Each symbol represents 1024 bytes of output.
    .................................................. Size: 49K
    .................................................. Size: 99K
    .................................................. Size: 149K
    .................................................. Size: 199K
    .................................................. Size: 249K
    .................................................. Size: 299K
    .................................................. Size: 349K
    .................................................. Size: 399K
    .................................................. Size: 449K
    .................................................. Size: 499K
    .................................................. Size: 549K
    .................................................. Size: 599K
    .................................................. Size: 649K
    .................................................. Size: 699K
    .................................................. Size: 749K
    .................................................. Size: 799K
    .................................................. Size: 849K
    .................................................. Size: 899K
    .................................................. Size: 950K
    .................................................. Size: 999K
    .................................................. Size: 1049K
    .................................................. Size: 1099K
    .................................................. Size: 1149K
    .................................................. Size: 1199K
    .................................................. Size: 1249K
    .................................................. Size: 1299K
    .................................................. Size: 1349K
    .................................................. Size: 1399K
    .................................Build was aborted
    Aborted by James M Willenbring
    ................. Size: 1450K
    ...................................Finished: ABORTED


CDash Test Results for PR# 3999.


Wiki: How to Reproduce PR Testing Builds and Errors.

@rppawlo
Copy link
Contributor Author

rppawlo commented Dec 5, 2018

@mhoemmen - changes have been pushed. Could you review and reapprove?

@trilinos-autotester
Copy link
Contributor

Status Flag 'Pre-Test Inspection' - Auto Inspected - Inspection Is Not Necessary for this Pull Request.

@trilinos-autotester
Copy link
Contributor

Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects:

Pull Request Auto Testing STARTING (click to expand)

Build Information

Test Name: Trilinos_pullrequest_intel_17.0.1

  • Build Num: 1798
  • Status: STARTED

Jenkins Parameters

Parameter Name Value
PULLREQUESTNUM 3999
TEST_REPO_ALIAS TRILINOS
TRILINOS_SOURCE_BRANCH 3968-tpetra-cuda-aware-absmax
TRILINOS_SOURCE_REPO https://github.com/rppawlo/Trilinos
TRILINOS_SOURCE_SHA 5f91b56
TRILINOS_TARGET_BRANCH develop
TRILINOS_TARGET_REPO https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA 436a132

Build Information

Test Name: Trilinos_pullrequest_gcc_4.9.3

  • Build Num: 2361
  • Status: STARTED

Jenkins Parameters

Parameter Name Value
COMPILER_MODULE sems-gcc/4.9.3
JENKINS_BUILD_TYPE Release
JENKINS_COMM_TYPE MPI
JENKINS_DO_COMPLEX OFF
JENKINS_JOB_TYPE Experimental
MPI_MODULE sems-openmpi/1.8.7
PULLREQUESTNUM 3999
TEST_REPO_ALIAS TRILINOS
TRILINOS_SOURCE_BRANCH 3968-tpetra-cuda-aware-absmax
TRILINOS_SOURCE_REPO https://github.com/rppawlo/Trilinos
TRILINOS_SOURCE_SHA 5f91b56
TRILINOS_TARGET_BRANCH develop
TRILINOS_TARGET_REPO https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA 436a132

Build Information

Test Name: Trilinos_pullrequest_gcc_4.8.4

  • Build Num: 2020
  • Status: STARTED

Jenkins Parameters

Parameter Name Value
COMPILER_MODULE sems-gcc/4.8.4
JENKINS_BUILD_TYPE Release
JENKINS_COMM_TYPE MPI
JENKINS_DO_COMPLEX OFF
JENKINS_JOB_TYPE Experimental
MPI_MODULE sems-openmpi/1.8.7
PULLREQUESTNUM 3999
TEST_REPO_ALIAS TRILINOS
TRILINOS_SOURCE_BRANCH 3968-tpetra-cuda-aware-absmax
TRILINOS_SOURCE_REPO https://github.com/rppawlo/Trilinos
TRILINOS_SOURCE_SHA 5f91b56
TRILINOS_TARGET_BRANCH develop
TRILINOS_TARGET_REPO https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA 436a132

Build Information

Test Name: Trilinos_pullrequest_gcc_4.9.3_SERIAL

  • Build Num: 329
  • Status: STARTED

Jenkins Parameters

Parameter Name Value
PULLREQUESTNUM 3999
TEST_REPO_ALIAS TRILINOS
TRILINOS_SOURCE_BRANCH 3968-tpetra-cuda-aware-absmax
TRILINOS_SOURCE_REPO https://github.com/rppawlo/Trilinos
TRILINOS_SOURCE_SHA 5f91b56
TRILINOS_TARGET_BRANCH develop
TRILINOS_TARGET_REPO https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA 436a132

Using Repos:

Repo: TRILINOS (rppawlo/Trilinos)
  • Branch: 3968-tpetra-cuda-aware-absmax
  • SHA: 5f91b56
  • Mode: TEST_REPO

Pull Request Author: rppawlo

auto mv_size = values.extent(0);
Kokkos::parallel_reduce(mv_size,panzer::dof_functors::SumRank2<GO,KV>(values),localsum);
Kokkos::parallel_reduce(mv_size,panzer::dof_functors::SumRank2<GO,decltype(values)>(values),localsum);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the future, please consider using an explicit RangePolicy instead of putting the dimension as the first argument of Kokkos::parallel_reduce. This would let you execute on the execution_space corresponding to Tpetra's data, rather than on Kokkos' default execution space. Tpetra's default execution space could differ from Kokkos' default execution space. (Some people really want Serial for Tpetra but CUDA for everything else.)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will do. Looks like this functor was written in 2015 - before execution policies even existed :)

@trilinos-autotester
Copy link
Contributor

Status Flag 'Pull Request AutoTester' - Jenkins Testing: all Jobs PASSED

Pull Request Auto Testing has PASSED (click to expand)

Build Information

Test Name: Trilinos_pullrequest_intel_17.0.1

  • Build Num: 1798
  • Status: PASSED

Jenkins Parameters

Parameter Name Value
PULLREQUESTNUM 3999
TEST_REPO_ALIAS TRILINOS
TRILINOS_SOURCE_BRANCH 3968-tpetra-cuda-aware-absmax
TRILINOS_SOURCE_REPO https://github.com/rppawlo/Trilinos
TRILINOS_SOURCE_SHA 5f91b56
TRILINOS_TARGET_BRANCH develop
TRILINOS_TARGET_REPO https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA 436a132

Build Information

Test Name: Trilinos_pullrequest_gcc_4.9.3

  • Build Num: 2361
  • Status: PASSED

Jenkins Parameters

Parameter Name Value
COMPILER_MODULE sems-gcc/4.9.3
JENKINS_BUILD_TYPE Release
JENKINS_COMM_TYPE MPI
JENKINS_DO_COMPLEX OFF
JENKINS_JOB_TYPE Experimental
MPI_MODULE sems-openmpi/1.8.7
PULLREQUESTNUM 3999
TEST_REPO_ALIAS TRILINOS
TRILINOS_SOURCE_BRANCH 3968-tpetra-cuda-aware-absmax
TRILINOS_SOURCE_REPO https://github.com/rppawlo/Trilinos
TRILINOS_SOURCE_SHA 5f91b56
TRILINOS_TARGET_BRANCH develop
TRILINOS_TARGET_REPO https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA 436a132

Build Information

Test Name: Trilinos_pullrequest_gcc_4.8.4

  • Build Num: 2020
  • Status: PASSED

Jenkins Parameters

Parameter Name Value
COMPILER_MODULE sems-gcc/4.8.4
JENKINS_BUILD_TYPE Release
JENKINS_COMM_TYPE MPI
JENKINS_DO_COMPLEX OFF
JENKINS_JOB_TYPE Experimental
MPI_MODULE sems-openmpi/1.8.7
PULLREQUESTNUM 3999
TEST_REPO_ALIAS TRILINOS
TRILINOS_SOURCE_BRANCH 3968-tpetra-cuda-aware-absmax
TRILINOS_SOURCE_REPO https://github.com/rppawlo/Trilinos
TRILINOS_SOURCE_SHA 5f91b56
TRILINOS_TARGET_BRANCH develop
TRILINOS_TARGET_REPO https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA 436a132

Build Information

Test Name: Trilinos_pullrequest_gcc_4.9.3_SERIAL

  • Build Num: 329
  • Status: PASSED

Jenkins Parameters

Parameter Name Value
PULLREQUESTNUM 3999
TEST_REPO_ALIAS TRILINOS
TRILINOS_SOURCE_BRANCH 3968-tpetra-cuda-aware-absmax
TRILINOS_SOURCE_REPO https://github.com/rppawlo/Trilinos
TRILINOS_SOURCE_SHA 5f91b56
TRILINOS_TARGET_BRANCH develop
TRILINOS_TARGET_REPO https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA 436a132


CDash Test Results for PR# 3999.

@trilinos-autotester
Copy link
Contributor

Status Flag 'Pre-Merge Inspection' - SUCCESS: The last commit to this Pull Request has been INSPECTED AND APPROVED by [ mhoemmen ]!

@trilinos-autotester
Copy link
Contributor

Status Flag 'Pull Request AutoTester' - Pull Request will be Automerged

@trilinos-autotester trilinos-autotester merged commit 3eabe8d into trilinos:develop Dec 5, 2018
@trilinos-autotester
Copy link
Contributor

Merge on Pull Request# 3999: IS A SUCCESS - Pull Request successfully merged

@trilinos-autotester trilinos-autotester removed the AT: AUTOMERGE Causes the PR autotester to automatically merge the PR branch once approvals are completed label Dec 5, 2018
@bartlettroscoe bartlettroscoe removed the stage: in progress Work on the issue has started label Dec 5, 2018
@rppawlo rppawlo deleted the 3968-tpetra-cuda-aware-absmax branch December 6, 2018 17:46
mhoemmen pushed a commit to mhoemmen/Trilinos that referenced this pull request Dec 10, 2018
@trilinos/tpetra

This was originally part of PR trilinos#3996.  @rppawlo pushed an improved fix
to trilinos#3968 first in PR trilinos#3999, which got merged.  I thus deleted PR trilinos#3996.
This commit includes just the unit test that was part of PR trilinos#3996, but
not part of PR trilinos#3999.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants