Transition to new site testing-vm.sandia.gov/cdash/

[bartlettroscoe]

CC: @trilinos/framework

Next Action Status

PR #3305 merged on 8/16/2018 that submits only to testing-vm.sandia.gov/cdash/. Redirect from https://testing.sandia.gov/cdash/ to https://testing-vm.sandia.gov/cdash/ completed on 8/30/2018. Next: Change all wiki references of https://testing-vm.sandia.gov/cdash/ to https://testing.sandia.gov/cdash/?

Description

This Story is to plan and track efforts to switch from the existing testing.sandia.gov/cdash/ site to the new just approved testing-vm.sandia.gov/cdash/ site (see https://gitlab.kitware.com/snl/project-1/issues/33). The machine testing.sandia.gov is old and needs to be retired and the new site testing-vm.sandia.gov/cdash/ supports all-at-once configure/build/test/submit and lots of other improvements and bug fixes.

Tasks

• Test and approve testing-vm.sandia.gov/cdash/ (see https://gitlab.kitware.com/snl/project-1/issues/33)
• Turn on CDash emails from testing-vm.sandia.gov/cdash/
• Send out email to trilinos-developers warning about the transition and stopping of sending CDash data to testing.sandia.gov shortly.
• Submit and merge to 'develop' a PR that stops dual submits to testing.sandia.gov/cdash/ by updating Trilinos/CTestConfig.cmake.
• Update Trilinos wiki pages to change from testing.sandia.gov to testing-vm.sandia.gov (can do this with a git clone of the wiki repo).
• Update 'master' from 'develop' with the updated Trilinos/CTestConfig.cmake file.
• Wait a day or more and verify that no data is submitted to testing.sandia.gov/cdash/ over a full 24-hour testing day period (to any CDash project on that site). (Or verify that no dual submits will occur after putting in redirect testing.sandia.gov/cdash/ to testing-vm.sandia.gov/cdash/.)
• Transition to testing-vm.sandia.gov/cdash/ and turn off testing.sandia.gov/cdash/ (all together)
  • Set up a URL redirect from testing.sandia.gov/cdash/ to testing-vm.sandia.gov/cdash/ (effectively taking testing.sandia.gov offline)
  • Send out email to Trilinos developers announcing the transition is complete

[bartlettroscoe]

FYI: I just turned off emails coming from the "Continuous" Group for the Trilinos project on the site testing.sandia.gov/cdash/ in order to accommodate switching the post-push CI build to all-at-once mode (see #3283). The "Continuous" Group has been configured to send out CDash emails for the Trilinos project on the new site testing-vm.sandia.gov/cdash/ for some time so nothing is lost by doing this.

[bartlettroscoe]

@jwillenbring, should we go ahead and turn off all CDash emails coming from testing.sandia.gov/cdash/?

[bartlettroscoe]

With @jwillenbring's approval, I am pulling the trigger on this. I just realized that we need to update the Trilinos/CMakeConfig.cmake file to push the commit that stops submitting to testing.sandia.gov/cdash/ at least a day before we set up the redirect from testing.sandia.gov/cdash/ to testing-vm.sandia.gov/cdash/ (effectively taking the old testing.sandia.gov machine offline). The reason is that we need to make sure that there will not existing automated builds against the 'develop' branch will not be submitting to both testing.sandia.gov/cdash/ and testing-vm.sandia.gov/cdash/ since I don't know what CDash will do with duplicate data. This really concerns me now that I think about this.

I would also like to ensure that the 'master' branch is updated from 'develop' that contains the change to Trilinos/CMakeConfig.cmake at least a day before we set up the redirect from testing.sandia.gov/cdash/ to testing-vm.sandia.gov/cdash/ in case someone is submitting to CDash using a version of Trilinos on 'master' (e.g. like Kitware might be doing).

[bartlettroscoe]

@trilinos/framework,

I created PR #3305 to submit to testing-vm.sandia.gov/cdash/ by default and stop submitting to testing.sandia.gov/cdash/. Can one of you please approve that PR? We need to get this merged to 'develop' and update 'master' at least a full day before we redirect from testing.sandia.gov/cdash/ to testing-vm.sandia.gov/cdash/ to avoid submitting the same data twice to the same CDash site.

[bartlettroscoe]

I sent out the below email announcing this transition. Because results will just be missing on testing.sandia.gov/cdash/ for several days, I will go ahead and update all of the references to the former on the trilinos GitHub wiki to smooth the transition. Also, I have noticed that filters for older CDash sites don't always work on newer CDash sites so I will check those as well as part of the update.

---

From: Bartlett, Roscoe A
Sent: Wednesday, August 15, 2018 7:08 PM
To: [email protected]
Cc: [email protected]
Subject: Transitioning from testing.sandia.gov to testing-vm.sandia.gov CDash sites

Hello Trilinos Developers,

FYI:

We will be transitioning from the testing.sandia.gov/cdash/ site to the testing-vm.sandia.gov/cdash/ site over the next few days. During this transition, we will update the configuration files in Trilinos to stop submitting to testing.sandia.gov/cdash/ and results will only be seen on testing-vm.sandia.gov/cdash/. Once the transition is complete, we will put in a web URL redirect from testing.sandia.gov/cdash/ to testing-vm.sandia.gov/cdash/ for those how may have pointers to the prior site.

For more details and to track progress on this activity, see:

#3285

Cheers,

-Ross

[bartlettroscoe]

The below email response fro Zack G. at Kitware confirms that we need to be careful and not submit results twice. Therefore, I will put in a step to watch testing.sandia.gov/cdash/ for a full day to make sure that nothing gets submitted to this once 'master' has been updated for a full 24 hours time period.

---

From: Zack Galbreath
Sent: Wednesday, August 15, 2018 7:28 PM
To: Bartlett, Roscoe A
Cc: ...
Subject: [EXTERNAL] Re: Submitting same data to same CDash site twice?

On Wed, Aug 15, 2018 at 6:58 PM, Bartlett, Roscoe A wrote:

Hello Zack, Bryon and Betsy,

What would happen if a ctest -S process submitted the same data for the same build to the same CDash site twice? What would CDash do with the duplicate submissions?

I am asking because when we switch from testing.sandia.gov/cdash/ to testing-vm.sandia.gov/cdash/ we will put in a redirect from testing.sandia.gov/cdash/ to testing-vm.sandia.gov/cdash/ and if older versions of Trilinos try to submit to CDash with dual submits still set up then they will be submitting the same data twice to the same CDash site. I am trying to figure out how bad that would be.

For the most part, numbers & such will get doubled. For instance, if you submit the same Test.xml file twice then its build will have two copies of each test. I believe this behavior is the same for build warnings & errors.

[bartlettroscoe]

PR #3305 was just merged which changes to submit Trilinos configure, build, and test results only to testing-vm.sandia.gov/cdash/ by default and change the submission of results for the old TrilinosDriver project using the old Trilinos Dashboard Driver process from the TrilinosDriver project on testing.sandia.gov/cdash/ to testing-vm.sandia.gov/cdash/. Now we wait and watch the CDash sites and ensure that there are no submissions to any testing.sandia.gov/cdash/ project in a couple of days (hopefully).

[bartlettroscoe]

FYI: I don't know if this is related to this transition and changes in the Trilinos/CTestConfig.cmake file, but we are seeing duplicate test results showing up for the ATDM builds this morning. For example, see the duplicate failing tests for the same build and same site shown in this query today.

I will wait and see if this clears up tomorrow and after that. If not, then I will dig into this on Monday morning.

[bartlettroscoe]

FYI: I updated all of the references for testing.sandia.gov/cdash/ on the Trilinos wiki to testing-vm.sandia.gov/cdash/ in the Trilinos.wiki repo commit:

d0c1901 "Change all references to testing-vm.sandia.gov (trilinos/Trilinos#3285)"
Author: Roscoe A. Bartlett <[email protected]>
Date:   Fri Aug 17 18:02:09 2018 -0600 (86 seconds ago)

M       Policies--|-Testing.md
M       Policies-|-Test-Dashboard-Policy.md

[bartlettroscoe]

FYI: The only build being submitted to the Trilinos CDash project on the old CDash site testing.sandia.gov/cdash/ is the build CMakeContract-m28b6a3c7 showin here. As described in:

• https://gitlab.kitware.com/snl/project-1/issues/71

that build is using a very old version of Trilinos from 2013 and it is not submitting builds to the CDash site as shown here. Therefore, when we put in the redirect from testing.sandia.gov to testing-vm.sandia.gov, the build will start submitting to the new site automatically through the redirect and it will not submit duplicate results.

Also, there is still one build submitting to the TrilinosDriver project on testing.sandia.gov/cdash/ as shown yesterday here. It is the build Linux-TDD-sems-srn-rhel6-slave-01 and it is just fundamentally broken. Also, we never set up dual submits of the TrilinosDriver job so these will get updated to submit to the new CDash site automatically after putting in the redirect.

Therefore, it looks like we are ready to put in the redirect from testing.sandia.gov to testing-vm.sandia.gov. I will let John P. know that he can put this in whenever he has the time.

[bartlettroscoe]

FYI: Looking at:

$ git log-short -2 --name-status github/master -- CTestConfig.cmake

21f7acc "Change from testing-vm.sandia.gov/CDash/ to /cdash/ (#1746)"
Author: Roscoe A. Bartlett <[email protected]>
Date:   Wed Oct 11 09:16:58 2017 -0600 (10 months ago)

M       CTestConfig.cmake

0c7793c "Set up dual submits to also submit to testing-vm.sandia.gov/cdash/ (#1746)"
Author: Roscoe A. Bartlett <[email protected]>
Date:   Tue Oct 10 14:23:47 2017 -0600 (10 months ago)

M       CTestConfig.cmake

(where the remote github points to [email protected]:trilinos/Trilinos.git) so the master branch has been updated with these changes.

[bartlettroscoe]

FYI: John Perseo just put in the redirect from https://testing.sandia.gov/cdash/ to https://testing-vm.sandia.gov/cdash/ and now it seems to be working.

We should go back and change all of the references on the wiki pointing to https://testing-vm.sandia.gov/cdash/ to point back to https://testing.sandia.gov/cdash/ at some point.

@trilinos/framework, anyone want to do that?

[bartlettroscoe]

From: Bartlett, Roscoe A
Sent: Thursday, August 30, 2018 2:09 PM
To: '[email protected]' [email protected]
Cc: '[email protected]' [email protected]
Subject: RE: Transitioning from testing.sandia.gov to testing-vm.sandia.gov CDash sites

FYI:

The transition to the new CDash site https://testing-vm.sandia.gov/cdash/ is complete and https://testing.sandia.gov/cdash/ now redirects to the new CDsah site.

-Ross