Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adding Regression Tests Baseline Syncing Capabilities #2057

Closed
Closed
Changes from all commits
Commits
Show all changes
43 commits
Select commit Hold shift + click to select a range
50a91ab
feature/add-RT-logs, Jenkins testing.
kbooker79 Dec 14, 2023
c41ca97
Updating to fix typo.
kbooker79 Dec 14, 2023
eb20bd0
Updating with new stage to set variables
kbooker79 Dec 14, 2023
4b1d48a
Updating with required node.
kbooker79 Dec 14, 2023
38a6f48
Removing stage.
kbooker79 Dec 14, 2023
c1340b9
Testing.
kbooker79 Dec 14, 2023
32cd6c5
Update
kbooker79 Dec 14, 2023
d98d3c5
Moving to appropriate section
kbooker79 Dec 14, 2023
671b832
Updating to test pushes.
kbooker79 Dec 14, 2023
d3eda0e
Adjusting pathing
kbooker79 Dec 14, 2023
153b6ca
Updating to test again.
kbooker79 Dec 14, 2023
f3caab8
Updating.
kbooker79 Dec 14, 2023
e96e7f8
Updating with bash profile
kbooker79 Dec 14, 2023
8b4c024
Looking at env.
kbooker79 Dec 14, 2023
d3b5ac7
Updating to use explicit bash
kbooker79 Dec 14, 2023
6ebc918
Removing.
kbooker79 Dec 14, 2023
93c4f2b
Updating.
kbooker79 Dec 14, 2023
6400960
Merge commit '766ae9955a0374c0c07c2abf6026b6fb672479e3' into HEAD
yashikno Dec 14, 2023
da94d57
[AutoRT] Jet Job Completed.
epic-cicd-jenkins Dec 14, 2023
a7b72cb
[AutoRT] Hera Job Completed.
epic-cicd-jenkins Dec 14, 2023
6e2d895
[AutoRT] Orion Job Completed.
epic-cicd-jenkins Dec 14, 2023
a5ebae7
Trying something new on Gaea
kbooker79 Dec 14, 2023
bab90a3
Removing
kbooker79 Dec 14, 2023
15bd725
Updating
kbooker79 Dec 14, 2023
0a989cf
Fixing LD_LIBRARY_PATH issue.
kbooker79 Dec 14, 2023
624b327
Merge commit '10635efe47e18296de8ab538e61c6d6af6a143bc' into HEAD
yashikno Dec 14, 2023
cde038a
[AutoRT] Gaea Job Completed.
epic-cicd-jenkins Dec 14, 2023
4458a0b
[AutoRT] Hercules Job Completed.
epic-cicd-jenkins Dec 15, 2023
1fecbfb
Updating logname
kbooker79 Dec 15, 2023
c4633b1
Removing bash sourcing.
kbooker79 Dec 15, 2023
ccd6d77
Updating
kbooker79 Dec 15, 2023
ea5d8d7
[AutoRT] Derecho Job Completed.
epic-cicd-jenkins Dec 15, 2023
71fbf80
[AutoRT] Gaea Job Completed.
epic-cicd-jenkins Dec 15, 2023
5fd3d84
Removing logs
kbooker79 Dec 15, 2023
0717ee8
Fixing logs
kbooker79 Dec 15, 2023
03f64d1
feature/add-RT-logs, removing comments and preparing for final PR
kbooker79 Dec 15, 2023
8e75982
feature/add-RT-logs, removing extraneous line
kbooker79 Dec 15, 2023
0f227db
Reconciling logs
kbooker79 Dec 15, 2023
03d767d
Removing a comment from testing.
kbooker79 Dec 15, 2023
5c64a93
feature/baseline-sync, adding baselines
kbooker79 Dec 19, 2023
73c87c7
feature/baseline-sync, adding baseline syncing for Hera.
kbooker79 Dec 19, 2023
3ff28d4
Merge branch 'develop' into feature/baseline-sync
kbooker79 Dec 21, 2023
1143523
feature/baseline-sync, updating as per Zach's recommendations
kbooker79 Jan 2, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
220 changes: 220 additions & 0 deletions tests/ci/Jenkinsfile.combined
Original file line number Diff line number Diff line change
@@ -0,0 +1,220 @@
matchedNode = []
generateBaselineNode = []
for (label in pullRequest.labels) {
listOfLabelNodeNames = jenkins.model.Jenkins.instance.nodes.collect {
node -> node.getLabelString().contains(label) ? node.name : null

if ((label.matches(node.getLabelString()+"-(.*)"))) {
matchedNode += node.getLabelString()
}

if ((label.matches(node.getLabelString()+"(.*)-BL"))) {
generateBaselineNode += node.getLabelString()
}

}
}

modifiedLabels = matchedNode.collect{"'" + it + "'"}
baselineLabels = generateBaselineNode.collect{"'" + it + "'"}
def generateStage(nodeLabel) {
return {
stage("Running on ${nodeLabel}") {
node(nodeLabel) {
cleanWs()
checkout scm
script {
echo "Running on ${nodeLabel}"
if (baselineLabels.contains(nodeLabel)) {
sh '''
git submodule update --init --recursive
pwd
cd tests
export machine=${NODE_NAME}

if [[ $machine =~ "Jet" ]]
then
echo "Creating baselines on $machine"
export dprefix=/lfs1/NAGAPE/$ACCNR/$USER
./rt.sh -a ${ACCNR} -c -r -l rt.conf
elif [[ $machine =~ "Hercules" ]]
then
echo "Creating baselines on $machine"
export dprefix=/work2/noaa/$ACCNR/$USER
sed "s|/noaa/stmp/|/noaa/$ACCNR/stmp/|g" -i rt.sh
./rt.sh -a ${ACCNR} -c -e -l rt.conf
source bl_date.conf
export DISKNM=/work/noaa/epic/hercules/UFS-WM_RT
cd ${DISKNM}/NEMSfv3gfs/
mkdir develop-${BL_DATE}
cd /work2/noaa/epic/stmp/role-epic/stmp/role-epic/FV3_RT
rsync -a REGRESSION_TEST/ ${DISKNM}/NEMSfv3gfs/develop-${BL_DATE}
cd ${DISKNM}/NEMSfv3gfs/
./adjust_permissions.sh hercules develop-${BL_DATE}
chgrp noaa-hpc develop-${BL_DATE}
elif [[ $machine =~ "Orion" ]]
then
sed -i 's|/work/noaa/stmp/${USER}|/work/noaa/epic/stmp/role-epic/|g' rt.sh
export ACCNR=epic
./rt.sh -a ${ACCNR} -c -e -l rt.conf
source bl_date.conf
export DISKNM=/work/noaa/epic/UFS-WM_RT
cd ${DISKNM}/NEMSfv3gfs/
mkdir develop-${BL_DATE}
cd /work/noaa/epic/stmp/role-epic/stmp/role-epic/FV3_RT/
rsync -a REGRESSION_TEST/ ${DISKNM}/NEMSfv3gfs/develop-${BL_DATE}
cd ${DISKNM}/NEMSfv3gfs/
./adjust_permissions.sh orion develop-${BL_DATE}
chgrp noaa-hpc develop-${BL_DATE}
elif [[ $machine =~ "Gaea" ]]
then
echo "Creating baselines on $machine"
./rt.sh -a ${ACCNR} -c -e -l rt.conf
source bl_date.conf
export DISKNM=/lustre/f2/pdata/ncep/role.epic/C5/RT
cd ${DISKNM}/NEMSfv3gfs/
mkdir develop-${BL_DATE}
cd /lustre/f2/scratch/role.epic/FV3_RT
rsync -a REGRESSION_TEST/ ${DISKNM}/NEMSfv3gfs/develop-${BL_DATE}
cd ${DISKNM}/NEMSfv3gfs/
chgrp ncep develop-${BL_DATE}
elif [[ $machine =~ "Hera" ]]
then
echo "Creating baselines on $machine"
export ACCNR=nems
./rt.sh -a ${ACCNR} -c -r -l rt.conf
source bl_date.conf
export DISKNM=/scratch2/NAGAPE/epic/UFS-WM_RT
cd ${DISKNM}/NEMSfv3gfs/
mkdir develop-${BL_DATE}
cd /scratch1/NCEPDEV/stmp4/role.epic/FV3_RT
rsync -a REGRESSION_TEST/ ${DISKNM}/NEMSfv3gfs/develop-${BL_DATE}
elif [[ $machine =~ "Derecho" ]]
then
echo "Creating baselines on $machine"
export ACCNR=nral0032
./rt.sh -a ${ACCNR} -c -e -l rt.conf
source bl_date.conf
export DISKNM=/glade/derecho/scratch/epicufsrt/ufs-weather-model/RT/
cd ${DISKNM}/NEMSfv3gfs/
mkdir develop-${BL_DATE}
cd /glade/derecho/scratch/epicufsrt/FV3_RT
rsync -a REGRESSION_TEST/ ${DISKNM}/NEMSfv3gfs/develop-${BL_DATE}
else
echo "Creating baselines on $machine"
./rt.sh -a ${ACCNR} -c -r -l rt.conf
fi
git config user.email "[email protected]"
git config user.name "epic-cicd-jenkins"
echo "Testing concluded...removing labels for $machine from $GIT_URL"

GIT_OWNER=$(echo $GIT_URL | cut -d '/' -f4)
GIT_REPO_NAME=$(echo $GIT_URL | cut -d '/' -f5 | cut -d '.' -f1)

curl --silent -X DELETE -H "Accept: application/vnd.github.v3+json" -H "Authorization: Bearer ${GITHUB_TOKEN}" https://api.github.com/repos/${GIT_OWNER}/${GIT_REPO_NAME}/issues/${CHANGE_ID}/labels/$machine-BL
'''
}
else {
sh '''
git submodule update --init --recursive
pwd
cd tests
export machine=${NODE_NAME}

if [[ $machine =~ "Jet" ]]
then
echo "Running regression tests on $machine"
export dprefix=/lfs1/NAGAPE/$ACCNR/$USER
./rt.sh -a ${ACCNR} -r -l rt.conf
elif [[ $machine =~ "Hercules" ]]
then
echo "Running regression tests on $machine"
export dprefix=/work2/noaa/$ACCNR/$USER
sed "s|/noaa/stmp/|/noaa/$ACCNR/stmp/|g" -i rt.sh
./rt.sh -a ${ACCNR} -e -l rt.conf
elif [[ $machine =~ "Orion" ]]
then
echo "Running regression tests on $machine"
export dprefix=/work2/noaa/$ACCNR/$USER
sed "s|/noaa/stmp/|/noaa/$ACCNR/stmp/|g" -i rt.sh
./rt.sh -a ${ACCNR} -e -l rt.conf
elif [[ $machine =~ "Gaea" ]]
then
echo "Running regression tests on $machine"
./rt.sh -a ${ACCNR} -e -l rt.conf
elif [[ $machine =~ "Hera" ]]
then
echo "Running regression tests on $machine"
export ACCNR=nems
./rt.sh -a ${ACCNR} -r -l rt.conf
else
echo "Running regression tests on $machine"
./rt.sh -a ${ACCNR} -r -l rt.conf
fi

git config user.email "[email protected]"
git config user.name "epic-cicd-jenkins"
echo "Testing concluded...removing labels for $machine from $GIT_URL"

GIT_OWNER=$(echo $GIT_URL | cut -d '/' -f4)
GIT_REPO_NAME=$(echo $GIT_URL | cut -d '/' -f5 | cut -d '.' -f1)

curl --silent -X DELETE -H "Accept: application/vnd.github.v3+json" -H "Authorization: Bearer ${GITHUB_TOKEN}" https://api.github.com/repos/${GIT_OWNER}/${GIT_REPO_NAME}/issues/${CHANGE_ID}/labels/$machine-RT

'''
}
}
}
}
}
}

def parallelStagesMap = modifiedLabels.collectEntries {
["${it}" : generateStage(it)]
}

pipeline {
agent none
environment {
ACCNR = 'epic'
GITHUB_TOKEN = credentials('GithubJenkinsNew')
GIT_URL = 'https://github.com/ufs-community/ufs-weather-model.git'
}
stages {
stage('Run SonarQube Testing') {
steps {
script {
build job: '/ufs-weather-model/ufs-wm-sonarqube', parameters: [
string(name: 'BRANCH_NAME', value: env.CHANGE_BRANCH ?: 'develop'),
string(name: 'FORK_NAME', value: env.CHANGE_FORK ?: '')
], wait: false
}
}
}
stage('Run Regression Tests in Parallel') {
steps {
script {
parallel parallelStagesMap
}
}
}
}
post {
success {
node('built-in') {
echo 'This will run only if successful.'
sh '''
aws sns publish --topic-arn "arn:aws:sns:us-east-1:211527314271:Jenkins-CICD-Notifications" --region us-east-1 --message '{"version":"1.0","source":"custom","content":{"description":":sunny: Jenkins build *'"$JOB_NAME"' '"$BUILD_NUMBER"'* with *PR-'"$CHANGE_ID"'* *succeeded*"}}'
'''
}
}
failure {
node('built-in') {
echo 'This will run only if the run was marked as unstable.'
sh '''
aws sns publish --topic-arn "arn:aws:sns:us-east-1:211527314271:Jenkins-CICD-Notifications" --region us-east-1 --message '{"version":"1.0","source":"custom","content":{"description":":warning: Jenkins build *'"$JOB_NAME"' '"$BUILD_NUMBER"'* with *PR-'"$CHANGE_ID"'* *failed!*"}}'
'''
}
}
}
}