Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make export-dir, source and target databases dynamic to enable parallel runs #2019

Merged
merged 26 commits into from
Jan 9, 2025
Merged

Make export-dir, source and target databases dynamic to enable parallel runs #2019

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
26 commits
Select commit Hold shift + click to select a range
f4da0f4
Initial changes to enable parallel tests
shubham-yb Dec 1, 2024
ad054cb
Fixed Sakila and Chinook
shubham-yb Dec 1, 2024
1918713
Test fixes
shubham-yb Dec 1, 2024
335247a
Test fixes
shubham-yb Dec 1, 2024
1ea24a7
Added the usage of SKIP_DB_CREATION
shubham-yb Dec 1, 2024
23942d0
Added the limit to the schema names for Oracle
shubham-yb Dec 2, 2024
7835cd3
Added extra logs for debugging
shubham-yb Dec 3, 2024
41e0b18
Fix log name
shubham-yb Dec 3, 2024
4962c24
Added the NOCACHE clause to the sequences tables
shubham-yb Dec 4, 2024
f903d09
Merge branch 'main' into shubham/unique-export-dir
shubham-yb Dec 8, 2024
eecd0e9
Added uniqueness to ssl tests
shubham-yb Dec 8, 2024
f70ea8b
Merge branch 'main' into shubham/unique-export-dir
shubham-yb Dec 17, 2024
901d774
Fix for missing operand basename error
shubham-yb Dec 18, 2024
d478e31
Added verbose to ora2pg output
shubham-yb Dec 18, 2024
ccece7e
Changed the source db name for PG Partitions with Index test
shubham-yb Dec 19, 2024
fa67610
Use debug with ora2pg commands
shubham-yb Dec 20, 2024
bf31156
Use debug with ora2pg commands
shubham-yb Dec 20, 2024
c6c16bc
Merge branch 'main' into shubham/unique-export-dir
shubham-yb Dec 31, 2024
ff7373a
Cleanup
shubham-yb Dec 31, 2024
be69d26
Merge branch 'main' into shubham/unique-export-dir
shubham-yb Jan 7, 2025
c49b3d1
Review comments
shubham-yb Jan 7, 2025
a9035b7
Merge branch 'main' into shubham/unique-export-dir
shubham-yb Jan 7, 2025
a582746
Cleanup
shubham-yb Jan 7, 2025
0c88b8a
Addressed review comment
shubham-yb Jan 9, 2025
c6fa662
Revert "Addressed review comment"
shubham-yb Jan 9, 2025
dcdc035
Added comments
shubham-yb Jan 9, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
31 changes: 30 additions & 1 deletion migtests/scripts/functions.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1071,4 +1071,33 @@ EOF
exit 1
;;
esac
}
}

normalize_and_export_vars() {
local test_suffix=$1

# Normalize TEST_NAME
export NORMALIZED_TEST_NAME="$(echo "$TEST_NAME" | tr '/-' '_')"

# Set EXPORT_DIR
export EXPORT_DIR=${EXPORT_DIR:-"${TEST_DIR}/${NORMALIZED_TEST_NAME}_${test_suffix}_export-dir"}
shubham-yb marked this conversation as resolved.
Show resolved Hide resolved
if [ -n "${SOURCE_DB_SSL_MODE}" ]; then
EXPORT_DIR="${EXPORT_DIR}_ssl"
fi

# Set database-specific variables
case "${SOURCE_DB_TYPE}" in
postgresql|mysql)
export SOURCE_DB_NAME=${SOURCE_DB_NAME:-"${NORMALIZED_TEST_NAME}_${test_suffix}"}
;;
oracle)
# Limit schema name to 10 characters for Oracle/Debezium due to 30 character limit
export SOURCE_DB_SCHEMA=${SOURCE_DB_SCHEMA:-"${NORMALIZED_TEST_NAME:0:10}_${test_suffix}"}
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: you can do the truncate 0:10 also above when generating NORMALIZED_TEST_NAME.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Keeping it specific to Oracle for now. The longer export-dir name brought out some issues. It'll be good to have that test case.

export SOURCE_DB_SCHEMA=${SOURCE_DB_SCHEMA^^}
;;
*)
echo "ERROR: Unsupported SOURCE_DB_TYPE: ${SOURCE_DB_TYPE}"
exit 1
;;
esac
}
25 changes: 13 additions & 12 deletions migtests/scripts/live-migration-fallb-run-test.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,9 +18,7 @@ export REPO_ROOT="${PWD}"
export SCRIPTS="${REPO_ROOT}/migtests/scripts"
export TESTS_DIR="${REPO_ROOT}/migtests/tests"
export TEST_DIR="${TESTS_DIR}/${TEST_NAME}"
export NORMALIZED_TEST_NAME="$(echo "$TEST_NAME" | tr '/-' '_')"
export QUEUE_SEGMENT_MAX_BYTES=400

export PYTHONPATH="${REPO_ROOT}/migtests/lib"

# Order of env.sh import matters.
Expand All @@ -42,19 +40,22 @@ else
source ${SCRIPTS}/${SOURCE_DB_TYPE}/env.sh
fi

if [[ "${SOURCE_DB_TYPE}" == "postgresql" || "${SOURCE_DB_TYPE}" == "mysql" ]]; then
export SOURCE_DB_NAME=${SOURCE_DB_NAME:-"${NORMALIZED_TEST_NAME}_fallb"}
elif [[ "${SOURCE_DB_TYPE}" == "oracle" ]]; then
export SOURCE_DB_SCHEMA=${SOURCE_DB_SCHEMA:-"${NORMALIZED_TEST_NAME:0:10}_fallb"} && export SOURCE_DB_SCHEMA=${SOURCE_DB_SCHEMA^^}
else
echo "ERROR: Unsupported SOURCE_DB_TYPE: ${SOURCE_DB_TYPE}"
exit 1
fi

source ${SCRIPTS}/yugabytedb/env.sh
# export NORMALIZED_TEST_NAME="$(echo "$TEST_NAME" | tr '/-' '_')"
# if [[ "${SOURCE_DB_TYPE}" == "postgresql" || "${SOURCE_DB_TYPE}" == "mysql" ]]; then
# export SOURCE_DB_NAME=${SOURCE_DB_NAME:-"${NORMALIZED_TEST_NAME}_fallb"}
# elif [[ "${SOURCE_DB_TYPE}" == "oracle" ]]; then
# export SOURCE_DB_SCHEMA=${SOURCE_DB_SCHEMA:-"${NORMALIZED_TEST_NAME:0:10}_fallb"} && export SOURCE_DB_SCHEMA=${SOURCE_DB_SCHEMA^^}
# else
# echo "ERROR: Unsupported SOURCE_DB_TYPE: ${SOURCE_DB_TYPE}"
# exit 1
# fi

source ${SCRIPTS}/functions.sh

normalize_and_export_vars "fallb"

source ${SCRIPTS}/yugabytedb/env.sh

main() {

echo "Deleting the parent export-dir present in the test directory"
Expand Down
36 changes: 20 additions & 16 deletions migtests/scripts/live-migration-fallf-run-test.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,6 @@ export REPO_ROOT="${PWD}"
export SCRIPTS="${REPO_ROOT}/migtests/scripts"
export TESTS_DIR="${REPO_ROOT}/migtests/tests"
export TEST_DIR="${TESTS_DIR}/${TEST_NAME}"
export NORMALIZED_TEST_NAME="$(echo "$TEST_NAME" | tr '/-' '_')"

export PYTHONPATH="${REPO_ROOT}/migtests/lib"
export PATH="${PATH}:/usr/lib/oracle/21/client64/bin"
Expand All @@ -31,33 +30,37 @@ else
source "${TEST_DIR}/env.sh"
fi

export EXPORT_DIR=${EXPORT_DIR:-"${TEST_DIR}/${NORMALIZED_TEST_NAME}_fallf_export-dir"}
if [ -n "${SOURCE_DB_SSL_MODE}" ]; then
EXPORT_DIR="${EXPORT_DIR}_ssl"
fi

if [ "${SOURCE_DB_TYPE}" = "oracle" ]
then
source ${SCRIPTS}/${SOURCE_DB_TYPE}/live_env.sh
else
source ${SCRIPTS}/${SOURCE_DB_TYPE}/env.sh
fi

if [[ "${SOURCE_DB_TYPE}" == "postgresql" || "${SOURCE_DB_TYPE}" == "mysql" ]]; then
export SOURCE_DB_NAME=${SOURCE_DB_NAME:-"${NORMALIZED_TEST_NAME}_fallf"}
elif [[ "${SOURCE_DB_TYPE}" == "oracle" ]]; then
export SOURCE_DB_SCHEMA=${SOURCE_DB_SCHEMA:-"${NORMALIZED_TEST_NAME:0:10}_fallf"} && export SOURCE_DB_SCHEMA=${SOURCE_DB_SCHEMA^^}
else
echo "ERROR: Unsupported SOURCE_DB_TYPE: ${SOURCE_DB_TYPE}"
exit 1
fi
# export NORMALIZED_TEST_NAME="$(echo "$TEST_NAME" | tr '/-' '_')"
# export EXPORT_DIR=${EXPORT_DIR:-"${TEST_DIR}/${NORMALIZED_TEST_NAME}_fallf_export-dir"}
# if [ -n "${SOURCE_DB_SSL_MODE}" ]; then
# EXPORT_DIR="${EXPORT_DIR}_ssl"
# fi

source ${SCRIPTS}/${SOURCE_DB_TYPE}/ff_env.sh
# if [[ "${SOURCE_DB_TYPE}" == "postgresql" || "${SOURCE_DB_TYPE}" == "mysql" ]]; then
# export SOURCE_DB_NAME=${SOURCE_DB_NAME:-"${NORMALIZED_TEST_NAME}_fallf"}
# elif [[ "${SOURCE_DB_TYPE}" == "oracle" ]]; then
# export SOURCE_DB_SCHEMA=${SOURCE_DB_SCHEMA:-"${NORMALIZED_TEST_NAME:0:10}_fallf"} && export SOURCE_DB_SCHEMA=${SOURCE_DB_SCHEMA^^}
# else
# echo "ERROR: Unsupported SOURCE_DB_TYPE: ${SOURCE_DB_TYPE}"
# exit 1
# fi

source ${SCRIPTS}/yugabytedb/env.sh

source ${SCRIPTS}/functions.sh

normalize_and_export_vars "fallf"

source ${SCRIPTS}/${SOURCE_DB_TYPE}/ff_env.sh

source ${SCRIPTS}/yugabytedb/env.sh

main() {

echo "Deleting the parent export-dir present in the test directory"
Expand All @@ -77,6 +80,7 @@ main() {
if [ "${SOURCE_DB_TYPE}" = "oracle" ]
then
create_source_db ${SOURCE_DB_SCHEMA}
# TODO: Add dynamic Fall Forward schema creation
create_source_db ${SOURCE_REPLICA_DB_SCHEMA}
run_sqlplus_as_sys ${SOURCE_REPLICA_DB_NAME} ${SCRIPTS}/oracle/create_metadata_tables.sql
fi
Expand Down
30 changes: 16 additions & 14 deletions migtests/scripts/live-migration-run-test.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,23 +37,25 @@ else
source ${SCRIPTS}/${SOURCE_DB_TYPE}/env.sh
fi

export EXPORT_DIR=${EXPORT_DIR:-"${TEST_DIR}/${NORMALIZED_TEST_NAME}_live_export-dir"}
if [ -n "${SOURCE_DB_SSL_MODE}" ]; then
EXPORT_DIR="${EXPORT_DIR}_ssl"
fi
# export EXPORT_DIR=${EXPORT_DIR:-"${TEST_DIR}/${NORMALIZED_TEST_NAME}_live_export-dir"}
# if [ -n "${SOURCE_DB_SSL_MODE}" ]; then
# EXPORT_DIR="${EXPORT_DIR}_ssl"
# fi

# if [[ "${SOURCE_DB_TYPE}" == "postgresql" || "${SOURCE_DB_TYPE}" == "mysql" ]]; then
# export SOURCE_DB_NAME=${SOURCE_DB_NAME:-"${NORMALIZED_TEST_NAME}_live"}
# elif [[ "${SOURCE_DB_TYPE}" == "oracle" ]]; then
# export SOURCE_DB_SCHEMA=${SOURCE_DB_SCHEMA:-"${NORMALIZED_TEST_NAME:0:10}_live"} && export SOURCE_DB_SCHEMA=${SOURCE_DB_SCHEMA^^}
# else
# echo "ERROR: Unsupported SOURCE_DB_TYPE: ${SOURCE_DB_TYPE}"
# exit 1
# fi

if [[ "${SOURCE_DB_TYPE}" == "postgresql" || "${SOURCE_DB_TYPE}" == "mysql" ]]; then
export SOURCE_DB_NAME=${SOURCE_DB_NAME:-"${NORMALIZED_TEST_NAME}_live"}
elif [[ "${SOURCE_DB_TYPE}" == "oracle" ]]; then
export SOURCE_DB_SCHEMA=${SOURCE_DB_SCHEMA:-"${NORMALIZED_TEST_NAME:0:10}_live"} && export SOURCE_DB_SCHEMA=${SOURCE_DB_SCHEMA^^}
else
echo "ERROR: Unsupported SOURCE_DB_TYPE: ${SOURCE_DB_TYPE}"
exit 1
fi
source ${SCRIPTS}/functions.sh

source ${SCRIPTS}/yugabytedb/env.sh
normalize_and_export_vars "live"

source ${SCRIPTS}/functions.sh
source ${SCRIPTS}/yugabytedb/env.sh

main() {

Expand Down
29 changes: 16 additions & 13 deletions migtests/scripts/run-test.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,22 +36,25 @@ fi

source ${SCRIPTS}/${SOURCE_DB_TYPE}/env.sh

export EXPORT_DIR=${EXPORT_DIR:-"${TEST_DIR}/${NORMALIZED_TEST_NAME}_offline_export-dir"}
if [ -n "${SOURCE_DB_SSL_MODE}" ]; then
EXPORT_DIR="${EXPORT_DIR}_ssl"
fi
# export EXPORT_DIR=${EXPORT_DIR:-"${TEST_DIR}/${NORMALIZED_TEST_NAME}_offline_export-dir"}
# if [ -n "${SOURCE_DB_SSL_MODE}" ]; then
# EXPORT_DIR="${EXPORT_DIR}_ssl"
# fi

# if [[ "${SOURCE_DB_TYPE}" == "postgresql" || "${SOURCE_DB_TYPE}" == "mysql" ]]; then
# export SOURCE_DB_NAME=${SOURCE_DB_NAME:-"${NORMALIZED_TEST_NAME}_offline"}
# elif [[ "${SOURCE_DB_TYPE}" == "oracle" ]]; then
# export SOURCE_DB_SCHEMA=${SOURCE_DB_SCHEMA:-"${NORMALIZED_TEST_NAME:0:10}_offline"} && export SOURCE_DB_SCHEMA=${SOURCE_DB_SCHEMA^^}
# else
# echo "ERROR: Unsupported SOURCE_DB_TYPE: ${SOURCE_DB_TYPE}"
# exit 1
# fi

if [[ "${SOURCE_DB_TYPE}" == "postgresql" || "${SOURCE_DB_TYPE}" == "mysql" ]]; then
export SOURCE_DB_NAME=${SOURCE_DB_NAME:-"${NORMALIZED_TEST_NAME}_offline"}
elif [[ "${SOURCE_DB_TYPE}" == "oracle" ]]; then
export SOURCE_DB_SCHEMA=${SOURCE_DB_SCHEMA:-"${NORMALIZED_TEST_NAME:0:10}_offline"} && export SOURCE_DB_SCHEMA=${SOURCE_DB_SCHEMA^^}
else
echo "ERROR: Unsupported SOURCE_DB_TYPE: ${SOURCE_DB_TYPE}"
exit 1
fi
source ${SCRIPTS}/functions.sh

normalize_and_export_vars "offline"

source ${SCRIPTS}/yugabytedb/env.sh
source ${SCRIPTS}/functions.sh

main() {
echo "Deleting the parent export-dir present in the test directory"
Expand Down