Skip to content

Commit

Permalink
Introduce TestContainer interface to streamline test setup of support…
Browse files Browse the repository at this point in the history
…ed databases (#2053)

- Any package in voyager code, for eg srcdb, tgtdb, connpool, yugabyted etc.. can spin up the required source db(oracle, pg, mysql) with a specific version to write unit tests.
- Every call to TestContainer.Start() check if a required database(dbtype + version) is already run then reuse it, otherwise start a new one

- In every package add a TestMain() function, to setup the environment needed specifically for that test.
- Added a new Github actions workflow to run the integration tests separately(required some oracle instance client libraries).
- Moving forward it will be good to have separation between go tests as unit or integration

* Bug fix: GetNonPKTables() for PG and YB returned non table objects like sequences and pk constraint objects
- fixed to return only table names
  • Loading branch information
sanyamsinghal authored Dec 24, 2024
1 parent 3c45ffc commit 43aaeeb
Show file tree
Hide file tree
Showing 36 changed files with 1,862 additions and 210 deletions.
4 changes: 2 additions & 2 deletions .github/workflows/go.yml
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ on:

jobs:

build:
build-and-test:
runs-on: ubuntu-22.04
steps:
- uses: actions/checkout@v3
Expand All @@ -26,7 +26,7 @@ jobs:
- name: Test
run: |
cd yb-voyager
go test -v -skip '^(TestDDLIssuesInYBVersion|TestDMLIssuesInYBVersion)$' ./...
go test -v -skip '^(TestDDLIssuesInYBVersion|TestDMLIssuesInYBVersion)$' ./... -tags '!integration'
- name: Vet
run: |
Expand Down
53 changes: 53 additions & 0 deletions .github/workflows/integration-tests.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,53 @@
name: Go

on:
push:
branches: ['main', '*.*-dev', '*.*.*-dev']
pull_request:
branches: [main]

env:
ORACLE_INSTANT_CLIENT_VERSION: "21.5.0.0.0-1"

jobs:
integration-tests:
strategy:
fail-fast: false

runs-on: ubuntu-22.04
steps:
- uses: actions/checkout@v3

- name: Set up Go
uses: actions/setup-go@v3
with:
go-version: "1.23.1"

- name: Build
run: |
cd yb-voyager
go build -v ./...
# required by godror driver used in the tests
- name: Install Oracle Instant Clients
run: |
# Download and install the YB APT repository package
wget https://s3.us-west-2.amazonaws.com/downloads.yugabyte.com/repos/reporpms/yb-apt-repo_1.0.0_all.deb
sudo apt-get install -y ./yb-apt-repo_1.0.0_all.deb
sudo apt-get update -y
# Install Oracle Instant Client packages using the defined version
sudo apt-get install -y oracle-instantclient-tools=${{ env.ORACLE_INSTANT_CLIENT_VERSION }}
sudo apt-get install -y oracle-instantclient-basic=${{ env.ORACLE_INSTANT_CLIENT_VERSION }}
sudo apt-get install -y oracle-instantclient-devel=${{ env.ORACLE_INSTANT_CLIENT_VERSION }}
sudo apt-get install -y oracle-instantclient-jdbc=${{ env.ORACLE_INSTANT_CLIENT_VERSION }}
sudo apt-get install -y oracle-instantclient-sqlplus=${{ env.ORACLE_INSTANT_CLIENT_VERSION }}
# Clean up the YB APT repository package
sudo apt-get remove -y yb-apt-repo
rm -f yb-apt-repo_1.0.0_all.deb
- name: Run Integration Tests
run: |
cd yb-voyager
go test -v -skip '^(TestDDLIssuesInYBVersion|TestDMLIssuesInYBVersion)$' ./... -tags 'integration'
3 changes: 1 addition & 2 deletions .github/workflows/issue-tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -47,5 +47,4 @@ jobs:
- name: Test Issues Against YB Version
run: |
cd yb-voyager
go test -v -run '^(TestDDLIssuesInYBVersion|TestDMLIssuesInYBVersion)$' ./...
go test -v -run '^(TestDDLIssuesInYBVersion|TestDMLIssuesInYBVersion)$' ./... -tags '!integration'
2 changes: 1 addition & 1 deletion yb-voyager/cmd/common_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -8,9 +8,9 @@ import (
"testing"

"github.com/yugabyte/yb-voyager/yb-voyager/src/migassessment"
"github.com/yugabyte/yb-voyager/yb-voyager/src/testutils"
"github.com/yugabyte/yb-voyager/yb-voyager/src/utils"
"github.com/yugabyte/yb-voyager/yb-voyager/src/ybversion"
testutils "github.com/yugabyte/yb-voyager/yb-voyager/test/utils"
)

func TestAssessmentReportStructs(t *testing.T) {
Expand Down
2 changes: 1 addition & 1 deletion yb-voyager/cmd/exportDataStatus_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -6,8 +6,8 @@ import (
"reflect"
"testing"

"github.com/yugabyte/yb-voyager/yb-voyager/src/testutils"
"github.com/yugabyte/yb-voyager/yb-voyager/src/utils/sqlname"
testutils "github.com/yugabyte/yb-voyager/yb-voyager/test/utils"
)

func TestExportSnapshotStatusStructs(t *testing.T) {
Expand Down
11 changes: 6 additions & 5 deletions yb-voyager/go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -22,8 +22,8 @@ require (
github.com/gosuri/uilive v0.0.4
github.com/gosuri/uitable v0.0.4
github.com/hashicorp/go-version v1.7.0
github.com/jackc/pgconn v1.13.0
github.com/jackc/pgx/v4 v4.17.2
github.com/jackc/pgconn v1.14.3
github.com/jackc/pgx/v4 v4.18.3
github.com/jackc/pgx/v5 v5.0.3
github.com/lib/pq v1.10.9
github.com/mattn/go-sqlite3 v1.14.17
Expand All @@ -45,6 +45,7 @@ require (
golang.org/x/term v0.24.0
google.golang.org/api v0.169.0
gopkg.in/natefinch/lumberjack.v2 v2.2.1
gotest.tools v2.2.0+incompatible
)

require (
Expand Down Expand Up @@ -143,9 +144,9 @@ require (
github.com/jackc/pgio v1.0.0 // indirect
github.com/jackc/pglogrepl v0.0.0-20231111135425-1627ab1b5780
github.com/jackc/pgpassfile v1.0.0 // indirect
github.com/jackc/pgproto3/v2 v2.3.1 // indirect
github.com/jackc/pgservicefile v0.0.0-20200714003250-2b9c44734f2b // indirect
github.com/jackc/pgtype v1.12.0 // indirect
github.com/jackc/pgproto3/v2 v2.3.3 // indirect
github.com/jackc/pgservicefile v0.0.0-20221227161230-091c0ba34f0a // indirect
github.com/jackc/pgtype v1.14.0 // indirect
github.com/kylelemons/godebug v1.1.0 // indirect
github.com/magiconair/properties v1.8.7
github.com/mattn/go-colorable v0.1.13 // indirect
Expand Down
15 changes: 10 additions & 5 deletions yb-voyager/go.sum
Original file line number Diff line number Diff line change
Expand Up @@ -1361,8 +1361,9 @@ github.com/jackc/pgconn v0.0.0-20190831204454-2fabfa3c18b7/go.mod h1:ZJKsE/KZfsU
github.com/jackc/pgconn v1.8.0/go.mod h1:1C2Pb36bGIP9QHGBYCjnyhqu7Rv3sGshaQUvmfGIB/o=
github.com/jackc/pgconn v1.9.0/go.mod h1:YctiPyvzfU11JFxoXokUOOKQXQmDMoJL9vJzHH8/2JY=
github.com/jackc/pgconn v1.9.1-0.20210724152538-d89c8390a530/go.mod h1:4z2w8XhRbP1hYxkpTuBjTS3ne3J48K83+u0zoyvg2pI=
github.com/jackc/pgconn v1.13.0 h1:3L1XMNV2Zvca/8BYhzcRFS70Lr0WlDg16Di6SFGAbys=
github.com/jackc/pgconn v1.13.0/go.mod h1:AnowpAqO4CMIIJNZl2VJp+KrkAZciAkhEl0W0JIobpI=
github.com/jackc/pgconn v1.14.3 h1:bVoTr12EGANZz66nZPkMInAV/KHD2TxH9npjXXgiB3w=
github.com/jackc/pgconn v1.14.3/go.mod h1:RZbme4uasqzybK2RK5c65VsHxoyaml09lx3tXOcO/VM=
github.com/jackc/pgio v1.0.0 h1:g12B9UwVnzGhueNavwioyEEpAmqMe1E/BN9ES+8ovkE=
github.com/jackc/pgio v1.0.0/go.mod h1:oP+2QK2wFfUWgr+gxjoBH9KGBb31Eio69xUb0w5bYf8=
github.com/jackc/pglogrepl v0.0.0-20231111135425-1627ab1b5780 h1:pNK2AKKIRC1MMMvpa6UiNtdtOebpiIloX7q2JZDkfsk=
Expand All @@ -1380,22 +1381,26 @@ github.com/jackc/pgproto3/v2 v2.0.0-rc3/go.mod h1:ryONWYqW6dqSg1Lw6vXNMXoBJhpzvW
github.com/jackc/pgproto3/v2 v2.0.0-rc3.0.20190831210041-4c03ce451f29/go.mod h1:ryONWYqW6dqSg1Lw6vXNMXoBJhpzvWKnT95C46ckYeM=
github.com/jackc/pgproto3/v2 v2.0.6/go.mod h1:WfJCnwN3HIg9Ish/j3sgWXnAfK8A9Y0bwXYU5xKaEdA=
github.com/jackc/pgproto3/v2 v2.1.1/go.mod h1:WfJCnwN3HIg9Ish/j3sgWXnAfK8A9Y0bwXYU5xKaEdA=
github.com/jackc/pgproto3/v2 v2.3.1 h1:nwj7qwf0S+Q7ISFfBndqeLwSwxs+4DPsbRFjECT1Y4Y=
github.com/jackc/pgproto3/v2 v2.3.1/go.mod h1:WfJCnwN3HIg9Ish/j3sgWXnAfK8A9Y0bwXYU5xKaEdA=
github.com/jackc/pgservicefile v0.0.0-20200714003250-2b9c44734f2b h1:C8S2+VttkHFdOOCXJe+YGfa4vHYwlt4Zx+IVXQ97jYg=
github.com/jackc/pgproto3/v2 v2.3.3 h1:1HLSx5H+tXR9pW3in3zaztoEwQYRC9SQaYUHjTSUOag=
github.com/jackc/pgproto3/v2 v2.3.3/go.mod h1:WfJCnwN3HIg9Ish/j3sgWXnAfK8A9Y0bwXYU5xKaEdA=
github.com/jackc/pgservicefile v0.0.0-20200714003250-2b9c44734f2b/go.mod h1:vsD4gTJCa9TptPL8sPkXrLZ+hDuNrZCnj29CQpr4X1E=
github.com/jackc/pgservicefile v0.0.0-20221227161230-091c0ba34f0a h1:bbPeKD0xmW/Y25WS6cokEszi5g+S0QxI/d45PkRi7Nk=
github.com/jackc/pgservicefile v0.0.0-20221227161230-091c0ba34f0a/go.mod h1:5TJZWKEWniPve33vlWYSoGYefn3gLQRzjfDlhSJ9ZKM=
github.com/jackc/pgtype v0.0.0-20190421001408-4ed0de4755e0/go.mod h1:hdSHsc1V01CGwFsrv11mJRHWJ6aifDLfdV3aVjFF0zg=
github.com/jackc/pgtype v0.0.0-20190824184912-ab885b375b90/go.mod h1:KcahbBH1nCMSo2DXpzsoWOAfFkdEtEJpPbVLq8eE+mc=
github.com/jackc/pgtype v0.0.0-20190828014616-a8802b16cc59/go.mod h1:MWlu30kVJrUS8lot6TQqcg7mtthZ9T0EoIBFiJcmcyw=
github.com/jackc/pgtype v1.8.1-0.20210724151600-32e20a603178/go.mod h1:C516IlIV9NKqfsMCXTdChteoXmwgUceqaLfjg2e3NlM=
github.com/jackc/pgtype v1.12.0 h1:Dlq8Qvcch7kiehm8wPGIW0W3KsCCHJnRacKW0UM8n5w=
github.com/jackc/pgtype v1.12.0/go.mod h1:LUMuVrfsFfdKGLw+AFFVv6KtHOFMwRgDDzBt76IqCA4=
github.com/jackc/pgtype v1.14.0 h1:y+xUdabmyMkJLyApYuPj38mW+aAIqCe5uuBB51rH3Vw=
github.com/jackc/pgtype v1.14.0/go.mod h1:LUMuVrfsFfdKGLw+AFFVv6KtHOFMwRgDDzBt76IqCA4=
github.com/jackc/pgx/v4 v4.0.0-20190420224344-cc3461e65d96/go.mod h1:mdxmSJJuR08CZQyj1PVQBHy9XOp5p8/SHH6a0psbY9Y=
github.com/jackc/pgx/v4 v4.0.0-20190421002000-1b8f0016e912/go.mod h1:no/Y67Jkk/9WuGR0JG/JseM9irFbnEPbuWV2EELPNuM=
github.com/jackc/pgx/v4 v4.0.0-pre1.0.20190824185557-6972a5742186/go.mod h1:X+GQnOEnf1dqHGpw7JmHqHc1NxDoalibchSk9/RWuDc=
github.com/jackc/pgx/v4 v4.12.1-0.20210724153913-640aa07df17c/go.mod h1:1QD0+tgSXP7iUjYm9C1NxKhny7lq6ee99u/z+IHFcgs=
github.com/jackc/pgx/v4 v4.17.2 h1:0Ut0rpeKwvIVbMQ1KbMBU4h6wxehBI535LK6Flheh8E=
github.com/jackc/pgx/v4 v4.17.2/go.mod h1:lcxIZN44yMIrWI78a5CpucdD14hX0SBDbNRvjDBItsw=
github.com/jackc/pgx/v4 v4.18.3 h1:dE2/TrEsGX3RBprb3qryqSV9Y60iZN1C6i8IrmW9/BA=
github.com/jackc/pgx/v4 v4.18.3/go.mod h1:Ey4Oru5tH5sB6tV7hDmfWFahwF15Eb7DNXlRKx2CkVw=
github.com/jackc/pgx/v5 v5.0.3 h1:4flM5ecR/555F0EcnjdaZa6MhBU+nr0QbZIo5vaKjuM=
github.com/jackc/pgx/v5 v5.0.3/go.mod h1:JBbvW3Hdw77jKl9uJrEDATUZIFM2VFPzRq4RWIhkF4o=
github.com/jackc/puddle v0.0.0-20190413234325-e4ced69a3a2b/go.mod h1:m4B5Dj62Y0fbyuIc15OsIqK0+JU8nkqQjsgx7dvjSWk=
Expand Down
2 changes: 1 addition & 1 deletion yb-voyager/src/callhome/diagnostics_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -5,8 +5,8 @@ import (
"testing"

"github.com/google/uuid"
"github.com/yugabyte/yb-voyager/yb-voyager/src/testutils"
"github.com/yugabyte/yb-voyager/yb-voyager/src/ybversion"
testutils "github.com/yugabyte/yb-voyager/yb-voyager/test/utils"
)

func TestCallhomeStructs(t *testing.T) {
Expand Down
22 changes: 13 additions & 9 deletions yb-voyager/src/cp/yugabyted/yugabyted_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -14,29 +14,33 @@ import (

"github.com/google/uuid"
"github.com/jackc/pgx/v4/pgxpool"
_ "github.com/lib/pq" // PostgreSQL driver
_ "github.com/jackc/pgx/v5/stdlib"
"github.com/stretchr/testify/assert"
controlPlane "github.com/yugabyte/yb-voyager/yb-voyager/src/cp"
"github.com/yugabyte/yb-voyager/yb-voyager/src/testutils"
"github.com/yugabyte/yb-voyager/yb-voyager/testcontainers"
"github.com/yugabyte/yb-voyager/yb-voyager/src/utils"
testcontainers "github.com/yugabyte/yb-voyager/yb-voyager/test/containers"
testutils "github.com/yugabyte/yb-voyager/yb-voyager/test/utils"
)

func TestYugabyteDTableSchema(t *testing.T) {
ctx := context.Background()

// Start a YugabyteDB container
ybContainer, host, port, err := testcontainers.StartDBContainer(ctx, testcontainers.YUGABYTEDB)
yugabyteDBContainer := testcontainers.NewTestContainer("yugabytedb", nil)
err := yugabyteDBContainer.Start(ctx)
if err != nil {
utils.ErrExit("Failed to start yugabytedb container: %v", err)
}
defer testcontainers.TerminateAllContainers()
assert.NoError(t, err, "Failed to start YugabyteDB container")
defer ybContainer.Terminate(ctx)

// Connect to the database
dsn := fmt.Sprintf("host=%s port=%s user=yugabyte password=yugabyte dbname=yugabyte sslmode=disable", host, port.Port())
db, err := sql.Open("postgres", dsn)
dsn := yugabyteDBContainer.GetConnectionString()
db, err := sql.Open("pgx", dsn)
assert.NoError(t, err)
defer db.Close()

// Wait for the database to be ready
err = testcontainers.WaitForDBToBeReady(db)
err = testutils.WaitForDBToBeReady(db)
assert.NoError(t, err)
// Export the database connection string to env variable YUGABYTED_DB_CONN_STRING
err = os.Setenv("YUGABYTED_DB_CONN_STRING", dsn)
Expand Down
2 changes: 1 addition & 1 deletion yb-voyager/src/datafile/descriptor_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ import (
"reflect"
"testing"

"github.com/yugabyte/yb-voyager/yb-voyager/src/testutils"
testutils "github.com/yugabyte/yb-voyager/yb-voyager/test/utils"
)

func TestDescriptorStructs(t *testing.T) {
Expand Down
2 changes: 1 addition & 1 deletion yb-voyager/src/dbzm/status_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ import (
"reflect"
"testing"

"github.com/yugabyte/yb-voyager/yb-voyager/src/testutils"
testutils "github.com/yugabyte/yb-voyager/yb-voyager/test/utils"
)

func TestExportStatusStructs(t *testing.T) {
Expand Down
2 changes: 1 addition & 1 deletion yb-voyager/src/metadb/metadataDB_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ import (
"testing"

_ "github.com/mattn/go-sqlite3"
"github.com/yugabyte/yb-voyager/yb-voyager/src/testutils"
testutils "github.com/yugabyte/yb-voyager/yb-voyager/test/utils"
)

// Test the initMetaDB function
Expand Down
2 changes: 1 addition & 1 deletion yb-voyager/src/migassessment/assessmentDB_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ import (
"testing"

_ "github.com/mattn/go-sqlite3"
"github.com/yugabyte/yb-voyager/yb-voyager/src/testutils"
testutils "github.com/yugabyte/yb-voyager/yb-voyager/test/utils"
)

func TestInitAssessmentDB(t *testing.T) {
Expand Down
2 changes: 1 addition & 1 deletion yb-voyager/src/namereg/namereg_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -13,8 +13,8 @@ import (
"github.com/stretchr/testify/require"

"github.com/yugabyte/yb-voyager/yb-voyager/src/constants"
"github.com/yugabyte/yb-voyager/yb-voyager/src/testutils"
"github.com/yugabyte/yb-voyager/yb-voyager/src/utils/sqlname"
testutils "github.com/yugabyte/yb-voyager/yb-voyager/test/utils"
)

var oracleToYBNameRegistry = &NameRegistry{
Expand Down
Loading

0 comments on commit 43aaeeb

Please sign in to comment.