fix: Fixes config flaky tests #1680

lantoli · 2023-11-29T11:21:06Z

Description

Jira ticket: INTMDB-1342

Fixes config flaky tests.

Type of change:

Bug fix (non-breaking change which fixes an issue). Please, add the "bug" label to the PR.
New feature (non-breaking change which adds functionality). Please, add the "enhancement" label to the PR.
Breaking change (fix or feature that would cause existing functionality to not work as expected). Please, add the "breaking change" label to the PR.
This change requires a documentation update
Documentation fix/enhancement

Required Checklist:

I have signed the MongoDB CLA
I have read the contribution guidelines
I have checked that this change does not generate any credentials and that they are NOT accidentally logged anywhere.
I have added tests that prove my fix is effective or that my feature works per HashiCorp requirements
I have added any necessary documentation (if appropriate)
I have run make fmt and formatted my code
If changes include deprecations or removals, I defined an isolated PR with a relevant title as it will be used in the auto-generated changelog.

Further comments

This reverts commit bc77edf.

This reverts commit a2bd6e7.

This reverts commit ae6d707.

lantoli · 2023-11-29T14:08:00Z

.github/workflows/acceptance-tests.yml

+    concurrency:
+      group: acctest-config
+      cancel-in-progress: false


run config jobs sequentially so they don't interfere each other

lantoli · 2023-11-29T14:08:33Z

internal/service/atlasuser/data_source_atlas_users_test.go

@@ -222,10 +222,13 @@ func dataSourceChecksForUsers(dataSourceName, orgID string, users *admin.Paginat
 		resource.TestCheckResourceAttr(dataSourceName, "org_id", orgID),
 		resource.TestCheckResourceAttr(dataSourceName, "total_count", fmt.Sprintf("%d", totalCountValue)),
 	}
+	skipUsername := os.Getenv("MONGODB_ATLAS_USERNAME_CLOUD_DEV") // this user can be changed by other tests


don't check the user that might be changed by other tests in this execution

lantoli · 2023-11-29T15:21:45Z

internal/service/atlasuser/data_source_atlas_user_test.go

 		resource.TestCheckResourceAttr(dataSourceName, fmt.Sprintf("%steam_ids.#", attrPrefix), fmt.Sprintf("%d", len(user.TeamIds))),
 		resource.TestCheckResourceAttr(dataSourceName, fmt.Sprintf("%slinks.#", attrPrefix), fmt.Sprintf("%d", len(user.Links))),
+		resource.TestCheckResourceAttr(dataSourceName, fmt.Sprintf("%sroles.#", attrPrefix), fmt.Sprintf("%d", len(user.Roles))),


change order in case it fails to know if team_ids or links fail before

This reverts commit 41f86a3.

This reverts commit f382cf9.

This reverts commit be71b22.

This reverts commit 8e7f8ee.

lantoli · 2023-11-29T15:52:41Z

internal/service/atlasuser/data_source_atlas_user_test.go

@@ -66,7 +66,6 @@ func dataSourceChecksForUser(dataSourceName, attrPrefix string, user *admin.Clou
 		resource.TestCheckResourceAttr(dataSourceName, fmt.Sprintf("%smobile_number", attrPrefix), user.MobileNumber),
 		resource.TestCheckResourceAttr(dataSourceName, fmt.Sprintf("%scountry", attrPrefix), user.Country),
 		resource.TestCheckResourceAttr(dataSourceName, fmt.Sprintf("%screated_at", attrPrefix), *conversion.TimePtrToStringPtr(user.CreatedAt)),
-		resource.TestCheckResourceAttr(dataSourceName, fmt.Sprintf("%sroles.#", attrPrefix), fmt.Sprintf("%d", len(user.Roles))),


roles are changed by other tests. (still singular datasource is checking roles and it uses the same code as plural one so removing this check shouldn't be a problem)

I believe singular data source is using the same dataSourceChecksForUser for generating checks. Nevertheless agree with the approach, or we can simply check that roles attribute is defined.

AgustinBettati · 2023-11-29T16:41:59Z

internal/service/atlasuser/data_source_atlas_user_test.go

@@ -66,7 +66,6 @@ func dataSourceChecksForUser(dataSourceName, attrPrefix string, user *admin.Clou
 		resource.TestCheckResourceAttr(dataSourceName, fmt.Sprintf("%smobile_number", attrPrefix), user.MobileNumber),
 		resource.TestCheckResourceAttr(dataSourceName, fmt.Sprintf("%scountry", attrPrefix), user.Country),
 		resource.TestCheckResourceAttr(dataSourceName, fmt.Sprintf("%screated_at", attrPrefix), *conversion.TimePtrToStringPtr(user.CreatedAt)),
-		resource.TestCheckResourceAttr(dataSourceName, fmt.Sprintf("%sroles.#", attrPrefix), fmt.Sprintf("%d", len(user.Roles))),


I believe singular data source is using the same dataSourceChecksForUser for generating checks. Nevertheless agree with the approach, or we can simply check that roles attribute is defined.

lantoli · 2023-11-29T17:22:32Z

internal/service/atlasuser/data_source_atlas_user_test.go

 				),
 			},
 		},
 	})
 }

-func dataSourceChecksForUser(dataSourceName, attrPrefix string, user *admin.CloudAppUser) []resource.TestCheckFunc {
+func dataSourceChecksForUser(dataSourceName, attrPrefix string, user *admin.CloudAppUser, hasToCheckCountRoles bool) []resource.TestCheckFunc {


@AgustinBettati you're right, changing to keep role count in singular, just check attribute exists in plural

github-actions · 2023-11-29T17:24:46Z

Package	Line Rate	Health
github.com/mongodb/terraform-provider-mongodbatlas/internal/common/conversion	31%	❌
github.com/mongodb/terraform-provider-mongodbatlas/internal/common/validate	68%	➖
github.com/mongodb/terraform-provider-mongodbatlas/internal/provider	5%	❌
github.com/mongodb/terraform-provider-mongodbatlas/internal/service/advancedcluster	3%	❌
github.com/mongodb/terraform-provider-mongodbatlas/internal/service/alertconfiguration	0%	❌
github.com/mongodb/terraform-provider-mongodbatlas/internal/service/atlasuser	0%	❌
github.com/mongodb/terraform-provider-mongodbatlas/internal/service/cluster	1%	❌
github.com/mongodb/terraform-provider-mongodbatlas/internal/service/databaseuser	0%	❌
github.com/mongodb/terraform-provider-mongodbatlas/internal/service/encryptionatrest	0%	❌
github.com/mongodb/terraform-provider-mongodbatlas/internal/service/project	0%	❌
github.com/mongodb/terraform-provider-mongodbatlas/internal/service/projectapikey	1%	❌
github.com/mongodb/terraform-provider-mongodbatlas/internal/service/projectipaccesslist	0%	❌
github.com/mongodb/terraform-provider-mongodbatlas/internal/service/searchdeployment	23%	❌
github.com/mongodb/terraform-provider-mongodbatlas/internal/testutil/acc	5%	❌
github.com/mongodb/terraform-provider-mongodbatlas/mongodbatlas	2%	❌
Summary	3% (315 / 10714)	❌

AgustinBettati

LGTM

TEMPORARY - run only the flaky test

bc77edf

lantoli requested a review from a team as a code owner November 29, 2023 11:21

lantoli marked this pull request as draft November 29, 2023 11:21

github-actions bot added the bug label Nov 29, 2023

lantoli added 4 commits November 29, 2023 12:24

TEMPORARY - don't wait to start acc tests

ae6d707

Revert "TEMPORARY - run only the flaky test"

575597c

This reverts commit bc77edf.

some change

a2bd6e7

Revert "some change"

c1ba8d5

This reverts commit a2bd6e7.

lantoli added the run-testacc To run acceptance tests label Nov 29, 2023

exclude user

8e7f8ee

lantoli removed the run-testacc To run acceptance tests label Nov 29, 2023

lantoli added 2 commits November 29, 2023 14:09

Revert "TEMPORARY - don't wait to start acc tests"

1d8db81

This reverts commit ae6d707.

extract skip user name

be71b22

lantoli added run-testacc To run acceptance tests run-testacc-project run-testacc-config labels Nov 29, 2023

lantoli added 2 commits November 29, 2023 14:49

only allow one config acctest at a time

f382cf9

only allow one config acctest at a time

41f86a3

lantoli removed run-testacc To run acceptance tests run-testacc-project run-testacc-config labels Nov 29, 2023

lantoli commented Nov 29, 2023

View reviewed changes

lantoli marked this pull request as ready for review November 29, 2023 14:27

check roles at the end to see if team ids or links fail first

baab2da

lantoli commented Nov 29, 2023

View reviewed changes

lantoli added 4 commits November 29, 2023 16:48

Revert "only allow one config acctest at a time"

cd3af42

This reverts commit 41f86a3.

Revert "only allow one config acctest at a time"

a1a13c5

This reverts commit f382cf9.

Revert "extract skip user name"

50993bf

This reverts commit be71b22.

Revert "exclude user"

9fe1306

This reverts commit 8e7f8ee.

don't check roles as they're changed by other tests

bb8de18

lantoli commented Nov 29, 2023

View reviewed changes

oarbusi approved these changes Nov 29, 2023

View reviewed changes

AgustinBettati approved these changes Nov 29, 2023

View reviewed changes

check role count in singular ds

f91dd93

lantoli commented Nov 29, 2023

View reviewed changes

AgustinBettati approved these changes Nov 29, 2023

View reviewed changes

lantoli merged commit 5fc2064 into master Nov 29, 2023

lantoli deleted the INTMDB-1342_config_tests branch November 29, 2023 17:56

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix: Fixes config flaky tests #1680

fix: Fixes config flaky tests #1680

lantoli commented Nov 29, 2023

lantoli Nov 29, 2023

lantoli Nov 29, 2023

lantoli Nov 29, 2023

lantoli Nov 29, 2023 •

edited

Loading

AgustinBettati Nov 29, 2023

AgustinBettati Nov 29, 2023

lantoli Nov 29, 2023

github-actions bot commented Nov 29, 2023

AgustinBettati left a comment

fix: Fixes config flaky tests #1680

fix: Fixes config flaky tests #1680

Conversation

lantoli commented Nov 29, 2023

Description

Type of change:

Required Checklist:

Further comments

lantoli Nov 29, 2023

Choose a reason for hiding this comment

lantoli Nov 29, 2023

Choose a reason for hiding this comment

lantoli Nov 29, 2023

Choose a reason for hiding this comment

lantoli Nov 29, 2023 • edited Loading

Choose a reason for hiding this comment

AgustinBettati Nov 29, 2023

Choose a reason for hiding this comment

AgustinBettati Nov 29, 2023

Choose a reason for hiding this comment

lantoli Nov 29, 2023

Choose a reason for hiding this comment

github-actions bot commented Nov 29, 2023

AgustinBettati left a comment

Choose a reason for hiding this comment

lantoli Nov 29, 2023 •

edited

Loading