Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Tech debt: Replace service-specific Region acceptance tests with acctest.PreCheckRegion #29964

Merged
merged 8 commits into from
Mar 13, 2023
Merged

Tech debt: Replace service-specific Region acceptance tests with acctest.PreCheckRegion #29964

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
88cc890
FMS: Require 'us-east-1' for acceptance tests.
ewbankkit Mar 12, 2023
599c22b
Lightsail Domains: Require 'us-east-1' for acceptance tests.
ewbankkit Mar 12, 2023
6780cae
WAF WebACL Logging: Require 'us-east-1' for acceptance tests.
ewbankkit Mar 12, 2023
e942755
Route 53 Zone Signing Keys: Require 'us-east-1' for acceptance tests.
ewbankkit Mar 12, 2023
3fa0bf2
Cost and Usage Reports: Require 'us-east-1' for acceptance tests.
ewbankkit Mar 12, 2023
56cce3c
Pricing: Require 'us-east-1' or 'ap-south-1' for acceptance tests.
ewbankkit Mar 12, 2023
d39eaa3
Log supported Regions in 'acctest.PreCheckRegion'.
ewbankkit Mar 12, 2023
8ed39c6
Remove 'Service-Specific Region Acceptance Testing' section from cont…
ewbankkit Mar 12, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
117 changes: 5 additions & 112 deletions docs/running-and-writing-acceptance-tests.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -571,6 +571,11 @@ If you add a new test that has preconditions which are checked by an existing pr

These are some of the standard provider PreChecks:

* `acctest.PreCheckRegion(t *testing.T, regions ...string)` checks that the test region is one of the specified AWS Regions.
* `acctest.PreCheckRegionNot(t *testing.T, regions ...string)` checks that the test region is not one of the specified AWS Regions.
* `acctest.PreCheckAlternateRegionIs(t *testing.T, region string)` checks that the alternate test region is the specified AWS Region.
* `acctest.PreCheckPartition(t *testing.T, partition string)` checks that the test partition is the specified partition.
* `acctest.PreCheckPartitionNot(t *testing.T, partitions ...string)` checks that the test partition is not one of the specified partitions.
* `acctest.PreCheckPartitionHasService(t *testing.T, serviceID string)` checks whether the current partition lists the service as part of its offerings. Note: AWS may not add new or public preview services to the service list immediately. This function will return a false positive in that case.
* `acctest.PreCheckOrganizationsAccount(ctx context.Context, t *testing.T)` checks whether the current account can perform AWS Organizations tests.
* `acctest.PreCheckAlternateAccount(t *testing.T)` checks whether the environment is set up for tests across accounts.
Expand Down Expand Up @@ -953,118 +958,6 @@ resource "aws_example" "test" {

Searching for usage of `acctest.PreCheckMultipleRegion` in the codebase will yield real world examples of this setup in action.

#### Service-Specific Region Acceptance Testing

Certain AWS service APIs are only available in specific AWS regions. For example as of this writing, the `pricing` service is available in `ap-south-1` and `us-east-1`, but no other regions or partitions. When encountering these types of services, the acceptance testing can be setup to automatically detect the correct region(s), while skipping the testing in unsupported partitions.

To prepare the shared service functionality, create a file named `internal/service/{SERVICE}/acc_test.go`. A starting example with the Pricing service (`internal/service/pricing/acc_test.go`):

```go
package aws

import (
"context"
"sync"
"testing"

"github.com/aws/aws-sdk-go/aws/endpoints"
"github.com/aws/aws-sdk-go/service/pricing"
"github.com/hashicorp/terraform-plugin-sdk/v2/diag"
"github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema"
"github.com/hashicorp/terraform-plugin-sdk/v2/terraform"
"github.com/hashicorp/terraform-provider-aws/internal/acctest"
"github.com/hashicorp/terraform-provider-aws/internal/provider"
)

// testAccPricingRegion is the chosen Pricing testing region
//
// Cached to prevent issues should multiple regions become available.
var testAccPricingRegion string

// testAccProviderPricing is the Pricing provider instance
//
// This Provider can be used in testing code for API calls without requiring
// the use of saving and referencing specific ProviderFactories instances.
//
// testAccPreCheckPricing(t) must be called before using this provider instance.
var testAccProviderPricing *schema.Provider

// testAccProviderPricingConfigure ensures the provider is only configured once
var testAccProviderPricingConfigure sync.Once

// testAccPreCheckPricing verifies AWS credentials and that Pricing is supported
func testAccPreCheckPricing(t *testing.T) {
acctest.PreCheckPartitionHasService(t, pricing.EndpointsID)

// Since we are outside the scope of the Terraform configuration we must
// call Configure() to properly initialize the provider configuration.
testAccProviderPricingConfigure.Do(func() {
testAccProviderPricing = provider.Provider()

config := map[string]interface{}{
"region": testAccGetPricingRegion(),
}

diags := testAccProviderPricing.Configure(context.Background(), terraform.NewResourceConfigRaw(config))

if diags != nil && diags.HasError() {
for _, d := range diags {
if d.Severity == diag.Error {
t.Fatalf("configuring Pricing provider: %s", d.Summary)
}
}
}
})
}

// testAccPricingRegionProviderConfig is the Terraform provider configuration for Pricing region testing
//
// Testing Pricing assumes no other provider configurations
// are necessary and overwrites the "aws" provider configuration.
func testAccPricingRegionProviderConfig() string {
return acctest.ConfigRegionalProvider(testAccGetPricingRegion())
}

// testAccGetPricingRegion returns the Pricing region for testing
func testAccGetPricingRegion() string {
if testAccPricingRegion != "" {
return testAccPricingRegion
}

if rs, ok := endpoints.RegionsForService(endpoints.DefaultPartitions(), testAccGetPartition(), pricing.ServiceName); ok {
// return available region (random if multiple)
for regionID := range rs {
testAccPricingRegion = regionID
return testAccPricingRegion
}
}

testAccPricingRegion = testAccGetRegion()

return testAccPricingRegion
}
```

For the resource or data source acceptance tests, the key items to adjust are:

* Ensure `TestCase` uses `ProtoV5ProviderFactories: acctest.ProtoV5ProviderFactories` instead of `ProviderFactories: acctest.ProviderFactories` or `Providers: acctest.Providers`
* Add the call for the new `PreCheck` function (keeping `acctest.PreCheck(t)`), e.g. `PreCheck: func() { acctest.PreCheck(t); testAccPreCheckPricing(t) },`
* If the testing is for a managed resource with a `CheckDestroy` function, ensure it uses the new provider instance, e.g. `testAccProviderPricing`, instead of `acctest.Provider`.
* If the testing is for a managed resource with a `Check...Exists` function, ensure it uses the new provider instance, e.g. `testAccProviderPricing`, instead of `acctest.Provider`.
* In each `TestStep` configuration, ensure the new provider configuration function is called, e.g.

```go
func testAccDataSourcePricingProductConfigRedshift() string {
return acctest.ConfigCompose(
testAccPricingRegionProviderConfig(),
`
# ... test configuration ...
`)
}
```

If the testing configurations require more than one region, reach out to the maintainers for further assistance.

#### Acceptance Test Concurrency

Certain AWS service APIs allow a limited number of a certain component, while the acceptance testing runs at a default concurrency of twenty tests at a time. For example as of this writing, the SageMaker service only allows one SageMaker Domain per AWS Region. Running the tests with the default concurrency will fail with API errors relating to the component quota being exceeded.
Expand Down
10 changes: 5 additions & 5 deletions internal/acctest/acctest.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -796,7 +796,7 @@ func PreCheckMultipleRegion(t *testing.T, regions int) {
}

if regions >= 3 {
if thirdRegionPartition() == "aws-us-gov" || Partition() == "aws-us-gov" {
if thirdRegionPartition() == endpoints.AwsUsGovPartitionID || Partition() == endpoints.AwsUsGovPartitionID {
t.Skipf("wanted %d regions, partition (%s) only has 2 regions", regions, Partition())
}

Expand All @@ -820,7 +820,7 @@ func PreCheckMultipleRegion(t *testing.T, regions int) {
}
}

// PreCheckRegion checks that the test region is one of the specified regions.
// PreCheckRegion checks that the test region is one of the specified AWS Regions.
func PreCheckRegion(t *testing.T, regions ...string) {
curr := Region()
var regionOK bool
Expand All @@ -833,11 +833,11 @@ func PreCheckRegion(t *testing.T, regions ...string) {
}

if !regionOK {
t.Skipf("skipping tests; %s (%s) not supported", envvar.DefaultRegion, curr)
t.Skipf("skipping tests; %s (%s) not supported. Supported: [%s]", envvar.DefaultRegion, curr, strings.Join(regions, ", "))
}
}

// PreCheckRegionNot checks that the test region is not one of the specified regions.
// PreCheckRegionNot checks that the test region is not one of the specified AWS Regions.
func PreCheckRegionNot(t *testing.T, regions ...string) {
curr := Region()

Expand All @@ -848,7 +848,7 @@ func PreCheckRegionNot(t *testing.T, regions ...string) {
}
}

// PreCheckAlternateRegionIs checks that the alternate test region is the specified region.
// PreCheckAlternateRegionIs checks that the alternate test region is the specified AWS Region.
func PreCheckAlternateRegionIs(t *testing.T, region string) {
if curr := AlternateRegion(); curr != region {
t.Skipf("skipping tests; %s (%s) does not equal %s", envvar.AlternateRegion, curr, region)
Expand Down
107 changes: 0 additions & 107 deletions internal/service/cur/acc_test.go
View file
Open in desktop

This file was deleted.

40 changes: 10 additions & 30 deletions internal/service/cur/report_definition_data_source_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,10 +4,10 @@ import (
"fmt"
"testing"

"github.com/aws/aws-sdk-go/service/costandusagereportservice"
"github.com/aws/aws-sdk-go/aws/endpoints"
cur "github.com/aws/aws-sdk-go/service/costandusagereportservice"
sdkacctest "github.com/hashicorp/terraform-plugin-sdk/v2/helper/acctest"
"github.com/hashicorp/terraform-plugin-sdk/v2/helper/resource"
"github.com/hashicorp/terraform-plugin-sdk/v2/terraform"
"github.com/hashicorp/terraform-provider-aws/internal/acctest"
)

Expand All @@ -20,15 +20,14 @@ func TestAccCURReportDefinitionDataSource_basic(t *testing.T) {
bucketName := fmt.Sprintf("tf-test-bucket-%d", sdkacctest.RandInt())

resource.ParallelTest(t, resource.TestCase{
PreCheck: func() { acctest.PreCheck(t); testAccPreCheck(ctx, t) },
ErrorCheck: acctest.ErrorCheck(t, costandusagereportservice.EndpointsID),
PreCheck: func() { acctest.PreCheck(t); acctest.PreCheckRegion(t, endpoints.UsEast1RegionID) },
ErrorCheck: acctest.ErrorCheck(t, cur.EndpointsID),
ProtoV5ProviderFactories: acctest.ProtoV5ProviderFactories,
CheckDestroy: testAccCheckReportDefinitionDestroy(ctx),
Steps: []resource.TestStep{
{
Config: testAccReportDefinitionDataSourceConfig_basic(reportName, bucketName),
Check: resource.ComposeTestCheckFunc(
testAccReportDefinitionCheckExistsDataSource(datasourceName, resourceName),
resource.TestCheckResourceAttrPair(datasourceName, "report_name", resourceName, "report_name"),
resource.TestCheckResourceAttrPair(datasourceName, "time_unit", resourceName, "time_unit"),
resource.TestCheckResourceAttrPair(datasourceName, "compression", resourceName, "compression"),
Expand All @@ -52,15 +51,14 @@ func TestAccCURReportDefinitionDataSource_additional(t *testing.T) {
bucketName := fmt.Sprintf("tf-test-bucket-%d", sdkacctest.RandInt())

resource.ParallelTest(t, resource.TestCase{
PreCheck: func() { acctest.PreCheck(t); testAccPreCheck(ctx, t) },
ErrorCheck: acctest.ErrorCheck(t, costandusagereportservice.EndpointsID),
PreCheck: func() { acctest.PreCheck(t); acctest.PreCheckRegion(t, endpoints.UsEast1RegionID) },
ErrorCheck: acctest.ErrorCheck(t, cur.EndpointsID),
ProtoV5ProviderFactories: acctest.ProtoV5ProviderFactories,
CheckDestroy: testAccCheckReportDefinitionDestroy(ctx),
Steps: []resource.TestStep{
{
Config: testAccReportDefinitionDataSourceConfig_additional(reportName, bucketName),
Check: resource.ComposeTestCheckFunc(
testAccReportDefinitionCheckExistsDataSource(datasourceName, resourceName),
resource.TestCheckResourceAttrPair(datasourceName, "report_name", resourceName, "report_name"),
resource.TestCheckResourceAttrPair(datasourceName, "time_unit", resourceName, "time_unit"),
resource.TestCheckResourceAttrPair(datasourceName, "compression", resourceName, "compression"),
Expand All @@ -77,24 +75,8 @@ func TestAccCURReportDefinitionDataSource_additional(t *testing.T) {
})
}

func testAccReportDefinitionCheckExistsDataSource(datasourceName, resourceName string) resource.TestCheckFunc {
return func(s *terraform.State) error {
_, ok := s.RootModule().Resources[datasourceName]
if !ok {
return fmt.Errorf("root module has no data source called %s", datasourceName)
}
_, ok = s.RootModule().Resources[resourceName]
if !ok {
return fmt.Errorf("root module has no resource called %s", resourceName)
}
return nil
}
}

func testAccReportDefinitionDataSourceConfig_basic(reportName string, bucketName string) string {
return acctest.ConfigCompose(
testAccRegionProviderConfig(),
fmt.Sprintf(`
return fmt.Sprintf(`
data "aws_billing_service_account" "test" {}

data "aws_partition" "current" {}
Expand Down Expand Up @@ -160,13 +142,11 @@ resource "aws_cur_report_definition" "test" {
data "aws_cur_report_definition" "test" {
report_name = aws_cur_report_definition.test.report_name
}
`, reportName, bucketName))
`, reportName, bucketName)
}

func testAccReportDefinitionDataSourceConfig_additional(reportName string, bucketName string) string {
return acctest.ConfigCompose(
testAccRegionProviderConfig(),
fmt.Sprintf(`
return fmt.Sprintf(`
data "aws_billing_service_account" "test" {}

data "aws_partition" "current" {}
Expand Down Expand Up @@ -234,5 +214,5 @@ resource "aws_cur_report_definition" "test" {
data "aws_cur_report_definition" "test" {
report_name = aws_cur_report_definition.test.report_name
}
`, reportName, bucketName))
`, reportName, bucketName)
}
Loading