Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add remaining fields within deidentify_config.record_transformations.field_transformations field of google_data_loss_prevention_deidentify_template resource #6886

Conversation

SarahFrench
Copy link
Contributor

@SarahFrench SarahFrench commented Nov 29, 2022

Part of hashicorp/terraform-provider-google#9836

This PR updates the google_data_loss_prevention_deidentify_template resource by adding multiple fields related to data transformations, underneath this parent field (deidentify_template.record_transformations.field_transformations.primitive_transformation - link to API docs)

These subfields are added:

  • crypto_replace_ffx_fpe_config
  • fixed_size_bucketing_config
  • bucketing_config
  • replace_with_info_type_config
  • time_part_config
  • crypto_hash_config
  • date_shift_config
  • crypto_deterministic_config
  • replace_dictionary_config

It does not include replace_with_info_type_config because this results in an error that the documentation doesn't hint at:

Error 400: replace_with_info_type_config is not valid as a primitive_transformations. It is only compati
ble with info_type_transformations.


If this PR is for Terraform, I acknowledge that I have:

  • Searched through the issue tracker for an open issue that this either resolves or contributes to, commented on it to claim it, and written "fixes {url}" or "part of {url}" in this PR description. If there were no relevant open issues, I opened one and commented that I would like to work on it (not necessary for very small changes).
  • Generated Terraform, and ran make test and make lint to ensure it passes unit and linter tests.
  • Ensured that all new fields I added that can be set by a user appear in at least one example (for generated resources) or third_party test (for handwritten resources or update tests).
  • Ran relevant acceptance tests (If the acceptance tests do not yet pass or you are unable to run them, please let your reviewer know).
  • Read the Release Notes Guide before writing my release note below.

Release Note Template for Downstream PRs (will be copied)

dlp: Added all subfields under `deidentify_template.record_transformations.field_transformations.primitive_transformation` to `google_data_loss_prevention_deidentify_template`

@modular-magician
Copy link
Collaborator

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR generated some diffs in downstreams - here they are.

Terraform GA: Diff ( 3 files changed, 9052 insertions(+), 940 deletions(-))
Terraform Beta: Diff ( 3 files changed, 9052 insertions(+), 940 deletions(-))
TF Validator: Diff ( 3 files changed, 2708 insertions(+), 3 deletions(-))

@modular-magician
Copy link
Collaborator

Tests analytics

Total tests: 2306
Passed tests 2052
Skipped tests: 244
Failed tests: 10

Action taken

Triggering VCR tests in RECORDING mode for the tests that failed during VCR. Click here to see the failed tests
TestAccLoggingBucketConfigProject_cmekSettings|TestAccDataLossPreventionDeidentifyTemplate_dlpDeidentifyTemplate_recordTransformations_redactConfig_Update|TestAccDataLossPreventionDeidentifyTemplate_dlpDeidentifyTemplate_recordTransformations_replaceConfig_Update|TestAccDataLossPreventionDeidentifyTemplate_dlpDeidentifyTemplate_infoTypeTransformationsUpdate|TestAccDataLossPreventionDeidentifyTemplate_dlpDeidentifyTemplate_recordTransformations_cryptoReplaceFfxFpeConfig_Update|TestAccDataLossPreventionDeidentifyTemplate_dlpDeidentifyTemplate_recordTransformations_characterMaskConfig_Update|TestAccCloudFunctions2Function_fullUpdate|TestAccCloudfunctions2function_cloudfunctions2BasicAuditlogsExample|TestAccCloudfunctions2function_cloudfunctions2BasicGcsExample|TestAccDataSourceGoogleServiceAccountIdToken_basic

@modular-magician
Copy link
Collaborator

Tests passed during RECORDING mode:
TestAccLoggingBucketConfigProject_cmekSettings[Debug log]
TestAccDataLossPreventionDeidentifyTemplate_dlpDeidentifyTemplate_recordTransformations_redactConfig_Update[Debug log]
TestAccDataLossPreventionDeidentifyTemplate_dlpDeidentifyTemplate_recordTransformations_replaceConfig_Update[Debug log]
TestAccDataLossPreventionDeidentifyTemplate_dlpDeidentifyTemplate_infoTypeTransformationsUpdate[Debug log]
TestAccDataLossPreventionDeidentifyTemplate_dlpDeidentifyTemplate_recordTransformations_cryptoReplaceFfxFpeConfig_Update[Debug log]
TestAccDataLossPreventionDeidentifyTemplate_dlpDeidentifyTemplate_recordTransformations_characterMaskConfig_Update[Debug log]
TestAccDataSourceGoogleServiceAccountIdToken_basic[Debug log]

Tests failed during RECORDING mode:
TestAccCloudFunctions2Function_fullUpdate[Error message] [Debug log]
TestAccCloudfunctions2function_cloudfunctions2BasicAuditlogsExample[Error message] [Debug log]
TestAccCloudfunctions2function_cloudfunctions2BasicGcsExample[Error message] [Debug log]

Please fix these to complete your PR
View the build log or the debug log for each test

@SarahFrench SarahFrench force-pushed the add-transformations-dlp-deidentify-templates-4 branch from 6b63666 to b1fbc79 Compare November 30, 2022 16:26
@modular-magician
Copy link
Collaborator

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR generated some diffs in downstreams - here they are.

Terraform GA: Diff ( 3 files changed, 8741 insertions(+), 940 deletions(-))
Terraform Beta: Diff ( 3 files changed, 8741 insertions(+), 940 deletions(-))
TF Validator: Diff ( 3 files changed, 2708 insertions(+), 3 deletions(-))

@modular-magician
Copy link
Collaborator

Tests analytics

Total tests: 2303
Passed tests 2054
Skipped tests: 244
Failed tests: 5

Action taken

Triggering VCR tests in RECORDING mode for the tests that failed during VCR. Click here to see the failed tests
TestAccLoggingBucketConfigProject_cmekSettings|TestAccDataLossPreventionDeidentifyTemplate_dlpDeidentifyTemplate_recordTransformationsUpdate|TestAccCloudfunctions2function_cloudfunctions2BasicAuditlogsExample|TestAccCloudfunctions2function_cloudfunctions2BasicGcsExample|TestAccCloudFunctions2Function_fullUpdate

@modular-magician
Copy link
Collaborator

Tests passed during RECORDING mode:
TestAccLoggingBucketConfigProject_cmekSettings[Debug log]
TestAccDataLossPreventionDeidentifyTemplate_dlpDeidentifyTemplate_recordTransformationsUpdate[Debug log]

Tests failed during RECORDING mode:
TestAccCloudfunctions2function_cloudfunctions2BasicAuditlogsExample[Error message] [Debug log]
TestAccCloudfunctions2function_cloudfunctions2BasicGcsExample[Error message] [Debug log]
TestAccCloudFunctions2Function_fullUpdate[Error message] [Debug log]

Please fix these to complete your PR
View the build log or the debug log for each test

@SarahFrench SarahFrench force-pushed the add-transformations-dlp-deidentify-templates-4 branch from b1fbc79 to 90bba68 Compare December 15, 2022 15:54
@modular-magician
Copy link
Collaborator

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR generated some diffs in downstreams - here they are.

Terraform GA: Diff ( 3 files changed, 8793 insertions(+), 972 deletions(-))
Terraform Beta: Diff ( 3 files changed, 8793 insertions(+), 972 deletions(-))
TF Validator: Diff ( 3 files changed, 2708 insertions(+), 3 deletions(-))

mmv1/products/dlp/api.yaml Outdated Show resolved Hide resolved
mmv1/products/dlp/api.yaml Outdated Show resolved Hide resolved
@modular-magician
Copy link
Collaborator

Tests analytics

Total tests: 2349
Passed tests 2094
Skipped tests: 246
Failed tests: 9

Action taken

Triggering VCR tests in RECORDING mode for the tests that failed during VCR. Click here to see the failed tests
TestAccFirebaserulesRelease_BasicRelease|TestAccDataLossPreventionDeidentifyTemplate_dlpDeidentifyTemplate_recordTransformationsUpdate|TestAccContainerCluster_withInvalidGatewayApiConfigChannel|TestAccComputeRouterNat_withNatRules|TestAccComputeRouterNat_withPortAllocationMethods|TestAccComputeRouterNat_withNatIpsAndDrainNatIps|TestAccComputeRouterNat_removeLogConfig|TestAccComputeRouterNat_update|TestAccLoggingBucketConfigProject_cmekSettings

@modular-magician
Copy link
Collaborator

Tests passed during RECORDING mode:
TestAccFirebaserulesRelease_BasicRelease[Debug log]
TestAccDataLossPreventionDeidentifyTemplate_dlpDeidentifyTemplate_recordTransformationsUpdate[Debug log]
TestAccContainerCluster_withInvalidGatewayApiConfigChannel[Debug log]
TestAccComputeRouterNat_withNatRules[Debug log]
TestAccComputeRouterNat_withPortAllocationMethods[Debug log]
TestAccComputeRouterNat_withNatIpsAndDrainNatIps[Debug log]
TestAccComputeRouterNat_removeLogConfig[Debug log]
TestAccComputeRouterNat_update[Debug log]
TestAccLoggingBucketConfigProject_cmekSettings[Debug log]

All tests passed
View the build log or the debug log for each test

Comment on lines +1718 to +1730
This can be used on data of type: double, long.

If the bound Value type differs from the type of data being transformed, we will first attempt converting the type of the data to be transformed to match the type of the bound before comparing.

See https://cloud.google.com/dlp/docs/concepts-bucketing to learn more.
properties:
- !ruby/object:Api::Type::NestedObject
name: lowerBound
description: |
Lower bound value of buckets.
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This type of transformation is specific to numeric data:

This can be used on data of type: double, long.

But, the description shows there's some kind of type conversion that occurs:

If the bound Value type differs from the type of data being transformed, we will first attempt converting the type of the data to be transformed to match the type of the bound before comparing.

Because of this complexity I think we should leave all the different data types present here (see Value entity in API docs)

@SarahFrench SarahFrench force-pushed the add-transformations-dlp-deidentify-templates-4 branch 2 times, most recently from 13696f0 to 5116d36 Compare December 19, 2022 17:31
@modular-magician
Copy link
Collaborator

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR generated some diffs in downstreams - here they are.

Terraform GA: Diff ( 3 files changed, 7921 insertions(+), 2184 deletions(-))
Terraform Beta: Diff ( 3 files changed, 7921 insertions(+), 2184 deletions(-))
TF Validator: Diff ( 3 files changed, 1925 insertions(+), 3 deletions(-))

@modular-magician
Copy link
Collaborator

Tests analytics

Total tests: 2355
Passed tests 2105
Skipped tests: 247
Failed tests: 3

Action taken

Triggering VCR tests in RECORDING mode for the tests that failed during VCR. Click here to see the failed tests
TestAccLoggingBucketConfigProject_cmekSettings|TestAccDataLossPreventionDeidentifyTemplate_dlpDeidentifyTemplate_recordTransformationsUpdate|TestAccContainerCluster_withInvalidGatewayApiConfigChannel

@modular-magician
Copy link
Collaborator

Tests passed during RECORDING mode:
TestAccLoggingBucketConfigProject_cmekSettings[Debug log]
TestAccDataLossPreventionDeidentifyTemplate_dlpDeidentifyTemplate_recordTransformationsUpdate[Debug log]
TestAccContainerCluster_withInvalidGatewayApiConfigChannel[Debug log]

All tests passed
View the build log or the debug log for each test

@SarahFrench SarahFrench marked this pull request as ready for review December 19, 2022 19:11
@SarahFrench SarahFrench requested a review from melinath December 19, 2022 19:11
@melinath
Copy link
Member

just checking in to let you know I haven't forgotten this. Hoping to get to it this week.

Copy link
Member

@melinath melinath left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM overall, a few questions re: docs & bootstrapping cryptokeys

mmv1/products/dlp/api.yaml Outdated Show resolved Hide resolved
mmv1/products/dlp/api.yaml Show resolved Hide resolved
mmv1/products/dlp/api.yaml Show resolved Hide resolved
mmv1/products/dlp/api.yaml Outdated Show resolved Hide resolved
mmv1/products/dlp/api.yaml Outdated Show resolved Hide resolved
@modular-magician
Copy link
Collaborator

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR generated some diffs in downstreams - here they are.

Terraform GA: Diff ( 3 files changed, 8351 insertions(+), 2652 deletions(-))
Terraform Beta: Diff ( 3 files changed, 8351 insertions(+), 2652 deletions(-))
TF Validator: Diff ( 3 files changed, 1925 insertions(+), 3 deletions(-))

@modular-magician
Copy link
Collaborator

Tests analytics

Total tests: 2408
Passed tests 2152
Skipped tests: 251
Failed tests: 5

Action taken

Triggering VCR tests in RECORDING mode for the tests that failed during VCR. Click here to see the failed tests
TestAccFirebaserulesRelease_BasicRelease|TestAccLoggingBucketConfigProject_cmekSettings|TestAccDataLossPreventionDeidentifyTemplate_dlpDeidentifyTemplate_recordTransformationsUpdate|TestAccDataLossPreventionDeidentifyTemplate_dlpDeidentifyTemplate_infoTypeTransformationsUpdate|TestAccRegionInstanceGroupManager_stateful

@modular-magician
Copy link
Collaborator

Tests passed during RECORDING mode:
TestAccFirebaserulesRelease_BasicRelease[Debug log]
TestAccLoggingBucketConfigProject_cmekSettings[Debug log]
TestAccDataLossPreventionDeidentifyTemplate_dlpDeidentifyTemplate_recordTransformationsUpdate[Debug log]
TestAccDataLossPreventionDeidentifyTemplate_dlpDeidentifyTemplate_infoTypeTransformationsUpdate[Debug log]
TestAccRegionInstanceGroupManager_stateful[Debug log]

All tests passed
View the build log or the debug log for each test

@modular-magician
Copy link
Collaborator

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR generated some diffs in downstreams - here they are.

Terraform GA: Diff ( 3 files changed, 8485 insertions(+), 2658 deletions(-))
Terraform Beta: Diff ( 3 files changed, 8485 insertions(+), 2658 deletions(-))
TF Validator: Diff ( 3 files changed, 1958 insertions(+), 3 deletions(-))

Copy link
Member

@melinath melinath left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@SarahFrench
Copy link
Contributor Author

Thanks! I'll wait for the tests to complete and then merge if it's all ok

@modular-magician
Copy link
Collaborator

Tests analytics

Total tests: 2408
Passed tests 2155
Skipped tests: 251
Failed tests: 2

Action taken

Triggering VCR tests in RECORDING mode for the tests that failed during VCR. Click here to see the failed tests
TestAccDataLossPreventionDeidentifyTemplate_dlpDeidentifyTemplate_recordTransformationsUpdate|TestAccDataLossPreventionDeidentifyTemplate_dlpDeidentifyTemplate_infoTypeTransformationsUpdate

@modular-magician
Copy link
Collaborator

Tests passed during RECORDING mode:
TestAccDataLossPreventionDeidentifyTemplate_dlpDeidentifyTemplate_recordTransformationsUpdate[Debug log]
TestAccDataLossPreventionDeidentifyTemplate_dlpDeidentifyTemplate_infoTypeTransformationsUpdate[Debug log]

All tests passed
View the build log or the debug log for each test

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants