From 37d6db76835ef0adb5d1851f19133cb59d0a99f2 Mon Sep 17 00:00:00 2001 From: Mend Renovate Date: Fri, 27 Jan 2023 19:14:15 +0000 Subject: [PATCH 01/12] chore(deps): update dependency google-cloud-documentai to v2.10.0 (#8966) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit [![Mend Renovate](https://app.renovatebot.com/images/banner.svg)](https://renovatebot.com) This PR contains the following updates: | Package | Change | Age | Adoption | Passing | Confidence | |---|---|---|---|---|---| | [google-cloud-documentai](https://togithub.com/googleapis/python-documentai) | `==2.6.0` -> `==2.10.0` | [![age](https://badges.renovateapi.com/packages/pypi/google-cloud-documentai/2.10.0/age-slim)](https://docs.renovatebot.com/merge-confidence/) | [![adoption](https://badges.renovateapi.com/packages/pypi/google-cloud-documentai/2.10.0/adoption-slim)](https://docs.renovatebot.com/merge-confidence/) | [![passing](https://badges.renovateapi.com/packages/pypi/google-cloud-documentai/2.10.0/compatibility-slim/2.6.0)](https://docs.renovatebot.com/merge-confidence/) | [![confidence](https://badges.renovateapi.com/packages/pypi/google-cloud-documentai/2.10.0/confidence-slim/2.6.0)](https://docs.renovatebot.com/merge-confidence/) | --- ### ⚠ Dependency Lookup Warnings ⚠ Warnings were logged while processing this repo. Please check the Dependency Dashboard for more information. --- ### Release Notes
googleapis/python-documentai ### [`v2.10.0`](https://togithub.com/googleapis/python-documentai/blob/HEAD/CHANGELOG.md#​2100-httpsgithubcomgoogleapispython-documentaicomparev291v2100-2023-01-25) [Compare Source](https://togithub.com/googleapis/python-documentai/compare/v2.9.1...v2.10.0) ##### Features - Added advanced_ocr_options field in OcrConfig ([#​451](https://togithub.com/googleapis/python-documentai/issues/451)) ([5e1bb96](https://togithub.com/googleapis/python-documentai/commit/5e1bb96b685981cba2e3e4fc319efb0eac757fbf)) ### [`v2.9.1`](https://togithub.com/googleapis/python-documentai/blob/HEAD/CHANGELOG.md#​291-httpsgithubcomgoogleapispython-documentaicomparev290v291-2023-01-20) [Compare Source](https://togithub.com/googleapis/python-documentai/compare/v2.9.0...v2.9.1) ##### Bug Fixes - Add context manager return types ([61256d1](https://togithub.com/googleapis/python-documentai/commit/61256d1728e17dc8b04cd4166e286a4f76c6018a)) ##### Documentation - Add documentation for enums ([61256d1](https://togithub.com/googleapis/python-documentai/commit/61256d1728e17dc8b04cd4166e286a4f76c6018a)) ### [`v2.9.0`](https://togithub.com/googleapis/python-documentai/blob/HEAD/CHANGELOG.md#​290-httpsgithubcomgoogleapispython-documentaicomparev280v290-2023-01-17) [Compare Source](https://togithub.com/googleapis/python-documentai/compare/v2.8.0...v2.9.0) ##### Features - **v1:** Exposed GetProcessorType ([#​446](https://togithub.com/googleapis/python-documentai/issues/446)) ([6c38227](https://togithub.com/googleapis/python-documentai/commit/6c38227279d889f760375dfe56f805da78e7bc68)) ### [`v2.8.0`](https://togithub.com/googleapis/python-documentai/blob/HEAD/CHANGELOG.md#​280-httpsgithubcomgoogleapispython-documentaicomparev270v280-2023-01-17) [Compare Source](https://togithub.com/googleapis/python-documentai/compare/v2.7.0...v2.8.0) ##### Features - Exposed GetProcessorType to v1beta3 ([#​444](https://togithub.com/googleapis/python-documentai/issues/444)) ([e5835f4](https://togithub.com/googleapis/python-documentai/commit/e5835f4b514b2b463478df33505ac08272a99fb7)) ##### Documentation - **samples:** Removed Samples after Migration to Mono Repo ([#​438](https://togithub.com/googleapis/python-documentai/issues/438)) ([8c7f52b](https://togithub.com/googleapis/python-documentai/commit/8c7f52bce317b6b54bcfc71cb193dc7e372b2812)) ### [`v2.7.0`](https://togithub.com/googleapis/python-documentai/blob/HEAD/CHANGELOG.md#​270-httpsgithubcomgoogleapispython-documentaicomparev260v270-2023-01-10) [Compare Source](https://togithub.com/googleapis/python-documentai/compare/v2.6.0...v2.7.0) ##### Features - Add support for python 3.11 ([#​440](https://togithub.com/googleapis/python-documentai/issues/440)) ([e3da68f](https://togithub.com/googleapis/python-documentai/commit/e3da68f1fb3e377eb0ccfbeacc588500ae0918ec))
--- ### Configuration 📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Never, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] If you want to rebase/retry this PR, check this box --- This PR has been generated by [Mend Renovate](https://www.mend.io/free-developer-tools/renovate/). View repository job log [here](https://app.renovatebot.com/dashboard#github/GoogleCloudPlatform/python-docs-samples). --- documentai/snippets/requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/documentai/snippets/requirements.txt b/documentai/snippets/requirements.txt index 21e4fc4a8288..2a939b382999 100644 --- a/documentai/snippets/requirements.txt +++ b/documentai/snippets/requirements.txt @@ -1,2 +1,2 @@ -google-cloud-documentai==2.6.0 +google-cloud-documentai==2.10.0 google-cloud-storage==2.7.0 From ae0ac8d4c8924b618d34733b3d68d166ccc783fe Mon Sep 17 00:00:00 2001 From: Charles Engelke Date: Fri, 27 Jan 2023 13:38:00 -0800 Subject: [PATCH 02/12] fix: remove unused assignment (#8979) Per issue #8977 this assignment is not needed as it is overwritten three statements later. Co-authored-by: Leah E. Cole <6719667+leahecole@users.noreply.github.com> Co-authored-by: gcf-merge-on-green[bot] <60162190+gcf-merge-on-green[bot]@users.noreply.github.com> Co-authored-by: Karl Weinmeister <11586922+kweinmeister@users.noreply.github.com> --- monitoring/snippets/v3/cloud-client/snippets.py | 1 - 1 file changed, 1 deletion(-) diff --git a/monitoring/snippets/v3/cloud-client/snippets.py b/monitoring/snippets/v3/cloud-client/snippets.py index cbb7d88bc48f..9fbfff56ac73 100644 --- a/monitoring/snippets/v3/cloud-client/snippets.py +++ b/monitoring/snippets/v3/cloud-client/snippets.py @@ -89,7 +89,6 @@ def list_time_series(project_id): client = monitoring_v3.MetricServiceClient() project_name = f"projects/{project_id}" - interval = monitoring_v3.TimeInterval() now = time.time() seconds = int(now) From a4b11b9438f049953fd4590d79471a0dc0d44d6a Mon Sep 17 00:00:00 2001 From: Mend Renovate Date: Sat, 28 Jan 2023 01:10:11 +0000 Subject: [PATCH 03/12] chore(deps): update dependency google-cloud-retail to v1.14.1 (#8881) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit [![Mend Renovate](https://app.renovatebot.com/images/banner.svg)](https://renovatebot.com) This PR contains the following updates: | Package | Change | Age | Adoption | Passing | Confidence | |---|---|---|---|---|---| | [google-cloud-retail](https://togithub.com/googleapis/python-retail) | `==1.10.2` -> `==1.14.1` | [![age](https://badges.renovateapi.com/packages/pypi/google-cloud-retail/1.14.1/age-slim)](https://docs.renovatebot.com/merge-confidence/) | [![adoption](https://badges.renovateapi.com/packages/pypi/google-cloud-retail/1.14.1/adoption-slim)](https://docs.renovatebot.com/merge-confidence/) | [![passing](https://badges.renovateapi.com/packages/pypi/google-cloud-retail/1.14.1/compatibility-slim/1.10.2)](https://docs.renovatebot.com/merge-confidence/) | [![confidence](https://badges.renovateapi.com/packages/pypi/google-cloud-retail/1.14.1/confidence-slim/1.10.2)](https://docs.renovatebot.com/merge-confidence/) | --- ### ⚠ Dependency Lookup Warnings ⚠ Warnings were logged while processing this repo. Please check the Dependency Dashboard for more information. --- ### Release Notes
googleapis/python-retail ### [`v1.14.1`](https://togithub.com/googleapis/python-retail/blob/HEAD/CHANGELOG.md#​1141-httpsgithubcomgoogleapispython-retailcomparev1140v1141-2023-01-20) [Compare Source](https://togithub.com/googleapis/python-retail/compare/v1.14.0...v1.14.1) ##### Bug Fixes - Add context manager return types ([f27cab3](https://togithub.com/googleapis/python-retail/commit/f27cab32eebe0d694d98bfa2d2ec75402ddbbf0a)) ##### Documentation - Add documentation for enums ([f27cab3](https://togithub.com/googleapis/python-retail/commit/f27cab32eebe0d694d98bfa2d2ec75402ddbbf0a)) ### [`v1.14.0`](https://togithub.com/googleapis/python-retail/blob/HEAD/CHANGELOG.md#​1140-httpsgithubcomgoogleapispython-retailcomparev1130v1140-2023-01-10) [Compare Source](https://togithub.com/googleapis/python-retail/compare/v1.13.0...v1.14.0) ##### Features - Add support for python 3.11 ([#​379](https://togithub.com/googleapis/python-retail/issues/379)) ([945fd66](https://togithub.com/googleapis/python-retail/commit/945fd66c014c8dedcdd6677f46fc7c54bf746130)) ### [`v1.13.0`](https://togithub.com/googleapis/python-retail/blob/HEAD/CHANGELOG.md#​1130-httpsgithubcomgoogleapispython-retailcomparev1120v1130-2023-01-05) [Compare Source](https://togithub.com/googleapis/python-retail/compare/v1.12.0...v1.13.0) ##### Features - **v2beta:** Allow set feed id in merchant center link ([78bf804](https://togithub.com/googleapis/python-retail/commit/78bf804e4d3aa622605730a5b8eb5613b7f694f8)) - **v2beta:** Deprecate retrievable_fields in product attribute ([78bf804](https://togithub.com/googleapis/python-retail/commit/78bf804e4d3aa622605730a5b8eb5613b7f694f8)) - **v2beta:** Support async write mode for WriteUserEvent API ([78bf804](https://togithub.com/googleapis/python-retail/commit/78bf804e4d3aa622605730a5b8eb5613b7f694f8)) - **v2beta:** Support collect and import GA4 event format with prebuilt whistle rule ([78bf804](https://togithub.com/googleapis/python-retail/commit/78bf804e4d3aa622605730a5b8eb5613b7f694f8)) - **v2beta:** Support data output to GCS ([78bf804](https://togithub.com/googleapis/python-retail/commit/78bf804e4d3aa622605730a5b8eb5613b7f694f8)) - **v2beta:** Support diversity type in serving config ([78bf804](https://togithub.com/googleapis/python-retail/commit/78bf804e4d3aa622605730a5b8eb5613b7f694f8)) - **v2beta:** Support exact searchable and retrievable in catalog attribute config ([78bf804](https://togithub.com/googleapis/python-retail/commit/78bf804e4d3aa622605730a5b8eb5613b7f694f8)) ##### Documentation - **v2beta:** Keep the API doc up-to-date with recent changes ([78bf804](https://togithub.com/googleapis/python-retail/commit/78bf804e4d3aa622605730a5b8eb5613b7f694f8)) ### [`v1.12.0`](https://togithub.com/googleapis/python-retail/blob/HEAD/CHANGELOG.md#​1120-httpsgithubcomgoogleapispython-retailcomparev1110v1120-2022-12-15) [Compare Source](https://togithub.com/googleapis/python-retail/compare/v1.11.0...v1.12.0) ##### Features - Deprecate retrievable_fields in product attribute ([1ba8e50](https://togithub.com/googleapis/python-retail/commit/1ba8e502166ceffb23a281f554bf04cd823c6868)) - Support async write mode for WriteUserEvent API ([1ba8e50](https://togithub.com/googleapis/python-retail/commit/1ba8e502166ceffb23a281f554bf04cd823c6868)) - Support collect GA4 event format with prebuilt whistle rule ([1ba8e50](https://togithub.com/googleapis/python-retail/commit/1ba8e502166ceffb23a281f554bf04cd823c6868)) - Support diversity type in serving config ([1ba8e50](https://togithub.com/googleapis/python-retail/commit/1ba8e502166ceffb23a281f554bf04cd823c6868)) - Support exact searchable and retrievable in catalog attribute config ([1ba8e50](https://togithub.com/googleapis/python-retail/commit/1ba8e502166ceffb23a281f554bf04cd823c6868)) ##### Documentation - Keep the API doc up-to-date with recent changes ([1ba8e50](https://togithub.com/googleapis/python-retail/commit/1ba8e502166ceffb23a281f554bf04cd823c6868)) ### [`v1.11.0`](https://togithub.com/googleapis/python-retail/blob/HEAD/CHANGELOG.md#​1110-httpsgithubcomgoogleapispython-retailcomparev1102v1110-2022-12-15) [Compare Source](https://togithub.com/googleapis/python-retail/compare/v1.10.2...v1.11.0) ##### Features - Add support for `google.cloud.retail.__version__` ([b113357](https://togithub.com/googleapis/python-retail/commit/b113357206652f4c6318c05c911055c1ec0ff7cb)) - Add typing to proto.Message based class attributes ([b113357](https://togithub.com/googleapis/python-retail/commit/b113357206652f4c6318c05c911055c1ec0ff7cb)) - **v2alpha:** Allow set feed id in merchant center link ([3ee1ee4](https://togithub.com/googleapis/python-retail/commit/3ee1ee45313a3642f6289117bf64749c422d1ad7)) - **v2alpha:** Deprecate retrievable_fields in product attribute ([3ee1ee4](https://togithub.com/googleapis/python-retail/commit/3ee1ee45313a3642f6289117bf64749c422d1ad7)) - **v2alpha:** Expose the local inventory data in product data retrieval ([3ee1ee4](https://togithub.com/googleapis/python-retail/commit/3ee1ee45313a3642f6289117bf64749c422d1ad7)) - **v2alpha:** Return personal product labels in search response ([3ee1ee4](https://togithub.com/googleapis/python-retail/commit/3ee1ee45313a3642f6289117bf64749c422d1ad7)) - **v2alpha:** Support async write mode for WriteUserEvent API ([3ee1ee4](https://togithub.com/googleapis/python-retail/commit/3ee1ee45313a3642f6289117bf64749c422d1ad7)) - **v2alpha:** Support attribute suggestion in autocomplete ([3ee1ee4](https://togithub.com/googleapis/python-retail/commit/3ee1ee45313a3642f6289117bf64749c422d1ad7)) - **v2alpha:** Support batch remove catalog attribute config ([3ee1ee4](https://togithub.com/googleapis/python-retail/commit/3ee1ee45313a3642f6289117bf64749c422d1ad7)) - **v2alpha:** Support collect and import GA4 event format with prebuilt whistle rule ([3ee1ee4](https://togithub.com/googleapis/python-retail/commit/3ee1ee45313a3642f6289117bf64749c422d1ad7)) - **v2alpha:** Support data output to GCS ([3ee1ee4](https://togithub.com/googleapis/python-retail/commit/3ee1ee45313a3642f6289117bf64749c422d1ad7)) - **v2alpha:** Support diversity type in serving config ([3ee1ee4](https://togithub.com/googleapis/python-retail/commit/3ee1ee45313a3642f6289117bf64749c422d1ad7)) - **v2alpha:** Support exact searchable and retrievable in catalog attribute config ([3ee1ee4](https://togithub.com/googleapis/python-retail/commit/3ee1ee45313a3642f6289117bf64749c422d1ad7)) ##### Bug Fixes - Add dict typing for client_options ([b113357](https://togithub.com/googleapis/python-retail/commit/b113357206652f4c6318c05c911055c1ec0ff7cb)) - **deps:** Require google-api-core >=1.34.0, >=2.11.0 ([6780c7a](https://togithub.com/googleapis/python-retail/commit/6780c7adc14fb7063597d6c9a514d0b33fc5dcd6)) - Drop usage of pkg_resources ([6780c7a](https://togithub.com/googleapis/python-retail/commit/6780c7adc14fb7063597d6c9a514d0b33fc5dcd6)) - Fix timeout default values ([6780c7a](https://togithub.com/googleapis/python-retail/commit/6780c7adc14fb7063597d6c9a514d0b33fc5dcd6)) ##### Documentation - **samples:** Snippetgen handling of repeated enum field ([b113357](https://togithub.com/googleapis/python-retail/commit/b113357206652f4c6318c05c911055c1ec0ff7cb)) - **samples:** Snippetgen should call await on the operation coroutine before calling result ([6780c7a](https://togithub.com/googleapis/python-retail/commit/6780c7adc14fb7063597d6c9a514d0b33fc5dcd6)) - **v2alpha:** Keep the API doc up-to-date with recent changes ([3ee1ee4](https://togithub.com/googleapis/python-retail/commit/3ee1ee45313a3642f6289117bf64749c422d1ad7))
--- ### Configuration 📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Never, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] If you want to rebase/retry this PR, check this box --- This PR has been generated by [Mend Renovate](https://www.mend.io/free-developer-tools/renovate/). View repository job log [here](https://app.renovatebot.com/dashboard#github/GoogleCloudPlatform/python-docs-samples). --- retail/interactive-tutorials/events/requirements.txt | 2 +- retail/interactive-tutorials/product/requirements.txt | 2 +- retail/interactive-tutorials/search/requirements.txt | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/retail/interactive-tutorials/events/requirements.txt b/retail/interactive-tutorials/events/requirements.txt index 547d882a7646..d24c907405d1 100644 --- a/retail/interactive-tutorials/events/requirements.txt +++ b/retail/interactive-tutorials/events/requirements.txt @@ -1,4 +1,4 @@ google==3.0.0 -google-cloud-retail==1.10.2 +google-cloud-retail==1.14.1 google-cloud-storage==2.6.0 google-cloud-bigquery==3.4.0 diff --git a/retail/interactive-tutorials/product/requirements.txt b/retail/interactive-tutorials/product/requirements.txt index 8162b3242f08..f57ba041500d 100644 --- a/retail/interactive-tutorials/product/requirements.txt +++ b/retail/interactive-tutorials/product/requirements.txt @@ -1,4 +1,4 @@ google==3.0.0 -google-cloud-retail==1.10.2 +google-cloud-retail==1.14.1 google-cloud-storage==2.6.0 google-cloud-bigquery==3.4.0 \ No newline at end of file diff --git a/retail/interactive-tutorials/search/requirements.txt b/retail/interactive-tutorials/search/requirements.txt index 8162b3242f08..f57ba041500d 100644 --- a/retail/interactive-tutorials/search/requirements.txt +++ b/retail/interactive-tutorials/search/requirements.txt @@ -1,4 +1,4 @@ google==3.0.0 -google-cloud-retail==1.10.2 +google-cloud-retail==1.14.1 google-cloud-storage==2.6.0 google-cloud-bigquery==3.4.0 \ No newline at end of file From 54b9cd6740b4dbc64db4d43a16de13c702b2364b Mon Sep 17 00:00:00 2001 From: Mend Renovate Date: Sat, 28 Jan 2023 18:44:11 +0000 Subject: [PATCH 04/12] chore(deps): update dependency grpc-google-iam-v1 to v0.12.6 (#8998) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit [![Mend Renovate](https://app.renovatebot.com/images/banner.svg)](https://renovatebot.com) This PR contains the following updates: | Package | Change | Age | Adoption | Passing | Confidence | |---|---|---|---|---|---| | [grpc-google-iam-v1](https://togithub.com/googleapis/python-grpc-google-iam-v1) | `==0.12.4` -> `==0.12.6` | [![age](https://badges.renovateapi.com/packages/pypi/grpc-google-iam-v1/0.12.6/age-slim)](https://docs.renovatebot.com/merge-confidence/) | [![adoption](https://badges.renovateapi.com/packages/pypi/grpc-google-iam-v1/0.12.6/adoption-slim)](https://docs.renovatebot.com/merge-confidence/) | [![passing](https://badges.renovateapi.com/packages/pypi/grpc-google-iam-v1/0.12.6/compatibility-slim/0.12.4)](https://docs.renovatebot.com/merge-confidence/) | [![confidence](https://badges.renovateapi.com/packages/pypi/grpc-google-iam-v1/0.12.6/confidence-slim/0.12.4)](https://docs.renovatebot.com/merge-confidence/) | --- ### ⚠ Dependency Lookup Warnings ⚠ Warnings were logged while processing this repo. Please check the Dependency Dashboard for more information. --- ### Release Notes
googleapis/python-grpc-google-iam-v1 ### [`v0.12.6`](https://togithub.com/googleapis/python-grpc-google-iam-v1/blob/HEAD/CHANGELOG.md#​0126-httpsgithubcomgoogleapispython-grpc-google-iam-v1comparev0125v0126-2023-01-06) [Compare Source](https://togithub.com/googleapis/python-grpc-google-iam-v1/compare/v0.12.4...v0.12.6) ##### Bug Fixes - **deps:** Allow protobuf>=3.19.5 ([1044946](https://togithub.com/googleapis/python-grpc-google-iam-v1/commit/10449467658fc6de2ea91a5ba0eccfc26871013b))
--- ### Configuration 📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Never, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] If you want to rebase/retry this PR, check this box --- This PR has been generated by [Mend Renovate](https://www.mend.io/free-developer-tools/renovate/). View repository job log [here](https://app.renovatebot.com/dashboard#github/GoogleCloudPlatform/python-docs-samples). --- iot/api-client/mqtt_example/requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/iot/api-client/mqtt_example/requirements.txt b/iot/api-client/mqtt_example/requirements.txt index 04f88f1c8bab..5643450b80dc 100644 --- a/iot/api-client/mqtt_example/requirements.txt +++ b/iot/api-client/mqtt_example/requirements.txt @@ -4,6 +4,6 @@ google-auth-httplib2==0.1.0 google-auth==2.6.2 google-cloud-pubsub==2.9.0 google-cloud-iot==2.5.1 -grpc-google-iam-v1==0.12.4 +grpc-google-iam-v1==0.12.6 pyjwt==2.6.0 paho-mqtt==1.5.1 From 20f66139587a606960f4371696833a50c459ec13 Mon Sep 17 00:00:00 2001 From: Charles Engelke Date: Sun, 29 Jan 2023 21:56:19 -0800 Subject: [PATCH 05/12] fix: wait longer for incoming mail (#9052) Co-authored-by: Karl Weinmeister <11586922+kweinmeister@users.noreply.github.com> --- .../bundled-services/mail/wsgi/main_test.py | 31 +++++++++++-------- 1 file changed, 18 insertions(+), 13 deletions(-) diff --git a/appengine/standard_python3/bundled-services/mail/wsgi/main_test.py b/appengine/standard_python3/bundled-services/mail/wsgi/main_test.py index b4e10f7de8ee..36cc59123078 100644 --- a/appengine/standard_python3/bundled-services/mail/wsgi/main_test.py +++ b/appengine/standard_python3/bundled-services/mail/wsgi/main_test.py @@ -101,19 +101,24 @@ def test_send_receive(version): assert "Successfully sent mail" in response.text assert response.status_code == 201 - # Give the mail some time to be delivered and logs to post - time.sleep(60) - - # Fetch logs to check messages on received mail - entries = gcloud_cli( - f'logging read "resource.type=gae_app AND resource.labels.version_id={version_id}"' - ) - - text_payloads = "" - for entry in entries: - if "textPayload" in entry: - text_payloads += entry["textPayload"] - text_payloads += "\n" + # External mail delivery and receipt can take varying lengths of time + for check in range(3): + # Give the mail some time to be delivered and logs to post + time.sleep(60) + + # Fetch logs to check messages on received mail + entries = gcloud_cli( + f'logging read "resource.type=gae_app AND resource.labels.version_id={version_id}"' + ) + + text_payloads = "" + for entry in entries: + if "textPayload" in entry: + text_payloads += entry["textPayload"] + text_payloads += "\n" + + if "Received" in text_payloads: + break expected = f"Received greeting for valid-user@{version_id}-dot-{project_id}.appspotmail.com" assert expected in text_payloads From d8cfe22db206af518df13fd82773cd7a896f566f Mon Sep 17 00:00:00 2001 From: Mend Renovate Date: Mon, 30 Jan 2023 17:35:33 +0000 Subject: [PATCH 06/12] chore(deps): update dependency pyarrow to v11 (#9056) * chore(deps): update dependency pyarrow to v11 * Accommodate Python 3.6 in pandas-gbq * Corrected wrong package name * Update requirements.txt * Update requirements.txt * Update requirements.txt * Update requirements-test.txt --------- Co-authored-by: Karl Weinmeister <11586922+kweinmeister@users.noreply.github.com> --- bigquery/bqml/requirements.txt | 4 ++-- bigquery/datalab-migration/requirements.txt | 2 +- bigquery/pandas-gbq-migration/requirements.txt | 6 ++++-- data-science-onramp/data-processing/requirements-test.txt | 2 +- 4 files changed, 8 insertions(+), 6 deletions(-) diff --git a/bigquery/bqml/requirements.txt b/bigquery/bqml/requirements.txt index 0f97e1e49124..8260b4e3a25e 100644 --- a/bigquery/bqml/requirements.txt +++ b/bigquery/bqml/requirements.txt @@ -2,7 +2,7 @@ google-cloud-bigquery[pandas,bqstorage]==2.34.4 google-cloud-bigquery-storage==2.14.0 pandas==1.1.5; python_version < '3.7' pandas==1.3.5; python_version > '3.6' -# pyarrow must stay under 7 -pyarrow==6.0.1 +# pyarrow must stay under 8 +pyarrow==7.0.0 flaky==3.7.0 mock==4.0.3 diff --git a/bigquery/datalab-migration/requirements.txt b/bigquery/datalab-migration/requirements.txt index 5f4217c02c50..8e255ffcbfbd 100644 --- a/bigquery/datalab-migration/requirements.txt +++ b/bigquery/datalab-migration/requirements.txt @@ -5,4 +5,4 @@ google-cloud-bigquery[pandas,pyarrow]==2.34.4 google-cloud-bigquery-storage==2.13.2 datalab==1.2.1 ipython==8.0.1 -pyarrow==6.0.1 +pyarrow==7.0.0 diff --git a/bigquery/pandas-gbq-migration/requirements.txt b/bigquery/pandas-gbq-migration/requirements.txt index f8fda74879c1..1d91da50ca74 100644 --- a/bigquery/pandas-gbq-migration/requirements.txt +++ b/bigquery/pandas-gbq-migration/requirements.txt @@ -2,8 +2,10 @@ google-cloud-bigquery==2.34.4 google-cloud-bigquery-storage==2.14.0 pandas==1.1.5; python_version < '3.7' pandas==1.3.5; python_version > '3.6' -pandas-gbq==0.16.0; python_version > '3.6' +pandas-gbq==0.17.2; python_version > '3.6' # pandas-gbq==0.14.1 is the latest compatible version for Python 3.6 pandas-gbq==0.14.1; python_version < '3.7' grpcio==1.47.0 -pyarrow==6.0.1 +pyarrow==7.0.0; python_version > '3.6' +# pyarrow==6.0.1 is the latest compatible version for pandas-gbq 0.14.1 +pyarrow==6.0.1; python_version < '3.7' diff --git a/data-science-onramp/data-processing/requirements-test.txt b/data-science-onramp/data-processing/requirements-test.txt index 1a0f5925a5fa..8de4d1b54823 100644 --- a/data-science-onramp/data-processing/requirements-test.txt +++ b/data-science-onramp/data-processing/requirements-test.txt @@ -2,5 +2,5 @@ pytest==7.0.1 pandas==1.1.5; python_version < '3.7' pandas==1.3.5; python_version > '3.6' pyarrow==6.0.1; python_version < '3.7' -pyarrow==8.0.0; python_version > '3.6' +pyarrow==11.0.0; python_version > '3.6' google-cloud-bigquery==2.34.4 From 457b8ca7042c6f95f7b73c21bbbf993a8c360ce0 Mon Sep 17 00:00:00 2001 From: Mend Renovate Date: Mon, 30 Jan 2023 17:36:00 +0000 Subject: [PATCH 07/12] chore(deps): update dependency google-cloud-contact-center-insights to v1.6.1 (#9041) --- contact-center-insights/snippets/requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/contact-center-insights/snippets/requirements.txt b/contact-center-insights/snippets/requirements.txt index c1672ab994cc..2fdf37db5ea5 100644 --- a/contact-center-insights/snippets/requirements.txt +++ b/contact-center-insights/snippets/requirements.txt @@ -1,3 +1,3 @@ google-api-core==2.11.0 google-cloud-bigquery==3.4.1 -google-cloud-contact-center-insights==1.6.0 +google-cloud-contact-center-insights==1.6.1 From 8823d962705313f6886b20d0d6ebdd2653060ae0 Mon Sep 17 00:00:00 2001 From: Mend Renovate Date: Mon, 30 Jan 2023 17:37:10 +0000 Subject: [PATCH 08/12] chore(deps): update dependency google-cloud-asset to v3.17.1 (#8965) --- asset/snippets/requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/asset/snippets/requirements.txt b/asset/snippets/requirements.txt index 48337dda253c..d10b4e0d1555 100644 --- a/asset/snippets/requirements.txt +++ b/asset/snippets/requirements.txt @@ -1,5 +1,5 @@ google-cloud-storage==2.6.0 -google-cloud-asset==3.14.2 +google-cloud-asset==3.17.1 google-cloud-resource-manager==1.6.3 google-cloud-pubsub==2.13.11 google-cloud-bigquery==3.4.0 From 142c96d6605764dab4cb2e49dcdb1b821ff99ce7 Mon Sep 17 00:00:00 2001 From: "Leah E. Cole" <6719667+leahecole@users.noreply.github.com> Date: Mon, 30 Jan 2023 13:15:02 -1000 Subject: [PATCH 09/12] Update vertex datascienceonramp (#8984) * update dependencies for python 3.11 * update datascionramp dependencies --------- Co-authored-by: Karl Weinmeister <11586922+kweinmeister@users.noreply.github.com> --- data-science-onramp/vertex-ai/requirements.txt | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/data-science-onramp/vertex-ai/requirements.txt b/data-science-onramp/vertex-ai/requirements.txt index 66e3eefd5284..af076dd1f356 100644 --- a/data-science-onramp/vertex-ai/requirements.txt +++ b/data-science-onramp/vertex-ai/requirements.txt @@ -1,2 +1,3 @@ -google-cloud-aiplatform==1.12.1 -google-cloud-storage==1.44.0 +google-cloud-aiplatform==1.21.0; python_version>'3.7' +google-cloud-aiplatform==1.20.0; python_version=='3.7' +google-cloud-storage==2.2.1 From f95ac0be70b759baad95d8cd8733eb13d2336790 Mon Sep 17 00:00:00 2001 From: Mend Renovate Date: Tue, 31 Jan 2023 15:56:50 +0000 Subject: [PATCH 10/12] chore(deps): update dependency google-cloud-container to v2.17.1 (#8848) --- container/snippets/requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/snippets/requirements.txt b/container/snippets/requirements.txt index 801bc8b41d87..5a68dad26a6e 100644 --- a/container/snippets/requirements.txt +++ b/container/snippets/requirements.txt @@ -1,3 +1,3 @@ -google-cloud-container==2.13.0 +google-cloud-container==2.17.1 backoff==2.2.1 pytest==7.2.0 \ No newline at end of file From e1603e50da9a243c597800205ecd6192f9937950 Mon Sep 17 00:00:00 2001 From: minherz Date: Wed, 1 Feb 2023 00:46:17 +0000 Subject: [PATCH 11/12] Add log redaction tutorial sample code (#9066) * doc(logging): add log redaction boilerplate code Create logging/redaction folder to store assets for log redaction tutorial. Assets will include: * documentation (README.md) * boilerplate code to start the tutorial * final version of the code * chore(fix): refactor dataflow fix package imports; add requirements.txt file to allow import google-cloud-logging that is not part of Beam SDK; fix logging_v2 client initialization; refactor some class, function and variable names; * chore(doc): add custom image Dockerfile Add a reference example for custom image Dockerfile * chore(fix): refactor code sample for tutorial Add "final" version of the code for tutorial. Format boilerplate code to keep same quote style and to add placeholder comments for code addition. * chore(fix): adding redaction step to pipeline Execute redaction function before sending logs in the pipeline. Refactoring argument names. * chore: fix compilation error * chore: comment text correction * chore: update codeowners * chore: change indent size from 2 to 4 * chore: fix lint errors * chore: minor text fix * chore: updating DEE observability ownership * chore: address style comments from dandhlee@ --- .github/CODEOWNERS | 7 +- .github/blunderbuss.yml | 2 + logging/redaction/Dockerfile | 6 + logging/redaction/README.md | 16 ++ logging/redaction/log_redaction.py | 150 +++++++++++++++ logging/redaction/log_redaction_final.py | 222 +++++++++++++++++++++++ logging/redaction/requirements.txt | 1 + 7 files changed, 401 insertions(+), 3 deletions(-) create mode 100644 logging/redaction/Dockerfile create mode 100644 logging/redaction/README.md create mode 100644 logging/redaction/log_redaction.py create mode 100644 logging/redaction/log_redaction_final.py create mode 100644 logging/redaction/requirements.txt diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS index 5d973f3691a8..6a72efec1851 100644 --- a/.github/CODEOWNERS +++ b/.github/CODEOWNERS @@ -62,12 +62,13 @@ /kubernetes_engine/**/* @GoogleCloudPlatform/python-samples-reviewers /kubernetes_engine/django_tutorial/**/* @glasnt @GoogleCloudPlatform/python-samples-reviewers /language/**/* @GoogleCloudPlatform/dee-data-ai @GoogleCloudPlatform/python-samples-reviewers +/logging/**/* @GoogleCloudPlatform/dee-observability @GoogleCloudPlatform/python-samples-reviewers /media_cdn/**/* @justin-mp @msampathkumar @GoogleCloudPlatform/python-samples-reviewers /memorystore/**/* @GoogleCloudPlatform/python-samples-reviewers /ml_engine/**/* @ivanmkc @GoogleCloudPlatform/python-samples-reviewers /monitoring/**/* @GoogleCloudPlatform/dee-observability @GoogleCloudPlatform/python-samples-reviewers -/monitoring/opencensus @yuriatgoogle @GoogleCloudPlatform/python-samples-reviewers -/monitoring/prometheus @yuriatgoogle @GoogleCloudPlatform/python-samples-reviewers +/monitoring/opencensus @yuriatgoogle @GoogleCloudPlatform/dee-observability @GoogleCloudPlatform/python-samples-reviewers +/monitoring/prometheus @yuriatgoogle @GoogleCloudPlatform/dee-observability @GoogleCloudPlatform/python-samples-reviewers /notebooks/**/* @alixhami @GoogleCloudPlatform/python-samples-reviewers /optimization/**/* @GoogleCloudPlatform/dee-data-ai @GoogleCloudPlatform/python-samples-reviewers /opencensus/**/* @GoogleCloudPlatform/python-samples-reviewers @@ -83,7 +84,7 @@ /storage/**/* @GoogleCloudPlatform/cloud-storage-dpes @GoogleCloudPlatform/python-samples-reviewers /storagetransfer/**/* @GoogleCloudPlatform/cloud-storage-dpes @GoogleCloudPlatform/python-samples-reviewers /texttospeech/**/* @GoogleCloudPlatform/dee-data-ai @GoogleCloudPlatform/python-samples-reviewers -/trace/**/* @ymotongpoo @GoogleCloudPlatform/python-samples-reviewers +/trace/**/* @ymotongpoo @GoogleCloudPlatform/dee-observability @GoogleCloudPlatform/python-samples-reviewers /translate/**/* @nicain @GoogleCloudPlatform/python-samples-reviewers /talent/**/* @GoogleCloudPlatform/python-samples-reviewers /vision/**/* @GoogleCloudPlatform/python-samples-reviewers diff --git a/.github/blunderbuss.yml b/.github/blunderbuss.yml index 60cfb68b3f43..46c13c453ca2 100644 --- a/.github/blunderbuss.yml +++ b/.github/blunderbuss.yml @@ -144,7 +144,9 @@ assign_issues_by: - GoogleCloudPlatform/python-samples-reviewers - ivanmkc - labels: + - 'api: logging' - 'api: monitoring' + - 'api: trace' to: - GoogleCloudPlatform/dee-observability - labels: diff --git a/logging/redaction/Dockerfile b/logging/redaction/Dockerfile new file mode 100644 index 000000000000..f3d1ced2d898 --- /dev/null +++ b/logging/redaction/Dockerfile @@ -0,0 +1,6 @@ +# From apache/beam_python3.9_sdk:2.43.0 +FROM apache/beam_python3.9_sdk@sha256:372abe1d342447118d517ed1325969115e2df03f09fc7604e606c7552380b2ce + +# Install google-cloud-logging package that is missing in Beam SDK +COPY requirements.txt /tmp +RUN pip3 install --upgrade pip && pip3 install -r /tmp/requirements.txt && pip3 check diff --git a/logging/redaction/README.md b/logging/redaction/README.md new file mode 100644 index 000000000000..8e62e5cbb396 --- /dev/null +++ b/logging/redaction/README.md @@ -0,0 +1,16 @@ +# Logging redaction tutorial code samples + +> **Warning** +> This section is still **W**ork **I**n **P**rogress. +> Cloud Shell button now opens this README. It will open the tutorial _AFTER_ its official launch. +> Tests to validate the code samples will be added. + +This section contains code that is used in the "Redact confidential information in logs" tutorial. +You can open the tutorial in Cloud Shell: + +[![Open in Cloud Shell][shell_img]][shell_link] + +_NOTE:_ You will need a valid Google Cloud credentials to open the tutorial. + +[shell_img]: http://gstatic.com/cloudssh/images/open-btn.png +[shell_link]: https://console.cloud.google.com/cloudshell/open?git_repo=https://github.com/GoogleCloudPlatform/python-docs-samples&page=editor&open_in_editor=logging/redaction/README.md diff --git a/logging/redaction/log_redaction.py b/logging/redaction/log_redaction.py new file mode 100644 index 000000000000..e5cd6161e116 --- /dev/null +++ b/logging/redaction/log_redaction.py @@ -0,0 +1,150 @@ +# Copyright 2023 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# https://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + + +import argparse +import json +import logging +from typing import List + +from apache_beam import CombineFn, CombineGlobally, DoFn, io, ParDo, Pipeline, WindowInto +from apache_beam.error import PipelineError +from apache_beam.options.pipeline_options import PipelineOptions +from apache_beam.transforms.window import FixedWindows + +from google.cloud import logging_v2 + + +# TODO: Place inspection and de-identification configurations + +class PayloadAsJson(DoFn): + '''Convert PubSub message payload to UTF-8 and return as JSON''' + def process(self, element): + yield json.loads(element.decode('utf-8')) + + +class BatchPayloads(CombineFn): + '''Opinionated way to batch all payloads in the window''' + + def create_accumulator(self): + return [] + + def add_input(self, accumulator, input): + accumulator.append(input) + return accumulator + + def merge_accumulators(self, accumulators): + merged = [ + item + for accumulator in accumulators + for item in accumulator + ] + return merged + + def extract_output(self, accumulator): + return accumulator + + +# TODO: Placeholder for LogRedaction class + + +class IngestLogs(DoFn): + '''Ingest payloads into destination log''' + + def __init__(self, destination_log_name): + self.destination_log_name = destination_log_name + self.logger = None + + def _replace_log_name(self, entry): + # update log name in the entry with destination log + entry['logName'] = self.logger.name + return entry + + def setup(self): + # initialize logging client + if self.logger: + return + + logging_client = logging_v2.Client() + if not logging_client: + logging.error('Cannot create GCP Logging Client') + raise PipelineError('Cannot create GCP Logging Client') + self.logger = logging_client.logger(self.destination_log_name) + if not self.logger: + logging.error('Google client library cannot create Logger object') + raise PipelineError('Google client library cannot create Logger object') + + def process(self, element): + if self.logger: + logs = list(map(self._replace_log_name, element)) + self.logger.client.logging_api.write_entries(logs) + yield logs + + +def run( + pubsub_subscription: str, + destination_log_name: str, + window_size: float, + pipeline_args: List[str] = None +) -> None: + '''Runs Dataflow pipeline''' + + pipeline_options = PipelineOptions( + pipeline_args, + streaming=True, + save_main_session=True + ) + pipeline = Pipeline(options=pipeline_options) + _ = ( + pipeline + | 'Read log entries from Pub/Sub' >> io.ReadFromPubSub(subscription=pubsub_subscription) + | 'Convert log entry payload to Json' >> ParDo(PayloadAsJson()) + | 'Aggregate payloads in fixed time intervals' >> WindowInto(FixedWindows(window_size)) + # Optimize Google API consumption and avoid possible throttling + # by calling APIs for batched data and not per each element + | 'Batch aggregated payloads' >> CombineGlobally(BatchPayloads()).without_defaults() + # TODO: Placeholder for redaction transformation + | 'Ingest to output log' >> ParDo(IngestLogs(destination_log_name)) + ) + pipeline.run() + + +if __name__ == '__main__': + logging.getLogger().setLevel(logging.INFO) + + parser = argparse.ArgumentParser() + parser.add_argument( + '--pubsub_subscription', + help='The Cloud Pub/Sub subscription to read from in the format ' + '"projects//subscription/".', + ) + parser.add_argument( + '--destination_log_name', + help='The log name to ingest log entries in the format ' + '"projects//logs/".', + ) + parser.add_argument( + '--window_size', + type=float, + default=60.0, + help='Output file\'s window size in seconds.', + ) + known_args, pipeline_args = parser.parse_known_args() + + run( + known_args.pubsub_subscription, + known_args.destination_log_name, + known_args.window_size, + pipeline_args, + ) diff --git a/logging/redaction/log_redaction_final.py b/logging/redaction/log_redaction_final.py new file mode 100644 index 000000000000..215cfe209bca --- /dev/null +++ b/logging/redaction/log_redaction_final.py @@ -0,0 +1,222 @@ +# Copyright 2023 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# https://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + + +import argparse +import json +import logging +from typing import List + +from apache_beam import CombineFn, CombineGlobally, DoFn, io, ParDo, Pipeline, WindowInto +from apache_beam.error import PipelineError +from apache_beam.options.pipeline_options import PipelineOptions +from apache_beam.transforms.window import FixedWindows + +from google.cloud import dlp_v2, logging_v2 + + +# For more details about info types format, please see +# https://cloud.google.com/dlp/docs/reference/rest/v2/InspectConfig +INSPECT_CFG = { + 'info_types': [ + {'name': 'US_SOCIAL_SECURITY_NUMBER'} + ] +} + +# For more details about transformation format, please see +# https://cloud.google.com/dlp/docs/reference/rest/v2/projects.deidentifyTemplates#DeidentifyTemplate.InfoTypeTransformations +REDACTION_CFG = { + 'info_type_transformations': { + 'transformations': [ + { + 'primitive_transformation': { + 'character_mask_config': { + 'masking_character': '#' + } + } + } + ] + } +} + + +class PayloadAsJson(DoFn): + '''Convert PubSub message payload to UTF-8 and return as JSON''' + def process(self, element): + yield json.loads(element.decode('utf-8')) + + +class BatchPayloads(CombineFn): + '''Collect all items in the windowed collection into single batch''' + + def create_accumulator(self): + return [] + + def add_input(self, accumulator, input): + accumulator.append(input) + return accumulator + + def merge_accumulators(self, accumulators): + merged = [ + item + for accumulator in accumulators + for item in accumulator + ] + return merged + + def extract_output(self, accumulator): + return accumulator + + +class LogRedaction(DoFn): + '''Apply inspection and redaction to textPayload field of log entries''' + + def __init__(self, project_id: str): + self.project_id = project_id + self.dlp_client = None + + def _log_to_row(self, entry): + # Make `Row` from `textPayload`. For more details on the row, please see + # https://cloud.google.com/dlp/docs/reference/rest/v2/ContentItem#Row + payload = entry.get('textPayload', '') + return {'values': [{'string_value': payload}]} + + def setup(self): + '''Initialize DLP client''' + if self.dlp_client: + return + self.dlp_client = dlp_v2.DlpServiceClient() + if not self.dlp_client: + logging.error('Cannot create Google DLP Client') + raise PipelineError('Cannot create Google DLP Client') + + def process(self, logs): + # Construct the `table`. For more details on the table schema, please see + # https://cloud.google.com/dlp/docs/reference/rest/v2/ContentItem#Table + table = { + 'table': { + 'headers': [{'name': 'textPayload'}], + 'rows': map(self._log_to_row, logs) + } + } + + response = self.dlp_client.deidentify_content( + request={ + 'parent': f'projects/{self.project_id}', + 'inspect_config': INSPECT_CFG, + 'deidentify_config': REDACTION_CFG, + 'item': table, + }) + + # replace payload with redacted version + modified_logs = [] + for index, log in enumerate(logs): + log['textPayload'] = response.item.table.rows[index].values[0].string_value + # you may consider changing insert ID if the project already has a copy + # of this log (e.g. log['insertId'] = 'deid-' + log['insertId']) + # For more details about insert ID, please see: + # https://cloud.google.com/logging/docs/reference/v2/rest/v2/LogEntry#FIELDS.insert_id + modified_logs.append(log) + yield modified_logs + + +class IngestLogs(DoFn): + '''Ingest payloads into destination log''' + + def __init__(self, destination_log_name): + self.destination_log_name = destination_log_name + self.logger = None + + def _replace_log_name(self, entry): + # updates log name in the entry to logger name + entry['logName'] = self.logger.name + return entry + + def setup(self): + # initialize logging client + if self.logger: + return + + logging_client = logging_v2.Client() + if not logging_client: + logging.error('Cannot create Google Logging Client') + raise PipelineError('Cannot create Google Logging Client') + self.logger = logging_client.logger(self.destination_log_name) + if not self.logger: + logging.error('Google client library cannot create Logger object') + raise PipelineError('Google client library cannot create Logger object') + + def process(self, element): + if self.logger: + logs = list(map(self._replace_log_name, element)) + self.logger.client.logging_api.write_entries(logs) + yield logs + + +def run( + pubsub_subscription: str, + destination_log_name: str, + window_size: float, + pipeline_args: List[str] = None +) -> None: + '''Runs Dataflow pipeline''' + + pipeline_options = PipelineOptions( + pipeline_args, + streaming=True, + save_main_session=True + ) + pipeline = Pipeline(options=pipeline_options) + _ = ( + pipeline + | 'Read log entries from Pub/Sub' >> io.ReadFromPubSub(subscription=pubsub_subscription) + | 'Convert log entry payload to Json' >> ParDo(PayloadAsJson()) + | 'Aggregate payloads in fixed time intervals' >> WindowInto(FixedWindows(window_size)) + # Optimize Google API consumption and avoid possible throttling + # by calling APIs for batched data and not per each element + | 'Batch aggregated payloads' >> CombineGlobally(BatchPayloads()).without_defaults() + | 'Redact SSN info from logs' >> ParDo(LogRedaction(destination_log_name.split('/')[1])) + | 'Ingest to output log' >> ParDo(IngestLogs(destination_log_name)) + ) + pipeline.run() + + +if __name__ == '__main__': + logging.getLogger().setLevel(logging.INFO) + + parser = argparse.ArgumentParser() + parser.add_argument( + '--pubsub_subscription', + help='The Cloud Pub/Sub subscription to read from in the format ' + '"projects//subscription/".', + ) + parser.add_argument( + '--destination_log_name', + help='The log name to ingest log entries in the format ' + '"projects//logs/".', + ) + parser.add_argument( + '--window_size', + type=float, + default=60.0, + help='Output file\'s window size in seconds.', + ) + known_args, pipeline_args = parser.parse_known_args() + + run( + known_args.pubsub_subscription, + known_args.destination_log_name, + known_args.window_size, + pipeline_args, + ) diff --git a/logging/redaction/requirements.txt b/logging/redaction/requirements.txt new file mode 100644 index 000000000000..e470934013ef --- /dev/null +++ b/logging/redaction/requirements.txt @@ -0,0 +1 @@ +google-cloud-logging>=3.4.0 From 78c86ecc6d7972f5a46dd52b8a43ec1397aea0ff Mon Sep 17 00:00:00 2001 From: Sita Lakshmi Sangameswaran Date: Wed, 1 Feb 2023 07:05:35 +0530 Subject: [PATCH 12/12] docs(samples): modified region tag to match apikey service (#8869) --- auth/api-client/authenticate_with_api_key.py | 4 ++-- auth/api-client/create_api_key.py | 4 ++-- auth/api-client/delete_api_key.py | 4 ++-- auth/api-client/lookup_api_key.py | 4 ++-- auth/api-client/restrict_api_key_android.py | 4 ++-- auth/api-client/restrict_api_key_api.py | 4 ++-- auth/api-client/restrict_api_key_http.py | 4 ++-- auth/api-client/restrict_api_key_ios.py | 4 ++-- auth/api-client/restrict_api_key_server.py | 4 ++-- 9 files changed, 18 insertions(+), 18 deletions(-) diff --git a/auth/api-client/authenticate_with_api_key.py b/auth/api-client/authenticate_with_api_key.py index 1e2b32ca1458..8ef697fc6f13 100644 --- a/auth/api-client/authenticate_with_api_key.py +++ b/auth/api-client/authenticate_with_api_key.py @@ -12,7 +12,7 @@ # See the License for the specific language governing permissions and # limitations under the License. -# [START auth_cloud_authenticate_api_key] +# [START apikeys_authenticate_api_key] from google.cloud import language_v1 @@ -46,4 +46,4 @@ def authenticate_with_api_key(quota_project_id: str, api_key_string: str) -> Non print(f"Sentiment: {sentiment.score}, {sentiment.magnitude}") print("Successfully authenticated using the API key") -# [END auth_cloud_authenticate_api_key] +# [END apikeys_authenticate_api_key] diff --git a/auth/api-client/create_api_key.py b/auth/api-client/create_api_key.py index b56e9ee158c4..0ecaf87094b1 100644 --- a/auth/api-client/create_api_key.py +++ b/auth/api-client/create_api_key.py @@ -12,7 +12,7 @@ # See the License for the specific language governing permissions and # limitations under the License. -# [START auth_cloud_create_api_key] +# [START apikeys_create_api_key] from google.cloud import api_keys_v2 from google.cloud.api_keys_v2 import Key @@ -53,4 +53,4 @@ def create_api_key(project_id: str) -> Key: # To restrict the usage of this API key, use the value in "response.name". return response -# [END auth_cloud_create_api_key] +# [END apikeys_create_api_key] diff --git a/auth/api-client/delete_api_key.py b/auth/api-client/delete_api_key.py index fe3657292572..49a9b3085601 100644 --- a/auth/api-client/delete_api_key.py +++ b/auth/api-client/delete_api_key.py @@ -12,7 +12,7 @@ # See the License for the specific language governing permissions and # limitations under the License. -# [START auth_cloud_delete_api_key] +# [START apikeys_delete_api_key] from google.cloud import api_keys_v2 @@ -42,4 +42,4 @@ def delete_api_key(project_id: str, key_id: str) -> None: result = client.delete_key(delete_key_request).result() print(f"Successfully deleted the API key: {result.name}") -# [END auth_cloud_delete_api_key] +# [END apikeys_delete_api_key] diff --git a/auth/api-client/lookup_api_key.py b/auth/api-client/lookup_api_key.py index b6ab728c6cc3..fece80df9c8f 100644 --- a/auth/api-client/lookup_api_key.py +++ b/auth/api-client/lookup_api_key.py @@ -12,7 +12,7 @@ # See the License for the specific language governing permissions and # limitations under the License. -# [START auth_cloud_lookup_api_key] +# [START apikeys_lookup_api_key] from google.cloud import api_keys_v2 @@ -45,4 +45,4 @@ def lookup_api_key(api_key_string: str) -> None: print(f"Successfully retrieved the API key name: {lookup_key_response.name}") -# [END auth_cloud_lookup_api_key] +# [END apikeys_lookup_api_key] diff --git a/auth/api-client/restrict_api_key_android.py b/auth/api-client/restrict_api_key_android.py index 9e5f657726e1..3c4dc23091c9 100644 --- a/auth/api-client/restrict_api_key_android.py +++ b/auth/api-client/restrict_api_key_android.py @@ -12,7 +12,7 @@ # See the License for the specific language governing permissions and # limitations under the License. -# [START auth_cloud_restrict_api_key_android] +# [START apikeys_restrict_api_key_android] from google.cloud import api_keys_v2 from google.cloud.api_keys_v2 import Key @@ -70,4 +70,4 @@ def restrict_api_key_android(project_id: str, key_id: str) -> Key: # Use response.key_string to authenticate. return response -# [END auth_cloud_restrict_api_key_android] +# [END apikeys_restrict_api_key_android] diff --git a/auth/api-client/restrict_api_key_api.py b/auth/api-client/restrict_api_key_api.py index bc3198e5c4d2..e2995ecc22e4 100644 --- a/auth/api-client/restrict_api_key_api.py +++ b/auth/api-client/restrict_api_key_api.py @@ -12,7 +12,7 @@ # See the License for the specific language governing permissions and # limitations under the License. -# [START auth_cloud_restrict_api_key_api] +# [START apikeys_restrict_api_key_api] from google.cloud import api_keys_v2 from google.cloud.api_keys_v2 import Key @@ -65,4 +65,4 @@ def restrict_api_key_api(project_id: str, key_id: str) -> Key: # Use response.key_string to authenticate. return response -# [END auth_cloud_restrict_api_key_api] +# [END apikeys_restrict_api_key_api] diff --git a/auth/api-client/restrict_api_key_http.py b/auth/api-client/restrict_api_key_http.py index f6c6cdf3461f..6e748d6261fd 100644 --- a/auth/api-client/restrict_api_key_http.py +++ b/auth/api-client/restrict_api_key_http.py @@ -12,7 +12,7 @@ # See the License for the specific language governing permissions and # limitations under the License. -# [START auth_cloud_restrict_api_key_http] +# [START apikeys_restrict_api_key_http] from google.cloud import api_keys_v2 from google.cloud.api_keys_v2 import Key @@ -64,4 +64,4 @@ def restrict_api_key_http(project_id: str, key_id: str) -> Key: # Use response.key_string to authenticate. return response -# [END auth_cloud_restrict_api_key_http] +# [END apikeys_restrict_api_key_http] diff --git a/auth/api-client/restrict_api_key_ios.py b/auth/api-client/restrict_api_key_ios.py index f808e2dcac8f..e5850b7aad92 100644 --- a/auth/api-client/restrict_api_key_ios.py +++ b/auth/api-client/restrict_api_key_ios.py @@ -12,7 +12,7 @@ # See the License for the specific language governing permissions and # limitations under the License. -# [START auth_cloud_restrict_api_key_ios] +# [START apikeys_restrict_api_key_ios] from google.cloud import api_keys_v2 from google.cloud.api_keys_v2 import Key @@ -64,4 +64,4 @@ def restrict_api_key_ios(project_id: str, key_id: str) -> Key: # Use response.key_string to authenticate. return response -# [END auth_cloud_restrict_api_key_ios] +# [END apikeys_restrict_api_key_ios] diff --git a/auth/api-client/restrict_api_key_server.py b/auth/api-client/restrict_api_key_server.py index aefe2a317513..5b4c56600dde 100644 --- a/auth/api-client/restrict_api_key_server.py +++ b/auth/api-client/restrict_api_key_server.py @@ -12,7 +12,7 @@ # See the License for the specific language governing permissions and # limitations under the License. -# [START auth_cloud_restrict_api_key_server] +# [START apikeys_restrict_api_key_server] from google.cloud import api_keys_v2 from google.cloud.api_keys_v2 import Key @@ -65,4 +65,4 @@ def restrict_api_key_server(project_id: str, key_id: str) -> Key: # Use response.key_string to authenticate. return response -# [END auth_cloud_restrict_api_key_server] +# [END apikeys_restrict_api_key_server]