(DS-2284, DS-2520)Creating Relay derived tables and views for Relay SaaSboard #3482

YeonjooSmith · 2022-12-30T03:23:23Z

Tried to mimic the data structure of mozilla vpn derived dataset to use the derived tables for Relay SaaSboard MVP
The attributions and some other parts in the vpn all subscriptions table are not included in the relay subscriptions table for simplicity for now. Please refer to commented out column names in subscriptions_v1 query.
There were error messages for using the bqetl tools to automatically validate and update schema and the error messages (in my understanding) were due to the creation of new files(nothing to compare against). But I was able to use the generate dag function to update the dag file.

Checklist for reviewer:

Commits should reference a bug or github issue, if relevant (if a bug is referenced, the pull request should include the bug number in the title).
If the PR comes from a fork, trigger integration CI tests by running the Push to upstream workflow and provide the <username>:<branch> of the fork as parameter. The parameter will also show up
in the logs of the manual-trigger-required-for-fork CI task together with more detailed instructions.
If adding a new field to a query, ensure that the schema and dependent downstream schemas have been updated.
When adding a new derived dataset, ensure that data is not available already (fully or partially) and recommend extending an existing dataset in favor of creating new ones. Data can be available in the bigquery-etl repository, looker-hub or in looker-spoke-default.

For modifications to schemas in restricted namespaces (see CODEOWNERS):

Follow the change control procedure

dataops-ci-bot · 2022-12-30T03:35:49Z

Integration report for "added country_name columns to all tables"

`sql.diff`

Click to expand!

diff -bur --no-dereference bigquery-etl-main/sql/moz-fx-cjms-prod-f3c7/cjms_bigquery/flows_v1/metadata.yaml /tmp/workspace/generated-sql/sql/moz-fx-cjms-prod-f3c7/cjms_bigquery/flows_v1/metadata.yaml
--- bigquery-etl-main/sql/moz-fx-cjms-prod-f3c7/cjms_bigquery/flows_v1/metadata.yaml	2022-12-30 03:34:55.841321869 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-cjms-prod-f3c7/cjms_bigquery/flows_v1/metadata.yaml	2022-12-30 03:28:05.000000000 +0000
@@ -13,4 +13,4 @@
 # Generated by bigquery_etl.dependency
 references:
   query.sql:
-  - moz-fx-data-shared-prod.firefox_accounts.fxa_all_events
+  - mozdata.firefox_accounts.fxa_content_auth_stdout_events
diff -bur --no-dereference bigquery-etl-main/sql/moz-fx-cjms-prod-f3c7/cjms_bigquery/flows_v1/query.sql /tmp/workspace/generated-sql/sql/moz-fx-cjms-prod-f3c7/cjms_bigquery/flows_v1/query.sql
--- bigquery-etl-main/sql/moz-fx-cjms-prod-f3c7/cjms_bigquery/flows_v1/query.sql	2022-12-30 03:30:44.920227771 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-cjms-prod-f3c7/cjms_bigquery/flows_v1/query.sql	2022-12-30 03:24:14.000000000 +0000
@@ -1,14 +1,3 @@
-WITH fxa_events AS (
-  SELECT
-    `timestamp`,
-    user_id,
-    flow_id,
-  FROM
-    `moz-fx-data-shared-prod.firefox_accounts.fxa_all_events`
-  WHERE
-    DATE(`timestamp`) = @submission_date
-    AND event_category IN ('content', 'auth', 'stdout')
-)
 SELECT
   DATE(`timestamp`) AS submission_date,
   flow_id,
@@ -25,9 +14,10 @@
       1
   )[SAFE_OFFSET(0)].*,
 FROM
-  fxa_events
+  mozdata.firefox_accounts.fxa_content_auth_stdout_events
 WHERE
-  flow_id IS NOT NULL
+  DATE(`timestamp`) = @submission_date
+  AND flow_id IS NOT NULL
 GROUP BY
   submission_date,
   flow_id
diff -bur --no-dereference bigquery-etl-main/sql/moz-fx-data-shared-prod/bedrock/interaction/schema.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/bedrock/interaction/schema.yaml
--- bigquery-etl-main/sql/moz-fx-data-shared-prod/bedrock/interaction/schema.yaml	2022-12-30 03:31:51.464475456 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/bedrock/interaction/schema.yaml	2022-12-30 03:24:37.000000000 +0000
@@ -226,9 +226,6 @@
         '
       mode: NULLABLE
       name: page_viewed
-      type: TIMESTAMP
-    - mode: NULLABLE
-      name: raw_page_viewed
       type: STRING
     mode: NULLABLE
     name: datetime
diff -bur --no-dereference bigquery-etl-main/sql/moz-fx-data-shared-prod/bedrock/interaction/view.sql /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/bedrock/interaction/view.sql
--- bigquery-etl-main/sql/moz-fx-data-shared-prod/bedrock/interaction/view.sql	2022-12-30 03:31:49.912470547 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/bedrock/interaction/view.sql	2022-12-30 03:24:36.000000000 +0000
@@ -5,16 +5,7 @@
 SELECT
   * REPLACE (
     mozfun.norm.metadata(metadata) AS metadata,
-    mozfun.norm.glean_ping_info(ping_info) AS ping_info,
-    (
-      SELECT AS STRUCT
-        metrics.* REPLACE (
-          STRUCT(
-            mozfun.glean.parse_datetime(metrics.datetime.page_viewed) AS page_viewed,
-            metrics.datetime.page_viewed AS raw_page_viewed
-          ) AS datetime
-        )
-    ) AS metrics
+    mozfun.norm.glean_ping_info(ping_info) AS ping_info
   )
 FROM
   `moz-fx-data-shared-prod.bedrock_stable.interaction_v1`
diff -bur --no-dereference bigquery-etl-main/sql/moz-fx-data-shared-prod/bedrock/non_interaction/schema.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/bedrock/non_interaction/schema.yaml
--- bigquery-etl-main/sql/moz-fx-data-shared-prod/bedrock/non_interaction/schema.yaml	2022-12-30 03:31:51.940477397 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/bedrock/non_interaction/schema.yaml	2022-12-30 03:24:37.000000000 +0000
@@ -226,9 +226,6 @@
         '
       mode: NULLABLE
       name: page_viewed
-      type: TIMESTAMP
-    - mode: NULLABLE
-      name: raw_page_viewed
       type: STRING
     mode: NULLABLE
     name: datetime
diff -bur --no-dereference bigquery-etl-main/sql/moz-fx-data-shared-prod/bedrock/non_interaction/view.sql /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/bedrock/non_interaction/view.sql
--- bigquery-etl-main/sql/moz-fx-data-shared-prod/bedrock/non_interaction/view.sql	2022-12-30 03:31:51.468475472 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/bedrock/non_interaction/view.sql	2022-12-30 03:24:37.000000000 +0000
@@ -5,16 +5,7 @@
 SELECT
   * REPLACE (
     mozfun.norm.metadata(metadata) AS metadata,
-    mozfun.norm.glean_ping_info(ping_info) AS ping_info,
-    (
-      SELECT AS STRUCT
-        metrics.* REPLACE (
-          STRUCT(
-            mozfun.glean.parse_datetime(metrics.datetime.page_viewed) AS page_viewed,
-            metrics.datetime.page_viewed AS raw_page_viewed
-          ) AS datetime
-        )
-    ) AS metrics
+    mozfun.norm.glean_ping_info(ping_info) AS ping_info
   )
 FROM
   `moz-fx-data-shared-prod.bedrock_stable.non_interaction_v1`
diff -bur --no-dereference bigquery-etl-main/sql/moz-fx-data-shared-prod/bedrock/page_view/schema.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/bedrock/page_view/schema.yaml
--- bigquery-etl-main/sql/moz-fx-data-shared-prod/bedrock/page_view/schema.yaml	2022-12-30 03:31:52.540479431 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/bedrock/page_view/schema.yaml	2022-12-30 03:24:38.000000000 +0000
@@ -226,9 +226,6 @@
         '
       mode: NULLABLE
       name: page_viewed
-      type: TIMESTAMP
-    - mode: NULLABLE
-      name: raw_page_viewed
       type: STRING
     mode: NULLABLE
     name: datetime
diff -bur --no-dereference bigquery-etl-main/sql/moz-fx-data-shared-prod/bedrock/page_view/view.sql /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/bedrock/page_view/view.sql
--- bigquery-etl-main/sql/moz-fx-data-shared-prod/bedrock/page_view/view.sql	2022-12-30 03:31:51.944477414 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/bedrock/page_view/view.sql	2022-12-30 03:24:37.000000000 +0000
@@ -5,16 +5,7 @@
 SELECT
   * REPLACE (
     mozfun.norm.metadata(metadata) AS metadata,
-    mozfun.norm.glean_ping_info(ping_info) AS ping_info,
-    (
-      SELECT AS STRUCT
-        metrics.* REPLACE (
-          STRUCT(
-            mozfun.glean.parse_datetime(metrics.datetime.page_viewed) AS page_viewed,
-            metrics.datetime.page_viewed AS raw_page_viewed
-          ) AS datetime
-        )
-    ) AS metrics
+    mozfun.norm.glean_ping_info(ping_info) AS ping_info
   )
 FROM
   `moz-fx-data-shared-prod.bedrock_stable.page_view_v1`
diff -bur --no-dereference bigquery-etl-main/sql/moz-fx-data-shared-prod/burnham/baseline/schema.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/burnham/baseline/schema.yaml
--- bigquery-etl-main/sql/moz-fx-data-shared-prod/burnham/baseline/schema.yaml	2022-12-30 03:31:53.196481481 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/burnham/baseline/schema.yaml	2022-12-30 03:24:38.000000000 +0000
@@ -358,9 +358,6 @@
         '
       mode: NULLABLE
       name: glean_validation_first_run_hour
-      type: TIMESTAMP
-    - mode: NULLABLE
-      name: raw_glean_validation_first_run_hour
       type: STRING
     mode: NULLABLE
     name: datetime
diff -bur --no-dereference bigquery-etl-main/sql/moz-fx-data-shared-prod/burnham/baseline/view.sql /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/burnham/baseline/view.sql
--- bigquery-etl-main/sql/moz-fx-data-shared-prod/burnham/baseline/view.sql	2022-12-30 03:31:52.540479431 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/burnham/baseline/view.sql	2022-12-30 03:24:38.000000000 +0000
@@ -6,18 +6,7 @@
   * REPLACE (
     mozfun.norm.metadata(metadata) AS metadata,
     mozfun.norm.glean_ping_info(ping_info) AS ping_info,
-    mozfun.norm.glean_baseline_client_info(client_info, metrics) AS client_info,
-    (
-      SELECT AS STRUCT
-        metrics.* REPLACE (
-          STRUCT(
-            mozfun.glean.parse_datetime(
-              metrics.datetime.glean_validation_first_run_hour
-            ) AS glean_validation_first_run_hour,
-            metrics.datetime.glean_validation_first_run_hour AS raw_glean_validation_first_run_hour
-          ) AS datetime
-        )
-    ) AS metrics
+    mozfun.norm.glean_baseline_client_info(client_info, metrics) AS client_info
   )
 FROM
   `moz-fx-data-shared-prod.burnham_stable.baseline_v1`
diff -bur --no-dereference bigquery-etl-main/sql/moz-fx-data-shared-prod/burnham/metrics/schema.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/burnham/metrics/schema.yaml
--- bigquery-etl-main/sql/moz-fx-data-shared-prod/burnham/metrics/schema.yaml	2022-12-30 03:31:52.136478169 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/burnham/metrics/schema.yaml	2022-12-30 03:24:38.000000000 +0000
@@ -515,9 +515,6 @@
         '
       mode: NULLABLE
       name: glean_validation_first_run_hour
-      type: TIMESTAMP
-    - mode: NULLABLE
-      name: raw_glean_validation_first_run_hour
       type: STRING
     mode: NULLABLE
     name: datetime
diff -bur --no-dereference bigquery-etl-main/sql/moz-fx-data-shared-prod/burnham/metrics/view.sql /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/burnham/metrics/view.sql
--- bigquery-etl-main/sql/moz-fx-data-shared-prod/burnham/metrics/view.sql	2022-12-30 03:31:51.484475538 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/burnham/metrics/view.sql	2022-12-30 03:24:38.000000000 +0000
@@ -5,18 +5,7 @@
 SELECT
   * REPLACE (
     mozfun.norm.metadata(metadata) AS metadata,
-    mozfun.norm.glean_ping_info(ping_info) AS ping_info,
-    (
-      SELECT AS STRUCT
-        metrics.* REPLACE (
-          STRUCT(
-            mozfun.glean.parse_datetime(
-              metrics.datetime.glean_validation_first_run_hour
-            ) AS glean_validation_first_run_hour,
-            metrics.datetime.glean_validation_first_run_hour AS raw_glean_validation_first_run_hour
-          ) AS datetime
-        )
-    ) AS metrics
+    mozfun.norm.glean_ping_info(ping_info) AS ping_info
   )
 FROM
   `moz-fx-data-shared-prod.burnham_stable.metrics_v1`
Only in bigquery-etl-main/sql/moz-fx-data-shared-prod/fenix: firefox_android_clients
diff -bur --no-dereference bigquery-etl-main/sql/moz-fx-data-shared-prod/fenix_derived/attributable_clients_v1/init.sql /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/fenix_derived/attributable_clients_v1/init.sql
--- bigquery-etl-main/sql/moz-fx-data-shared-prod/fenix_derived/attributable_clients_v1/init.sql	2022-12-30 03:30:44.932227834 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/fenix_derived/attributable_clients_v1/init.sql	2022-12-30 03:24:14.000000000 +0000
@@ -63,15 +63,28 @@
     ),
     adjust_client AS (
       SELECT
-        client_id,
-        adjust_network,
-        adjust_ad_group AS adjust_adgroup,
-        adjust_campaign,
-        adjust_creative,
+        client_info.client_id AS client_id,
+        ARRAY_AGG(metrics.string.first_session_network ORDER BY submission_timestamp)[
+          SAFE_OFFSET(0)
+        ] AS adjust_network,
+        ARRAY_AGG(metrics.string.first_session_adgroup ORDER BY submission_timestamp)[
+          SAFE_OFFSET(0)
+        ] AS adjust_adgroup,
+        ARRAY_AGG(metrics.string.first_session_campaign ORDER BY submission_timestamp)[
+          SAFE_OFFSET(0)
+        ] AS adjust_campaign,
+        ARRAY_AGG(metrics.string.first_session_creative ORDER BY submission_timestamp)[
+          SAFE_OFFSET(0)
+        ] AS adjust_creative,
+        MIN(DATE(submission_timestamp)) AS first_session_date
       FROM
-        mozdata.fenix.firefox_android_clients
+        `mozdata.fenix.first_session`
       WHERE
-        adjust_network != 'Unknown'
+        DATE(submission_timestamp) >= first_date
+        AND metrics.string.first_session_network IS NOT NULL
+        AND metrics.string.first_session_network != ''
+      GROUP BY
+        client_id
     )
     SELECT
       submission_date,
diff -bur --no-dereference bigquery-etl-main/sql/moz-fx-data-shared-prod/fenix_derived/attributable_clients_v1/metadata.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/fenix_derived/attributable_clients_v1/metadata.yaml
--- bigquery-etl-main/sql/moz-fx-data-shared-prod/fenix_derived/attributable_clients_v1/metadata.yaml	2022-12-30 03:34:56.197322666 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/fenix_derived/attributable_clients_v1/metadata.yaml	2022-12-30 03:28:05.000000000 +0000
@@ -23,7 +23,7 @@
     execution_delta: 1h
   - task_id: search_derived__mobile_search_clients_daily__v1
     dag_name: bqetl_mobile_search
-    execution_delta: 0h
+    execution_delta: 1h
 bigquery:
   time_partitioning:
     field: submission_date
@@ -41,9 +41,9 @@
   - moz-fx-data-shared-prod.search_derived.mobile_search_clients_daily_v1
   - mozdata.fenix.baseline
   - mozdata.fenix.baseline_clients_first_seen
-  - mozdata.fenix.firefox_android_clients
+  - mozdata.fenix.first_session
   query.sql:
   - moz-fx-data-shared-prod.search_derived.mobile_search_clients_daily_v1
   - mozdata.fenix.baseline
   - mozdata.fenix.baseline_clients_first_seen
-  - mozdata.fenix.firefox_android_clients
+  - mozdata.fenix.first_session
diff -bur --no-dereference bigquery-etl-main/sql/moz-fx-data-shared-prod/fenix_derived/attributable_clients_v1/query.sql /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/fenix_derived/attributable_clients_v1/query.sql
--- bigquery-etl-main/sql/moz-fx-data-shared-prod/fenix_derived/attributable_clients_v1/query.sql	2022-12-30 03:30:44.932227834 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/fenix_derived/attributable_clients_v1/query.sql	2022-12-30 03:24:14.000000000 +0000
@@ -53,15 +53,28 @@
 ),
 adjust_client AS (
   SELECT
-    client_id,
-    adjust_network,
-    adjust_ad_group AS adjust_adgroup,
-    adjust_campaign,
-    adjust_creative,
+    client_info.client_id AS client_id,
+    ARRAY_AGG(metrics.string.first_session_network ORDER BY submission_timestamp)[
+      SAFE_OFFSET(0)
+    ] AS adjust_network,
+    ARRAY_AGG(metrics.string.first_session_adgroup ORDER BY submission_timestamp)[
+      SAFE_OFFSET(0)
+    ] AS adjust_adgroup,
+    ARRAY_AGG(metrics.string.first_session_campaign ORDER BY submission_timestamp)[
+      SAFE_OFFSET(0)
+    ] AS adjust_campaign,
+    ARRAY_AGG(metrics.string.first_session_creative ORDER BY submission_timestamp)[
+      SAFE_OFFSET(0)
+    ] AS adjust_creative,
+    MIN(DATE(submission_timestamp)) AS first_session_date
   FROM
-    mozdata.fenix.firefox_android_clients
+    `mozdata.fenix.first_session`
   WHERE
-    adjust_network != 'Unknown'
+    DATE(submission_timestamp) >= "2021-01-01"
+    AND metrics.string.first_session_network IS NOT NULL
+    AND metrics.string.first_session_network != ''
+  GROUP BY
+    client_id
 )
 SELECT
   submission_date,
Only in bigquery-etl-main/sql/moz-fx-data-shared-prod/fenix_derived: firefox_android_clients_v1
diff -bur --no-dereference bigquery-etl-main/sql/moz-fx-data-shared-prod/firefox_accounts/fxa_all_events/metadata.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_accounts/fxa_all_events/metadata.yaml
--- bigquery-etl-main/sql/moz-fx-data-shared-prod/firefox_accounts/fxa_all_events/metadata.yaml	2022-12-30 03:34:56.241322764 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_accounts/fxa_all_events/metadata.yaml	2022-12-30 03:28:05.000000000 +0000
@@ -6,4 +6,3 @@
   - moz-fx-data-shared-prod.firefox_accounts_derived.fxa_auth_events_v1
   - moz-fx-data-shared-prod.firefox_accounts_derived.fxa_content_events_v1
   - moz-fx-data-shared-prod.firefox_accounts_derived.fxa_oauth_events_v1
-  - moz-fx-data-shared-prod.firefox_accounts_derived.fxa_stdout_events_v1
diff -bur --no-dereference bigquery-etl-main/sql/moz-fx-data-shared-prod/firefox_accounts/fxa_all_events/view.sql /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_accounts/fxa_all_events/view.sql
--- bigquery-etl-main/sql/moz-fx-data-shared-prod/firefox_accounts/fxa_all_events/view.sql	2022-12-30 03:30:44.936227854 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_accounts/fxa_all_events/view.sql	2022-12-30 03:24:14.000000000 +0000
@@ -1,15 +1,9 @@
--- NOTE regarding oauth events:
--- oauth events have been merged into the auth table.
--- However, the oauth events table still contains around
--- 162 days of data from 2019H2 hence why it's still available
--- via this view.
 CREATE OR REPLACE VIEW
   `moz-fx-data-shared-prod.firefox_accounts.fxa_all_events`
 AS
 WITH fxa_auth_events AS (
   SELECT
-    `timestamp`,
-    receiveTimestamp,
+    timestamp AS submission_timestamp,
     jsonPayload.fields.user_id,
     jsonPayload.fields.country,
     jsonPayload.fields.language,
@@ -17,10 +11,7 @@
     jsonPayload.fields.os_name,
     jsonPayload.fields.os_version,
     jsonPayload.fields.event_type,
-    jsonPayload.logger,
-    jsonPayload.fields.user_properties,
-    jsonPayload.fields.event_properties,
-    jsonPayload.fields.device_id,
+    JSON_VALUE(jsonPayload.fields.event_properties, '$.service') AS service
   FROM
     `moz-fx-data-shared-prod.firefox_accounts_derived.fxa_auth_events_v1`
 ),
@@ -28,28 +19,23 @@
   -- but should always be included for a complete raw event log.
 fxa_auth_bounce_events AS (
   SELECT
-    `timestamp`,
-    receiveTimestamp,
+    timestamp AS submission_timestamp,
     jsonPayload.fields.user_id,
     CAST(
       NULL AS STRING
     ) AS country,  -- No country field in auth_bounces
     jsonPayload.fields.language,
     jsonPayload.fields.app_version,
-    CAST(NULL AS STRING) AS os_name,
-    CAST(NULL AS STRING) AS os_version,
+    CAST(NULL AS STRING),
+    CAST(NULL AS STRING),
     jsonPayload.fields.event_type,
-    jsonPayload.logger,
-    jsonPayload.fields.user_properties,
-    jsonPayload.fields.event_properties,
-    CAST(NULL AS STRING) AS device_id,
+    JSON_VALUE(jsonPayload.fields.event_properties, '$.service') AS service
   FROM
     `moz-fx-data-shared-prod.firefox_accounts_derived.fxa_auth_bounce_events_v1`
 ),
 fxa_content_events AS (
   SELECT
-    `timestamp`,
-    receiveTimestamp,
+    timestamp AS submission_timestamp,
     jsonPayload.fields.user_id,
     jsonPayload.fields.country,
     jsonPayload.fields.language,
@@ -57,18 +43,13 @@
     jsonPayload.fields.os_name,
     jsonPayload.fields.os_version,
     jsonPayload.fields.event_type,
-    jsonPayload.logger,
-    jsonPayload.fields.user_properties,
-    jsonPayload.fields.event_properties,
-    jsonPayload.fields.device_id,
+    JSON_VALUE(jsonPayload.fields.event_properties, '$.service') AS service
   FROM
     `moz-fx-data-shared-prod.firefox_accounts_derived.fxa_content_events_v1`
 ),
--- oauth events, see the note on top
 fxa_oauth_events AS (
   SELECT
-    `timestamp`,
-    receiveTimestamp,
+    timestamp AS submission_timestamp,
     jsonPayload.fields.user_id,
     CAST(NULL AS STRING) AS country,
     CAST(NULL AS STRING) AS language,
@@ -76,109 +57,26 @@
     CAST(NULL AS STRING) AS os_name,
     CAST(NULL AS STRING) AS os_version,
     jsonPayload.fields.event_type,
-    jsonPayload.logger,
-    jsonPayload.fields.user_properties,
-    jsonPayload.fields.event_properties,
-    CAST(NULL AS STRING) AS device_id,
+    JSON_VALUE(jsonPayload.fields.event_properties, '$.service') AS service
   FROM
     `moz-fx-data-shared-prod.firefox_accounts_derived.fxa_oauth_events_v1`
-),
-fxa_stdout_events AS (
-  SELECT
-    `timestamp`,
-    receiveTimestamp,
-    jsonPayload.fields.user_id,
-    CAST(NULL AS STRING) AS country,
-    jsonPayload.fields.language,
-    jsonPayload.fields.app_version,
-    jsonPayload.fields.os_name,
-    jsonPayload.fields.os_version,
-    jsonPayload.fields.event_type,
-    jsonPayload.logger,
-    jsonPayload.fields.user_properties,
-    jsonPayload.fields.event_properties,
-    jsonPayload.fields.device_id,
-  FROM
-    `moz-fx-data-shared-prod.firefox_accounts_derived.fxa_stdout_events_v1`
-),
-unioned AS (
-  SELECT
-    *,
-    'auth' AS event_category,
-  FROM
+)
+SELECT
+  *
+FROM
     fxa_auth_events
-  UNION ALL
-  SELECT
-    *,
-    'auth_bounce' AS event_category,
-  FROM
+UNION ALL
+SELECT
+  *
+FROM
     fxa_auth_bounce_events
-  UNION ALL
-  SELECT
-    *,
-    'content' AS event_category,
-  FROM
+UNION ALL
+SELECT
+  *
+FROM
     fxa_content_events
-  UNION ALL
-  -- oauth events, see the note on top
-  SELECT
-    *,
-    'oauth' AS event_category,
-  FROM
-    fxa_oauth_events
-  UNION ALL
-  SELECT
-    *,
-    'stdout' AS event_category,
-  FROM
-    fxa_stdout_events
-)
+UNION ALL
 SELECT
-  `timestamp`,
-  receiveTimestamp,
-  event_category,
-  event_type,
-  user_id,
-  device_id,
-  country,
-  `language`,
-  app_version,
-  os_name,
-  os_version,
-  user_properties,
-  event_properties,
-  -- extract user properties
-  JSON_VALUE(user_properties, "$.utm_term") AS utm_term,
-  JSON_VALUE(user_properties, "$.utm_source") AS utm_source,
-  JSON_VALUE(user_properties, "$.utm_medium") AS utm_medium,
-  JSON_VALUE(user_properties, "$.utm_campaign") AS utm_campaign,
-  JSON_VALUE(user_properties, "$.utm_content") AS utm_content,
-  JSON_VALUE(user_properties, "$.ua_version") AS ua_version,
-  JSON_VALUE(user_properties, "$.ua_browser") AS ua_browser,
-  JSON_VALUE(user_properties, "$.entrypoint") AS entrypoint,
-  JSON_VALUE(user_properties, "$.entrypoint_experiment") AS entrypoint_experiment,
-  JSON_VALUE(user_properties, "$.entrypoint_variation") AS entrypoint_variation,
-  JSON_VALUE(user_properties, "$.flow_id") AS flow_id,
-  JSON_VALUE(user_properties, "$.sync_device_count") AS sync_device_count,
-  JSON_VALUE(user_properties, "$.sync_active_devices_day") AS sync_active_devices_day,
-  JSON_VALUE(user_properties, "$.sync_active_devices_week") AS sync_active_devices_week,
-  JSON_VALUE(user_properties, "$.sync_active_devices_month") AS sync_active_devices_month,
-  -- extract event properties
-  JSON_VALUE(event_properties, "$.service") AS service,
-  JSON_VALUE(event_properties, "$.email_type") AS email_type,
-  JSON_VALUE(event_properties, "$.email_provider") AS email_provider,
-  JSON_VALUE(event_properties, "$.oauth_client_id") AS oauth_client_id,
-  JSON_VALUE(event_properties, "$.connect_device_flow") AS connect_device_flow,
-  JSON_VALUE(event_properties, "$.connect_device_os") AS connect_device_os,
-  JSON_VALUE(event_properties, "$.email_sender") AS email_sender,
-  JSON_VALUE(event_properties, "$.email_service") AS email_service,
-  JSON_VALUE(event_properties, "$.email_template") AS email_template,
-  JSON_VALUE(event_properties, "$.email_version") AS email_version,
-  JSON_VALUE(event_properties, "$.plan_id") AS plan_id,
-  JSON_VALUE(event_properties, "$.product_id") AS product_id,
-  JSON_VALUE(event_properties, "$.promotionCode") AS promotion_code,
-  JSON_VALUE(event_properties, "$.payment_provider") AS payment_provider,
-  JSON_VALUE(event_properties, "$.checkout_type") AS checkout_type,
-  JSON_VALUE(event_properties, "$.source_country") AS source_country,
+  *
 FROM
-  unioned
+  fxa_oauth_events
diff -bur --no-dereference bigquery-etl-main/sql/moz-fx-data-shared-prod/firefox_accounts/fxa_content_auth_events/view.sql /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_accounts/fxa_content_auth_events/view.sql
--- bigquery-etl-main/sql/moz-fx-data-shared-prod/firefox_accounts/fxa_content_auth_events/view.sql	2022-12-30 03:30:44.936227854 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_accounts/fxa_content_auth_events/view.sql	2022-12-30 03:24:14.000000000 +0000
@@ -1,25 +1,3 @@
------
---- ** DEPRECATION NOTICE **
----
---- There is currently an ongoing effort to deprecate this view
---- please use `firefox_accounts.fxa_all_events` view instead
---- in new queries.
----
---- Please filter on `event_category` field to limit your results
---- to events coming only from a specific fxa server like so:
---- WHERE event_category IN ('auth', 'stdout', ...)
---- Options include:
----   content
----   auth
----   stdout
----   oauth -- this has been deprecated and merged into fxa_auth_event
----   auth_bounce
---- to replicate results of this view use:
---- WHERE event_category IN (
----  'content',
----  'auth'
---- )
------
 CREATE OR REPLACE VIEW
   `moz-fx-data-shared-prod.firefox_accounts.fxa_content_auth_events`
 AS
diff -bur --no-dereference bigquery-etl-main/sql/moz-fx-data-shared-prod/firefox_accounts/fxa_content_auth_oauth_events/view.sql /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_accounts/fxa_content_auth_oauth_events/view.sql
--- bigquery-etl-main/sql/moz-fx-data-shared-prod/firefox_accounts/fxa_content_auth_oauth_events/view.sql	2022-12-30 03:30:44.936227854 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_accounts/fxa_content_auth_oauth_events/view.sql	2022-12-30 03:24:14.000000000 +0000
@@ -1,26 +1,3 @@
------
---- ** DEPRECATION NOTICE **
----
---- There is currently an ongoing effort to deprecate this view
---- please use `firefox_accounts.fxa_all_events` view instead
---- in new queries.
----
---- Please filter on `event_category` field to limit your results
---- to events coming only from a specific fxa server like so:
---- WHERE event_category IN ('auth', 'stdout', ...)
---- Options include:
----   content
----   auth
----   stdout
----   oauth -- this has been deprecated and merged into fxa_auth_event
----   auth_bounce
---- to replicate results of this view use:
---- WHERE event_category IN (
----  'content',
----  'auth',
---   'oauth'
---- )
------
 CREATE OR REPLACE VIEW
   `moz-fx-data-shared-prod.firefox_accounts.fxa_content_auth_oauth_events`
 AS
diff -bur --no-dereference bigquery-etl-main/sql/moz-fx-data-shared-prod/firefox_accounts/fxa_content_auth_stdout_events/view.sql /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_accounts/fxa_content_auth_stdout_events/view.sql
--- bigquery-etl-main/sql/moz-fx-data-shared-prod/firefox_accounts/fxa_content_auth_stdout_events/view.sql	2022-12-30 03:30:44.936227854 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_accounts/fxa_content_auth_stdout_events/view.sql	2022-12-30 03:24:14.000000000 +0000
@@ -1,26 +1,3 @@
------
---- ** DEPRECATION NOTICE **
----
---- There is currently an ongoing effort to deprecate this view
---- please use `firefox_accounts.fxa_all_events` view instead
---- in new queries.
----
---- Please filter on `event_category` field to limit your results
---- to events coming only from a specific fxa server like so:
---- WHERE event_category IN ('auth', 'stdout', ...)
---- Options include:
----   content
----   auth
----   stdout
----   oauth -- this has been deprecated and merged into fxa_auth_event
----   auth_bounce
---- to replicate results of this view use:
---- WHERE event_category IN (
----  'content',
----  'auth',
---   'stdout'
---- )
------
 CREATE OR REPLACE VIEW
   `moz-fx-data-shared-prod.firefox_accounts.fxa_content_auth_stdout_events`
 AS
diff -bur --no-dereference bigquery-etl-main/sql/moz-fx-data-shared-prod/firefox_accounts_derived/funnel_events_source_v1/metadata.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_accounts_derived/funnel_events_source_v1/metadata.yaml
--- bigquery-etl-main/sql/moz-fx-data-shared-prod/firefox_accounts_derived/funnel_events_source_v1/metadata.yaml	2022-12-30 03:34:56.281322854 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_accounts_derived/funnel_events_source_v1/metadata.yaml	2022-12-30 03:28:05.000000000 +0000
@@ -2,7 +2,7 @@
 description: |-
   The materialized events used to create the events_daily rollup.
   Not exposed as a view since it's not meant for consumption;
-  use firefox_accounts.fxa_all_events
+  use firefox_accounts.fxa_content_auth_oauth_events
   instead. This is materialized solely because
   the events_daily queries are too complex for BigQuery
   otherwise.
@@ -18,4 +18,4 @@
 # Generated by bigquery_etl.dependency
 references:
   query.sql:
-  - moz-fx-data-shared-prod.firefox_accounts.fxa_all_events
+  - moz-fx-data-shared-prod.firefox_accounts.fxa_content_auth_oauth_events
diff -bur --no-dereference bigquery-etl-main/sql/moz-fx-data-shared-prod/firefox_accounts_derived/funnel_events_source_v1/query.sql /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_accounts_derived/funnel_events_source_v1/query.sql
--- bigquery-etl-main/sql/moz-fx-data-shared-prod/firefox_accounts_derived/funnel_events_source_v1/query.sql	2022-12-30 03:30:44.936227854 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_accounts_derived/funnel_events_source_v1/query.sql	2022-12-30 03:24:14.000000000 +0000
@@ -1,26 +1,5 @@
-WITH fxa_events AS (
-  SELECT
-    `timestamp`,
-    user_id,
-    event_type,
-    service,
-    email_type,
-    oauth_client_id,
-    connect_device_flow,
-    connect_device_os,
-    sync_device_count,
-    email_sender,
-    email_service,
-    email_template,
-    email_version,
-  FROM
-    `moz-fx-data-shared-prod.firefox_accounts.fxa_all_events`
-  WHERE
-    DATE(`timestamp`) = @submission_date
-    AND event_category IN ('content', 'auth', 'oauth')
-)
 SELECT
-  DATE(`timestamp`) AS submission_date,
+  DATE(timestamp) AS submission_date,
   user_id AS client_id,
   `moz-fx-data-shared-prod`.udf.safe_sample_id(user_id) AS sample_id,
   SPLIT(event_type, ' - ')[OFFSET(0)] AS category,
@@ -40,4 +19,6 @@
   CAST([] AS ARRAY<STRUCT<key STRING, value STRING>>) AS experiments,
   *,
 FROM
-  fxa_events
+  `moz-fx-data-shared-prod.firefox_accounts.fxa_content_auth_oauth_events`
+WHERE
+  DATE(timestamp) = @submission_date
diff -bur --no-dereference bigquery-etl-main/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_users_daily_v1/query.sql /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_users_daily_v1/query.sql
--- bigquery-etl-main/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_users_daily_v1/query.sql	2022-12-30 03:30:44.940227875 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_users_daily_v1/query.sql	2022-12-30 03:24:14.000000000 +0000
@@ -35,7 +35,7 @@
 
 WITH windowed AS (
   SELECT
-    DATE(`timestamp`) AS submission_date,
+    DATE(submission_timestamp) AS submission_date,
     user_id,
     ROW_NUMBER() OVER w1_unframed AS _n,
     udf.mode_last(ARRAY_AGG(country) OVER w1) AS country,
@@ -49,10 +49,9 @@
       NOT (event_type = 'fxa_rp - engage' AND service = 'fx-monitor')
     ) OVER w1 = 0 AS monitor_only
   FROM
-    `firefox_accounts.fxa_all_events`
+    firefox_accounts.fxa_all_events
   WHERE
-    event_category IN ('auth', 'auth_bounce', 'content', 'oauth')
-    AND user_id IS NOT NULL
+    user_id IS NOT NULL
     AND event_type NOT IN ( --
       'fxa_email - bounced',
       'fxa_email - click',
@@ -69,14 +68,14 @@
       'sync - repair_triggered'
     )
     -- Reprocess all dates by running this query with --parameter=submission_date:DATE:NULL
-    AND (@submission_date IS NULL OR @submission_date = DATE(`timestamp`))
+    AND (@submission_date IS NULL OR @submission_date = DATE(submission_timestamp))
   WINDOW
     w1 AS (
       PARTITION BY
         user_id,
-        DATE(`timestamp`)
+        DATE(submission_timestamp)
       ORDER BY
-        `timestamp` --
+        submission_timestamp --
       ROWS BETWEEN
         UNBOUNDED PRECEDING
         AND UNBOUNDED FOLLOWING
@@ -85,9 +84,9 @@
     w1_unframed AS (
       PARTITION BY
         user_id,
-        DATE(`timestamp`)
+        DATE(submission_timestamp)
       ORDER BY
-        `timestamp`
+        submission_timestamp
     )
 )
 SELECT
diff -bur --no-dereference bigquery-etl-main/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_users_first_seen_v1/init.sql /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_users_first_seen_v1/init.sql
--- bigquery-etl-main/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_users_first_seen_v1/init.sql	2022-12-30 03:30:44.940227875 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_users_first_seen_v1/init.sql	2022-12-30 03:24:14.000000000 +0000
@@ -7,12 +7,11 @@
 AS
 SELECT
   user_id,
-  DATE(MIN(`timestamp`)) AS first_seen_date,
+  DATE(MIN(submission_timestamp)) AS first_seen_date,
   ARRAY_AGG(DISTINCT service IGNORE NULLS ORDER BY service) AS services_used,
 FROM
   firefox_accounts.fxa_all_events
 WHERE
-  `timestamp` > '2010-01-01'
-  AND event_category IN ('auth', 'auth_bounce', 'content', 'oauth')
+  submission_timestamp > '2010-01-01'
 GROUP BY
   user_id
diff -bur --no-dereference bigquery-etl-main/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_users_first_seen_v1/query.sql /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_users_first_seen_v1/query.sql
--- bigquery-etl-main/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_users_first_seen_v1/query.sql	2022-12-30 03:30:44.940227875 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_users_first_seen_v1/query.sql	2022-12-30 03:24:14.000000000 +0000
@@ -5,8 +5,7 @@
   FROM
     firefox_accounts.fxa_all_events
   WHERE
-    DATE(`timestamp`) = @submission_date
-    AND event_category IN ('auth', 'auth_bounce', 'content', 'oauth')
+    DATE(submission_timestamp) = @submission_date
   GROUP BY
     user_id
 ),
diff -bur --no-dereference bigquery-etl-main/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_users_services_daily_v1/metadata.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_users_services_daily_v1/metadata.yaml
--- bigquery-etl-main/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_users_services_daily_v1/metadata.yaml	2022-12-30 03:34:56.329322961 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_users_services_daily_v1/metadata.yaml	2022-12-30 03:28:05.000000000 +0000
@@ -9,9 +9,8 @@
   schedule: daily
 scheduling:
   dag_name: bqetl_fxa_events
-  date_partition_parameter: submission_date
 
 # Generated by bigquery_etl.dependency
 references:
   query.sql:
-  - moz-fx-data-shared-prod.firefox_accounts.fxa_all_events
+  - firefox_accounts.fxa_content_auth_oauth_events
diff -bur --no-dereference bigquery-etl-main/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_users_services_daily_v1/query.sql /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_users_services_daily_v1/query.sql
--- bigquery-etl-main/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_users_services_daily_v1/query.sql	2022-12-30 03:30:44.940227875 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_users_services_daily_v1/query.sql	2022-12-30 03:24:14.000000000 +0000
@@ -34,24 +34,18 @@
   )
 );
 
+  --
 WITH base AS (
   SELECT
-    `timestamp`,
-    event_type,
-    user_id,
-    app_version,
-    os_name,
-    os_version,
-    country,
-    `language`,
+    * REPLACE (
     -- cert_signed is specific to sync, but these events do not have the
     -- 'service' field populated, so we fill in the service name for this special case.
-    IF(service IS NULL AND event_type = 'fxa_activity - cert_signed', 'sync', service) AS service,
+      IF(service IS NULL AND event_type = 'fxa_activity - cert_signed', 'sync', service) AS service
+    )
   FROM
-    `moz-fx-data-shared-prod.firefox_accounts.fxa_all_events`
+    firefox_accounts.fxa_content_auth_oauth_events
   WHERE
-    event_category IN ('content', 'auth', 'oauth')
-    AND user_id IS NOT NULL
+    user_id IS NOT NULL
     AND event_type NOT IN ( --
       'fxa_email - bounced',
       'fxa_email - click',
diff -bur --no-dereference bigquery-etl-main/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_users_services_devices_daily_v1/metadata.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_users_services_devices_daily_v1/metadata.yaml
--- bigquery-etl-main/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_users_services_devices_daily_v1/metadata.yaml	2022-12-30 03:34:56.333322970 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_users_services_devices_daily_v1/metadata.yaml	2022-12-30 03:28:05.000000000 +0000
@@ -40,4 +40,4 @@
 # Generated by bigquery_etl.dependency
 references:
   query.sql:
-  - firefox_accounts.fxa_all_events
+  - moz-fx-data-shared-prod.firefox_accounts.fxa_content_auth_oauth_events
diff -bur --no-dereference bigquery-etl-main/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_users_services_devices_daily_v1/query.sql /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_users_services_devices_daily_v1/query.sql
--- bigquery-etl-main/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_users_services_devices_daily_v1/query.sql	2022-12-30 03:30:44.940227875 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_users_services_devices_daily_v1/query.sql	2022-12-30 03:24:14.000000000 +0000
@@ -18,12 +18,11 @@
     ua_version,
     ua_browser,
   FROM
-    `firefox_accounts.fxa_all_events`
+    `moz-fx-data-shared-prod.firefox_accounts.fxa_content_auth_oauth_events` -- TODO: this will need updated to fxa_all_events once unified
   WHERE
     DATE(`timestamp`)
     BETWEEN DATE_SUB(@submission_date, INTERVAL 1 DAY)
     AND @submission_date
-    AND event_category IN ('content', 'auth', 'oauth')
     -- re-using the filter from users_services_daily_v1 for consistency across the models
     -- at some point in the future we should re-evaluate this list
     AND event_type NOT IN ( --
diff -bur --no-dereference bigquery-etl-main/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_users_services_first_seen_v1/init.sql /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_users_services_first_seen_v1/init.sql
--- bigquery-etl-main/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_users_services_first_seen_v1/init.sql	2022-12-30 03:30:44.940227875 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_users_services_first_seen_v1/init.sql	2022-12-30 03:24:14.000000000 +0000
@@ -1,25 +1,18 @@
 CREATE OR REPLACE TABLE
-  `moz-fx-data-shared-prod.firefox_accounts_derived.fxa_users_services_first_seen_v1`
+  fxa_users_services_first_seen_v1
 PARTITION BY
   DATE(first_service_timestamp)
 CLUSTER BY
   service,
   user_id
 AS
-WITH fxa_content_auth_oauth AS (
+WITH base AS (
   SELECT
-    `timestamp`,
-    user_id,
-    IF(service IS NULL AND event_type = 'fxa_activity - cert_signed', 'sync', service) AS service,
-    os_name,
-    flow_id,
-    event_type,
-    country,
-    entrypoint,
+    * REPLACE (
+      IF(service IS NULL AND event_type = 'fxa_activity - cert_signed', 'sync', service) AS service
+    )
   FROM
-    `moz-fx-data-shared-prod.firefox_accounts.fxa_all_events`
-  WHERE
-    event_category IN ('content', 'auth', 'oauth')
+    firefox_accounts.fxa_content_auth_oauth_events
 ),
   -- use a window function to look within each USER and SERVICE for the first value of service, os, and country.
   -- also, get the first value of flow_id for later use and create a boolean column that is true if the first instance of a service usage includes a registration.
@@ -36,7 +29,7 @@
     udf.mode_last(ARRAY_AGG(flow_id) OVER w1_reversed) AS first_service_flow,
     LOGICAL_OR(IFNULL(event_type = 'fxa_reg - complete', FALSE)) OVER w1_reversed AS did_register
   FROM
-    fxa_content_auth_oauth
+    base
   WHERE
     (
       (event_type IN ('fxa_login - complete', 'fxa_reg - complete') AND service IS NOT NULL)
@@ -121,7 +114,7 @@
   FROM
     first_services_g s
   INNER JOIN
-    fxa_content_auth_oauth AS f
+    firefox_accounts.fxa_content_auth_oauth_events AS f
   ON
     s.first_service_flow = f.flow_id
   WHERE
diff -bur --no-dereference bigquery-etl-main/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_users_services_first_seen_v1/metadata.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_users_services_first_seen_v1/metadata.yaml
--- bigquery-etl-main/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_users_services_first_seen_v1/metadata.yaml	2022-12-30 03:34:56.349323006 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_users_services_first_seen_v1/metadata.yaml	2022-12-30 03:28:05.000000000 +0000
@@ -19,6 +19,6 @@
 # Generated by bigquery_etl.dependency
 references:
   init.sql:
-  - moz-fx-data-shared-prod.firefox_accounts.fxa_all_events
+  - firefox_accounts.fxa_content_auth_oauth_events
   query.sql:
-  - moz-fx-data-shared-prod.firefox_accounts.fxa_all_events
+  - firefox_accounts.fxa_content_auth_oauth_events
diff -bur --no-dereference bigquery-etl-main/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_users_services_first_seen_v1/query.sql /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_users_services_first_seen_v1/query.sql
--- bigquery-etl-main/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_users_services_first_seen_v1/query.sql	2022-12-30 03:30:44.940227875 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_users_services_first_seen_v1/query.sql	2022-12-30 03:24:14.000000000 +0000
@@ -1,17 +1,10 @@
-WITH fxa_content_auth_oauth AS (
+WITH base AS (
   SELECT
-    `timestamp`,
-    user_id,
-    IF(service IS NULL AND event_type = 'fxa_activity - cert_signed', 'sync', service) AS service,
-    os_name,
-    flow_id,
-    event_type,
-    country,
-    entrypoint,
+    * REPLACE (
+      IF(service IS NULL AND event_type = 'fxa_activity - cert_signed', 'sync', service) AS service
+    )
   FROM
-    `moz-fx-data-shared-prod.firefox_accounts.fxa_all_events`
-  WHERE
-    event_category IN ('content', 'auth', 'oauth')
+    firefox_accounts.fxa_content_auth_oauth_events
 ),
   -- use a window function to look within each USER and SERVICE for the first value of service, os, and country.
   -- also, get the first value of flow_id for later use and create a boolean column that is true if the first instance of a service usage includes a registration.
@@ -28,7 +21,7 @@
     udf.mode_last(ARRAY_AGG(flow_id) OVER w1_reversed) AS first_service_flow,
     LOGICAL_OR(IFNULL(event_type = 'fxa_reg - complete', FALSE)) OVER w1_reversed AS did_register
   FROM
-    fxa_content_auth_oauth
+    base
   WHERE
     (
       (event_type IN ('fxa_login - complete', 'fxa_reg - complete') AND service IS NOT NULL)
@@ -113,7 +106,7 @@
   FROM
     first_services_g s
   INNER JOIN
-    fxa_content_auth_oauth AS f
+    firefox_accounts.fxa_content_auth_oauth_events AS f
   ON
     s.first_service_flow = f.flow_id
   WHERE
diff -bur --no-dereference bigquery-etl-main/sql/moz-fx-data-shared-prod/firefox_desktop/baseline/schema.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_desktop/baseline/schema.yaml
--- bigquery-etl-main/sql/moz-fx-data-shared-prod/firefox_desktop/baseline/schema.yaml	2022-12-30 03:31:49.728469997 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_desktop/baseline/schema.yaml	2022-12-30 03:24:36.000000000 +0000
@@ -392,9 +392,6 @@
         '
       mode: NULLABLE
       name: glean_validation_first_run_hour
-      type: TIMESTAMP
-    - mode: NULLABLE
-      name: raw_glean_validation_first_run_hour
       type: STRING
     mode: NULLABLE
     name: datetime
diff -bur --no-dereference bigquery-etl-main/sql/moz-fx-data-shared-prod/firefox_desktop/baseline/view.sql /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_desktop/baseline/view.sql
--- bigquery-etl-main/sql/moz-fx-data-shared-prod/firefox_desktop/baseline/view.sql	2022-12-30 03:31:48.916467572 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_desktop/baseline/view.sql	2022-12-30 03:24:33.000000000 +0000
@@ -7,17 +7,6 @@
     mozfun.norm.metadata(metadata) AS metadata,
     mozfun.norm.glean_ping_info(ping_info) AS ping_info,
     mozfun.norm.glean_baseline_client_info(client_info, metrics) AS client_info,
-    (
-      SELECT AS STRUCT
-        metrics.* REPLACE (
-          STRUCT(
-            mozfun.glean.parse_datetime(
-              metrics.datetime.glean_validation_first_run_hour
-            ) AS glean_validation_first_run_hour,
-            metrics.datetime.glean_validation_first_run_hour AS raw_glean_validation_first_run_hour
-          ) AS datetime
-        )
-    ) AS metrics,
     'Firefox' AS normalized_app_name
   )
 FROM
diff -bur --no-dereference bigquery-etl-main/sql/moz-fx-data-shared-prod/firefox_desktop/metrics/schema.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_desktop/metrics/schema.yaml
--- bigquery-etl-main/sql/moz-fx-data-shared-prod/firefox_desktop/metrics/schema.yaml	2022-12-30 03:31:53.472482343 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_desktop/metrics/schema.yaml	2022-12-30 03:24:40.000000000 +0000
@@ -1329,9 +1329,6 @@
         '
       mode: NULLABLE
       name: glean_validation_first_run_hour
-      type: TIMESTAMP
-    - mode: NULLABLE
-      name: raw_glean_validation_first_run_hour
       type: STRING
     mode: NULLABLE
     name: datetime
diff -bur --no-dereference bigquery-etl-main/sql/moz-fx-data-shared-prod/firefox_desktop/metrics/view.sql /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_desktop/metrics/view.sql
--- bigquery-etl-main/sql/moz-fx-data-shared-prod/firefox_desktop/metrics/view.sql	2022-12-30 03:31:51.896477218 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_desktop/metrics/view.sql	2022-12-30 03:24:39.000000000 +0000
@@ -6,17 +6,6 @@
   * REPLACE (
     mozfun.norm.metadata(metadata) AS metadata,
     mozfun.norm.glean_ping_info(ping_info) AS ping_info,
-    (
-      SELECT AS STRUCT
-        metrics.* REPLACE (
-          STRUCT(
-            mozfun.glean.parse_datetime(
-              metrics.datetime.glean_validation_first_run_hour
-            ) AS glean_validation_first_run_hour,
-            metrics.datetime.glean_validation_first_run_hour AS raw_glean_validation_first_run_hour
-          ) AS datetime
-        )
-    ) AS metrics,
     'Firefox' AS normalized_app_name
   )
 FROM
diff -bur --no-dereference bigquery-etl-main/sql/moz-fx-data-shared-prod/firefox_desktop_background_update/background_update/schema.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_desktop_background_update/background_update/schema.yaml
--- bigquery-etl-main/sql/moz-fx-data-shared-prod/firefox_desktop_background_update/background_update/schema.yaml	2022-12-30 03:31:50.524472374 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_desktop_background_update/background_update/schema.yaml	2022-12-30 03:24:36.000000000 +0000
@@ -479,9 +479,6 @@
         '
       mode: NULLABLE
       name: background_update_targeting_env_current_date
-      type: TIMESTAMP
-    - mode: NULLABLE
-      name: raw_background_update_targeting_env_current_date
       type: STRING
     - description: 'The `environment.profileAgeCreated` of the default profile''s
         serialized targeting snapshot.
@@ -489,9 +486,6 @@
         '
       mode: NULLABLE
       name: background_update_targeting_env_profile_age
-      type: TIMESTAMP
-    - mode: NULLABLE
-      name: raw_background_update_targeting_env_profile_age
       type: STRING
     mode: NULLABLE
     name: datetime
diff -bur --no-dereference bigquery-etl-main/sql/moz-fx-data-shared-prod/firefox_desktop_background_update/background_update/view.sql /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_desktop_background_update/background_update/view.sql
--- bigquery-etl-main/sql/moz-fx-data-shared-prod/firefox_desktop_background_update/background_update/view.sql	2022-12-30 03:31:49.816470260 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_desktop_background_update/background_update/view.sql	2022-12-30 03:24:36.000000000 +0000
@@ -5,22 +5,7 @@
 SELECT
   * REPLACE (
     mozfun.norm.metadata(metadata) AS metadata,
-    mozfun.norm.glean_ping_info(ping_info) AS ping_info,
-    (
-      SELECT AS STRUCT
-        metrics.* REPLACE (
-          STRUCT(
-            mozfun.glean.parse_datetime(
-              metrics.datetime.background_update_targeting_env_current_date
-            ) AS background_update_targeting_env_current_date,
-            metrics.datetime.background_update_targeting_env_current_date AS raw_background_update_targeting_env_current_date,
-            mozfun.glean.parse_datetime(
-              metrics.datetime.background_update_targeting_env_profile_age
-            ) AS background_update_targeting_env_profile_age,
-            metrics.datetime.background_update_targeting_env_profile_age AS raw_background_update_targeting_env_profile_age
-          ) AS datetime
-        )
-    ) AS metrics
+    mozfun.norm.glean_ping_info(ping_info) AS ping_info
   )
 FROM
   `moz-fx-data-shared-prod.firefox_desktop_background_update_stable.background_update_v1`
diff -bur --no-dereference bigquery-etl-main/sql/moz-fx-data-shared-prod/firefox_desktop_background_update/baseline/schema.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_desktop_background_update/baseline/schema.yaml
--- bigquery-etl-main/sql/moz-fx-data-shared-prod/firefox_desktop_background_update/baseline/schema.yaml	2022-12-30 03:31:51.256474607 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_desktop_background_update/baseline/schema.yaml	2022-12-30 03:24:38.000000000 +0000
@@ -226,9 +226,6 @@
         '
       mode: NULLABLE
       name: glean_validation_first_run_hour
-      type: TIMESTAMP
-    - mode: NULLABLE
-      name: raw_glean_validation_first_run_hour
       type: STRING
     mode: NULLABLE
     name: datetime
diff -bur --no-dereference bigquery-etl-main/sql/moz-fx-data-shared-prod/firefox_desktop_background_update/baseline/view.sql /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_desktop_background_update/baseline/view.sql
--- bigquery-etl-main/sql/moz-fx-data-shared-prod/firefox_desktop_background_update/baseline/view.sql	2022-12-30 03:31:50.524472374 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_desktop_background_update/baseline/view.sql	2022-12-30 03:24:36.000000000 +0000
@@ -6,18 +6,7 @@
   * REPLACE (
     mozfun.norm.metadata(metadata) AS metadata,
     mozfun.norm.glean_ping_info(ping_info) AS ping_info,
-    mozfun.norm.glean_baseline_client_info(client_info, metrics) AS client_info,
-    (
-      SELECT AS STRUCT
-        metrics.* REPLACE (
-          STRUCT(
-            mozfun.glean.parse_datetime(
-              metrics.datetime.glean_validation_first_run_hour
-            ) AS glean_validation_first_run_hour,
-            metrics.datetime.glean_validation_first_run_hour AS raw_glean_validation_first_run_hour
-          ) AS datetime
-        )
-    ) AS metrics
+    mozfun.norm.glean_baseline_client_info(client_info, metrics) AS client_info
   )
 FROM
   `moz-fx-data-shared-prod.firefox_desktop_background_update_stable.baseline_v1`
diff -bur --no-dereference bigquery-etl-main/sql/moz-fx-data-shared-prod/firefox_desktop_background_update/metrics/schema.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_desktop_background_update/metrics/schema.yaml
--- bigquery-etl-main/sql/moz-fx-data-shared-prod/firefox_desktop_background_update/metrics/schema.yaml	2022-12-30 03:31:53.356481981 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_desktop_background_update/metrics/schema.yaml	2022-12-30 03:24:40.000000000 +0000
@@ -299,9 +299,6 @@
         '
       mode: NULLABLE
       name: glean_validation_first_run_hour
-      type: TIMESTAMP
-    - mode: NULLABLE
-      name: raw_glean_validation_first_run_hour
       type: STRING
     mode: NULLABLE
     name: datetime
diff -bur --no-dereference bigquery-etl-main/sql/moz-fx-data-shared-prod/firefox_desktop_background_update/metrics/view.sql /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_desktop_background_update/metrics/view.sql
--- bigquery-etl-main/sql/moz-fx-data-shared-prod/firefox_desktop_background_update/metrics/view.sql	2022-12-30 03:31:52.644479757 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_desktop_background_update/metrics/view.sql	2022-12-30 03:24:39.000000000 +0000
@@ -5,18 +5,7 @@
 SELECT
   * REPLACE (
     mozfun.norm.metadata(metadata) AS metadata,
-    mozfun.norm.glean_ping_info(ping_info) AS ping_info,
-    (
-      SELECT AS STRUCT
-        metrics.* REPLACE (
-          STRUCT(
-            mozfun.glean.parse_datetime(
-              metrics.datetime.glean_validation_first_run_hour
-            ) AS glean_validation_first_run_hour,
-            metrics.datetime.glean_validation_first_run_hour AS raw_glean_validation_first_run_hour
-          ) AS datetime
-        )
-    ) AS metrics
+    mozfun.norm.glean_ping_info(ping_info) AS ping_info
   )
 FROM
   `moz-fx-data-shared-prod.firefox_desktop_background_update_stable.metrics_v1`
diff -bur --no-dereference bigquery-etl-main/sql/moz-fx-data-shared-prod/glean_dictionary/page_view/schema.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/glean_dictionary/page_view/schema.yaml
--- bigquery-etl-main/sql/moz-fx-data-shared-prod/glean_dictionary/page_view/schema.yaml	2022-12-30 03:31:51.380475113 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/glean_dictionary/page_view/schema.yaml	2022-12-30 03:24:38.000000000 +0000
@@ -226,9 +226,6 @@
         '
       mode: NULLABLE
       name: page_loaded
-      type: TIMESTAMP
-    - mode: NULLABLE
-      name: raw_page_loaded
       type: STRING
     mode: NULLABLE
     name: datetime
diff -bur --no-dereference bigquery-etl-main/sql/moz-fx-data-shared-prod/glean_dictionary/page_view/view.sql /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/glean_dictionary/page_view/view.sql
--- bigquery-etl-main/sql/moz-fx-data-shared-prod/glean_dictionary/page_view/view.sql	2022-12-30 03:31:50.952473653 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/glean_dictionary/page_view/view.sql	2022-12-30 03:24:38.000000000 +0000
@@ -5,16 +5,7 @@
 SELECT
   * REPLACE (
     mozfun.norm.metadata(metadata) AS metadata,
-    mozfun.norm.glean_ping_info(ping_info) AS ping_info,
-    (
-      SELECT AS STRUCT
-        metrics.* REPLACE (
-          STRUCT(
-            mozfun.glean.parse_datetime(metrics.datetime.page_loaded) AS page_loaded,
-            metrics.datetime.page_loaded AS raw_page_loaded
-          ) AS datetime
-        )
-    ) AS metrics
+    mozfun.norm.glean_ping_info(ping_info) AS ping_info
   )
 FROM
   `moz-fx-data-shared-prod.glean_dictionary_stable.page_view_v1`
diff -bur --no-dereference bigquery-etl-main/sql/moz-fx-data-shared-prod/mlhackweek_search/action/view.sql /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/mlhackweek_search/action/view.sql
--- bigquery-etl-main/sql/moz-fx-data-shared-prod/mlhackweek_search/action/view.sql	2022-12-30 03:31:49.012467859 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/mlhackweek_search/action/view.sql	2022-12-30 03:24:33.000000000 +0000
@@ -5,22 +5,7 @@
 SELECT
   * REPLACE (
     mozfun.norm.metadata(metadata) AS metadata,
-    mozfun.norm.glean_ping_info(ping_info) AS ping_info,
-    (
-      SELECT AS STRUCT
-        metrics.* REPLACE (
-          STRUCT(
-            mozfun.glean.parse_datetime(
-              metrics.datetime.search_meta_url_select_timestamp
-            ) AS search_meta_url_select_timestamp,
-            metrics.datetime.search_meta_url_select_timestamp AS raw_search_meta_url_select_timestamp,
-            mozfun.glean.parse_datetime(
-              metrics.datetime.search_url_select_timestamp
-            ) AS search_url_select_timestamp,
-            metrics.datetime.search_url_select_timestamp AS raw_search_url_select_timestamp
-          ) AS datetime
-        )
-    ) AS metrics
+    mozfun.norm.glean_ping_info(ping_info) AS ping_info
   )
 FROM
   `moz-fx-data-shared-prod.mlhackweek_search_stable.action_v1`
diff -bur --no-dereference bigquery-etl-main/sql/moz-fx-data-shared-prod/mlhackweek_search/baseline/view.sql /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/mlhackweek_search/baseline/view.sql
--- bigquery-etl-main/sql/moz-fx-data-shared-prod/mlhackweek_search/baseline/view.sql	2022-12-30 03:31:49.468469221 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/mlhackweek_search/baseline/view.sql	2022-12-30 03:24:36.000000000 +0000
@@ -6,18 +6,7 @@
   * REPLACE (
     mozfun.norm.metadata(metadata) AS metadata,
     mozfun.norm.glean_ping_info(ping_info) AS ping_info,
-    mozfun.norm.glean_baseline_client_info(client_info, metrics) AS client_info,
-    (
-      SELECT AS STRUCT
-        metrics.* REPLACE (
-          STRUCT(
-            mozfun.glean.parse_datetime(
-              metrics.datetime.glean_validation_first_run_hour
-            ) AS glean_validation_first_run_hour,
-            metrics.datetime.glean_validation_first_run_hour AS raw_glean_validation_first_run_hour
-          ) AS datetime
-        )
-    ) AS metrics
+    mozfun.norm.glean_baseline_client_info(client_info, metrics) AS client_info
   )
 FROM
   `moz-fx-data-shared-prod.mlhackweek_search_stable.baseline_v1`
diff -bur --no-dereference bigquery-etl-main/sql/moz-fx-data-shared-prod/mlhackweek_search/custom/view.sql /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/mlhackweek_search/custom/view.sql
--- bigquery-etl-main/sql/moz-fx-data-shared-prod/mlhackweek_search/custom/view.sql	2022-12-30 03:31:49.876470439 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/mlhackweek_search/custom/view.sql	2022-12-30 03:24:36.000000000 +0000
@@ -5,18 +5,7 @@
 SELECT
   * REPLACE (
     mozfun.norm.metadata(metadata) AS metadata,
-    mozfun.norm.glean_ping_info(ping_info) AS ping_info,
-    (
-      SELECT AS STRUCT
-        metrics.* REPLACE (
-          STRUCT(
-            mozfun.glean.parse_datetime(
-              metrics.datetime.search_url_select_timestamp
-            ) AS search_url_select_timestamp,
-            metrics.datetime.search_url_select_timestamp AS raw_search_url_select_timestamp
-          ) AS datetime
-        )
-    ) AS metrics
+    mozfun.norm.glean_ping_info(ping_info) AS ping_info
   )
 FROM
   `moz-fx-data-shared-prod.mlhackweek_search_stable.custom_v1`
diff -bur --no-dereference bigquery-etl-main/sql/moz-fx-data-shared-prod/mlhackweek_search/metrics/view.sql /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/mlhackweek_search/metrics/view.sql
--- bigquery-etl-main/sql/moz-fx-data-shared-prod/mlhackweek_search/metrics/view.sql	2022-12-30 03:31:51.172474309 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/mlhackweek_search/metrics/view.sql	2022-12-30 03:24:37.000000000 +0000
@@ -5,18 +5,7 @@
 SELECT
   * REPLACE (
     mozfun.norm.metadata(metadata) AS metadata,
-    mozfun.norm.glean_ping_info(ping_info) AS ping_info,
-    (
-      SELECT AS STRUCT
-        metrics.* REPLACE (
-          STRUCT(
-            mozfun.glean.parse_datetime(
-              metrics.datetime.glean_validation_first_run_hour
-            ) AS glean_validation_first_run_hour,
-            metrics.datetime.glean_validation_first_run_hour AS raw_glean_validation_first_run_hour
-          ) AS datetime
-        )
-    ) AS metrics
+    mozfun.norm.glean_ping_info(ping_info) AS ping_info
   )
 FROM
   `moz-fx-data-shared-prod.mlhackweek_search_stable.metrics_v1`
diff -bur --no-dereference bigquery-etl-main/sql/moz-fx-data-shared-prod/monitoring/bigquery_table_storage/schema.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/monitoring/bigquery_table_storage/schema.yaml
--- bigquery-etl-main/sql/moz-fx-data-shared-prod/monitoring/bigquery_table_storage/schema.yaml	2022-12-30 03:33:10.304781299 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/monitoring/bigquery_table_storage/schema.yaml	2022-12-30 03:26:36.000000000 +0000
@@ -19,35 +19,35 @@
   mode: NULLABLE
   name: total_rows
   type: INTEGER
-- description: The total number of partitions present in the table
+- description: The number of partitions present in the table
   mode: NULLABLE
   name: total_partitions
   type: INTEGER
-- description: Total number of uncompressed bytes in the table
+- description: Total number of logical bytes in the table
   mode: NULLABLE
   name: total_logical_bytes
   type: INTEGER
-- description: Number of uncompressed bytes less than 90 days old
+- description: Number of logical bytes less than 90 days old
   mode: NULLABLE
   name: active_logical_bytes
   type: INTEGER
-- description: Number of uncompressed bytes more than 90 days old
+- description: Number of logical bytes more than 90 days old
   mode: NULLABLE
   name: long_term_logical_bytes
   type: INTEGER
-- description: Total number of compressed bytes used for storage
+- description: Total number of physical bytes used for storage
   mode: NULLABLE
   name: total_physical_bytes
   type: INTEGER
-- description: Number of compressed bytes less than 90 days old
+- description: Number of physical bytes less than 90 days old
   mode: NULLABLE
   name: active_physical_bytes
   type: INTEGER
-- description: Number of compressed bytes more than 90 days old
+- description: Number of physical bytes more than 90 days old
   mode: NULLABLE
   name: long_term_physical_bytes
   type: INTEGER
-- description: Number of compressed bytes for deleted or changed data
+- description: Number of physical bytes used by time travel
   mode: NULLABLE
   name: time_travel_physical_bytes
   type: INTEGER
diff -bur --no-dereference bigquery-etl-main/sql/moz-fx-data-shared-prod/monitoring/bigquery_table_storage_timeline_daily/schema.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/monitoring/bigquery_table_storage_timeline_daily/schema.yaml
--- bigquery-etl-main/sql/moz-fx-data-shared-prod/monitoring/bigquery_table_storage_timeline_daily/schema.yaml	2022-12-30 03:33:10.360781542 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/monitoring/bigquery_table_storage_timeline_daily/schema.yaml	2022-12-30 03:26:37.000000000 +0000
@@ -1,5 +1,5 @@
 fields:
-- description: Date of change to the data in the table.
+- description: Date of when storage was last recalculated
   mode: NULLABLE
   name: change_date
   type: DATE
@@ -23,7 +23,7 @@
   mode: NULLABLE
   name: creation_date
   type: DATE
-- description: The total number of changes to a table in one day
+- description: The total number of changes
   mode: NULLABLE
   name: change_count
   type: INTEGER
@@ -35,31 +35,31 @@
   mode: NULLABLE
   name: avg_total_partitions
   type: FLOAT
-- description: Average total number of uncompressed bytes in the table
+- description: Average total number of logical bytes in the table
   mode: NULLABLE
   name: avg_total_logical_bytes
   type: FLOAT
-- description: Average number of uncompressed bytes less than 90 days old
+- description: Average number of logical bytes less than 90 days old
   mode: NULLABLE
   name: avg_active_logical_bytes
   type: FLOAT
-- description: Average number of uncompressed bytes more than 90 days old
+- description: Average number of logical bytes more than 90 days old
   mode: NULLABLE
   name: avg_long_term_logical_bytes
   type: FLOAT
-- description: Average total number of compressed  bytes used for storage
+- description: Average total number of physical bytes used for storage
   mode: NULLABLE
   name: avg_total_physical_bytes
   type: FLOAT
-- description: Average number of compressed bytes less than 90 days old
+- description: Average number of physical bytes less than 90 days old
   mode: NULLABLE
   name: avg_active_physical_bytes
   type: FLOAT
-- description: Average number of compressed bytes more than 90 days old
+- description: Average number of physical bytes more than 90 days old
   mode: NULLABLE
   name: avg_long_term_physical_bytes
   type: FLOAT
-- description: Average number of compressed bytes for deleted or changed data
+- description: Average number of physical bytes used by time travel
   mode: NULLABLE
   name: avg_time_travel_physical_bytes
   type: FLOAT
Only in bigquery-etl-main/sql/moz-fx

⚠️ Only part of the diff is displayed.

dataops-ci-bot · 2022-12-30T13:58:43Z

Integration report for "reformat query files for circle ci error fixing"

`sql.diff`

Click to expand!

diff -bur --no-dereference bigquery-etl-main/sql/moz-fx-cjms-prod-f3c7/cjms_bigquery/flows_v1/metadata.yaml /tmp/workspace/generated-sql/sql/moz-fx-cjms-prod-f3c7/cjms_bigquery/flows_v1/metadata.yaml
--- bigquery-etl-main/sql/moz-fx-cjms-prod-f3c7/cjms_bigquery/flows_v1/metadata.yaml	2022-12-30 13:57:46.893877827 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-cjms-prod-f3c7/cjms_bigquery/flows_v1/metadata.yaml	2022-12-30 13:50:43.000000000 +0000
@@ -13,4 +13,4 @@
 # Generated by bigquery_etl.dependency
 references:
   query.sql:
-  - moz-fx-data-shared-prod.firefox_accounts.fxa_all_events
+  - mozdata.firefox_accounts.fxa_content_auth_stdout_events
diff -bur --no-dereference bigquery-etl-main/sql/moz-fx-cjms-prod-f3c7/cjms_bigquery/flows_v1/query.sql /tmp/workspace/generated-sql/sql/moz-fx-cjms-prod-f3c7/cjms_bigquery/flows_v1/query.sql
--- bigquery-etl-main/sql/moz-fx-cjms-prod-f3c7/cjms_bigquery/flows_v1/query.sql	2022-12-30 13:53:30.857312700 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-cjms-prod-f3c7/cjms_bigquery/flows_v1/query.sql	2022-12-30 13:45:37.000000000 +0000
@@ -1,14 +1,3 @@
-WITH fxa_events AS (
-  SELECT
-    `timestamp`,
-    user_id,
-    flow_id,
-  FROM
-    `moz-fx-data-shared-prod.firefox_accounts.fxa_all_events`
-  WHERE
-    DATE(`timestamp`) = @submission_date
-    AND event_category IN ('content', 'auth', 'stdout')
-)
 SELECT
   DATE(`timestamp`) AS submission_date,
   flow_id,
@@ -25,9 +14,10 @@
       1
   )[SAFE_OFFSET(0)].*,
 FROM
-  fxa_events
+  mozdata.firefox_accounts.fxa_content_auth_stdout_events
 WHERE
-  flow_id IS NOT NULL
+  DATE(`timestamp`) = @submission_date
+  AND flow_id IS NOT NULL
 GROUP BY
   submission_date,
   flow_id
diff -bur --no-dereference bigquery-etl-main/sql/moz-fx-data-shared-prod/bedrock/interaction/schema.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/bedrock/interaction/schema.yaml
--- bigquery-etl-main/sql/moz-fx-data-shared-prod/bedrock/interaction/schema.yaml	2022-12-30 13:54:34.153452947 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/bedrock/interaction/schema.yaml	2022-12-30 13:46:00.000000000 +0000
@@ -226,9 +226,6 @@
         '
       mode: NULLABLE
       name: page_viewed
-      type: TIMESTAMP
-    - mode: NULLABLE
-      name: raw_page_viewed
       type: STRING
     mode: NULLABLE
     name: datetime
diff -bur --no-dereference bigquery-etl-main/sql/moz-fx-data-shared-prod/bedrock/interaction/view.sql /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/bedrock/interaction/view.sql
--- bigquery-etl-main/sql/moz-fx-data-shared-prod/bedrock/interaction/view.sql	2022-12-30 13:54:33.625451777 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/bedrock/interaction/view.sql	2022-12-30 13:46:00.000000000 +0000
@@ -5,16 +5,7 @@
 SELECT
   * REPLACE (
     mozfun.norm.metadata(metadata) AS metadata,
-    mozfun.norm.glean_ping_info(ping_info) AS ping_info,
-    (
-      SELECT AS STRUCT
-        metrics.* REPLACE (
-          STRUCT(
-            mozfun.glean.parse_datetime(metrics.datetime.page_viewed) AS page_viewed,
-            metrics.datetime.page_viewed AS raw_page_viewed
-          ) AS datetime
-        )
-    ) AS metrics
+    mozfun.norm.glean_ping_info(ping_info) AS ping_info
   )
 FROM
   `moz-fx-data-shared-prod.bedrock_stable.interaction_v1`
diff -bur --no-dereference bigquery-etl-main/sql/moz-fx-data-shared-prod/bedrock/non_interaction/schema.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/bedrock/non_interaction/schema.yaml
--- bigquery-etl-main/sql/moz-fx-data-shared-prod/bedrock/non_interaction/schema.yaml	2022-12-30 13:54:34.601453940 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/bedrock/non_interaction/schema.yaml	2022-12-30 13:46:01.000000000 +0000
@@ -226,9 +226,6 @@
         '
       mode: NULLABLE
       name: page_viewed
-      type: TIMESTAMP
-    - mode: NULLABLE
-      name: raw_page_viewed
       type: STRING
     mode: NULLABLE
     name: datetime
diff -bur --no-dereference bigquery-etl-main/sql/moz-fx-data-shared-prod/bedrock/non_interaction/view.sql /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/bedrock/non_interaction/view.sql
--- bigquery-etl-main/sql/moz-fx-data-shared-prod/bedrock/non_interaction/view.sql	2022-12-30 13:54:34.153452947 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/bedrock/non_interaction/view.sql	2022-12-30 13:46:00.000000000 +0000
@@ -5,16 +5,7 @@
 SELECT
   * REPLACE (
     mozfun.norm.metadata(metadata) AS metadata,
-    mozfun.norm.glean_ping_info(ping_info) AS ping_info,
-    (
-      SELECT AS STRUCT
-        metrics.* REPLACE (
-          STRUCT(
-            mozfun.glean.parse_datetime(metrics.datetime.page_viewed) AS page_viewed,
-            metrics.datetime.page_viewed AS raw_page_viewed
-          ) AS datetime
-        )
-    ) AS metrics
+    mozfun.norm.glean_ping_info(ping_info) AS ping_info
   )
 FROM
   `moz-fx-data-shared-prod.bedrock_stable.non_interaction_v1`
diff -bur --no-dereference bigquery-etl-main/sql/moz-fx-data-shared-prod/bedrock/page_view/schema.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/bedrock/page_view/schema.yaml
--- bigquery-etl-main/sql/moz-fx-data-shared-prod/bedrock/page_view/schema.yaml	2022-12-30 13:54:35.157455172 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/bedrock/page_view/schema.yaml	2022-12-30 13:46:01.000000000 +0000
@@ -226,9 +226,6 @@
         '
       mode: NULLABLE
       name: page_viewed
-      type: TIMESTAMP
-    - mode: NULLABLE
-      name: raw_page_viewed
       type: STRING
     mode: NULLABLE
     name: datetime
diff -bur --no-dereference bigquery-etl-main/sql/moz-fx-data-shared-prod/bedrock/page_view/view.sql /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/bedrock/page_view/view.sql
--- bigquery-etl-main/sql/moz-fx-data-shared-prod/bedrock/page_view/view.sql	2022-12-30 13:54:34.601453940 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/bedrock/page_view/view.sql	2022-12-30 13:46:01.000000000 +0000
@@ -5,16 +5,7 @@
 SELECT
   * REPLACE (
     mozfun.norm.metadata(metadata) AS metadata,
-    mozfun.norm.glean_ping_info(ping_info) AS ping_info,
-    (
-      SELECT AS STRUCT
-        metrics.* REPLACE (
-          STRUCT(
-            mozfun.glean.parse_datetime(metrics.datetime.page_viewed) AS page_viewed,
-            metrics.datetime.page_viewed AS raw_page_viewed
-          ) AS datetime
-        )
-    ) AS metrics
+    mozfun.norm.glean_ping_info(ping_info) AS ping_info
   )
 FROM
   `moz-fx-data-shared-prod.bedrock_stable.page_view_v1`
diff -bur --no-dereference bigquery-etl-main/sql/moz-fx-data-shared-prod/burnham/baseline/schema.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/burnham/baseline/schema.yaml
--- bigquery-etl-main/sql/moz-fx-data-shared-prod/burnham/baseline/schema.yaml	2022-12-30 13:54:35.629456218 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/burnham/baseline/schema.yaml	2022-12-30 13:46:02.000000000 +0000
@@ -358,9 +358,6 @@
         '
       mode: NULLABLE
       name: glean_validation_first_run_hour
-      type: TIMESTAMP
-    - mode: NULLABLE
-      name: raw_glean_validation_first_run_hour
       type: STRING
     mode: NULLABLE
     name: datetime
diff -bur --no-dereference bigquery-etl-main/sql/moz-fx-data-shared-prod/burnham/baseline/view.sql /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/burnham/baseline/view.sql
--- bigquery-etl-main/sql/moz-fx-data-shared-prod/burnham/baseline/view.sql	2022-12-30 13:54:35.157455172 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/burnham/baseline/view.sql	2022-12-30 13:46:01.000000000 +0000
@@ -6,18 +6,7 @@
   * REPLACE (
     mozfun.norm.metadata(metadata) AS metadata,
     mozfun.norm.glean_ping_info(ping_info) AS ping_info,
-    mozfun.norm.glean_baseline_client_info(client_info, metrics) AS client_info,
-    (
-      SELECT AS STRUCT
-        metrics.* REPLACE (
-          STRUCT(
-            mozfun.glean.parse_datetime(
-              metrics.datetime.glean_validation_first_run_hour
-            ) AS glean_validation_first_run_hour,
-            metrics.datetime.glean_validation_first_run_hour AS raw_glean_validation_first_run_hour
-          ) AS datetime
-        )
-    ) AS metrics
+    mozfun.norm.glean_baseline_client_info(client_info, metrics) AS client_info
   )
 FROM
   `moz-fx-data-shared-prod.burnham_stable.baseline_v1`
diff -bur --no-dereference bigquery-etl-main/sql/moz-fx-data-shared-prod/burnham/metrics/schema.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/burnham/metrics/schema.yaml
--- bigquery-etl-main/sql/moz-fx-data-shared-prod/burnham/metrics/schema.yaml	2022-12-30 13:54:36.209457504 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/burnham/metrics/schema.yaml	2022-12-30 13:46:02.000000000 +0000
@@ -515,9 +515,6 @@
         '
       mode: NULLABLE
       name: glean_validation_first_run_hour
-      type: TIMESTAMP
-    - mode: NULLABLE
-      name: raw_glean_validation_first_run_hour
       type: STRING
     mode: NULLABLE
     name: datetime
diff -bur --no-dereference bigquery-etl-main/sql/moz-fx-data-shared-prod/burnham/metrics/view.sql /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/burnham/metrics/view.sql
--- bigquery-etl-main/sql/moz-fx-data-shared-prod/burnham/metrics/view.sql	2022-12-30 13:54:35.613456183 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/burnham/metrics/view.sql	2022-12-30 13:46:02.000000000 +0000
@@ -5,18 +5,7 @@
 SELECT
   * REPLACE (
     mozfun.norm.metadata(metadata) AS metadata,
-    mozfun.norm.glean_ping_info(ping_info) AS ping_info,
-    (
-      SELECT AS STRUCT
-        metrics.* REPLACE (
-          STRUCT(
-            mozfun.glean.parse_datetime(
-              metrics.datetime.glean_validation_first_run_hour
-            ) AS glean_validation_first_run_hour,
-            metrics.datetime.glean_validation_first_run_hour AS raw_glean_validation_first_run_hour
-          ) AS datetime
-        )
-    ) AS metrics
+    mozfun.norm.glean_ping_info(ping_info) AS ping_info
   )
 FROM
   `moz-fx-data-shared-prod.burnham_stable.metrics_v1`
Only in bigquery-etl-main/sql/moz-fx-data-shared-prod/fenix: firefox_android_clients
diff -bur --no-dereference bigquery-etl-main/sql/moz-fx-data-shared-prod/fenix_derived/attributable_clients_v1/init.sql /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/fenix_derived/attributable_clients_v1/init.sql
--- bigquery-etl-main/sql/moz-fx-data-shared-prod/fenix_derived/attributable_clients_v1/init.sql	2022-12-30 13:53:30.873312735 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/fenix_derived/attributable_clients_v1/init.sql	2022-12-30 13:45:37.000000000 +0000
@@ -63,15 +63,28 @@
     ),
     adjust_client AS (
       SELECT
-        client_id,
-        adjust_network,
-        adjust_ad_group AS adjust_adgroup,
-        adjust_campaign,
-        adjust_creative,
+        client_info.client_id AS client_id,
+        ARRAY_AGG(metrics.string.first_session_network ORDER BY submission_timestamp)[
+          SAFE_OFFSET(0)
+        ] AS adjust_network,
+        ARRAY_AGG(metrics.string.first_session_adgroup ORDER BY submission_timestamp)[
+          SAFE_OFFSET(0)
+        ] AS adjust_adgroup,
+        ARRAY_AGG(metrics.string.first_session_campaign ORDER BY submission_timestamp)[
+          SAFE_OFFSET(0)
+        ] AS adjust_campaign,
+        ARRAY_AGG(metrics.string.first_session_creative ORDER BY submission_timestamp)[
+          SAFE_OFFSET(0)
+        ] AS adjust_creative,
+        MIN(DATE(submission_timestamp)) AS first_session_date
       FROM
-        mozdata.fenix.firefox_android_clients
+        `mozdata.fenix.first_session`
       WHERE
-        adjust_network != 'Unknown'
+        DATE(submission_timestamp) >= first_date
+        AND metrics.string.first_session_network IS NOT NULL
+        AND metrics.string.first_session_network != ''
+      GROUP BY
+        client_id
     )
     SELECT
       submission_date,
diff -bur --no-dereference bigquery-etl-main/sql/moz-fx-data-shared-prod/fenix_derived/attributable_clients_v1/metadata.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/fenix_derived/attributable_clients_v1/metadata.yaml
--- bigquery-etl-main/sql/moz-fx-data-shared-prod/fenix_derived/attributable_clients_v1/metadata.yaml	2022-12-30 13:57:47.405878986 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/fenix_derived/attributable_clients_v1/metadata.yaml	2022-12-30 13:50:44.000000000 +0000
@@ -23,7 +23,7 @@
     execution_delta: 1h
   - task_id: search_derived__mobile_search_clients_daily__v1
     dag_name: bqetl_mobile_search
-    execution_delta: 0h
+    execution_delta: 1h
 bigquery:
   time_partitioning:
     field: submission_date
@@ -41,9 +41,9 @@
   - moz-fx-data-shared-prod.search_derived.mobile_search_clients_daily_v1
   - mozdata.fenix.baseline
   - mozdata.fenix.baseline_clients_first_seen
-  - mozdata.fenix.firefox_android_clients
+  - mozdata.fenix.first_session
   query.sql:
   - moz-fx-data-shared-prod.search_derived.mobile_search_clients_daily_v1
   - mozdata.fenix.baseline
   - mozdata.fenix.baseline_clients_first_seen
-  - mozdata.fenix.firefox_android_clients
+  - mozdata.fenix.first_session
diff -bur --no-dereference bigquery-etl-main/sql/moz-fx-data-shared-prod/fenix_derived/attributable_clients_v1/query.sql /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/fenix_derived/attributable_clients_v1/query.sql
--- bigquery-etl-main/sql/moz-fx-data-shared-prod/fenix_derived/attributable_clients_v1/query.sql	2022-12-30 13:53:30.873312735 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/fenix_derived/attributable_clients_v1/query.sql	2022-12-30 13:45:37.000000000 +0000
@@ -53,15 +53,28 @@
 ),
 adjust_client AS (
   SELECT
-    client_id,
-    adjust_network,
-    adjust_ad_group AS adjust_adgroup,
-    adjust_campaign,
-    adjust_creative,
+    client_info.client_id AS client_id,
+    ARRAY_AGG(metrics.string.first_session_network ORDER BY submission_timestamp)[
+      SAFE_OFFSET(0)
+    ] AS adjust_network,
+    ARRAY_AGG(metrics.string.first_session_adgroup ORDER BY submission_timestamp)[
+      SAFE_OFFSET(0)
+    ] AS adjust_adgroup,
+    ARRAY_AGG(metrics.string.first_session_campaign ORDER BY submission_timestamp)[
+      SAFE_OFFSET(0)
+    ] AS adjust_campaign,
+    ARRAY_AGG(metrics.string.first_session_creative ORDER BY submission_timestamp)[
+      SAFE_OFFSET(0)
+    ] AS adjust_creative,
+    MIN(DATE(submission_timestamp)) AS first_session_date
   FROM
-    mozdata.fenix.firefox_android_clients
+    `mozdata.fenix.first_session`
   WHERE
-    adjust_network != 'Unknown'
+    DATE(submission_timestamp) >= "2021-01-01"
+    AND metrics.string.first_session_network IS NOT NULL
+    AND metrics.string.first_session_network != ''
+  GROUP BY
+    client_id
 )
 SELECT
   submission_date,
Only in bigquery-etl-main/sql/moz-fx-data-shared-prod/fenix_derived: firefox_android_clients_v1
diff -bur --no-dereference bigquery-etl-main/sql/moz-fx-data-shared-prod/firefox_accounts/fxa_all_events/metadata.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_accounts/fxa_all_events/metadata.yaml
--- bigquery-etl-main/sql/moz-fx-data-shared-prod/firefox_accounts/fxa_all_events/metadata.yaml	2022-12-30 13:57:47.465879122 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_accounts/fxa_all_events/metadata.yaml	2022-12-30 13:50:44.000000000 +0000
@@ -6,4 +6,3 @@
   - moz-fx-data-shared-prod.firefox_accounts_derived.fxa_auth_events_v1
   - moz-fx-data-shared-prod.firefox_accounts_derived.fxa_content_events_v1
   - moz-fx-data-shared-prod.firefox_accounts_derived.fxa_oauth_events_v1
-  - moz-fx-data-shared-prod.firefox_accounts_derived.fxa_stdout_events_v1
diff -bur --no-dereference bigquery-etl-main/sql/moz-fx-data-shared-prod/firefox_accounts/fxa_all_events/view.sql /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_accounts/fxa_all_events/view.sql
--- bigquery-etl-main/sql/moz-fx-data-shared-prod/firefox_accounts/fxa_all_events/view.sql	2022-12-30 13:53:30.877312744 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_accounts/fxa_all_events/view.sql	2022-12-30 13:45:37.000000000 +0000
@@ -1,15 +1,9 @@
--- NOTE regarding oauth events:
--- oauth events have been merged into the auth table.
--- However, the oauth events table still contains around
--- 162 days of data from 2019H2 hence why it's still available
--- via this view.
 CREATE OR REPLACE VIEW
   `moz-fx-data-shared-prod.firefox_accounts.fxa_all_events`
 AS
 WITH fxa_auth_events AS (
   SELECT
-    `timestamp`,
-    receiveTimestamp,
+    timestamp AS submission_timestamp,
     jsonPayload.fields.user_id,
     jsonPayload.fields.country,
     jsonPayload.fields.language,
@@ -17,10 +11,7 @@
     jsonPayload.fields.os_name,
     jsonPayload.fields.os_version,
     jsonPayload.fields.event_type,
-    jsonPayload.logger,
-    jsonPayload.fields.user_properties,
-    jsonPayload.fields.event_properties,
-    jsonPayload.fields.device_id,
+    JSON_VALUE(jsonPayload.fields.event_properties, '$.service') AS service
   FROM
     `moz-fx-data-shared-prod.firefox_accounts_derived.fxa_auth_events_v1`
 ),
@@ -28,28 +19,23 @@
   -- but should always be included for a complete raw event log.
 fxa_auth_bounce_events AS (
   SELECT
-    `timestamp`,
-    receiveTimestamp,
+    timestamp AS submission_timestamp,
     jsonPayload.fields.user_id,
     CAST(
       NULL AS STRING
     ) AS country,  -- No country field in auth_bounces
     jsonPayload.fields.language,
     jsonPayload.fields.app_version,
-    CAST(NULL AS STRING) AS os_name,
-    CAST(NULL AS STRING) AS os_version,
+    CAST(NULL AS STRING),
+    CAST(NULL AS STRING),
     jsonPayload.fields.event_type,
-    jsonPayload.logger,
-    jsonPayload.fields.user_properties,
-    jsonPayload.fields.event_properties,
-    CAST(NULL AS STRING) AS device_id,
+    JSON_VALUE(jsonPayload.fields.event_properties, '$.service') AS service
   FROM
     `moz-fx-data-shared-prod.firefox_accounts_derived.fxa_auth_bounce_events_v1`
 ),
 fxa_content_events AS (
   SELECT
-    `timestamp`,
-    receiveTimestamp,
+    timestamp AS submission_timestamp,
     jsonPayload.fields.user_id,
     jsonPayload.fields.country,
     jsonPayload.fields.language,
@@ -57,18 +43,13 @@
     jsonPayload.fields.os_name,
     jsonPayload.fields.os_version,
     jsonPayload.fields.event_type,
-    jsonPayload.logger,
-    jsonPayload.fields.user_properties,
-    jsonPayload.fields.event_properties,
-    jsonPayload.fields.device_id,
+    JSON_VALUE(jsonPayload.fields.event_properties, '$.service') AS service
   FROM
     `moz-fx-data-shared-prod.firefox_accounts_derived.fxa_content_events_v1`
 ),
--- oauth events, see the note on top
 fxa_oauth_events AS (
   SELECT
-    `timestamp`,
-    receiveTimestamp,
+    timestamp AS submission_timestamp,
     jsonPayload.fields.user_id,
     CAST(NULL AS STRING) AS country,
     CAST(NULL AS STRING) AS language,
@@ -76,109 +57,26 @@
     CAST(NULL AS STRING) AS os_name,
     CAST(NULL AS STRING) AS os_version,
     jsonPayload.fields.event_type,
-    jsonPayload.logger,
-    jsonPayload.fields.user_properties,
-    jsonPayload.fields.event_properties,
-    CAST(NULL AS STRING) AS device_id,
+    JSON_VALUE(jsonPayload.fields.event_properties, '$.service') AS service
   FROM
     `moz-fx-data-shared-prod.firefox_accounts_derived.fxa_oauth_events_v1`
-),
-fxa_stdout_events AS (
-  SELECT
-    `timestamp`,
-    receiveTimestamp,
-    jsonPayload.fields.user_id,
-    CAST(NULL AS STRING) AS country,
-    jsonPayload.fields.language,
-    jsonPayload.fields.app_version,
-    jsonPayload.fields.os_name,
-    jsonPayload.fields.os_version,
-    jsonPayload.fields.event_type,
-    jsonPayload.logger,
-    jsonPayload.fields.user_properties,
-    jsonPayload.fields.event_properties,
-    jsonPayload.fields.device_id,
-  FROM
-    `moz-fx-data-shared-prod.firefox_accounts_derived.fxa_stdout_events_v1`
-),
-unioned AS (
-  SELECT
-    *,
-    'auth' AS event_category,
-  FROM
+)
+SELECT
+  *
+FROM
     fxa_auth_events
-  UNION ALL
-  SELECT
-    *,
-    'auth_bounce' AS event_category,
-  FROM
+UNION ALL
+SELECT
+  *
+FROM
     fxa_auth_bounce_events
-  UNION ALL
-  SELECT
-    *,
-    'content' AS event_category,
-  FROM
+UNION ALL
+SELECT
+  *
+FROM
     fxa_content_events
-  UNION ALL
-  -- oauth events, see the note on top
-  SELECT
-    *,
-    'oauth' AS event_category,
-  FROM
-    fxa_oauth_events
-  UNION ALL
-  SELECT
-    *,
-    'stdout' AS event_category,
-  FROM
-    fxa_stdout_events
-)
+UNION ALL
 SELECT
-  `timestamp`,
-  receiveTimestamp,
-  event_category,
-  event_type,
-  user_id,
-  device_id,
-  country,
-  `language`,
-  app_version,
-  os_name,
-  os_version,
-  user_properties,
-  event_properties,
-  -- extract user properties
-  JSON_VALUE(user_properties, "$.utm_term") AS utm_term,
-  JSON_VALUE(user_properties, "$.utm_source") AS utm_source,
-  JSON_VALUE(user_properties, "$.utm_medium") AS utm_medium,
-  JSON_VALUE(user_properties, "$.utm_campaign") AS utm_campaign,
-  JSON_VALUE(user_properties, "$.utm_content") AS utm_content,
-  JSON_VALUE(user_properties, "$.ua_version") AS ua_version,
-  JSON_VALUE(user_properties, "$.ua_browser") AS ua_browser,
-  JSON_VALUE(user_properties, "$.entrypoint") AS entrypoint,
-  JSON_VALUE(user_properties, "$.entrypoint_experiment") AS entrypoint_experiment,
-  JSON_VALUE(user_properties, "$.entrypoint_variation") AS entrypoint_variation,
-  JSON_VALUE(user_properties, "$.flow_id") AS flow_id,
-  JSON_VALUE(user_properties, "$.sync_device_count") AS sync_device_count,
-  JSON_VALUE(user_properties, "$.sync_active_devices_day") AS sync_active_devices_day,
-  JSON_VALUE(user_properties, "$.sync_active_devices_week") AS sync_active_devices_week,
-  JSON_VALUE(user_properties, "$.sync_active_devices_month") AS sync_active_devices_month,
-  -- extract event properties
-  JSON_VALUE(event_properties, "$.service") AS service,
-  JSON_VALUE(event_properties, "$.email_type") AS email_type,
-  JSON_VALUE(event_properties, "$.email_provider") AS email_provider,
-  JSON_VALUE(event_properties, "$.oauth_client_id") AS oauth_client_id,
-  JSON_VALUE(event_properties, "$.connect_device_flow") AS connect_device_flow,
-  JSON_VALUE(event_properties, "$.connect_device_os") AS connect_device_os,
-  JSON_VALUE(event_properties, "$.email_sender") AS email_sender,
-  JSON_VALUE(event_properties, "$.email_service") AS email_service,
-  JSON_VALUE(event_properties, "$.email_template") AS email_template,
-  JSON_VALUE(event_properties, "$.email_version") AS email_version,
-  JSON_VALUE(event_properties, "$.plan_id") AS plan_id,
-  JSON_VALUE(event_properties, "$.product_id") AS product_id,
-  JSON_VALUE(event_properties, "$.promotionCode") AS promotion_code,
-  JSON_VALUE(event_properties, "$.payment_provider") AS payment_provider,
-  JSON_VALUE(event_properties, "$.checkout_type") AS checkout_type,
-  JSON_VALUE(event_properties, "$.source_country") AS source_country,
+  *
 FROM
-  unioned
+  fxa_oauth_events
diff -bur --no-dereference bigquery-etl-main/sql/moz-fx-data-shared-prod/firefox_accounts/fxa_content_auth_events/view.sql /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_accounts/fxa_content_auth_events/view.sql
--- bigquery-etl-main/sql/moz-fx-data-shared-prod/firefox_accounts/fxa_content_auth_events/view.sql	2022-12-30 13:53:30.877312744 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_accounts/fxa_content_auth_events/view.sql	2022-12-30 13:45:37.000000000 +0000
@@ -1,25 +1,3 @@
------
---- ** DEPRECATION NOTICE **
----
---- There is currently an ongoing effort to deprecate this view
---- please use `firefox_accounts.fxa_all_events` view instead
---- in new queries.
----
---- Please filter on `event_category` field to limit your results
---- to events coming only from a specific fxa server like so:
---- WHERE event_category IN ('auth', 'stdout', ...)
---- Options include:
----   content
----   auth
----   stdout
----   oauth -- this has been deprecated and merged into fxa_auth_event
----   auth_bounce
---- to replicate results of this view use:
---- WHERE event_category IN (
----  'content',
----  'auth'
---- )
------
 CREATE OR REPLACE VIEW
   `moz-fx-data-shared-prod.firefox_accounts.fxa_content_auth_events`
 AS
diff -bur --no-dereference bigquery-etl-main/sql/moz-fx-data-shared-prod/firefox_accounts/fxa_content_auth_oauth_events/view.sql /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_accounts/fxa_content_auth_oauth_events/view.sql
--- bigquery-etl-main/sql/moz-fx-data-shared-prod/firefox_accounts/fxa_content_auth_oauth_events/view.sql	2022-12-30 13:53:30.877312744 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_accounts/fxa_content_auth_oauth_events/view.sql	2022-12-30 13:45:37.000000000 +0000
@@ -1,26 +1,3 @@
------
---- ** DEPRECATION NOTICE **
----
---- There is currently an ongoing effort to deprecate this view
---- please use `firefox_accounts.fxa_all_events` view instead
---- in new queries.
----
---- Please filter on `event_category` field to limit your results
---- to events coming only from a specific fxa server like so:
---- WHERE event_category IN ('auth', 'stdout', ...)
---- Options include:
----   content
----   auth
----   stdout
----   oauth -- this has been deprecated and merged into fxa_auth_event
----   auth_bounce
---- to replicate results of this view use:
---- WHERE event_category IN (
----  'content',
----  'auth',
---   'oauth'
---- )
------
 CREATE OR REPLACE VIEW
   `moz-fx-data-shared-prod.firefox_accounts.fxa_content_auth_oauth_events`
 AS
diff -bur --no-dereference bigquery-etl-main/sql/moz-fx-data-shared-prod/firefox_accounts/fxa_content_auth_stdout_events/view.sql /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_accounts/fxa_content_auth_stdout_events/view.sql
--- bigquery-etl-main/sql/moz-fx-data-shared-prod/firefox_accounts/fxa_content_auth_stdout_events/view.sql	2022-12-30 13:53:30.877312744 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_accounts/fxa_content_auth_stdout_events/view.sql	2022-12-30 13:45:37.000000000 +0000
@@ -1,26 +1,3 @@
------
---- ** DEPRECATION NOTICE **
----
---- There is currently an ongoing effort to deprecate this view
---- please use `firefox_accounts.fxa_all_events` view instead
---- in new queries.
----
---- Please filter on `event_category` field to limit your results
---- to events coming only from a specific fxa server like so:
---- WHERE event_category IN ('auth', 'stdout', ...)
---- Options include:
----   content
----   auth
----   stdout
----   oauth -- this has been deprecated and merged into fxa_auth_event
----   auth_bounce
---- to replicate results of this view use:
---- WHERE event_category IN (
----  'content',
----  'auth',
---   'stdout'
---- )
------
 CREATE OR REPLACE VIEW
   `moz-fx-data-shared-prod.firefox_accounts.fxa_content_auth_stdout_events`
 AS
diff -bur --no-dereference bigquery-etl-main/sql/moz-fx-data-shared-prod/firefox_accounts_derived/funnel_events_source_v1/metadata.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_accounts_derived/funnel_events_source_v1/metadata.yaml
--- bigquery-etl-main/sql/moz-fx-data-shared-prod/firefox_accounts_derived/funnel_events_source_v1/metadata.yaml	2022-12-30 13:57:47.609879447 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_accounts_derived/funnel_events_source_v1/metadata.yaml	2022-12-30 13:50:44.000000000 +0000
@@ -2,7 +2,7 @@
 description: |-
   The materialized events used to create the events_daily rollup.
   Not exposed as a view since it's not meant for consumption;
-  use firefox_accounts.fxa_all_events
+  use firefox_accounts.fxa_content_auth_oauth_events
   instead. This is materialized solely because
   the events_daily queries are too complex for BigQuery
   otherwise.
@@ -18,4 +18,4 @@
 # Generated by bigquery_etl.dependency
 references:
   query.sql:
-  - moz-fx-data-shared-prod.firefox_accounts.fxa_all_events
+  - moz-fx-data-shared-prod.firefox_accounts.fxa_content_auth_oauth_events
diff -bur --no-dereference bigquery-etl-main/sql/moz-fx-data-shared-prod/firefox_accounts_derived/funnel_events_source_v1/query.sql /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_accounts_derived/funnel_events_source_v1/query.sql
--- bigquery-etl-main/sql/moz-fx-data-shared-prod/firefox_accounts_derived/funnel_events_source_v1/query.sql	2022-12-30 13:53:30.877312744 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_accounts_derived/funnel_events_source_v1/query.sql	2022-12-30 13:45:37.000000000 +0000
@@ -1,26 +1,5 @@
-WITH fxa_events AS (
-  SELECT
-    `timestamp`,
-    user_id,
-    event_type,
-    service,
-    email_type,
-    oauth_client_id,
-    connect_device_flow,
-    connect_device_os,
-    sync_device_count,
-    email_sender,
-    email_service,
-    email_template,
-    email_version,
-  FROM
-    `moz-fx-data-shared-prod.firefox_accounts.fxa_all_events`
-  WHERE
-    DATE(`timestamp`) = @submission_date
-    AND event_category IN ('content', 'auth', 'oauth')
-)
 SELECT
-  DATE(`timestamp`) AS submission_date,
+  DATE(timestamp) AS submission_date,
   user_id AS client_id,
   `moz-fx-data-shared-prod`.udf.safe_sample_id(user_id) AS sample_id,
   SPLIT(event_type, ' - ')[OFFSET(0)] AS category,
@@ -40,4 +19,6 @@
   CAST([] AS ARRAY<STRUCT<key STRING, value STRING>>) AS experiments,
   *,
 FROM
-  fxa_events
+  `moz-fx-data-shared-prod.firefox_accounts.fxa_content_auth_oauth_events`
+WHERE
+  DATE(timestamp) = @submission_date
diff -bur --no-dereference bigquery-etl-main/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_users_daily_v1/query.sql /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_users_daily_v1/query.sql
--- bigquery-etl-main/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_users_daily_v1/query.sql	2022-12-30 13:53:30.881312753 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_users_daily_v1/query.sql	2022-12-30 13:45:37.000000000 +0000
@@ -35,7 +35,7 @@
 
 WITH windowed AS (
   SELECT
-    DATE(`timestamp`) AS submission_date,
+    DATE(submission_timestamp) AS submission_date,
     user_id,
     ROW_NUMBER() OVER w1_unframed AS _n,
     udf.mode_last(ARRAY_AGG(country) OVER w1) AS country,
@@ -49,10 +49,9 @@
       NOT (event_type = 'fxa_rp - engage' AND service = 'fx-monitor')
     ) OVER w1 = 0 AS monitor_only
   FROM
-    `firefox_accounts.fxa_all_events`
+    firefox_accounts.fxa_all_events
   WHERE
-    event_category IN ('auth', 'auth_bounce', 'content', 'oauth')
-    AND user_id IS NOT NULL
+    user_id IS NOT NULL
     AND event_type NOT IN ( --
       'fxa_email - bounced',
       'fxa_email - click',
@@ -69,14 +68,14 @@
       'sync - repair_triggered'
     )
     -- Reprocess all dates by running this query with --parameter=submission_date:DATE:NULL
-    AND (@submission_date IS NULL OR @submission_date = DATE(`timestamp`))
+    AND (@submission_date IS NULL OR @submission_date = DATE(submission_timestamp))
   WINDOW
     w1 AS (
       PARTITION BY
         user_id,
-        DATE(`timestamp`)
+        DATE(submission_timestamp)
       ORDER BY
-        `timestamp` --
+        submission_timestamp --
       ROWS BETWEEN
         UNBOUNDED PRECEDING
         AND UNBOUNDED FOLLOWING
@@ -85,9 +84,9 @@
     w1_unframed AS (
       PARTITION BY
         user_id,
-        DATE(`timestamp`)
+        DATE(submission_timestamp)
       ORDER BY
-        `timestamp`
+        submission_timestamp
     )
 )
 SELECT
diff -bur --no-dereference bigquery-etl-main/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_users_first_seen_v1/init.sql /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_users_first_seen_v1/init.sql
--- bigquery-etl-main/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_users_first_seen_v1/init.sql	2022-12-30 13:53:30.881312753 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_users_first_seen_v1/init.sql	2022-12-30 13:45:37.000000000 +0000
@@ -7,12 +7,11 @@
 AS
 SELECT
   user_id,
-  DATE(MIN(`timestamp`)) AS first_seen_date,
+  DATE(MIN(submission_timestamp)) AS first_seen_date,
   ARRAY_AGG(DISTINCT service IGNORE NULLS ORDER BY service) AS services_used,
 FROM
   firefox_accounts.fxa_all_events
 WHERE
-  `timestamp` > '2010-01-01'
-  AND event_category IN ('auth', 'auth_bounce', 'content', 'oauth')
+  submission_timestamp > '2010-01-01'
 GROUP BY
   user_id
diff -bur --no-dereference bigquery-etl-main/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_users_first_seen_v1/query.sql /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_users_first_seen_v1/query.sql
--- bigquery-etl-main/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_users_first_seen_v1/query.sql	2022-12-30 13:53:30.881312753 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_users_first_seen_v1/query.sql	2022-12-30 13:45:37.000000000 +0000
@@ -5,8 +5,7 @@
   FROM
     firefox_accounts.fxa_all_events
   WHERE
-    DATE(`timestamp`) = @submission_date
-    AND event_category IN ('auth', 'auth_bounce', 'content', 'oauth')
+    DATE(submission_timestamp) = @submission_date
   GROUP BY
     user_id
 ),
diff -bur --no-dereference bigquery-etl-main/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_users_services_daily_v1/metadata.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_users_services_daily_v1/metadata.yaml
--- bigquery-etl-main/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_users_services_daily_v1/metadata.yaml	2022-12-30 13:57:47.825879937 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_users_services_daily_v1/metadata.yaml	2022-12-30 13:50:44.000000000 +0000
@@ -9,9 +9,8 @@
   schedule: daily
 scheduling:
   dag_name: bqetl_fxa_events
-  date_partition_parameter: submission_date
 
 # Generated by bigquery_etl.dependency
 references:
   query.sql:
-  - moz-fx-data-shared-prod.firefox_accounts.fxa_all_events
+  - firefox_accounts.fxa_content_auth_oauth_events
diff -bur --no-dereference bigquery-etl-main/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_users_services_daily_v1/query.sql /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_users_services_daily_v1/query.sql
--- bigquery-etl-main/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_users_services_daily_v1/query.sql	2022-12-30 13:53:30.881312753 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_users_services_daily_v1/query.sql	2022-12-30 13:45:37.000000000 +0000
@@ -34,24 +34,18 @@
   )
 );
 
+  --
 WITH base AS (
   SELECT
-    `timestamp`,
-    event_type,
-    user_id,
-    app_version,
-    os_name,
-    os_version,
-    country,
-    `language`,
+    * REPLACE (
     -- cert_signed is specific to sync, but these events do not have the
     -- 'service' field populated, so we fill in the service name for this special case.
-    IF(service IS NULL AND event_type = 'fxa_activity - cert_signed', 'sync', service) AS service,
+      IF(service IS NULL AND event_type = 'fxa_activity - cert_signed', 'sync', service) AS service
+    )
   FROM
-    `moz-fx-data-shared-prod.firefox_accounts.fxa_all_events`
+    firefox_accounts.fxa_content_auth_oauth_events
   WHERE
-    event_category IN ('content', 'auth', 'oauth')
-    AND user_id IS NOT NULL
+    user_id IS NOT NULL
     AND event_type NOT IN ( --
       'fxa_email - bounced',
       'fxa_email - click',
diff -bur --no-dereference bigquery-etl-main/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_users_services_devices_daily_v1/metadata.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_users_services_devices_daily_v1/metadata.yaml
--- bigquery-etl-main/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_users_services_devices_daily_v1/metadata.yaml	2022-12-30 13:57:47.833879955 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_users_services_devices_daily_v1/metadata.yaml	2022-12-30 13:50:44.000000000 +0000
@@ -40,4 +40,4 @@
 # Generated by bigquery_etl.dependency
 references:
   query.sql:
-  - firefox_accounts.fxa_all_events
+  - moz-fx-data-shared-prod.firefox_accounts.fxa_content_auth_oauth_events
diff -bur --no-dereference bigquery-etl-main/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_users_services_devices_daily_v1/query.sql /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_users_services_devices_daily_v1/query.sql
--- bigquery-etl-main/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_users_services_devices_daily_v1/query.sql	2022-12-30 13:53:30.881312753 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_users_services_devices_daily_v1/query.sql	2022-12-30 13:45:37.000000000 +0000
@@ -18,12 +18,11 @@
     ua_version,
     ua_browser,
   FROM
-    `firefox_accounts.fxa_all_events`
+    `moz-fx-data-shared-prod.firefox_accounts.fxa_content_auth_oauth_events` -- TODO: this will need updated to fxa_all_events once unified
   WHERE
     DATE(`timestamp`)
     BETWEEN DATE_SUB(@submission_date, INTERVAL 1 DAY)
     AND @submission_date
-    AND event_category IN ('content', 'auth', 'oauth')
     -- re-using the filter from users_services_daily_v1 for consistency across the models
     -- at some point in the future we should re-evaluate this list
     AND event_type NOT IN ( --
diff -bur --no-dereference bigquery-etl-main/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_users_services_first_seen_v1/init.sql /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_users_services_first_seen_v1/init.sql
--- bigquery-etl-main/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_users_services_first_seen_v1/init.sql	2022-12-30 13:53:30.881312753 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_users_services_first_seen_v1/init.sql	2022-12-30 13:45:37.000000000 +0000
@@ -1,25 +1,18 @@
 CREATE OR REPLACE TABLE
-  `moz-fx-data-shared-prod.firefox_accounts_derived.fxa_users_services_first_seen_v1`
+  fxa_users_services_first_seen_v1
 PARTITION BY
   DATE(first_service_timestamp)
 CLUSTER BY
   service,
   user_id
 AS
-WITH fxa_content_auth_oauth AS (
+WITH base AS (
   SELECT
-    `timestamp`,
-    user_id,
-    IF(service IS NULL AND event_type = 'fxa_activity - cert_signed', 'sync', service) AS service,
-    os_name,
-    flow_id,
-    event_type,
-    country,
-    entrypoint,
+    * REPLACE (
+      IF(service IS NULL AND event_type = 'fxa_activity - cert_signed', 'sync', service) AS service
+    )
   FROM
-    `moz-fx-data-shared-prod.firefox_accounts.fxa_all_events`
-  WHERE
-    event_category IN ('content', 'auth', 'oauth')
+    firefox_accounts.fxa_content_auth_oauth_events
 ),
   -- use a window function to look within each USER and SERVICE for the first value of service, os, and country.
   -- also, get the first value of flow_id for later use and create a boolean column that is true if the first instance of a service usage includes a registration.
@@ -36,7 +29,7 @@
     udf.mode_last(ARRAY_AGG(flow_id) OVER w1_reversed) AS first_service_flow,
     LOGICAL_OR(IFNULL(event_type = 'fxa_reg - complete', FALSE)) OVER w1_reversed AS did_register
   FROM
-    fxa_content_auth_oauth
+    base
   WHERE
     (
       (event_type IN ('fxa_login - complete', 'fxa_reg - complete') AND service IS NOT NULL)
@@ -121,7 +114,7 @@
   FROM
     first_services_g s
   INNER JOIN
-    fxa_content_auth_oauth AS f
+    firefox_accounts.fxa_content_auth_oauth_events AS f
   ON
     s.first_service_flow = f.flow_id
   WHERE
diff -bur --no-dereference bigquery-etl-main/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_users_services_first_seen_v1/metadata.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_users_services_first_seen_v1/metadata.yaml
--- bigquery-etl-main/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_users_services_first_seen_v1/metadata.yaml	2022-12-30 13:57:47.845879982 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_users_services_first_seen_v1/metadata.yaml	2022-12-30 13:50:44.000000000 +0000
@@ -19,6 +19,6 @@
 # Generated by bigquery_etl.dependency
 references:
   init.sql:
-  - moz-fx-data-shared-prod.firefox_accounts.fxa_all_events
+  - firefox_accounts.fxa_content_auth_oauth_events
   query.sql:
-  - moz-fx-data-shared-prod.firefox_accounts.fxa_all_events
+  - firefox_accounts.fxa_content_auth_oauth_events
diff -bur --no-dereference bigquery-etl-main/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_users_services_first_seen_v1/query.sql /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_users_services_first_seen_v1/query.sql
--- bigquery-etl-main/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_users_services_first_seen_v1/query.sql	2022-12-30 13:53:30.881312753 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_users_services_first_seen_v1/query.sql	2022-12-30 13:45:37.000000000 +0000
@@ -1,17 +1,10 @@
-WITH fxa_content_auth_oauth AS (
+WITH base AS (
   SELECT
-    `timestamp`,
-    user_id,
-    IF(service IS NULL AND event_type = 'fxa_activity - cert_signed', 'sync', service) AS service,
-    os_name,
-    flow_id,
-    event_type,
-    country,
-    entrypoint,
+    * REPLACE (
+      IF(service IS NULL AND event_type = 'fxa_activity - cert_signed', 'sync', service) AS service
+    )
   FROM
-    `moz-fx-data-shared-prod.firefox_accounts.fxa_all_events`
-  WHERE
-    event_category IN ('content', 'auth', 'oauth')
+    firefox_accounts.fxa_content_auth_oauth_events
 ),
   -- use a window function to look within each USER and SERVICE for the first value of service, os, and country.
   -- also, get the first value of flow_id for later use and create a boolean column that is true if the first instance of a service usage includes a registration.
@@ -28,7 +21,7 @@
     udf.mode_last(ARRAY_AGG(flow_id) OVER w1_reversed) AS first_service_flow,
     LOGICAL_OR(IFNULL(event_type = 'fxa_reg - complete', FALSE)) OVER w1_reversed AS did_register
   FROM
-    fxa_content_auth_oauth
+    base
   WHERE
     (
       (event_type IN ('fxa_login - complete', 'fxa_reg - complete') AND service IS NOT NULL)
@@ -113,7 +106,7 @@
   FROM
     first_services_g s
   INNER JOIN
-    fxa_content_auth_oauth AS f
+    firefox_accounts.fxa_content_auth_oauth_events AS f
   ON
     s.first_service_flow = f.flow_id
   WHERE
diff -bur --no-dereference bigquery-etl-main/sql/moz-fx-data-shared-prod/firefox_desktop/baseline/schema.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_desktop/baseline/schema.yaml
--- bigquery-etl-main/sql/moz-fx-data-shared-prod/firefox_desktop/baseline/schema.yaml	2022-12-30 13:54:33.841452255 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_desktop/baseline/schema.yaml	2022-12-30 13:45:59.000000000 +0000
@@ -392,9 +392,6 @@
         '
       mode: NULLABLE
       name: glean_validation_first_run_hour
-      type: TIMESTAMP
-    - mode: NULLABLE
-      name: raw_glean_validation_first_run_hour
       type: STRING
     mode: NULLABLE
     name: datetime
diff -bur --no-dereference bigquery-etl-main/sql/moz-fx-data-shared-prod/firefox_desktop/baseline/view.sql /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_desktop/baseline/view.sql
--- bigquery-etl-main/sql/moz-fx-data-shared-prod/firefox_desktop/baseline/view.sql	2022-12-30 13:54:32.577449454 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_desktop/baseline/view.sql	2022-12-30 13:45:56.000000000 +0000
@@ -7,17 +7,6 @@
     mozfun.norm.metadata(metadata) AS metadata,
     mozfun.norm.glean_ping_info(ping_info) AS ping_info,
     mozfun.norm.glean_baseline_client_info(client_info, metrics) AS client_info,
-    (
-      SELECT AS STRUCT
-        metrics.* REPLACE (
-          STRUCT(
-            mozfun.glean.parse_datetime(
-              metrics.datetime.glean_validation_first_run_hour
-            ) AS glean_validation_first_run_hour,
-            metrics.datetime.glean_validation_first_run_hour AS raw_glean_validation_first_run_hour
-          ) AS datetime
-        )
-    ) AS metrics,
     'Firefox' AS normalized_app_name
   )
 FROM
diff -bur --no-dereference bigquery-etl-main/sql/moz-fx-data-shared-prod/firefox_desktop/metrics/schema.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_desktop/metrics/schema.yaml
--- bigquery-etl-main/sql/moz-fx-data-shared-prod/firefox_desktop/metrics/schema.yaml	2022-12-30 13:54:38.761463159 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_desktop/metrics/schema.yaml	2022-12-30 13:46:03.000000000 +0000
@@ -1329,9 +1329,6 @@
         '
       mode: NULLABLE
       name: glean_validation_first_run_hour
-      type: TIMESTAMP
-    - mode: NULLABLE
-      name: raw_glean_validation_first_run_hour
       type: STRING
     mode: NULLABLE
     name: datetime
diff -bur --no-dereference bigquery-etl-main/sql/moz-fx-data-shared-prod/firefox_desktop/metrics/view.sql /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_desktop/metrics/view.sql
--- bigquery-etl-main/sql/moz-fx-data-shared-prod/firefox_desktop/metrics/view.sql	2022-12-30 13:54:36.749458700 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_desktop/metrics/view.sql	2022-12-30 13:46:02.000000000 +0000
@@ -6,17 +6,6 @@
   * REPLACE (
     mozfun.norm.metadata(metadata) AS metadata,
     mozfun.norm.glean_ping_info(ping_info) AS ping_info,
-    (
-      SELECT AS STRUCT
-        metrics.* REPLACE (
-          STRUCT(
-            mozfun.glean.parse_datetime(
-              metrics.datetime.glean_validation_first_run_hour
-            ) AS glean_validation_first_run_hour,
-            metrics.datetime.glean_validation_first_run_hour AS raw_glean_validation_first_run_hour
-          ) AS datetime
-        )
-    ) AS metrics,
     'Firefox' AS normalized_app_name
   )
 FROM
diff -bur --no-dereference bigquery-etl-main/sql/moz-fx-data-shared-prod/firefox_desktop_background_update/background_update/schema.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_desktop_background_update/background_update/schema.yaml
--- bigquery-etl-main/sql/moz-fx-data-shared-prod/firefox_desktop_background_update/background_update/schema.yaml	2022-12-30 13:54:34.177453000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_desktop_background_update/background_update/schema.yaml	2022-12-30 13:45:59.000000000 +0000
@@ -479,9 +479,6 @@
         '
       mode: NULLABLE
       name: background_update_targeting_env_current_date
-      type: TIMESTAMP
-    - mode: NULLABLE
-      name: raw_background_update_targeting_env_current_date
       type: STRING
     - description: 'The `environment.profileAgeCreated` of the default profile''s
         serialized targeting snapshot.
@@ -489,9 +486,6 @@
         '
       mode: NULLABLE
       name: background_update_targeting_env_profile_age
-      type: TIMESTAMP
-    - mode: NULLABLE
-      name: raw_background_update_targeting_env_profile_age
       type: STRING
     mode: NULLABLE
     name: datetime
diff -bur --no-dereference bigquery-etl-main/sql/moz-fx-data-shared-prod/firefox_desktop_background_update/background_update/view.sql /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_desktop_background_update/background_update/view.sql
--- bigquery-etl-main/sql/moz-fx-data-shared-prod/firefox_desktop_background_update/background_update/view.sql	2022-12-30 13:54:33.441451370 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_desktop_background_update/background_update/view.sql	2022-12-30 13:45:58.000000000 +0000
@@ -5,22 +5,7 @@
 SELECT
   * REPLACE (
     mozfun.norm.metadata(metadata) AS metadata,
-    mozfun.norm.glean_ping_info(ping_info) AS ping_info,
-    (
-      SELECT AS STRUCT
-        metrics.* REPLACE (
-          STRUCT(
-            mozfun.glean.parse_datetime(
-              metrics.datetime.background_update_targeting_env_current_date
-            ) AS background_update_targeting_env_current_date,
-            metrics.datetime.background_update_targeting_env_current_date AS raw_background_update_targeting_env_current_date,
-            mozfun.glean.parse_datetime(
-              metrics.datetime.background_update_targeting_env_profile_age
-            ) AS background_update_targeting_env_profile_age,
-            metrics.datetime.background_update_targeting_env_profile_age AS raw_background_update_targeting_env_profile_age
-          ) AS datetime
-        )
-    ) AS metrics
+    mozfun.norm.glean_ping_info(ping_info) AS ping_info
   )
 FROM
   `moz-fx-data-shared-prod.firefox_desktop_background_update_stable.background_update_v1`
diff -bur --no-dereference bigquery-etl-main/sql/moz-fx-data-shared-prod/firefox_desktop_background_update/baseline/schema.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_desktop_background_update/baseline/schema.yaml
--- bigquery-etl-main/sql/moz-fx-data-shared-prod/firefox_desktop_background_update/baseline/schema.yaml	2022-12-30 13:54:34.889454578 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_desktop_background_update/baseline/schema.yaml	2022-12-30 13:46:00.000000000 +0000
@@ -226,9 +226,6 @@
         '
       mode: NULLABLE
       name: glean_validation_first_run_hour
-      type: TIMESTAMP
-    - mode: NULLABLE
-      name: raw_glean_validation_first_run_hour
       type: STRING
     mode: NULLABLE
     name: datetime
diff -bur --no-dereference bigquery-etl-main/sql/moz-fx-data-shared-prod/firefox_desktop_background_update/baseline/view.sql /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_desktop_background_update/baseline/view.sql
--- bigquery-etl-main/sql/moz-fx-data-shared-prod/firefox_desktop_background_update/baseline/view.sql	2022-12-30 13:54:34.177453000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_desktop_background_update/baseline/view.sql	2022-12-30 13:45:59.000000000 +0000
@@ -6,18 +6,7 @@
   * REPLACE (
     mozfun.norm.metadata(metadata) AS metadata,
     mozfun.norm.glean_ping_info(ping_info) AS ping_info,
-    mozfun.norm.glean_baseline_client_info(client_info, metrics) AS client_info,
-    (
-      SELECT AS STRUCT
-        metrics.* REPLACE (
-          STRUCT(
-            mozfun.glean.parse_datetime(
-              metrics.datetime.glean_validation_first_run_hour
-            ) AS glean_validation_first_run_hour,
-            metrics.datetime.glean_validation_first_run_hour AS raw_glean_validation_first_run_hour
-          ) AS datetime
-        )
-    ) AS metrics
+    mozfun.norm.glean_baseline_client_info(client_info, metrics) AS client_info
   )
 FROM
   `moz-fx-data-shared-prod.firefox_desktop_background_update_stable.baseline_v1`
diff -bur --no-dereference bigquery-etl-main/sql/moz-fx-data-shared-prod/firefox_desktop_background_update/metrics/schema.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_desktop_background_update/metrics/schema.yaml
--- bigquery-etl-main/sql/moz-fx-data-shared-prod/firefox_desktop_background_update/metrics/schema.yaml	2022-12-30 13:54:37.489460340 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_desktop_background_update/metrics/schema.yaml	2022-12-30 13:46:02.000000000 +0000
@@ -299,9 +299,6 @@
         '
       mode: NULLABLE
       name: glean_validation_first_run_hour
-      type: TIMESTAMP
-    - mode: NULLABLE
-      name: raw_glean_validation_first_run_hour
       type: STRING
     mode: NULLABLE
     name: datetime
diff -bur --no-dereference bigquery-etl-main/sql/moz-fx-data-shared-prod/firefox_desktop_background_update/metrics/view.sql /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_desktop_background_update/metrics/view.sql
--- bigquery-etl-main/sql/moz-fx-data-shared-prod/firefox_desktop_background_update/metrics/view.sql	2022-12-30 13:54:36.545458248 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_desktop_background_update/metrics/view.sql	2022-12-30 13:46:02.000000000 +0000
@@ -5,18 +5,7 @@
 SELECT
   * REPLACE (
     mozfun.norm.metadata(metadata) AS metadata,
-    mozfun.norm.glean_ping_info(ping_info) AS ping_info,
-    (
-      SELECT AS STRUCT
-        metrics.* REPLACE (
-          STRUCT(
-            mozfun.glean.parse_datetime(
-              metrics.datetime.glean_validation_first_run_hour
-            ) AS glean_validation_first_run_hour,
-            metrics.datetime.glean_validation_first_run_hour AS raw_glean_validation_first_run_hour
-          ) AS datetime
-        )
-    ) AS metrics
+    mozfun.norm.glean_ping_info(ping_info) AS ping_info
   )
 FROM
   `moz-fx-data-shared-prod.firefox_desktop_background_update_stable.metrics_v1`
diff -bur --no-dereference bigquery-etl-main/sql/moz-fx-data-shared-prod/glean_dictionary/page_view/schema.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/glean_dictionary/page_view/schema.yaml
--- bigquery-etl-main/sql/moz-fx-data-shared-prod/glean_dictionary/page_view/schema.yaml	2022-12-30 13:54:35.377455659 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/glean_dictionary/page_view/schema.yaml	2022-12-30 13:46:01.000000000 +0000
@@ -226,9 +226,6 @@
         '
       mode: NULLABLE
       name: page_loaded
-      type: TIMESTAMP
-    - mode: NULLABLE
-      name: raw_page_loaded
       type: STRING
     mode: NULLABLE
     name: datetime
diff -bur --no-dereference bigquery-etl-main/sql/moz-fx-data-shared-prod/glean_dictionary/page_view/view.sql /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/glean_dictionary/page_view/view.sql
--- bigquery-etl-main/sql/moz-fx-data-shared-prod/glean_dictionary/page_view/view.sql	2022-12-30 13:54:34.857454507 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/glean_dictionary/page_view/view.sql	2022-12-30 13:46:00.000000000 +0000
@@ -5,16 +5,7 @@
 SELECT
   * REPLACE (
     mozfun.norm.metadata(metadata) AS metadata,
-    mozfun.norm.glean_ping_info(ping_info) AS ping_info,
-    (
-      SELECT AS STRUCT
-        metrics.* REPLACE (
-          STRUCT(
-            mozfun.glean.parse_datetime(metrics.datetime.page_loaded) AS page_loaded,
-            metrics.datetime.page_loaded AS raw_page_loaded
-          ) AS datetime
-        )
-    ) AS metrics
+    mozfun.norm.glean_ping_info(ping_info) AS ping_info
   )
 FROM
   `moz-fx-data-shared-prod.glean_dictionary_stable.page_view_v1`
diff -bur --no-dereference bigquery-etl-main/sql/moz-fx-data-shared-prod/mlhackweek_search/action/view.sql /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/mlhackweek_search/action/view.sql
--- bigquery-etl-main/sql/moz-fx-data-shared-prod/mlhackweek_search/action/view.sql	2022-12-30 13:54:32.681449685 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/mlhackweek_search/action/view.sql	2022-12-30 13:45:56.000000000 +0000
@@ -5,22 +5,7 @@
 SELECT
   * REPLACE (
     mozfun.norm.metadata(metadata) AS metadata,
-    mozfun.norm.glean_ping_info(ping_info) AS ping_info,
-    (
-      SELECT AS STRUCT
-        metrics.* REPLACE (
-          STRUCT(
-            mozfun.glean.parse_datetime(
-              metrics.datetime.search_meta_url_select_timestamp
-            ) AS search_meta_url_select_timestamp,
-            metrics.datetime.search_meta_url_select_timestamp AS raw_search_meta_url_select_timestamp,
-            mozfun.glean.parse_datetime(
-              metrics.datetime.search_url_select_timestamp
-            ) AS search_url_select_timestamp,
-            metrics.datetime.search_url_select_timestamp AS raw_search_url_select_timestamp
-          ) AS datetime
-        )
-    ) AS metrics
+    mozfun.norm.glean_ping_info(ping_info) AS ping_info
   )
 FROM
   `moz-fx-data-shared-prod.mlhackweek_search_stable.action_v1`
diff -bur --no-dereference bigquery-etl-main/sql/moz-fx-data-shared-prod/mlhackweek_search/baseline/view.sql /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/mlhackweek_search/baseline/view.sql
--- bigquery-etl-main/sql/moz-fx-data-shared-prod/mlhackweek_search/baseline/view.sql	2022-12-30 13:54:33.173450775 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/mlhackweek_search/baseline/view.sql	2022-12-30 13:45:58.000000000 +0000
@@ -6,18 +6,7 @@
   * REPLACE (
     mozfun.norm.metadata(metadata) AS metadata,
     mozfun.norm.glean_ping_info(ping_info) AS ping_info,
-    mozfun.norm.glean_baseline_client_info(client_info, metrics) AS client_info,
-    (
-      SELECT AS STRUCT
-        metrics.* REPLACE (
-          STRUCT(
-            mozfun.glean.parse_datetime(
-              metrics.datetime.glean_validation_first_run_hour
-            ) AS glean_validation_first_run_hour,
-            metrics.datetime.glean_validation_first_run_hour AS raw_glean_validation_first_run_hour
-          ) AS datetime
-        )
-    ) AS metrics
+    mozfun.norm.glean_baseline_client_info(client_info, metrics) AS client_info
   )
 FROM
   `moz-fx-data-shared-prod.mlhackweek_search_stable.baseline_v1`
diff -bur --no-dereference bigquery-etl-main/sql/moz-fx-data-shared-prod/mlhackweek_search/custom/view.sql /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/mlhackweek_search/custom/view.sql
--- bigquery-etl-main/sql/moz-fx-data-shared-prod/mlhackweek_search/custom/view.sql	2022-12-30 13:54:33.681451901 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/mlhackweek_search/custom/view.sql	2022-12-30 13:45:59.000000000 +0000
@@ -5,18 +5,7 @@
 SELECT
   * REPLACE (
     mozfun.norm.metadata(metadata) AS metadata,
-    mozfun.norm.glean_ping_info(ping_info) AS ping_info,
-    (
-      SELECT AS STRUCT
-        metrics.* REPLACE (
-          STRUCT(
-            mozfun.glean.parse_datetime(
-              metrics.datetime.search_url_select_timestamp
-            ) AS search_url_select_timestamp,
-            metrics.datetime.search_url_select_timestamp AS raw_search_url_select_timestamp
-          ) AS datetime
-        )
-    ) AS metrics
+    mozfun.norm.glean_ping_info(ping_info) AS ping_info
   )
 FROM
   `moz-fx-data-shared-prod.mlhackweek_search_stable.custom_v1`
diff -bur --no-dereference bigquery-etl-main/sql/moz-fx-data-shared-prod/mlhackweek_search/metrics/view.sql /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/mlhackweek_search/metrics/view.sql
--- bigquery-etl-main/sql/moz-fx-data-shared-prod/mlhackweek_search/metrics/view.sql	2022-12-30 13:54:34.573453878 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/mlhackweek_search/metrics/view.sql	2022-12-30 13:46:01.000000000 +0000
@@ -5,18 +5,7 @@
 SELECT
   * REPLACE (
     mozfun.norm.metadata(metadata) AS metadata,
-    mozfun.norm.glean_ping_info(ping_info) AS ping_info,
-    (
-      SELECT AS STRUCT
-        metrics.* REPLACE (
-          STRUCT(
-            mozfun.glean.parse_datetime(
-              metrics.datetime.glean_validation_first_run_hour
-            ) AS glean_validation_first_run_hour,
-            metrics.datetime.glean_validation_first_run_hour AS raw_glean_validation_first_run_hour
-          ) AS datetime
-        )
-    ) AS metrics
+    mozfun.norm.glean_ping_info(ping_info) AS ping_info
   )
 FROM
   `moz-fx-data-shared-prod.mlhackweek_search_stable.metrics_v1`
diff -bur --no-dereference bigquery-etl-main/sql/moz-fx-data-shared-prod/monitoring/bigquery_table_storage/schema.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/monitoring/bigquery_table_storage/schema.yaml
--- bigquery-etl-main/sql/moz-fx-data-shared-prod/monitoring/bigquery_table_storage/schema.yaml	2022-12-30 13:55:59.869643100 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/monitoring/bigquery_table_storage/schema.yaml	2022-12-30 13:49:12.000000000 +0000
@@ -19,35 +19,35 @@
   mode: NULLABLE
   name: total_rows
   type: INTEGER
-- description: The total number of partitions present in the table
+- description: The number of partitions present in the table
   mode: NULLABLE
   name: total_partitions
   type: INTEGER
-- description: Total number of uncompressed bytes in the table
+- description: Total number of logical bytes in the table
   mode: NULLABLE
   name: total_logical_bytes
   type: INTEGER
-- description: Number of uncompressed bytes less than 90 days old
+- description: Number of logical bytes less than 90 days old
   mode: NULLABLE
   name: active_logical_bytes
   type: INTEGER
-- description: Number of uncompressed bytes more than 90 days old
+- description: Number of logical bytes more than 90 days old
   mode: NULLABLE
   name: long_term_logical_bytes
   type: INTEGER
-- description: Total number of compressed bytes used for storage
+- description: Total number of physical bytes used for storage
   mode: NULLABLE
   name: total_physical_bytes
   type: INTEGER
-- description: Number of compressed bytes less than 90 days old
+- description: Number of physical bytes less than 90 days old
   mode: NULLABLE
   name: active_physical_bytes
   type: INTEGER
-- description: Number of compressed bytes more than 90 days old
+- description: Number of physical bytes more than 90 days old
   mode: NULLABLE
   name: long_term_physical_bytes
   type: INTEGER
-- description: Number of compressed bytes for deleted or changed data
+- description: Number of physical bytes used by time travel
   mode: NULLABLE
   name: time_travel_physical_bytes
   type: INTEGER
diff -bur --no-dereference bigquery-etl-main/sql/moz-fx-data-shared-prod/monitoring/bigquery_table_storage_timeline_daily/schema.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/monitoring/bigquery_table_storage_timeline_daily/schema.yaml
--- bigquery-etl-main/sql/moz-fx-data-shared-prod/monitoring/bigquery_table_storage_timeline_daily/schema.yaml	2022-12-30 13:56:00.233643874 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/monitoring/bigquery_table_storage_timeline_daily/schema.yaml	2022-12-30 13:49:12.000000000 +0000
@@ -1,5 +1,5 @@
 fields:
-- description: Date of change to the data in the table.
+- description: Date of when storage was last recalculated
   mode: NULLABLE
   name: change_date
   type: DATE
@@ -23,7 +23,7 @@
   mode: NULLABLE
   name: creation_date
   type: DATE
-- description: The total number of changes to a table in one day
+- description: The total number of changes
   mode: NULLABLE
   name: change_count
   type: INTEGER
@@ -35,31 +35,31 @@
   mode: NULLABLE
   name: avg_total_partitions
   type: FLOAT
-- description: Average total number of uncompressed bytes in the table
+- description: Average total number of logical bytes in the table
   mode: NULLABLE
   name: avg_total_logical_bytes
   type: FLOAT
-- description: Average number of uncompressed bytes less than 90 days old
+- description: Average number of logical bytes less than 90 days old
   mode: NULLABLE
   name: avg_active_logical_bytes
   type: FLOAT
-- description: Average number of uncompressed bytes more than 90 days old
+- description: Average number of logical bytes more than 90 days old
   mode: NULLABLE
   name: avg_long_term_logical_bytes
   type: FLOAT
-- description: Average total number of compressed  bytes used for storage
+- description: Average total number of physical bytes used for storage
   mode: NULLABLE
   name: avg_total_physical_bytes
   type: FLOAT
-- description: Average number of compressed bytes less than 90 days old
+- description: Average number of physical bytes less than 90 days old
   mode: NULLABLE
   name: avg_active_physical_bytes
   type: FLOAT
-- description: Average number of compressed bytes more than 90 days old
+- description: Average number of physical bytes more than 90 days old
   mode: NULLABLE
   name: avg_long_term_physical_bytes
   type: FLOAT
-- description: Average number of compressed bytes for deleted or changed data
+- description: Average number of physical bytes used by time travel
   mode: NULLABLE
   name: avg_time_travel_physical_bytes
   type: FLOAT
Only in bigquery-etl-main/sql/moz-fx

⚠️ Only part of the diff is displayed.

dataops-ci-bot · 2022-12-30T14:23:51Z

Integration report for "Merge branch 'main' into yeonjoo_relay_saasboard_mvp"

`sql.diff`

Click to expand!

Only in /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod: relay
Only in /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod: relay_derived
Only in bigquery-etl-main/sql/moz-fx-data-shared-prod/telemetry/regrets_reporter_update: schema.yaml
Only in bigquery-etl-main/sql/moz-fx-data-shared-prod/telemetry/xfocsp_error_report: schema.yaml
diff -bur --no-dereference bigquery-etl-main/dags/bqetl_subplat.py /tmp/workspace/generated-sql/dags/bqetl_subplat.py
--- bigquery-etl-main/dags/bqetl_subplat.py	2022-12-30 14:23:19.133844073 +0000
+++ /tmp/workspace/generated-sql/dags/bqetl_subplat.py	2022-12-30 14:17:18.000000000 +0000
@@ -567,6 +567,54 @@
         ],
     )
 
+    relay_derived__active_subscription_ids__v1 = bigquery_etl_query(
+        task_id="relay_derived__active_subscription_ids__v1",
+        destination_table='active_subscription_ids_v1${{ macros.ds_format(macros.ds_add(ds, -7), "%Y-%m-%d", "%Y%m%d") }}',
+        dataset_id="relay_derived",
+        project_id="moz-fx-data-shared-prod",
+        owner="[email protected]",
+        email=["[email protected]", "[email protected]"],
+        date_partition_parameter=None,
+        depends_on_past=True,
+        parameters=["date:DATE:{{macros.ds_add(ds, -7)}}"],
+    )
+
+    relay_derived__active_subscriptions__v1 = bigquery_etl_query(
+        task_id="relay_derived__active_subscriptions__v1",
+        destination_table='active_subscriptions_v1${{ macros.ds_format(macros.ds_add(ds, -7), "%Y-%m-%d", "%Y%m%d") }}',
+        dataset_id="relay_derived",
+        project_id="moz-fx-data-shared-prod",
+        owner="[email protected]",
+        email=["[email protected]", "[email protected]"],
+        date_partition_parameter=None,
+        depends_on_past=False,
+        parameters=["date:DATE:{{macros.ds_add(ds, -7)}}"],
+    )
+
+    relay_derived__subscription_events__v1 = bigquery_etl_query(
+        task_id="relay_derived__subscription_events__v1",
+        destination_table='subscription_events_v1${{ macros.ds_format(macros.ds_add(ds, -8), "%Y-%m-%d", "%Y%m%d") }}',
+        dataset_id="relay_derived",
+        project_id="moz-fx-data-shared-prod",
+        owner="[email protected]",
+        email=["[email protected]", "[email protected]"],
+        date_partition_parameter=None,
+        depends_on_past=False,
+        parameters=["date:DATE:{{macros.ds_add(ds, -8)}}"],
+    )
+
+    relay_derived__subscriptions__v1 = bigquery_etl_query(
+        task_id="relay_derived__subscriptions__v1",
+        destination_table="subscriptions_v1",
+        dataset_id="relay_derived",
+        project_id="moz-fx-data-shared-prod",
+        owner="[email protected]",
+        email=["[email protected]", "[email protected]"],
+        date_partition_parameter=None,
+        depends_on_past=False,
+        task_concurrency=1,
+    )
+
     stripe_external__itemized_payout_reconciliation__v5 = gke_command(
         task_id="stripe_external__itemized_payout_reconciliation__v5",
         command=[
@@ -762,3 +810,25 @@
     )
 
     mozilla_vpn_derived__users__v1.set_upstream(mozilla_vpn_external__users__v1)
+
+    relay_derived__active_subscription_ids__v1.set_upstream(
+        relay_derived__subscriptions__v1
+    )
+
+    relay_derived__active_subscriptions__v1.set_upstream(
+        relay_derived__active_subscription_ids__v1
+    )
+
+    relay_derived__active_subscriptions__v1.set_upstream(
+        relay_derived__subscriptions__v1
+    )
+
+    relay_derived__subscription_events__v1.set_upstream(
+        relay_derived__active_subscription_ids__v1
+    )
+
+    relay_derived__subscription_events__v1.set_upstream(
+        relay_derived__subscriptions__v1
+    )
+
+    relay_derived__subscriptions__v1.set_upstream(fivetran_stripe_sync_wait)

sean-rose

A couple of additional changes to make:

This is missing a schema.yaml file for relay_derived.active_subscription_ids_v1.
sql/moz-fx-data-shared-prod/relay_derived/subscriptions_v1/query.sql should be added to the "Access Denied" section of bigquery_etl/dryrun.py, because the account that does the dry runs won't have permission to query the Stripe tables.

sql/moz-fx-data-shared-prod/relay_derived/active_subscriptions_live/view.sql

sql/moz-fx-data-shared-prod/relay_derived/subscription_events_live/view.sql

sql/moz-fx-data-shared-prod/relay_derived/subscription_events_v1/query.sql

sql/moz-fx-data-shared-prod/relay_derived/subscriptions_v1/metadata.yaml

sql/moz-fx-data-shared-prod/relay_derived/subscriptions_v1/query.sql

dataops-ci-bot · 2022-12-31T02:27:31Z

Integration report for "Schema edited and dry run list updated"

`sql.diff`

Click to expand!

Only in /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod: relay
Only in /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod: relay_derived
Only in bigquery-etl-main/sql/moz-fx-data-shared-prod/telemetry/regrets_reporter_update: schema.yaml
Only in bigquery-etl-main/sql/moz-fx-data-shared-prod/telemetry/xfocsp_error_report: schema.yaml
diff -bur --no-dereference bigquery-etl-main/dags/bqetl_subplat.py /tmp/workspace/generated-sql/dags/bqetl_subplat.py
--- bigquery-etl-main/dags/bqetl_subplat.py	2022-12-31 02:27:01.119204831 +0000
+++ /tmp/workspace/generated-sql/dags/bqetl_subplat.py	2022-12-31 02:21:27.000000000 +0000
@@ -567,6 +567,54 @@
         ],
     )
 
+    relay_derived__active_subscription_ids__v1 = bigquery_etl_query(
+        task_id="relay_derived__active_subscription_ids__v1",
+        destination_table='active_subscription_ids_v1${{ macros.ds_format(macros.ds_add(ds, -7), "%Y-%m-%d", "%Y%m%d") }}',
+        dataset_id="relay_derived",
+        project_id="moz-fx-data-shared-prod",
+        owner="[email protected]",
+        email=["[email protected]", "[email protected]"],
+        date_partition_parameter=None,
+        depends_on_past=True,
+        parameters=["date:DATE:{{macros.ds_add(ds, -7)}}"],
+    )
+
+    relay_derived__active_subscriptions__v1 = bigquery_etl_query(
+        task_id="relay_derived__active_subscriptions__v1",
+        destination_table='active_subscriptions_v1${{ macros.ds_format(macros.ds_add(ds, -7), "%Y-%m-%d", "%Y%m%d") }}',
+        dataset_id="relay_derived",
+        project_id="moz-fx-data-shared-prod",
+        owner="[email protected]",
+        email=["[email protected]", "[email protected]"],
+        date_partition_parameter=None,
+        depends_on_past=False,
+        parameters=["date:DATE:{{macros.ds_add(ds, -7)}}"],
+    )
+
+    relay_derived__subscription_events__v1 = bigquery_etl_query(
+        task_id="relay_derived__subscription_events__v1",
+        destination_table='subscription_events_v1${{ macros.ds_format(macros.ds_add(ds, -8), "%Y-%m-%d", "%Y%m%d") }}',
+        dataset_id="relay_derived",
+        project_id="moz-fx-data-shared-prod",
+        owner="[email protected]",
+        email=["[email protected]", "[email protected]"],
+        date_partition_parameter=None,
+        depends_on_past=False,
+        parameters=["date:DATE:{{macros.ds_add(ds, -8)}}"],
+    )
+
+    relay_derived__subscriptions__v1 = bigquery_etl_query(
+        task_id="relay_derived__subscriptions__v1",
+        destination_table="subscriptions_v1",
+        dataset_id="relay_derived",
+        project_id="moz-fx-data-shared-prod",
+        owner="[email protected]",
+        email=["[email protected]", "[email protected]"],
+        date_partition_parameter=None,
+        depends_on_past=False,
+        task_concurrency=1,
+    )
+
     stripe_external__itemized_payout_reconciliation__v5 = gke_command(
         task_id="stripe_external__itemized_payout_reconciliation__v5",
         command=[
@@ -762,3 +810,25 @@
     )
 
     mozilla_vpn_derived__users__v1.set_upstream(mozilla_vpn_external__users__v1)
+
+    relay_derived__active_subscription_ids__v1.set_upstream(
+        relay_derived__subscriptions__v1
+    )
+
+    relay_derived__active_subscriptions__v1.set_upstream(
+        relay_derived__active_subscription_ids__v1
+    )
+
+    relay_derived__active_subscriptions__v1.set_upstream(
+        relay_derived__subscriptions__v1
+    )
+
+    relay_derived__subscription_events__v1.set_upstream(
+        relay_derived__active_subscription_ids__v1
+    )
+
+    relay_derived__subscription_events__v1.set_upstream(
+        relay_derived__subscriptions__v1
+    )
+
+    relay_derived__subscriptions__v1.set_upstream(fivetran_stripe_sync_wait)

Co-authored-by: Sean Rose <[email protected]>

dataops-ci-bot · 2022-12-31T02:44:09Z

Integration report for "removing status column in multiple places"

`sql.diff`

Click to expand!

Only in /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod: relay
Only in /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod: relay_derived
Only in bigquery-etl-main/sql/moz-fx-data-shared-prod/telemetry/regrets_reporter_update: schema.yaml
Only in bigquery-etl-main/sql/moz-fx-data-shared-prod/telemetry/xfocsp_error_report: schema.yaml
diff -bur --no-dereference bigquery-etl-main/dags/bqetl_subplat.py /tmp/workspace/generated-sql/dags/bqetl_subplat.py
--- bigquery-etl-main/dags/bqetl_subplat.py	2022-12-31 02:43:44.438743528 +0000
+++ /tmp/workspace/generated-sql/dags/bqetl_subplat.py	2022-12-31 02:38:09.000000000 +0000
@@ -567,6 +567,54 @@
         ],
     )
 
+    relay_derived__active_subscription_ids__v1 = bigquery_etl_query(
+        task_id="relay_derived__active_subscription_ids__v1",
+        destination_table='active_subscription_ids_v1${{ macros.ds_format(macros.ds_add(ds, -7), "%Y-%m-%d", "%Y%m%d") }}',
+        dataset_id="relay_derived",
+        project_id="moz-fx-data-shared-prod",
+        owner="[email protected]",
+        email=["[email protected]", "[email protected]"],
+        date_partition_parameter=None,
+        depends_on_past=True,
+        parameters=["date:DATE:{{macros.ds_add(ds, -7)}}"],
+    )
+
+    relay_derived__active_subscriptions__v1 = bigquery_etl_query(
+        task_id="relay_derived__active_subscriptions__v1",
+        destination_table='active_subscriptions_v1${{ macros.ds_format(macros.ds_add(ds, -7), "%Y-%m-%d", "%Y%m%d") }}',
+        dataset_id="relay_derived",
+        project_id="moz-fx-data-shared-prod",
+        owner="[email protected]",
+        email=["[email protected]", "[email protected]"],
+        date_partition_parameter=None,
+        depends_on_past=False,
+        parameters=["date:DATE:{{macros.ds_add(ds, -7)}}"],
+    )
+
+    relay_derived__subscription_events__v1 = bigquery_etl_query(
+        task_id="relay_derived__subscription_events__v1",
+        destination_table='subscription_events_v1${{ macros.ds_format(macros.ds_add(ds, -8), "%Y-%m-%d", "%Y%m%d") }}',
+        dataset_id="relay_derived",
+        project_id="moz-fx-data-shared-prod",
+        owner="[email protected]",
+        email=["[email protected]", "[email protected]"],
+        date_partition_parameter=None,
+        depends_on_past=False,
+        parameters=["date:DATE:{{macros.ds_add(ds, -8)}}"],
+    )
+
+    relay_derived__subscriptions__v1 = bigquery_etl_query(
+        task_id="relay_derived__subscriptions__v1",
+        destination_table="subscriptions_v1",
+        dataset_id="relay_derived",
+        project_id="moz-fx-data-shared-prod",
+        owner="[email protected]",
+        email=["[email protected]", "[email protected]"],
+        date_partition_parameter=None,
+        depends_on_past=False,
+        task_concurrency=1,
+    )
+
     stripe_external__itemized_payout_reconciliation__v5 = gke_command(
         task_id="stripe_external__itemized_payout_reconciliation__v5",
         command=[
@@ -762,3 +810,25 @@
     )
 
     mozilla_vpn_derived__users__v1.set_upstream(mozilla_vpn_external__users__v1)
+
+    relay_derived__active_subscription_ids__v1.set_upstream(
+        relay_derived__subscriptions__v1
+    )
+
+    relay_derived__active_subscriptions__v1.set_upstream(
+        relay_derived__active_subscription_ids__v1
+    )
+
+    relay_derived__active_subscriptions__v1.set_upstream(
+        relay_derived__subscriptions__v1
+    )
+
+    relay_derived__subscription_events__v1.set_upstream(
+        relay_derived__active_subscription_ids__v1
+    )
+
+    relay_derived__subscription_events__v1.set_upstream(
+        relay_derived__subscriptions__v1
+    )
+
+    relay_derived__subscriptions__v1.set_upstream(fivetran_stripe_sync_wait)

dataops-ci-bot · 2022-12-31T02:50:06Z

Integration report for "matching date_partition_offset setting with the table"

`sql.diff`

Click to expand!

Only in /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod: relay
Only in /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod: relay_derived
Only in bigquery-etl-main/sql/moz-fx-data-shared-prod/telemetry/regrets_reporter_update: schema.yaml
Only in bigquery-etl-main/sql/moz-fx-data-shared-prod/telemetry/xfocsp_error_report: schema.yaml
diff -bur --no-dereference bigquery-etl-main/dags/bqetl_subplat.py /tmp/workspace/generated-sql/dags/bqetl_subplat.py
--- bigquery-etl-main/dags/bqetl_subplat.py	2022-12-31 02:49:35.389220238 +0000
+++ /tmp/workspace/generated-sql/dags/bqetl_subplat.py	2022-12-31 02:44:14.000000000 +0000
@@ -567,6 +567,54 @@
         ],
     )
 
+    relay_derived__active_subscription_ids__v1 = bigquery_etl_query(
+        task_id="relay_derived__active_subscription_ids__v1",
+        destination_table='active_subscription_ids_v1${{ macros.ds_format(macros.ds_add(ds, -7), "%Y-%m-%d", "%Y%m%d") }}',
+        dataset_id="relay_derived",
+        project_id="moz-fx-data-shared-prod",
+        owner="[email protected]",
+        email=["[email protected]", "[email protected]"],
+        date_partition_parameter=None,
+        depends_on_past=True,
+        parameters=["date:DATE:{{macros.ds_add(ds, -7)}}"],
+    )
+
+    relay_derived__active_subscriptions__v1 = bigquery_etl_query(
+        task_id="relay_derived__active_subscriptions__v1",
+        destination_table='active_subscriptions_v1${{ macros.ds_format(macros.ds_add(ds, -7), "%Y-%m-%d", "%Y%m%d") }}',
+        dataset_id="relay_derived",
+        project_id="moz-fx-data-shared-prod",
+        owner="[email protected]",
+        email=["[email protected]", "[email protected]"],
+        date_partition_parameter=None,
+        depends_on_past=False,
+        parameters=["date:DATE:{{macros.ds_add(ds, -7)}}"],
+    )
+
+    relay_derived__subscription_events__v1 = bigquery_etl_query(
+        task_id="relay_derived__subscription_events__v1",
+        destination_table='subscription_events_v1${{ macros.ds_format(macros.ds_add(ds, -8), "%Y-%m-%d", "%Y%m%d") }}',
+        dataset_id="relay_derived",
+        project_id="moz-fx-data-shared-prod",
+        owner="[email protected]",
+        email=["[email protected]", "[email protected]"],
+        date_partition_parameter=None,
+        depends_on_past=False,
+        parameters=["date:DATE:{{macros.ds_add(ds, -8)}}"],
+    )
+
+    relay_derived__subscriptions__v1 = bigquery_etl_query(
+        task_id="relay_derived__subscriptions__v1",
+        destination_table="subscriptions_v1",
+        dataset_id="relay_derived",
+        project_id="moz-fx-data-shared-prod",
+        owner="[email protected]",
+        email=["[email protected]", "[email protected]"],
+        date_partition_parameter=None,
+        depends_on_past=False,
+        task_concurrency=1,
+    )
+
     stripe_external__itemized_payout_reconciliation__v5 = gke_command(
         task_id="stripe_external__itemized_payout_reconciliation__v5",
         command=[
@@ -762,3 +810,25 @@
     )
 
     mozilla_vpn_derived__users__v1.set_upstream(mozilla_vpn_external__users__v1)
+
+    relay_derived__active_subscription_ids__v1.set_upstream(
+        relay_derived__subscriptions__v1
+    )
+
+    relay_derived__active_subscriptions__v1.set_upstream(
+        relay_derived__active_subscription_ids__v1
+    )
+
+    relay_derived__active_subscriptions__v1.set_upstream(
+        relay_derived__subscriptions__v1
+    )
+
+    relay_derived__subscription_events__v1.set_upstream(
+        relay_derived__active_subscription_ids__v1
+    )
+
+    relay_derived__subscription_events__v1.set_upstream(
+        relay_derived__subscriptions__v1
+    )
+
+    relay_derived__subscriptions__v1.set_upstream(fivetran_stripe_sync_wait)

dataops-ci-bot · 2022-12-31T02:52:33Z

Integration report for "keep the consistency of friendly_name"

`sql.diff`

Click to expand!

Only in /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod: relay
Only in /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod: relay_derived
Only in bigquery-etl-main/sql/moz-fx-data-shared-prod/telemetry/regrets_reporter_update: schema.yaml
Only in bigquery-etl-main/sql/moz-fx-data-shared-prod/telemetry/xfocsp_error_report: schema.yaml
diff -bur --no-dereference bigquery-etl-main/dags/bqetl_subplat.py /tmp/workspace/generated-sql/dags/bqetl_subplat.py
--- bigquery-etl-main/dags/bqetl_subplat.py	2022-12-31 02:52:09.353064708 +0000
+++ /tmp/workspace/generated-sql/dags/bqetl_subplat.py	2022-12-31 02:46:26.000000000 +0000
@@ -567,6 +567,54 @@
         ],
     )
 
+    relay_derived__active_subscription_ids__v1 = bigquery_etl_query(
+        task_id="relay_derived__active_subscription_ids__v1",
+        destination_table='active_subscription_ids_v1${{ macros.ds_format(macros.ds_add(ds, -7), "%Y-%m-%d", "%Y%m%d") }}',
+        dataset_id="relay_derived",
+        project_id="moz-fx-data-shared-prod",
+        owner="[email protected]",
+        email=["[email protected]", "[email protected]"],
+        date_partition_parameter=None,
+        depends_on_past=True,
+        parameters=["date:DATE:{{macros.ds_add(ds, -7)}}"],
+    )
+
+    relay_derived__active_subscriptions__v1 = bigquery_etl_query(
+        task_id="relay_derived__active_subscriptions__v1",
+        destination_table='active_subscriptions_v1${{ macros.ds_format(macros.ds_add(ds, -7), "%Y-%m-%d", "%Y%m%d") }}',
+        dataset_id="relay_derived",
+        project_id="moz-fx-data-shared-prod",
+        owner="[email protected]",
+        email=["[email protected]", "[email protected]"],
+        date_partition_parameter=None,
+        depends_on_past=False,
+        parameters=["date:DATE:{{macros.ds_add(ds, -7)}}"],
+    )
+
+    relay_derived__subscription_events__v1 = bigquery_etl_query(
+        task_id="relay_derived__subscription_events__v1",
+        destination_table='subscription_events_v1${{ macros.ds_format(macros.ds_add(ds, -8), "%Y-%m-%d", "%Y%m%d") }}',
+        dataset_id="relay_derived",
+        project_id="moz-fx-data-shared-prod",
+        owner="[email protected]",
+        email=["[email protected]", "[email protected]"],
+        date_partition_parameter=None,
+        depends_on_past=False,
+        parameters=["date:DATE:{{macros.ds_add(ds, -8)}}"],
+    )
+
+    relay_derived__subscriptions__v1 = bigquery_etl_query(
+        task_id="relay_derived__subscriptions__v1",
+        destination_table="subscriptions_v1",
+        dataset_id="relay_derived",
+        project_id="moz-fx-data-shared-prod",
+        owner="[email protected]",
+        email=["[email protected]", "[email protected]"],
+        date_partition_parameter=None,
+        depends_on_past=False,
+        task_concurrency=1,
+    )
+
     stripe_external__itemized_payout_reconciliation__v5 = gke_command(
         task_id="stripe_external__itemized_payout_reconciliation__v5",
         command=[
@@ -762,3 +810,25 @@
     )
 
     mozilla_vpn_derived__users__v1.set_upstream(mozilla_vpn_external__users__v1)
+
+    relay_derived__active_subscription_ids__v1.set_upstream(
+        relay_derived__subscriptions__v1
+    )
+
+    relay_derived__active_subscriptions__v1.set_upstream(
+        relay_derived__active_subscription_ids__v1
+    )
+
+    relay_derived__active_subscriptions__v1.set_upstream(
+        relay_derived__subscriptions__v1
+    )
+
+    relay_derived__subscription_events__v1.set_upstream(
+        relay_derived__active_subscription_ids__v1
+    )
+
+    relay_derived__subscription_events__v1.set_upstream(
+        relay_derived__subscriptions__v1
+    )
+
+    relay_derived__subscriptions__v1.set_upstream(fivetran_stripe_sync_wait)

dataops-ci-bot · 2022-12-31T03:11:13Z

Integration report for "removing unnecessary comment"

`sql.diff`

Click to expand!

Only in /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod: relay
Only in /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod: relay_derived
Only in bigquery-etl-main/sql/moz-fx-data-shared-prod/telemetry/regrets_reporter_update: schema.yaml
Only in bigquery-etl-main/sql/moz-fx-data-shared-prod/telemetry/xfocsp_error_report: schema.yaml
diff -bur --no-dereference bigquery-etl-main/dags/bqetl_subplat.py /tmp/workspace/generated-sql/dags/bqetl_subplat.py
--- bigquery-etl-main/dags/bqetl_subplat.py	2022-12-31 03:10:28.399504340 +0000
+++ /tmp/workspace/generated-sql/dags/bqetl_subplat.py	2022-12-31 03:04:31.000000000 +0000
@@ -567,6 +567,54 @@
         ],
     )
 
+    relay_derived__active_subscription_ids__v1 = bigquery_etl_query(
+        task_id="relay_derived__active_subscription_ids__v1",
+        destination_table='active_subscription_ids_v1${{ macros.ds_format(macros.ds_add(ds, -7), "%Y-%m-%d", "%Y%m%d") }}',
+        dataset_id="relay_derived",
+        project_id="moz-fx-data-shared-prod",
+        owner="[email protected]",
+        email=["[email protected]", "[email protected]"],
+        date_partition_parameter=None,
+        depends_on_past=True,
+        parameters=["date:DATE:{{macros.ds_add(ds, -7)}}"],
+    )
+
+    relay_derived__active_subscriptions__v1 = bigquery_etl_query(
+        task_id="relay_derived__active_subscriptions__v1",
+        destination_table='active_subscriptions_v1${{ macros.ds_format(macros.ds_add(ds, -7), "%Y-%m-%d", "%Y%m%d") }}',
+        dataset_id="relay_derived",
+        project_id="moz-fx-data-shared-prod",
+        owner="[email protected]",
+        email=["[email protected]", "[email protected]"],
+        date_partition_parameter=None,
+        depends_on_past=False,
+        parameters=["date:DATE:{{macros.ds_add(ds, -7)}}"],
+    )
+
+    relay_derived__subscription_events__v1 = bigquery_etl_query(
+        task_id="relay_derived__subscription_events__v1",
+        destination_table='subscription_events_v1${{ macros.ds_format(macros.ds_add(ds, -8), "%Y-%m-%d", "%Y%m%d") }}',
+        dataset_id="relay_derived",
+        project_id="moz-fx-data-shared-prod",
+        owner="[email protected]",
+        email=["[email protected]", "[email protected]"],
+        date_partition_parameter=None,
+        depends_on_past=False,
+        parameters=["date:DATE:{{macros.ds_add(ds, -8)}}"],
+    )
+
+    relay_derived__subscriptions__v1 = bigquery_etl_query(
+        task_id="relay_derived__subscriptions__v1",
+        destination_table="subscriptions_v1",
+        dataset_id="relay_derived",
+        project_id="moz-fx-data-shared-prod",
+        owner="[email protected]",
+        email=["[email protected]", "[email protected]"],
+        date_partition_parameter=None,
+        depends_on_past=False,
+        task_concurrency=1,
+    )
+
     stripe_external__itemized_payout_reconciliation__v5 = gke_command(
         task_id="stripe_external__itemized_payout_reconciliation__v5",
         command=[
@@ -762,3 +810,25 @@
     )
 
     mozilla_vpn_derived__users__v1.set_upstream(mozilla_vpn_external__users__v1)
+
+    relay_derived__active_subscription_ids__v1.set_upstream(
+        relay_derived__subscriptions__v1
+    )
+
+    relay_derived__active_subscriptions__v1.set_upstream(
+        relay_derived__active_subscription_ids__v1
+    )
+
+    relay_derived__active_subscriptions__v1.set_upstream(
+        relay_derived__subscriptions__v1
+    )
+
+    relay_derived__subscription_events__v1.set_upstream(
+        relay_derived__active_subscription_ids__v1
+    )
+
+    relay_derived__subscription_events__v1.set_upstream(
+        relay_derived__subscriptions__v1
+    )
+
+    relay_derived__subscriptions__v1.set_upstream(fivetran_stripe_sync_wait)

YeonjooSmith · 2022-12-31T03:34:50Z

First round of revisions are done. Let me know if there is anything to be changed before the approval.

dataops-ci-bot · 2022-12-31T03:43:12Z

Integration report for "subscriptions query updated"

`sql.diff`

Click to expand!

Only in /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod: relay
Only in /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod: relay_derived
Only in bigquery-etl-main/sql/moz-fx-data-shared-prod/telemetry/regrets_reporter_update: schema.yaml
Only in bigquery-etl-main/sql/moz-fx-data-shared-prod/telemetry/xfocsp_error_report: schema.yaml
diff -bur --no-dereference bigquery-etl-main/dags/bqetl_subplat.py /tmp/workspace/generated-sql/dags/bqetl_subplat.py
--- bigquery-etl-main/dags/bqetl_subplat.py	2022-12-31 03:42:45.364677904 +0000
+++ /tmp/workspace/generated-sql/dags/bqetl_subplat.py	2022-12-31 03:37:18.000000000 +0000
@@ -567,6 +567,54 @@
         ],
     )
 
+    relay_derived__active_subscription_ids__v1 = bigquery_etl_query(
+        task_id="relay_derived__active_subscription_ids__v1",
+        destination_table='active_subscription_ids_v1${{ macros.ds_format(macros.ds_add(ds, -7), "%Y-%m-%d", "%Y%m%d") }}',
+        dataset_id="relay_derived",
+        project_id="moz-fx-data-shared-prod",
+        owner="[email protected]",
+        email=["[email protected]", "[email protected]"],
+        date_partition_parameter=None,
+        depends_on_past=True,
+        parameters=["date:DATE:{{macros.ds_add(ds, -7)}}"],
+    )
+
+    relay_derived__active_subscriptions__v1 = bigquery_etl_query(
+        task_id="relay_derived__active_subscriptions__v1",
+        destination_table='active_subscriptions_v1${{ macros.ds_format(macros.ds_add(ds, -7), "%Y-%m-%d", "%Y%m%d") }}',
+        dataset_id="relay_derived",
+        project_id="moz-fx-data-shared-prod",
+        owner="[email protected]",
+        email=["[email protected]", "[email protected]"],
+        date_partition_parameter=None,
+        depends_on_past=False,
+        parameters=["date:DATE:{{macros.ds_add(ds, -7)}}"],
+    )
+
+    relay_derived__subscription_events__v1 = bigquery_etl_query(
+        task_id="relay_derived__subscription_events__v1",
+        destination_table='subscription_events_v1${{ macros.ds_format(macros.ds_add(ds, -8), "%Y-%m-%d", "%Y%m%d") }}',
+        dataset_id="relay_derived",
+        project_id="moz-fx-data-shared-prod",
+        owner="[email protected]",
+        email=["[email protected]", "[email protected]"],
+        date_partition_parameter=None,
+        depends_on_past=False,
+        parameters=["date:DATE:{{macros.ds_add(ds, -8)}}"],
+    )
+
+    relay_derived__subscriptions__v1 = bigquery_etl_query(
+        task_id="relay_derived__subscriptions__v1",
+        destination_table="subscriptions_v1",
+        dataset_id="relay_derived",
+        project_id="moz-fx-data-shared-prod",
+        owner="[email protected]",
+        email=["[email protected]", "[email protected]"],
+        date_partition_parameter=None,
+        depends_on_past=False,
+        task_concurrency=1,
+    )
+
     stripe_external__itemized_payout_reconciliation__v5 = gke_command(
         task_id="stripe_external__itemized_payout_reconciliation__v5",
         command=[
@@ -762,3 +810,25 @@
     )
 
     mozilla_vpn_derived__users__v1.set_upstream(mozilla_vpn_external__users__v1)
+
+    relay_derived__active_subscription_ids__v1.set_upstream(
+        relay_derived__subscriptions__v1
+    )
+
+    relay_derived__active_subscriptions__v1.set_upstream(
+        relay_derived__active_subscription_ids__v1
+    )
+
+    relay_derived__active_subscriptions__v1.set_upstream(
+        relay_derived__subscriptions__v1
+    )
+
+    relay_derived__subscription_events__v1.set_upstream(
+        relay_derived__active_subscription_ids__v1
+    )
+
+    relay_derived__subscription_events__v1.set_upstream(
+        relay_derived__subscriptions__v1
+    )
+
+    relay_derived__subscriptions__v1.set_upstream(fivetran_stripe_sync_wait)

sql/moz-fx-data-shared-prod/relay_derived/subscriptions_v1/query.sql

sql/moz-fx-data-shared-prod/relay_derived/subscriptions_v1/schema.yaml

Co-authored-by: Sean Rose <[email protected]>

dataops-ci-bot · 2023-01-06T20:09:00Z

Integration report for "subscriptions_v1 schema editted"

`sql.diff`

Click to expand!

Only in /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_accounts/fxa_content_auth_events: metadata.yaml
diff -bur --no-dereference bigquery-etl-main/sql/moz-fx-data-shared-prod/firefox_accounts/fxa_content_auth_events/view.sql /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_accounts/fxa_content_auth_events/view.sql
--- bigquery-etl-main/sql/moz-fx-data-shared-prod/firefox_accounts/fxa_content_auth_events/view.sql	2023-01-06 20:03:36.836501034 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_accounts/fxa_content_auth_events/view.sql	2023-01-06 19:57:30.000000000 +0000
@@ -23,85 +23,81 @@
 CREATE OR REPLACE VIEW
   `moz-fx-data-shared-prod.firefox_accounts.fxa_content_auth_events`
 AS
---   --
--- WITH content AS (
---   SELECT
---     jsonPayload.logger,
---     jsonPayload.fields.event_type,
---     jsonPayload.fields.app_version,
---     jsonPayload.fields.os_name,
---     jsonPayload.fields.os_version,
---     jsonPayload.fields.country,
---     jsonPayload.fields.language,
---     jsonPayload.fields.user_id,
---     jsonPayload.fields.user_properties,
---     jsonPayload.fields.event_properties,
---     `timestamp`,
---     receiveTimestamp,
---   FROM
---     `moz-fx-data-shared-prod.firefox_accounts_derived.fxa_content_events_v1`
--- ),
---   --
--- auth AS (
---   SELECT
---     jsonPayload.logger,
---     jsonPayload.fields.event_type,
---     jsonPayload.fields.app_version,
---     jsonPayload.fields.os_name,
---     jsonPayload.fields.os_version,
---     jsonPayload.fields.country,
---     jsonPayload.fields.language,
---     jsonPayload.fields.user_id,
---     jsonPayload.fields.user_properties,
---     jsonPayload.fields.event_properties,
---     `timestamp`,
---     receiveTimestamp,
---   FROM
---     `moz-fx-data-shared-prod.firefox_accounts_derived.fxa_auth_events_v1`
--- ),
---   --
--- unioned AS (
---   SELECT
---     *
---   FROM
---     auth
---   UNION ALL
---   SELECT
---     *
---   FROM
---     content
--- )
---   --
--- SELECT
---   * EXCEPT (user_properties, event_properties),
---   JSON_VALUE(user_properties, "$.utm_term") AS utm_term,
---   JSON_VALUE(user_properties, "$.utm_source") AS utm_source,
---   JSON_VALUE(user_properties, "$.utm_medium") AS utm_medium,
---   JSON_VALUE(user_properties, "$.utm_campaign") AS utm_campaign,
---   JSON_VALUE(user_properties, "$.utm_content") AS utm_content,
---   JSON_VALUE(user_properties, "$.ua_version") AS ua_version,
---   JSON_VALUE(user_properties, "$.ua_browser") AS ua_browser,
---   JSON_VALUE(user_properties, "$.entrypoint") AS entrypoint,
---   JSON_VALUE(user_properties, "$.entrypoint_experiment") AS entrypoint_experiment,
---   JSON_VALUE(user_properties, "$.entrypoint_variation") AS entrypoint_variation,
---   JSON_VALUE(user_properties, "$.flow_id") AS flow_id,
---   JSON_VALUE(event_properties, "$.service") AS service,
---   JSON_VALUE(event_properties, "$.email_type") AS email_type,
---   JSON_VALUE(event_properties, "$.email_provider") AS email_provider,
---   JSON_VALUE(event_properties, "$.oauth_client_id") AS oauth_client_id,
---   JSON_VALUE(event_properties, "$.connect_device_flow") AS connect_device_flow,
---   JSON_VALUE(event_properties, "$.connect_device_os") AS connect_device_os,
---   JSON_VALUE(user_properties, "$.sync_device_count") AS sync_device_count,
---   JSON_VALUE(user_properties, "$.sync_active_devices_day") AS sync_active_devices_day,
---   JSON_VALUE(user_properties, "$.sync_active_devices_week") AS sync_active_devices_week,
---   JSON_VALUE(user_properties, "$.sync_active_devices_month") AS sync_active_devices_month,
---   JSON_VALUE(event_properties, "$.email_sender") AS email_sender,
---   JSON_VALUE(event_properties, "$.email_service") AS email_service,
---   JSON_VALUE(event_properties, "$.email_template") AS email_template,
---   JSON_VALUE(event_properties, "$.email_version") AS email_version,
--- FROM
---   unioned
+  --
+WITH content AS (
+  SELECT
+    jsonPayload.logger,
+    jsonPayload.fields.event_type,
+    jsonPayload.fields.app_version,
+    jsonPayload.fields.os_name,
+    jsonPayload.fields.os_version,
+    jsonPayload.fields.country,
+    jsonPayload.fields.language,
+    jsonPayload.fields.user_id,
+    jsonPayload.fields.user_properties,
+    jsonPayload.fields.event_properties,
+    `timestamp`,
+    receiveTimestamp,
+  FROM
+    `moz-fx-data-shared-prod.firefox_accounts_derived.fxa_content_events_v1`
+),
+  --
+auth AS (
+  SELECT
+    jsonPayload.logger,
+    jsonPayload.fields.event_type,
+    jsonPayload.fields.app_version,
+    jsonPayload.fields.os_name,
+    jsonPayload.fields.os_version,
+    jsonPayload.fields.country,
+    jsonPayload.fields.language,
+    jsonPayload.fields.user_id,
+    jsonPayload.fields.user_properties,
+    jsonPayload.fields.event_properties,
+    `timestamp`,
+    receiveTimestamp,
+  FROM
+    `moz-fx-data-shared-prod.firefox_accounts_derived.fxa_auth_events_v1`
+),
+  --
+unioned AS (
+  SELECT
+    *
+  FROM
+    auth
+  UNION ALL
+  SELECT
+    *
+  FROM
+    content
+)
+  --
 SELECT
-  ERROR(
-    'VIEW DEPRECATED - This view will be completely deleted after 1st of February 2023, please use `fxa_all_events` with filter on `event_category` instead. See DENG-582 for more info.'
-  )
+  * EXCEPT (user_properties, event_properties),
+  JSON_VALUE(user_properties, "$.utm_term") AS utm_term,
+  JSON_VALUE(user_properties, "$.utm_source") AS utm_source,
+  JSON_VALUE(user_properties, "$.utm_medium") AS utm_medium,
+  JSON_VALUE(user_properties, "$.utm_campaign") AS utm_campaign,
+  JSON_VALUE(user_properties, "$.utm_content") AS utm_content,
+  JSON_VALUE(user_properties, "$.ua_version") AS ua_version,
+  JSON_VALUE(user_properties, "$.ua_browser") AS ua_browser,
+  JSON_VALUE(user_properties, "$.entrypoint") AS entrypoint,
+  JSON_VALUE(user_properties, "$.entrypoint_experiment") AS entrypoint_experiment,
+  JSON_VALUE(user_properties, "$.entrypoint_variation") AS entrypoint_variation,
+  JSON_VALUE(user_properties, "$.flow_id") AS flow_id,
+  JSON_VALUE(event_properties, "$.service") AS service,
+  JSON_VALUE(event_properties, "$.email_type") AS email_type,
+  JSON_VALUE(event_properties, "$.email_provider") AS email_provider,
+  JSON_VALUE(event_properties, "$.oauth_client_id") AS oauth_client_id,
+  JSON_VALUE(event_properties, "$.connect_device_flow") AS connect_device_flow,
+  JSON_VALUE(event_properties, "$.connect_device_os") AS connect_device_os,
+  JSON_VALUE(user_properties, "$.sync_device_count") AS sync_device_count,
+  JSON_VALUE(user_properties, "$.sync_active_devices_day") AS sync_active_devices_day,
+  JSON_VALUE(user_properties, "$.sync_active_devices_week") AS sync_active_devices_week,
+  JSON_VALUE(user_properties, "$.sync_active_devices_month") AS sync_active_devices_month,
+  JSON_VALUE(event_properties, "$.email_sender") AS email_sender,
+  JSON_VALUE(event_properties, "$.email_service") AS email_service,
+  JSON_VALUE(event_properties, "$.email_template") AS email_template,
+  JSON_VALUE(event_properties, "$.email_version") AS email_version,
+FROM
+  unioned
Only in /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_accounts/fxa_content_auth_oauth_events: metadata.yaml
diff -bur --no-dereference bigquery-etl-main/sql/moz-fx-data-shared-prod/firefox_accounts/fxa_content_auth_oauth_events/view.sql /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_accounts/fxa_content_auth_oauth_events/view.sql
--- bigquery-etl-main/sql/moz-fx-data-shared-prod/firefox_accounts/fxa_content_auth_oauth_events/view.sql	2023-01-06 20:03:36.836501034 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_accounts/fxa_content_auth_oauth_events/view.sql	2023-01-06 19:57:30.000000000 +0000
@@ -24,109 +24,105 @@
 CREATE OR REPLACE VIEW
   `moz-fx-data-shared-prod.firefox_accounts.fxa_content_auth_oauth_events`
 AS
---   --
--- WITH content AS (
---   SELECT
---     jsonPayload.logger,
---     jsonPayload.fields.event_type,
---     jsonPayload.fields.app_version,
---     jsonPayload.fields.os_name,
---     jsonPayload.fields.os_version,
---     jsonPayload.fields.country,
---     jsonPayload.fields.language,
---     jsonPayload.fields.user_id,
---     jsonPayload.fields.user_properties,
---     jsonPayload.fields.event_properties,
---     jsonPayload.fields.device_id,
---     `timestamp`,
---     receiveTimestamp,
---   FROM
---     `moz-fx-data-shared-prod.firefox_accounts_derived.fxa_content_events_v1`
--- ),
---     --
--- auth AS (
---   SELECT
---     jsonPayload.logger,
---     jsonPayload.fields.event_type,
---     jsonPayload.fields.app_version,
---     jsonPayload.fields.os_name,
---     jsonPayload.fields.os_version,
---     jsonPayload.fields.country,
---     jsonPayload.fields.language,
---     jsonPayload.fields.user_id,
---     jsonPayload.fields.user_properties,
---     jsonPayload.fields.event_properties,
---     jsonPayload.fields.device_id,
---     `timestamp`,
---     receiveTimestamp,
---   FROM
---     `moz-fx-data-shared-prod.firefox_accounts_derived.fxa_auth_events_v1`
--- ),
---     --
--- oauth AS (
---   SELECT
---     jsonPayload.logger,
---     jsonPayload.fields.event_type,
---     jsonPayload.fields.app_version,
---     CAST(NULL AS STRING) AS os_name,
---     CAST(NULL AS STRING) AS os_version,
---     CAST(NULL AS STRING) AS country,
---     CAST(NULL AS STRING) AS language,
---     jsonPayload.fields.user_id,
---     jsonPayload.fields.user_properties,
---     jsonPayload.fields.event_properties,
---     CAST(NULL AS STRING) AS device_id,
---     `timestamp`,
---     receiveTimestamp,
---   FROM
---     `moz-fx-data-shared-prod.firefox_accounts_derived.fxa_oauth_events_v1`
--- ),
---     --
--- unioned AS (
---   SELECT
---     *
---   FROM
---     auth
---   UNION ALL
---   SELECT
---     *
---   FROM
---     content
---   UNION ALL
---   SELECT
---     *
---   FROM
---     oauth
--- )
---     --
--- SELECT
---   * EXCEPT (user_properties, event_properties),
---   JSON_VALUE(user_properties, "$.utm_term") AS utm_term,
---   JSON_VALUE(user_properties, "$.utm_source") AS utm_source,
---   JSON_VALUE(user_properties, "$.utm_medium") AS utm_medium,
---   JSON_VALUE(user_properties, "$.utm_campaign") AS utm_campaign,
---   JSON_VALUE(user_properties, "$.utm_content") AS utm_content,
---   JSON_VALUE(user_properties, "$.ua_version") AS ua_version,
---   JSON_VALUE(user_properties, "$.ua_browser") AS ua_browser,
---   JSON_VALUE(user_properties, "$.entrypoint") AS entrypoint,
---   JSON_VALUE(user_properties, "$.flow_id") AS flow_id,
---   JSON_VALUE(event_properties, "$.service") AS service,
---   JSON_VALUE(event_properties, "$.email_type") AS email_type,
---   JSON_VALUE(event_properties, "$.email_provider") AS email_provider,
---   JSON_VALUE(event_properties, "$.oauth_client_id") AS oauth_client_id,
---   JSON_VALUE(event_properties, "$.connect_device_flow") AS connect_device_flow,
---   JSON_VALUE(event_properties, "$.connect_device_os") AS connect_device_os,
---   JSON_VALUE(user_properties, "$.sync_device_count") AS sync_device_count,
---   JSON_VALUE(user_properties, "$.sync_active_devices_day") AS sync_active_devices_day,
---   JSON_VALUE(user_properties, "$.sync_active_devices_week") AS sync_active_devices_week,
---   JSON_VALUE(user_properties, "$.sync_active_devices_month") AS sync_active_devices_month,
---   JSON_VALUE(event_properties, "$.email_sender") AS email_sender,
---   JSON_VALUE(event_properties, "$.email_service") AS email_service,
---   JSON_VALUE(event_properties, "$.email_template") AS email_template,
---   JSON_VALUE(event_properties, "$.email_version") AS email_version,
--- FROM
---   unioned
+  --
+WITH content AS (
+  SELECT
+    jsonPayload.logger,
+    jsonPayload.fields.event_type,
+    jsonPayload.fields.app_version,
+    jsonPayload.fields.os_name,
+    jsonPayload.fields.os_version,
+    jsonPayload.fields.country,
+    jsonPayload.fields.language,
+    jsonPayload.fields.user_id,
+    jsonPayload.fields.user_properties,
+    jsonPayload.fields.event_properties,
+    jsonPayload.fields.device_id,
+    `timestamp`,
+    receiveTimestamp,
+  FROM
+    `moz-fx-data-shared-prod.firefox_accounts_derived.fxa_content_events_v1`
+),
+    --
+auth AS (
+  SELECT
+    jsonPayload.logger,
+    jsonPayload.fields.event_type,
+    jsonPayload.fields.app_version,
+    jsonPayload.fields.os_name,
+    jsonPayload.fields.os_version,
+    jsonPayload.fields.country,
+    jsonPayload.fields.language,
+    jsonPayload.fields.user_id,
+    jsonPayload.fields.user_properties,
+    jsonPayload.fields.event_properties,
+    jsonPayload.fields.device_id,
+    `timestamp`,
+    receiveTimestamp,
+  FROM
+    `moz-fx-data-shared-prod.firefox_accounts_derived.fxa_auth_events_v1`
+),
+    --
+oauth AS (
+  SELECT
+    jsonPayload.logger,
+    jsonPayload.fields.event_type,
+    jsonPayload.fields.app_version,
+    CAST(NULL AS STRING) AS os_name,
+    CAST(NULL AS STRING) AS os_version,
+    CAST(NULL AS STRING) AS country,
+    CAST(NULL AS STRING) AS language,
+    jsonPayload.fields.user_id,
+    jsonPayload.fields.user_properties,
+    jsonPayload.fields.event_properties,
+    CAST(NULL AS STRING) AS device_id,
+    `timestamp`,
+    receiveTimestamp,
+  FROM
+    `moz-fx-data-shared-prod.firefox_accounts_derived.fxa_oauth_events_v1`
+),
+    --
+unioned AS (
+  SELECT
+    *
+  FROM
+    auth
+  UNION ALL
+  SELECT
+    *
+  FROM
+    content
+  UNION ALL
+  SELECT
+    *
+  FROM
+    oauth
+)
+    --
 SELECT
-  ERROR(
-    'VIEW DEPRECATED - This view will be completely deleted after 1st of February 2023, please use `fxa_all_events` with filter on `event_category` instead. See DENG-582 for more info.'
-  )
+  * EXCEPT (user_properties, event_properties),
+  JSON_VALUE(user_properties, "$.utm_term") AS utm_term,
+  JSON_VALUE(user_properties, "$.utm_source") AS utm_source,
+  JSON_VALUE(user_properties, "$.utm_medium") AS utm_medium,
+  JSON_VALUE(user_properties, "$.utm_campaign") AS utm_campaign,
+  JSON_VALUE(user_properties, "$.utm_content") AS utm_content,
+  JSON_VALUE(user_properties, "$.ua_version") AS ua_version,
+  JSON_VALUE(user_properties, "$.ua_browser") AS ua_browser,
+  JSON_VALUE(user_properties, "$.entrypoint") AS entrypoint,
+  JSON_VALUE(user_properties, "$.flow_id") AS flow_id,
+  JSON_VALUE(event_properties, "$.service") AS service,
+  JSON_VALUE(event_properties, "$.email_type") AS email_type,
+  JSON_VALUE(event_properties, "$.email_provider") AS email_provider,
+  JSON_VALUE(event_properties, "$.oauth_client_id") AS oauth_client_id,
+  JSON_VALUE(event_properties, "$.connect_device_flow") AS connect_device_flow,
+  JSON_VALUE(event_properties, "$.connect_device_os") AS connect_device_os,
+  JSON_VALUE(user_properties, "$.sync_device_count") AS sync_device_count,
+  JSON_VALUE(user_properties, "$.sync_active_devices_day") AS sync_active_devices_day,
+  JSON_VALUE(user_properties, "$.sync_active_devices_week") AS sync_active_devices_week,
+  JSON_VALUE(user_properties, "$.sync_active_devices_month") AS sync_active_devices_month,
+  JSON_VALUE(event_properties, "$.email_sender") AS email_sender,
+  JSON_VALUE(event_properties, "$.email_service") AS email_service,
+  JSON_VALUE(event_properties, "$.email_template") AS email_template,
+  JSON_VALUE(event_properties, "$.email_version") AS email_version,
+FROM
+  unioned
Only in /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_accounts/fxa_content_auth_stdout_events: metadata.yaml
diff -bur --no-dereference bigquery-etl-main/sql/moz-fx-data-shared-prod/firefox_accounts/fxa_content_auth_stdout_events/view.sql /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_accounts/fxa_content_auth_stdout_events/view.sql
--- bigquery-etl-main/sql/moz-fx-data-shared-prod/firefox_accounts/fxa_content_auth_stdout_events/view.sql	2023-01-06 20:03:36.836501034 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_accounts/fxa_content_auth_stdout_events/view.sql	2023-01-06 19:57:30.000000000 +0000
@@ -25,113 +25,109 @@
   `moz-fx-data-shared-prod.firefox_accounts.fxa_content_auth_stdout_events`
 AS
   --
--- WITH content AS (
---   SELECT
---     jsonPayload.logger,
---     jsonPayload.fields.event_type,
---     jsonPayload.fields.app_version,
---     jsonPayload.fields.os_name,
---     jsonPayload.fields.os_version,
---     jsonPayload.fields.country,
---     jsonPayload.fields.language,
---     jsonPayload.fields.user_id,
---     jsonPayload.fields.user_properties,
---     jsonPayload.fields.event_properties,
---     `timestamp`,
---     receiveTimestamp,
---   FROM
---     `moz-fx-data-shared-prod.firefox_accounts_derived.fxa_content_events_v1`
--- ),
---   --
--- auth AS (
---   SELECT
---     jsonPayload.logger,
---     jsonPayload.fields.event_type,
---     jsonPayload.fields.app_version,
---     jsonPayload.fields.os_name,
---     jsonPayload.fields.os_version,
---     jsonPayload.fields.country,
---     jsonPayload.fields.language,
---     jsonPayload.fields.user_id,
---     jsonPayload.fields.user_properties,
---     jsonPayload.fields.event_properties,
---     `timestamp`,
---     receiveTimestamp,
---   FROM
---     `moz-fx-data-shared-prod.firefox_accounts_derived.fxa_auth_events_v1`
--- ),
---   --
--- stdout AS (
---   SELECT
---     jsonPayload.logger,
---     jsonPayload.fields.event_type,
---     jsonPayload.fields.app_version,
---     jsonPayload.fields.os_name,
---     jsonPayload.fields.os_version,
---     CAST(NULL AS STRING) AS country,
---     jsonPayload.fields.language,
---     jsonPayload.fields.user_id,
---     jsonPayload.fields.user_properties,
---     jsonPayload.fields.event_properties,
---     `timestamp`,
---     receiveTimestamp,
---   FROM
---     `moz-fx-data-shared-prod.firefox_accounts_derived.fxa_stdout_events_v1`
--- ),
---   --
--- unioned AS (
---   SELECT
---     *
---   FROM
---     auth
---   UNION ALL
---   SELECT
---     *
---   FROM
---     content
---   UNION ALL
---   SELECT
---     *
---   FROM
---     stdout
--- )
---   --
--- SELECT
---   * EXCEPT (user_properties, event_properties),
---   JSON_VALUE(user_properties, "$.utm_term") AS utm_term,
---   JSON_VALUE(user_properties, "$.utm_source") AS utm_source,
---   JSON_VALUE(user_properties, "$.utm_medium") AS utm_medium,
---   JSON_VALUE(user_properties, "$.utm_campaign") AS utm_campaign,
---   JSON_VALUE(user_properties, "$.utm_content") AS utm_content,
---   JSON_VALUE(user_properties, "$.ua_version") AS ua_version,
---   JSON_VALUE(user_properties, "$.ua_browser") AS ua_browser,
---   JSON_VALUE(user_properties, "$.entrypoint") AS entrypoint,
---   JSON_VALUE(user_properties, "$.entrypoint_experiment") AS entrypoint_experiment,
---   JSON_VALUE(user_properties, "$.entrypoint_variation") AS entrypoint_variation,
---   JSON_VALUE(user_properties, "$.flow_id") AS flow_id,
---   JSON_VALUE(event_properties, "$.service") AS service,
---   JSON_VALUE(event_properties, "$.email_type") AS email_type,
---   JSON_VALUE(event_properties, "$.email_provider") AS email_provider,
---   JSON_VALUE(event_properties, "$.oauth_client_id") AS oauth_client_id,
---   JSON_VALUE(event_properties, "$.connect_device_flow") AS connect_device_flow,
---   JSON_VALUE(event_properties, "$.connect_device_os") AS connect_device_os,
---   JSON_VALUE(user_properties, "$.sync_device_count") AS sync_device_count,
---   JSON_VALUE(user_properties, "$.sync_active_devices_day") AS sync_active_devices_day,
---   JSON_VALUE(user_properties, "$.sync_active_devices_week") AS sync_active_devices_week,
---   JSON_VALUE(user_properties, "$.sync_active_devices_month") AS sync_active_devices_month,
---   JSON_VALUE(event_properties, "$.email_sender") AS email_sender,
---   JSON_VALUE(event_properties, "$.email_service") AS email_service,
---   JSON_VALUE(event_properties, "$.email_template") AS email_template,
---   JSON_VALUE(event_properties, "$.email_version") AS email_version,
---   JSON_VALUE(event_properties, "$.plan_id") AS plan_id,
---   JSON_VALUE(event_properties, "$.product_id") AS product_id,
---   JSON_VALUE(event_properties, "$.promotionCode") AS promotion_code,
---   JSON_VALUE(event_properties, "$.payment_provider") AS payment_provider,
---   JSON_VALUE(event_properties, "$.checkout_type") AS checkout_type,
---   JSON_VALUE(event_properties, "$.source_country") AS source_country,
--- FROM
---   unioned
+WITH content AS (
+  SELECT
+    jsonPayload.logger,
+    jsonPayload.fields.event_type,
+    jsonPayload.fields.app_version,
+    jsonPayload.fields.os_name,
+    jsonPayload.fields.os_version,
+    jsonPayload.fields.country,
+    jsonPayload.fields.language,
+    jsonPayload.fields.user_id,
+    jsonPayload.fields.user_properties,
+    jsonPayload.fields.event_properties,
+    `timestamp`,
+    receiveTimestamp,
+  FROM
+    `moz-fx-data-shared-prod.firefox_accounts_derived.fxa_content_events_v1`
+),
+  --
+auth AS (
+  SELECT
+    jsonPayload.logger,
+    jsonPayload.fields.event_type,
+    jsonPayload.fields.app_version,
+    jsonPayload.fields.os_name,
+    jsonPayload.fields.os_version,
+    jsonPayload.fields.country,
+    jsonPayload.fields.language,
+    jsonPayload.fields.user_id,
+    jsonPayload.fields.user_properties,
+    jsonPayload.fields.event_properties,
+    `timestamp`,
+    receiveTimestamp,
+  FROM
+    `moz-fx-data-shared-prod.firefox_accounts_derived.fxa_auth_events_v1`
+),
+  --
+stdout AS (
+  SELECT
+    jsonPayload.logger,
+    jsonPayload.fields.event_type,
+    jsonPayload.fields.app_version,
+    jsonPayload.fields.os_name,
+    jsonPayload.fields.os_version,
+    CAST(NULL AS STRING) AS country,
+    jsonPayload.fields.language,
+    jsonPayload.fields.user_id,
+    jsonPayload.fields.user_properties,
+    jsonPayload.fields.event_properties,
+    `timestamp`,
+    receiveTimestamp,
+  FROM
+    `moz-fx-data-shared-prod.firefox_accounts_derived.fxa_stdout_events_v1`
+),
+  --
+unioned AS (
+  SELECT
+    *
+  FROM
+    auth
+  UNION ALL
+  SELECT
+    *
+  FROM
+    content
+  UNION ALL
+  SELECT
+    *
+  FROM
+    stdout
+)
+  --
 SELECT
-  ERROR(
-    'VIEW DEPRECATED - This view will be completely deleted after 1st of February 2023, please use `fxa_all_events` with filter on `event_category` instead. See DENG-582 for more info.'
-  )
+  * EXCEPT (user_properties, event_properties),
+  JSON_VALUE(user_properties, "$.utm_term") AS utm_term,
+  JSON_VALUE(user_properties, "$.utm_source") AS utm_source,
+  JSON_VALUE(user_properties, "$.utm_medium") AS utm_medium,
+  JSON_VALUE(user_properties, "$.utm_campaign") AS utm_campaign,
+  JSON_VALUE(user_properties, "$.utm_content") AS utm_content,
+  JSON_VALUE(user_properties, "$.ua_version") AS ua_version,
+  JSON_VALUE(user_properties, "$.ua_browser") AS ua_browser,
+  JSON_VALUE(user_properties, "$.entrypoint") AS entrypoint,
+  JSON_VALUE(user_properties, "$.entrypoint_experiment") AS entrypoint_experiment,
+  JSON_VALUE(user_properties, "$.entrypoint_variation") AS entrypoint_variation,
+  JSON_VALUE(user_properties, "$.flow_id") AS flow_id,
+  JSON_VALUE(event_properties, "$.service") AS service,
+  JSON_VALUE(event_properties, "$.email_type") AS email_type,
+  JSON_VALUE(event_properties, "$.email_provider") AS email_provider,
+  JSON_VALUE(event_properties, "$.oauth_client_id") AS oauth_client_id,
+  JSON_VALUE(event_properties, "$.connect_device_flow") AS connect_device_flow,
+  JSON_VALUE(event_properties, "$.connect_device_os") AS connect_device_os,
+  JSON_VALUE(user_properties, "$.sync_device_count") AS sync_device_count,
+  JSON_VALUE(user_properties, "$.sync_active_devices_day") AS sync_active_devices_day,
+  JSON_VALUE(user_properties, "$.sync_active_devices_week") AS sync_active_devices_week,
+  JSON_VALUE(user_properties, "$.sync_active_devices_month") AS sync_active_devices_month,
+  JSON_VALUE(event_properties, "$.email_sender") AS email_sender,
+  JSON_VALUE(event_properties, "$.email_service") AS email_service,
+  JSON_VALUE(event_properties, "$.email_template") AS email_template,
+  JSON_VALUE(event_properties, "$.email_version") AS email_version,
+  JSON_VALUE(event_properties, "$.plan_id") AS plan_id,
+  JSON_VALUE(event_properties, "$.product_id") AS product_id,
+  JSON_VALUE(event_properties, "$.promotionCode") AS promotion_code,
+  JSON_VALUE(event_properties, "$.payment_provider") AS payment_provider,
+  JSON_VALUE(event_properties, "$.checkout_type") AS checkout_type,
+  JSON_VALUE(event_properties, "$.source_country") AS source_country,
+FROM
+  unioned
Only in bigquery-etl-main/sql/moz-fx-data-shared-prod/firefox_accounts/fxa_users_last_seen: schema.yaml
Only in bigquery-etl-main/sql/moz-fx-data-shared-prod/firefox_accounts_derived/docker_fxa_admin_server_sanitized_v1: init.sql
Only in /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_accounts_derived/docker_fxa_admin_server_sanitized_v1: inti.sql
diff -bur --no-dereference bigquery-etl-main/sql/moz-fx-data-shared-prod/firefox_accounts_derived/docker_fxa_admin_server_sanitized_v1/query.sql /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_accounts_derived/docker_fxa_admin_server_sanitized_v1/query.sql
--- bigquery-etl-main/sql/moz-fx-data-shared-prod/firefox_accounts_derived/docker_fxa_admin_server_sanitized_v1/query.sql	2023-01-06 20:03:36.840501016 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_accounts_derived/docker_fxa_admin_server_sanitized_v1/query.sql	2023-01-06 19:57:30.000000000 +0000
@@ -16,6 +16,6 @@
     ) AS jsonPayload
   )
 FROM
-  `moz-fx-fxa-prod-0712.fxa_prod_logs.docker_fxa_admin_server`
+  `moz-fx-fxa-prod-0712.fxa_prod_logs.docker_fxa_admin_server_20*`
 WHERE
-  DATE(`timestamp`) = @submission_date
+  _TABLE_SUFFIX = FORMAT_DATE('%y%m%d', @submission_date)
diff -bur --no-dereference bigquery-etl-main/sql/moz-fx-data-shared-prod/firefox_accounts_derived/docker_fxa_customs_sanitized_v1/init.sql /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_accounts_derived/docker_fxa_customs_sanitized_v1/init.sql
--- bigquery-etl-main/sql/moz-fx-data-shared-prod/firefox_accounts_derived/docker_fxa_customs_sanitized_v1/init.sql	2023-01-06 20:03:36.840501016 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_accounts_derived/docker_fxa_customs_sanitized_v1/init.sql	2023-01-06 19:57:30.000000000 +0000
@@ -13,6 +13,6 @@
     ) AS jsonPayload
   )
 FROM
-  `moz-fx-fxa-prod-0712.fxa_prod_logs.docker_fxa_customs`
+  `moz-fx-fxa-prod-0712.fxa_prod_logs.docker_fxa_customs*`
 WHERE
-  DATE(`timestamp`) = "2020-01-01"
+  _TABLE_SUFFIX >= FORMAT_DATE('%y%m%d', "2020-01-01")
diff -bur --no-dereference bigquery-etl-main/sql/moz-fx-data-shared-prod/firefox_accounts_derived/docker_fxa_customs_sanitized_v1/query.sql /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_accounts_derived/docker_fxa_customs_sanitized_v1/query.sql
--- bigquery-etl-main/sql/moz-fx-data-shared-prod/firefox_accounts_derived/docker_fxa_customs_sanitized_v1/query.sql	2023-01-06 20:03:36.840501016 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_accounts_derived/docker_fxa_customs_sanitized_v1/query.sql	2023-01-06 19:57:30.000000000 +0000
@@ -8,6 +8,6 @@
     ) AS jsonPayload
   )
 FROM
-  `moz-fx-fxa-prod-0712.fxa_prod_logs.docker_fxa_customs`
+  `moz-fx-fxa-prod-0712.fxa_prod_logs.docker_fxa_customs_20*`
 WHERE
-  DATE(`timestamp`) = @submission_date
+  _TABLE_SUFFIX = FORMAT_DATE('%y%m%d', @submission_date)
diff -bur --no-dereference bigquery-etl-main/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_amplitude_export_v1/metadata.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_amplitude_export_v1/metadata.yaml
--- bigquery-etl-main/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_amplitude_export_v1/metadata.yaml	2023-01-06 20:07:58.295409777 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_amplitude_export_v1/metadata.yaml	2023-01-06 20:00:33.000000000 +0000
@@ -16,4 +16,4 @@
   - firefox_accounts_derived.fxa_amplitude_export_v1
   - moz-fx-data-shared-prod.firefox_accounts_derived.encrypted_keys_v1
   - moz-fx-dataops-secrets.airflow_query_keys.fxa_prod
-  - moz-fx-fxa-prod-0712.fxa_prod_logs.docker_fxa_auth
+  - moz-fx-fxa-prod-0712.fxa_prod_logs.docker_fxa_auth_20*
diff -bur --no-dereference bigquery-etl-main/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_amplitude_export_v1/query.sql /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_amplitude_export_v1/query.sql
--- bigquery-etl-main/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_amplitude_export_v1/query.sql	2023-01-06 20:03:36.840501016 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_amplitude_export_v1/query.sql	2023-01-06 19:57:30.000000000 +0000
@@ -14,7 +14,7 @@
   SELECT
     *
   FROM
-    `moz-fx-fxa-prod-0712.fxa_prod_logs.docker_fxa_auth`
+    `moz-fx-fxa-prod-0712.fxa_prod_logs.docker_fxa_auth_20*`
   WHERE
     -- @submission_date is PDT, so we need two days of UTC-based
     -- data. We assume that we run immediately at the end of
@@ -22,8 +22,8 @@
     -- submission date.
     -- See https://console.cloud.google.com/bigquery?sq=768515352537:e63d2d2faa85431dbf0e5440021af837
     (
-      DATE(`timestamp`) = @submission_date
-      OR DATE(`timestamp`) = DATE_ADD(@submission_date, INTERVAL 1 DAY)
+      _TABLE_SUFFIX = FORMAT_DATE('%y%m%d', @submission_date)
+      OR _TABLE_SUFFIX = FORMAT_DATE('%y%m%d', DATE_ADD(@submission_date, INTERVAL 1 DAY))
     )
     AND DATE(`timestamp`, "America/Los_Angeles") = @submission_date
     AND jsonPayload.fields.event_type IN (
diff -bur --no-dereference bigquery-etl-main/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_amplitude_user_ids_v1/init.sql /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_amplitude_user_ids_v1/init.sql
--- bigquery-etl-main/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_amplitude_user_ids_v1/init.sql	2023-01-06 20:03:36.840501016 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_amplitude_user_ids_v1/init.sql	2023-01-06 19:57:30.000000000 +0000
@@ -18,4 +18,4 @@
     udf.hmac_sha256((SELECT * FROM hmac_key), CAST(jsonPayload.fields.user_id AS BYTES))
   ) AS user_id
 FROM
-  `moz-fx-fxa-prod-0712.fxa_prod_logs.docker_fxa_auth`
+  `moz-fx-fxa-prod-0712.fxa_prod_logs.docker_fxa_auth_20*`
diff -bur --no-dereference bigquery-etl-main/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_amplitude_user_ids_v1/metadata.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_amplitude_user_ids_v1/metadata.yaml
--- bigquery-etl-main/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_amplitude_user_ids_v1/metadata.yaml	2023-01-06 20:07:58.299409760 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_amplitude_user_ids_v1/metadata.yaml	2023-01-06 20:00:33.000000000 +0000
@@ -17,9 +17,9 @@
   init.sql:
   - moz-fx-data-shared-prod.firefox_accounts_derived.encrypted_keys_v1
   - moz-fx-dataops-secrets.airflow_query_keys.fxa_prod
-  - moz-fx-fxa-prod-0712.fxa_prod_logs.docker_fxa_auth
+  - moz-fx-fxa-prod-0712.fxa_prod_logs.docker_fxa_auth_20*
   query.sql:
   - fxa_amplitude_user_ids_v1
   - moz-fx-data-shared-prod.firefox_accounts_derived.encrypted_keys_v1
   - moz-fx-dataops-secrets.airflow_query_keys.fxa_prod
-  - moz-fx-fxa-prod-0712.fxa_prod_logs.docker_fxa_auth
+  - moz-fx-fxa-prod-0712.fxa_prod_logs.docker_fxa_auth_20*
diff -bur --no-dereference bigquery-etl-main/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_amplitude_user_ids_v1/query.sql /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_amplitude_user_ids_v1/query.sql
--- bigquery-etl-main/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_amplitude_user_ids_v1/query.sql	2023-01-06 20:03:36.840501016 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_amplitude_user_ids_v1/query.sql	2023-01-06 19:57:30.000000000 +0000
@@ -15,9 +15,9 @@
     udf.hmac_sha256((SELECT * FROM hmac_key), CAST(jsonPayload.fields.user_id AS BYTES))
   ) AS user_id
 FROM
-  `moz-fx-fxa-prod-0712.fxa_prod_logs.docker_fxa_auth`
+  `moz-fx-fxa-prod-0712.fxa_prod_logs.docker_fxa_auth_20*`
 WHERE
-  DATE(`timestamp`) = @submission_date
+  _TABLE_SUFFIX = FORMAT_DATE('%y%m%d', @submission_date)
 UNION DISTINCT
 SELECT
   user_id
diff -bur --no-dereference bigquery-etl-main/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_auth_bounce_events_v1/metadata.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_auth_bounce_events_v1/metadata.yaml
--- bigquery-etl-main/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_auth_bounce_events_v1/metadata.yaml	2023-01-06 20:07:58.303409744 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_auth_bounce_events_v1/metadata.yaml	2023-01-06 20:00:33.000000000 +0000
@@ -19,4 +19,4 @@
 # Generated by bigquery_etl.dependency
 references:
   query.sql:
-  - moz-fx-fxa-prod-0712.fxa_prod_logs.docker_fxa_auth_bounces
+  - moz-fx-fxa-prod-0712.fxa_prod_logs.docker_fxa_auth_bounces_20*
diff -bur --no-dereference bigquery-etl-main/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_auth_bounce_events_v1/query.sql /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_auth_bounce_events_v1/query.sql
--- bigquery-etl-main/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_auth_bounce_events_v1/query.sql	2023-01-06 20:03:36.840501016 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_auth_bounce_events_v1/query.sql	2023-01-06 19:57:30.000000000 +0000
@@ -13,9 +13,9 @@
     ) AS jsonPayload
   )
 FROM
-  `moz-fx-fxa-prod-0712.fxa_prod_logs.docker_fxa_auth_bounces`
+  `moz-fx-fxa-prod-0712.fxa_prod_logs.docker_fxa_auth_bounces_20*`
 WHERE
   jsonPayload.type = 'amplitudeEvent'
   AND jsonPayload.fields.event_type IS NOT NULL
   AND jsonPayload.fields.user_id IS NOT NULL
-  AND DATE(`timestamp`) = @submission_date
+  AND _TABLE_SUFFIX = FORMAT_DATE('%y%m%d', @submission_date)
diff -bur --no-dereference bigquery-etl-main/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_auth_events_v1/metadata.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_auth_events_v1/metadata.yaml
--- bigquery-etl-main/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_auth_events_v1/metadata.yaml	2023-01-06 20:07:58.307409728 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_auth_events_v1/metadata.yaml	2023-01-06 20:00:33.000000000 +0000
@@ -15,4 +15,4 @@
 # Generated by bigquery_etl.dependency
 references:
   query.sql:
-  - moz-fx-fxa-prod-0712.fxa_prod_logs.docker_fxa_auth
+  - moz-fx-fxa-prod-0712.fxa_prod_logs.docker_fxa_auth_20*
diff -bur --no-dereference bigquery-etl-main/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_auth_events_v1/query.sql /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_auth_events_v1/query.sql
--- bigquery-etl-main/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_auth_events_v1/query.sql	2023-01-06 20:03:36.840501016 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_auth_events_v1/query.sql	2023-01-06 19:57:30.000000000 +0000
@@ -27,9 +27,9 @@
       ) AS jsonPayload
     )
   FROM
-    `moz-fx-fxa-prod-0712.fxa_prod_logs.docker_fxa_auth`
+    `moz-fx-fxa-prod-0712.fxa_prod_logs.docker_fxa_auth_20*`
   WHERE
-    DATE(`timestamp`) = @submission_date
+    _TABLE_SUFFIX = FORMAT_DATE('%y%m%d', @submission_date)
 )
   --
 SELECT
diff -bur --no-dereference bigquery-etl-main/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_content_events_v1/metadata.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_content_events_v1/metadata.yaml
--- bigquery-etl-main/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_content_events_v1/metadata.yaml	2023-01-06 20:07:58.311409711 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_content_events_v1/metadata.yaml	2023-01-06 20:00:33.000000000 +0000
@@ -15,4 +15,4 @@
 # Generated by bigquery_etl.dependency
 references:
   query.sql:
-  - moz-fx-fxa-prod-0712.fxa_prod_logs.docker_fxa_content
+  - moz-fx-fxa-prod-0712.fxa_prod_logs.docker_fxa_content_20*
diff -bur --no-dereference bigquery-etl-main/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_content_events_v1/query.sql /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_content_events_v1/query.sql
--- bigquery-etl-main/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_content_events_v1/query.sql	2023-01-06 20:03:36.840501016 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_content_events_v1/query.sql	2023-01-06 19:57:30.000000000 +0000
@@ -13,8 +13,8 @@
     ) AS jsonPayload
   )
 FROM
-  `moz-fx-fxa-prod-0712.fxa_prod_logs.docker_fxa_content`
+  `moz-fx-fxa-prod-0712.fxa_prod_logs.docker_fxa_content_20*`
 WHERE
   jsonPayload.type = 'amplitudeEvent'
   AND jsonPayload.fields.event_type IS NOT NULL
-  AND DATE(`timestamp`) = @submission_date
+  AND _TABLE_SUFFIX = FORMAT_DATE('%y%m%d', @submission_date)
diff -bur --no-dereference bigquery-etl-main/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_delete_events_v1/metadata.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_delete_events_v1/metadata.yaml
--- bigquery-etl-main/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_delete_events_v1/metadata.yaml	2023-01-06 20:07:58.315409694 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_delete_events_v1/metadata.yaml	2023-01-06 20:00:33.000000000 +0000
@@ -22,4 +22,4 @@
   query.sql:
   - moz-fx-data-shared-prod.firefox_accounts_derived.encrypted_keys_v1
   - moz-fx-dataops-secrets.airflow_query_keys.fxa_prod
-  - moz-fx-fxa-prod-0712.fxa_prod_logs.docker_fxa_auth
+  - moz-fx-fxa-prod-0712.fxa_prod_logs.docker_fxa_auth_20*
diff -bur --no-dereference bigquery-etl-main/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_delete_events_v1/query.sql /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_delete_events_v1/query.sql
--- bigquery-etl-main/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_delete_events_v1/query.sql	2023-01-06 20:03:36.840501016 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_delete_events_v1/query.sql	2023-01-06 19:57:30.000000000 +0000
@@ -17,9 +17,9 @@
     udf.hmac_sha256((SELECT * FROM hmac_key), CAST(jsonPayload.fields.uid AS BYTES))
   ) AS hmac_user_id,
 FROM
-  `moz-fx-fxa-prod-0712.fxa_prod_logs.docker_fxa_auth`
+  `moz-fx-fxa-prod-0712.fxa_prod_logs.docker_fxa_auth_20*`
 WHERE
-  DATE(`timestamp`) = @submission_date
+  _TABLE_SUFFIX = FORMAT_DATE('%y%m%d', @submission_date)
   AND jsonPayload.type = 'activityEvent'
   AND jsonPayload.fields.event = 'account.deleted'
   AND jsonPayload.fields.uid IS NOT NULL
diff -bur --no-dereference bigquery-etl-main/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_log_auth_events_v1/metadata.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_log_auth_events_v1/metadata.yaml
--- bigquery-etl-main/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_log_auth_events_v1/metadata.yaml	2023-01-06 20:07:58.319409678 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_log_auth_events_v1/metadata.yaml	2023-01-06 20:00:33.000000000 +0000
@@ -18,4 +18,4 @@
 # Generated by bigquery_etl.dependency
 references:
   query.sql:
-  - moz-fx-fxa-prod-0712.fxa_prod_logs.docker_fxa_auth
+  - moz-fx-fxa-prod-0712.fxa_prod_logs.docker_fxa_auth_20*
diff -bur --no-dereference bigquery-etl-main/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_log_auth_events_v1/query.sql /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_log_auth_events_v1/query.sql
--- bigquery-etl-main/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_log_auth_events_v1/query.sql	2023-01-06 20:03:36.840501016 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_log_auth_events_v1/query.sql	2023-01-06 19:57:30.000000000 +0000
@@ -13,7 +13,7 @@
   jsonPayload.fields.utm_medium,
   jsonPayload.fields.utm_term
 FROM
-  `moz-fx-fxa-prod-0712.fxa_prod_logs.docker_fxa_auth`
+  `moz-fx-fxa-prod-0712.fxa_prod_logs.docker_fxa_auth_20*`
 WHERE
   -- We exclude event types that are Amplitude events;
   -- these are already captured in the fxa_auth_events_v1 table.
@@ -33,4 +33,4 @@
       'sms.installFirefox.sent'
     )
   )
-  AND DATE(`timestamp`) = @submission_date
+  AND _TABLE_SUFFIX = FORMAT_DATE('%y%m%d', @submission_date)
diff -bur --no-dereference bigquery-etl-main/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_log_content_events_v1/metadata.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_log_content_events_v1/metadata.yaml
--- bigquery-etl-main/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_log_content_events_v1/metadata.yaml	2023-01-06 20:07:58.323409661 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_log_content_events_v1/metadata.yaml	2023-01-06 20:00:33.000000000 +0000
@@ -18,4 +18,4 @@
 # Generated by bigquery_etl.dependency
 references:
   query.sql:
-  - moz-fx-fxa-prod-0712.fxa_prod_logs.docker_fxa_content
+  - moz-fx-fxa-prod-0712.fxa_prod_logs.docker_fxa_content_19700101
diff -bur --no-dereference bigquery-etl-main/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_log_content_events_v1/query.sql /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_log_content_events_v1/query.sql
--- bigquery-etl-main/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_log_content_events_v1/query.sql	2023-01-06 20:03:36.840501016 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_log_content_events_v1/query.sql	2023-01-06 19:57:30.000000000 +0000
@@ -15,9 +15,7 @@
     jsonPayload.utm_medium,
     jsonPayload.utm_source,
   FROM
-    `moz-fx-fxa-prod-0712.fxa_prod_logs.docker_fxa_content`
-  WHERE
-    DATE(`timestamp`) = "1970-01-01"
+    `moz-fx-fxa-prod-0712.fxa_prod_logs.docker_fxa_content_19700101`
 )
 SELECT
   *
diff -bur --no-dereference bigquery-etl-main/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_log_device_command_events_v1/metadata.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_log_device_command_events_v1/metadata.yaml
--- bigquery-etl-main/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_log_device_command_events_v1/metadata.yaml	2023-01-06 20:07:58.327409644 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_log_device_command_events_v1/metadata.yaml	2023-01-06 20:00:33.000000000 +0000
@@ -23,4 +23,4 @@
   query.sql:
   - moz-fx-data-shared-prod.firefox_accounts_derived.encrypted_keys_v1
   - moz-fx-dataops-secrets.airflow_query_keys.fxa_prod
-  - moz-fx-fxa-prod-0712.fxa_prod_logs.docker_fxa_auth
+  - moz-fx-fxa-prod-0712.fxa_prod_logs.docker_fxa_auth_20*
diff -bur --no-dereference bigquery-etl-main/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_log_device_command_events_v1/query.sql /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_log_device_command_events_v1/query.sql
--- bigquery-etl-main/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_log_device_command_events_v1/query.sql	2023-01-06 20:03:36.840501016 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_log_device_command_events_v1/query.sql	2023-01-06 19:57:30.000000000 +0000
@@ -35,10 +35,10 @@
   jsonPayload.fields.senderOS AS sender_os,
   jsonPayload.fields.senderType AS sender_type,
 FROM
-  `moz-fx-fxa-prod-0712.fxa_prod_logs.docker_fxa_auth`
+  `moz-fx-fxa-prod-0712.fxa_prod_logs.docker_fxa_auth_20*`
 WHERE
   jsonPayload.type LIKE 'device.command.%'
   -- Device command metrics were first deployed and stable on 2020-07-08;
   -- there is some data for earlier dates but it's from a failed deployment so we don't count it.
-  AND DATE(`timestamp`) >= "2020-07-08"
-  AND DATE(`timestamp`) = @submission_date
+  AND _TABLE_SUFFIX >= '200708'
+  AND _TABLE_SUFFIX = FORMAT_DATE('%y%m%d', @submission_date)
diff -bur --no-dereference bigquery-etl-main/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_stdout_events_v1/metadata.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_stdout_events_v1/metadata.yaml
--- bigquery-etl-main/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_stdout_events_v1/metadata.yaml	2023-01-06 20:07:58.335409611 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_stdout_events_v1/metadata.yaml	2023-01-06 20:00:33.000000000 +0000
@@ -14,4 +14,4 @@
 # Generated by bigquery_etl.dependency
 references:
   query.sql:
-  - moz-fx-fxa-prod-0712.fxa_prod_logs.stdout
+  - moz-fx-fxa-prod-0712.fxa_prod_logs.stdout_20*
diff -bur --no-dereference bigquery-etl-main/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_stdout_events_v1/query.sql /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_stdout_events_v1/query.sql
--- bigquery-etl-main/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_stdout_events_v1/query.sql	2023-01-06 20:03:36.840501016 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_stdout_events_v1/query.sql	2023-01-06 19:57:30.000000000 +0000
@@ -13,8 +13,8 @@
     ) AS jsonPayload
   )
 FROM
-  `moz-fx-fxa-prod-0712.fxa_prod_logs.stdout`
+  `moz-fx-fxa-prod-0712.fxa_prod_logs.stdout_20*`
 WHERE
   jsonPayload.type = 'amplitudeEvent'
   AND jsonPayload.fields.event_type IS NOT NULL
-  AND DATE(`timestamp`) = @submission_date
+  AND _TABLE_SUFFIX = FORMAT_DATE('%y%m%d', @submission_date)
diff -bur --no-dereference bigquery-etl-main/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_users_daily_v1/metadata.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_users_daily_v1/metadata.yaml
--- bigquery-etl-main/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_users_daily_v1/metadata.yaml	2023-01-06 20:07:58.339409595 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_users_daily_v1/metadata.yaml	2023-01-06 20:00:33.000000000 +0000
@@ -1,26 +1,14 @@
 ---
 friendly_name: FxA Users Daily
-description: |
-  Usage aggregations per FxA user per day.
-  Only the following category of events is used for building this models:
-  event_category IN ('auth', 'auth_bounce', 'content', 'oauth')
-  Partitioned by submission_date and clustered by user_id.
+description: Usage aggregations per FxA user per day
 owners:
-  - [email protected]
+  - [email protected]
 labels:
   application: fxa
   incremental: true
   schedule: daily
 scheduling:
   dag_name: bqetl_fxa_events
-bigquery:
-  time_partitioning:
-    field: submission_date
-    type: day
-    require_partition_filter: false
-  clustering:
-    fields:
-      - user_id
 
 # Generated by bigquery_etl.dependency
 references:
Only in bigquery-etl-main/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_users_daily_v1: schema.yaml
diff -bur --no-dereference bigquery-etl-main/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_users_last_seen_v1/init.sql /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_users_last_seen_v1/init.sql
--- bigquery-etl-main/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_users_last_seen_v1/init.sql	2023-01-06 20:03:36.840501016 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_users_last_seen_v1/init.sql	2023-01-06 19:57:30.000000000 +0000
@@ -1,5 +1,5 @@
 CREATE TABLE IF NOT EXISTS
-  `moz-fx-data-shared-prod.firefox_accounts_derived.fxa_users_last_seen_v1`
+  fxa_users_last_seen_v1
 PARTITION BY
   submission_date
 CLUSTER BY
diff -bur --no-dereference bigquery-etl-main/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_users_last_seen_v1/metadata.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_users_last_seen_v1/metadata.yaml
--- bigquery-etl-main/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_users_last_seen_v1/metadata.yaml	2023-01-06 20:07:58.355409528 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_users_last_seen_v1/metadata.yaml	2023-01-06 20:00:33.000000000 +0000
@@ -1,16 +1,8 @@
 ---
 friendly_name: FxA Users Last Seen
-description: |
-  Usage aggregations per FxA user per day over a 28-day window.
-  The fields containing the 28-day user activity info include:
-  days_seen_bits, days_seen_in_tier1_country_bits, days_registered_bits
-  days_seen_no_monitor_bits
-  These fields are calculated from `fxa_users_daily_v1` table
-  which only looks at the following category of events:
-  auth, auth_bounce, content, oauth.
-
+description: Usage aggregations per FxA user per day over a 28-day window
 owners:
-  - [email protected]
+  - [email protected]
 labels:
   application: fxa
   incremental: true
@@ -19,20 +11,11 @@
   dag_name: bqetl_fxa_events
   depends_on_past: true
   start_date: '2019-04-23'
-bigquery:
-  time_partitioning:
-    field: submission_date
-    type: day
-    require_partition_filter: true
-    expiration_days: null
-  clustering:
-    fields:
-      - user_id
 
 # Generated by bigquery_etl.dependency
 references:
   init.sql:
   - fxa_users_daily_v1
   query.sql:
-  - moz-fx-data-shared-prod.firefox_accounts.fxa_users_daily
-  - moz-fx-data-shared-prod.firefox_accounts_derived.fxa_users_last_seen_v1
+  - fxa_users_daily_v1
+  - fxa_users_last_seen_v1
diff -bur --no-dereference bigquery-etl-main/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_users_last_seen_v1/query.sql /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_users_last_seen_v1/query.sql
--- bigquery-etl-main/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_users_last_seen_v1/query.sql	2023-01-06 20:03:36.840501016 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_users_last_seen_v1/query.sql	2023-01-06 19:57:30.000000000 +0000
@@ -17,7 +17,7 @@
       IF(submission_date < '2019-11-25', TRUE, monitor_only IS FALSE) AS INT64
     ) AS days_seen_no_monitor_bits
   FROM
-    `moz-fx-data-shared-prod.firefox_accounts.fxa_users_daily`
+    fxa_users_daily_v1
   WHERE
     submission_date = @submission_date
 ),
@@ -31,7 +31,7 @@
       ) AS days_seen_no_monitor_bits
     )
   FROM
-    `moz-fx-data-shared-prod.firefox_accounts_derived.fxa_users_last_seen_v1`
+    fxa_users_last_seen_v1
   WHERE
     submission_date = DATE_SUB(@submission_date, INTERVAL 1 DAY)
     -- Filter out rows from yesterday that have now fallen outside the 28-day window.
Only in bigquery-etl-main/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_users_last_seen_v1: schema.yaml
Only in bigquery-etl-main/sql/moz-fx-data-shared-prod: google_ads
Only in bigquery-etl-main/sql/moz-fx-data-shared-prod: google_ads_derived
Only in bigquery-etl-main/sql/moz-fx-data-shared-prod/monitoring/bigquery_tables_inventory: schema.yaml
Only in /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/monitoring: bigquery_tables_last_modified
diff -bur --no-dereference bigquery-etl-main/sql/moz-fx-data-shared-prod/monitoring_derived/bigquery_table_storage_timeline_daily_v1/metadata.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/monitoring_derived/bigquery_table_storage_timeline_daily_v1/metadata.yaml
--- bigquery-etl-main/sql/moz-fx-data-shared-prod/monitoring_derived/bigquery_table_storage_timeline_daily_v1/metadata.yaml	2023-01-06 20:03:36.852500960 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/monitoring_derived/bigquery_table_storage_timeline_daily_v1/metadata.yaml	2023-01-06 19:57:30.000000000 +0000
@@ -1,4 +1,4 @@
-friendly_name: BigQuery Tables Storage Timeline Daily
+friendly_name: Bigquery Query Tables Storage Timeline Daily
 description: The daily timeline in table storage changes, partitioned by day.
 owners:
 - [email protected]
diff -bur --no-dereference bigquery-etl-main/sql/moz-fx-data-shared-prod/monitoring_derived/bigquery_table_storage_v1/metadata.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/monitoring_derived/bigquery_table_storage_v1/metadata.yaml
--- bigquery-etl-main/sql/moz-fx-data-shared-prod/monitoring_derived/bigquery_table_storage_v1/metadata.yaml	2023-01-06 20:03:36.852500960 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/monitoring_derived/bigquery_table_storage_v1/metadata.yaml	2023-01-06 19:57:30.000000000 +0000
@@ -1,5 +1,5 @@
-friendly_name: BigQuery Tables Storage
-description: BigQuery table storage size and costs.
+friendly_name: Bigquery Query Tables Storage
+description: Bigquery table storage size and costs.
 owners:
 - [email protected]
 labels:
diff -bur --no-dereference bigquery-etl-main/sql/moz-fx-data-shared-prod/monitoring_derived/bigquery_tables_inventory_v1/metadata.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/monitoring_derived/bigquery_tables_inventory_v1/metadata.yaml
--- bigquery-etl-main/sql/moz-fx-data-shared-prod/monitoring_derived/bigquery_tables_inventory_v1/metadata.yaml	2023-01-06 20:03:36.852500960 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/monitoring_derived/bigquery_tables_inventory_v1/metadata.yaml	2023-01-06 19:57:30.000000000 +0000
@@ -1,5 +1,5 @@
-friendly_name: BigQuery Tables Inventory
-description: Inventory of BigQuery tables.
+friendly_name: Bigquery Query Tables Inventory
+description: Inventory of big query tables.
 owners:
 - [email protected]
 labels:
@@ -7,8 +7,3 @@
   schedule: daily
 scheduling:
   dag_name: bqetl_monitoring
-  arguments: ["--date", "{{ ds }}"]
-bigquery:
-  time_partitioning:
-    field: submission_date
-    type: day
diff -bur --no-dereference bigquery-etl-main/sql/moz-fx-data-shared-prod/monitoring_derived/bigquery_tables_inventory_v1/query.py /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/monitoring_derived/bigquery_tables_inventory_v1/query.py
--- bigquery-etl-main/sql/moz-fx-data-shared-prod/monitoring_derived/bigquery_tables_inventory_v1/query.py	2023-01-06 20:03:36.852500960 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/monitoring_derived/bigquery_tables_inventory_v1/query.py	2023-01-06 19:57:30.000000000 +0000
@@ -16,72 +16,24 @@
 ]
 
 parser = ArgumentParser(description=__doc__)
-parser.add_argument("--date", required=True)  # expect string with format yyyy-mm-dd
 parser.add_argument("--project", default="moz-fx-data-shared-prod")
 # projects queries were run from that access table
 parser.add_argument("--source_projects", nargs="+", default=DEFAULT_PROJECTS)
 parser.add_argument("--destination_dataset", default="monitoring_derived")
 parser.add_argument("--destination_table", default="bigquery_tables_inventory_v1")
-parser.add_argument("--tmp_table", default="bigquery_tables_last_modified_tmp")
 
 
-def create_last_modified_tmp_table(project, tmp_table_name):
-    """Create temp table to capture last modified dates."""
-    # remove old table in case of re-run
-    client = bigquery.Client(project)
-    client.delete_table(tmp_table_name, not_found_ok=True)
-
-    tmp_table = bigquery.Table(tmp_table_name)
-    tmp_table.schema = (
-        bigquery.SchemaField("project_id", "STRING"),
-        bigquery.SchemaField("dataset_id", "STRING"),
-        bigquery.SchemaField("table_id", "STRING"),
-        bigquery.SchemaField("creation_date", "DATE"),
-        bigquery.SchemaField("last_modified_date", "DATE"),
-    )
-    client.create_table(tmp_table)
-
-    datasets = list(client.list_datasets())
-
-    for dataset in datasets:
-
-        query = f"""
-              SELECT
-                project_id,
-                dataset_id,
-                table_id,
-                DATE(TIMESTAMP_MILLIS(creation_time)) AS creation_date,
-                DATE(TIMESTAMP_MILLIS(last_modified_time)) AS last_modified_date
-                FROM `{project}.{dataset.dataset_id}.__TABLES__`
-        """
-
-        try:
-            job_config = bigquery.QueryJobConfig(
-                destination=tmp_table_name, write_disposition="WRITE_APPEND"
-            )
-            client.query(query, job_config=job_config).result()
-
-        except Exception as e:
-            print(f"Error querying dataset {dataset.dataset_id}: {e}")
-
-
-def create_query(date, source_project, tmp_table_name):
+def create_query(source_project):
     """Create query for a source project."""
     return f"""
-        SELECT *
-        FROM
-            (SELECT
-            DATE('{date}') AS submission_date,
+        SELECT
             DATE(creation_time) AS creation_date,
             table_catalog AS project_id,
             table_schema AS dataset_id,
             table_name AS table_id,
             table_type,
             FROM `{source_project}.region-us.INFORMATION_SCHEMA.TABLES`
-            )
-        LEFT JOIN {tmp_table_name}
-        USING (project_id, dataset_id, table_id, creation_date)
-        ORDER BY submission_date, project_id, dataset_id, table_id, table_type
+        ORDER BY creation_date, project_id, dataset_id, table_id, table_type
     """
 
 
@@ -89,29 +41,22 @@
     """Run query for each source project."""
     args = parser.parse_args()
 
-    partition = args.date.replace("-", "")
-
-    tmp_table_name = f"{args.project}.{args.destination_dataset}.{args.tmp_table}"
-
-    destination_table = f"{args.project}.{args.destination_dataset}.{args.destination_table}${partition}"
+    destination_table = (
+        f"{args.project}.{args.destination_dataset}.{args.destination_table}"
+    )
 
     # remove old table in case of re-run
     client = bigquery.Client(args.project)
     client.delete_table(destination_table, not_found_ok=True)
 
     for project in args.source_projects:
-
-        create_last_modified_tmp_table(project, tmp_table_name)
-
         client = bigquery.Client(project)
-        query = create_query(args.date, project, tmp_table_name)
+        query = create_query(project)
         job_config = bigquery.QueryJobConfig(
             destination=destination_table, write_disposition="WRITE_APPEND"
         )
         client.query(query, job_config=job_config).result()
 
-    client.delete_table(tmp_table_name, not_found_ok=True)
-
 
 if __name__ == "__main__":
     main()
diff -bur --no-dereference bigquery-etl-main/sql/moz-fx-data-shared-prod/monitoring_derived/bigquery_tables_inventory_v1/schema.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/monitoring_derived/bigquery_tables_inventory_v1/schema.yaml
--- bigquery-etl-main/sql/moz-fx-data-shared-prod/monitoring_derived/bigquery_tables_inventory_v1/schema.yaml	2023-01-06 20:03:36.852500960 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/monitoring_derived/bigquery_tables_inventory_v1/schema.yaml	2023-01-06 19:57:30.000000000 +0000
@@ -1,11 +1,6 @@
 fields:
 
 - mode: NULLABLE
-  name: submission_date
-  type: DATE
-  description: The date when data was captured
-
-- mode: NULLABLE
   name: creation_date
   type: DATE
   description: The table's creation date
@@ -29,8 +24,3 @@
   name: table_type
   type: STRING
   description: The table type
-
-- mode: NULLABLE
-  name: last_modified_date
-  type: DATE
-  description: The table's last modified date
Only in /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/monitoring_derived: bigquery_tables_last_modified_v1
diff -bur --no-dereference bigquery-etl-main/sql/moz-fx-data-shared-prod/monitoring_derived/bigquery_usage_v1/metadata.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/monitoring_derived/bigquery_usage_v1/metadata.yaml
--- bigquery-etl-main/sql/moz-fx-data-shared-prod/monitoring_derived/bigquery_usage_v1/metadata.yaml	2023-01-06 20:03:36.852500960 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/monitoring_derived/bigquery_usage_v1/metadata.yaml	2023-01-06 19:57:30.000000000 +0000
@@ -1,5 +1,5 @@
-friendly_name: BigQuery Query Usage
-description: BigQuery usage, partitioned by day.
+friendly_name: Bigquery Query Usage
+description: Big query usage, partitioned by day.
 owners:
 - [email protected]
 labels:
Only in /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod: relay
Only in /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod: relay_derived
Only in bigquery-etl-main/sql/moz-fx-data-shared-prod/telemetry/regrets_reporter_update: schema.yaml
Only in bigquery-etl-main/sql/moz-fx-data-shared-prod/telemetry/xfocsp_error_report: schema.yaml
diff -bur --no-dereference bigquery-etl-main/dags/bqetl_cjms_nonprod.py /tmp/workspace/generated-sql/dags/bqetl_cjms_nonprod.py
--- bigquery-etl-main/dags/bqetl_cjms_nonprod.py	2023-01-06 20:08:28.711283318 +0000
+++ /tmp/workspace/generated-sql/dags/bqetl_cjms_nonprod.py	2023-01-06 20:02:09.000000000 +0000
@@ -8,9 +8,8 @@
 from utils.constants import ALLOWED_STATES, FAILED_STATES
 from utils.gcp import bigquery_etl_query, gke_command
 
-from fivetran_provider.operators.fivetran import FivetranOperator
-from fivetran_provider.sensors.fivetran import FivetranSensor
-from utils.callbacks import retry_tasks_callback
+from operators.backport.fivetran.operator import FivetranOperator
+from operators.backport.fivetran.sensor import FivetranSensor
 
 docs = """
 ### bqetl_cjms_nonprod
@@ -98,10 +97,7 @@
     fivetran_stripe_nonprod_sync_wait = FivetranSensor(
         connector_id="{{ var.value.fivetran_stripe_nonprod_connector_id }}",
         task_id="fivetran_stripe_nonprod_sensor",
-        poke_interval=30,
-        xcom="{{ task_instance.xcom_pull('fivetran_stripe_nonprod_task') }}",
-        on_retry_callback=retry_tasks_callback,
-        params={"retry_tasks": ["fivetran_stripe_nonprod_task"]},
+        poke_interval=5,
     )
 
     fivetran_stripe_nonprod_sync_wait.set_upstream(fivetran_stripe_nonprod_sync_start)
Only in bigquery-etl-main/dags/: bqetl_fivetran_google_ads.py
diff -bur --no-dereference bigquery-etl-main/dags/bqetl_fxa_events.py /tmp/workspace/generated-sql/dags/bqetl_fxa_events.py
--- bigquery-etl-main/dags/bqetl_fxa_events.py	2023-01-06 20:08:27.723287429 +0000
+++ /tmp/workspace/generated-sql/dags/bqetl_fxa_events.py	2023-01-06 20:02:08.000000000 +0000
@@ -246,12 +246,8 @@
         destination_table="fxa_users_daily_v1",
         dataset_id="firefox_accounts_derived",
         project_id="moz-fx-data-shared-prod",
-        owner="[email protected]",
-        email=[
-            "[email protected]",
-            "[email protected]",
-            "[email protected]",
-        ],
+        owner="[email protected]",
+        email=["[email protected]", "[email protected]"],
         date_partition_parameter="submission_date",
         depends_on_past=False,
     )
@@ -279,12 +275,8 @@
         destination_table="fxa_users_last_seen_v1",
         dataset_id="firefox_accounts_derived",
         project_id="moz-fx-data-shared-prod",
-        owner="[email protected]",
-        email=[
-            "[email protected]",
-            "[email protected]",
-            "[email protected]",
-        ],
+        owner="[email protected]",
+        email=["[email protected]", "[email protected]"],
         start_date=datetime.datetime(2019, 4, 23, 0, 0),
         date_partition_parameter="submission_date",
         depends_on_past=True,
@@ -424,6 +416,10 @@
         firefox_accounts_derived__fxa_stdout_events__v1
     )
 
+    firefox_accounts_derived__fxa_users_last_seen__v1.set_upstream(
+        firefox_accounts_derived__fxa_users_daily__v1
+    )
+
     firefox_accounts_derived__fxa_users_services_daily__v1.set_upstream(
         firefox_accounts_derived__fxa_auth_events__v1
     )
diff -bur --no-derefere

⚠️ Only part of the diff is displayed.

dataops-ci-bot · 2023-01-09T18:27:08Z

Integration report for "Merge branch 'main' into yeonjoo_relay_saasboard_mvp"

`sql.diff`

Click to expand!

Only in /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod: relay
Only in /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod: relay_derived
Only in bigquery-etl-main/sql/moz-fx-data-shared-prod/telemetry/regrets_reporter_update: schema.yaml
Only in bigquery-etl-main/sql/moz-fx-data-shared-prod/telemetry/xfocsp_error_report: schema.yaml
diff -bur --no-dereference bigquery-etl-main/dags/bqetl_subplat.py /tmp/workspace/generated-sql/dags/bqetl_subplat.py
--- bigquery-etl-main/dags/bqetl_subplat.py	2023-01-09 18:26:39.043042459 +0000
+++ /tmp/workspace/generated-sql/dags/bqetl_subplat.py	2023-01-09 18:20:58.000000000 +0000
@@ -568,6 +568,54 @@
         ],
     )
 
+    relay_derived__active_subscription_ids__v1 = bigquery_etl_query(
+        task_id="relay_derived__active_subscription_ids__v1",
+        destination_table='active_subscription_ids_v1${{ macros.ds_format(macros.ds_add(ds, -7), "%Y-%m-%d", "%Y%m%d") }}',
+        dataset_id="relay_derived",
+        project_id="moz-fx-data-shared-prod",
+        owner="[email protected]",
+        email=["[email protected]", "[email protected]"],
+        date_partition_parameter=None,
+        depends_on_past=True,
+        parameters=["date:DATE:{{macros.ds_add(ds, -7)}}"],
+    )
+
+    relay_derived__active_subscriptions__v1 = bigquery_etl_query(
+        task_id="relay_derived__active_subscriptions__v1",
+        destination_table='active_subscriptions_v1${{ macros.ds_format(macros.ds_add(ds, -7), "%Y-%m-%d", "%Y%m%d") }}',
+        dataset_id="relay_derived",
+        project_id="moz-fx-data-shared-prod",
+        owner="[email protected]",
+        email=["[email protected]", "[email protected]"],
+        date_partition_parameter=None,
+        depends_on_past=False,
+        parameters=["date:DATE:{{macros.ds_add(ds, -7)}}"],
+    )
+
+    relay_derived__subscription_events__v1 = bigquery_etl_query(
+        task_id="relay_derived__subscription_events__v1",
+        destination_table='subscription_events_v1${{ macros.ds_format(macros.ds_add(ds, -8), "%Y-%m-%d", "%Y%m%d") }}',
+        dataset_id="relay_derived",
+        project_id="moz-fx-data-shared-prod",
+        owner="[email protected]",
+        email=["[email protected]", "[email protected]"],
+        date_partition_parameter=None,
+        depends_on_past=False,
+        parameters=["date:DATE:{{macros.ds_add(ds, -8)}}"],
+    )
+
+    relay_derived__subscriptions__v1 = bigquery_etl_query(
+        task_id="relay_derived__subscriptions__v1",
+        destination_table="subscriptions_v1",
+        dataset_id="relay_derived",
+        project_id="moz-fx-data-shared-prod",
+        owner="[email protected]",
+        email=["[email protected]", "[email protected]"],
+        date_partition_parameter=None,
+        depends_on_past=False,
+        task_concurrency=1,
+    )
+
     stripe_external__itemized_payout_reconciliation__v5 = gke_command(
         task_id="stripe_external__itemized_payout_reconciliation__v5",
         command=[
@@ -766,3 +814,25 @@
     )
 
     mozilla_vpn_derived__users__v1.set_upstream(mozilla_vpn_external__users__v1)
+
+    relay_derived__active_subscription_ids__v1.set_upstream(
+        relay_derived__subscriptions__v1
+    )
+
+    relay_derived__active_subscriptions__v1.set_upstream(
+        relay_derived__active_subscription_ids__v1
+    )
+
+    relay_derived__active_subscriptions__v1.set_upstream(
+        relay_derived__subscriptions__v1
+    )
+
+    relay_derived__subscription_events__v1.set_upstream(
+        relay_derived__active_subscription_ids__v1
+    )
+
+    relay_derived__subscription_events__v1.set_upstream(
+        relay_derived__subscriptions__v1
+    )
+
+    relay_derived__subscriptions__v1.set_upstream(fivetran_stripe_sync_wait)

dataops-ci-bot · 2023-01-09T21:14:52Z

Integration report for "add "relay-phones" to be included for filtering"

`sql.diff`

Click to expand!

Only in /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod: relay
Only in /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod: relay_derived
Only in bigquery-etl-main/sql/moz-fx-data-shared-prod/telemetry/regrets_reporter_update: schema.yaml
Only in bigquery-etl-main/sql/moz-fx-data-shared-prod/telemetry/xfocsp_error_report: schema.yaml
diff -bur --no-dereference bigquery-etl-main/dags/bqetl_subplat.py /tmp/workspace/generated-sql/dags/bqetl_subplat.py
--- bigquery-etl-main/dags/bqetl_subplat.py	2023-01-09 21:14:15.806918808 +0000
+++ /tmp/workspace/generated-sql/dags/bqetl_subplat.py	2023-01-09 21:08:17.000000000 +0000
@@ -568,6 +568,54 @@
         ],
     )
 
+    relay_derived__active_subscription_ids__v1 = bigquery_etl_query(
+        task_id="relay_derived__active_subscription_ids__v1",
+        destination_table='active_subscription_ids_v1${{ macros.ds_format(macros.ds_add(ds, -7), "%Y-%m-%d", "%Y%m%d") }}',
+        dataset_id="relay_derived",
+        project_id="moz-fx-data-shared-prod",
+        owner="[email protected]",
+        email=["[email protected]", "[email protected]"],
+        date_partition_parameter=None,
+        depends_on_past=True,
+        parameters=["date:DATE:{{macros.ds_add(ds, -7)}}"],
+    )
+
+    relay_derived__active_subscriptions__v1 = bigquery_etl_query(
+        task_id="relay_derived__active_subscriptions__v1",
+        destination_table='active_subscriptions_v1${{ macros.ds_format(macros.ds_add(ds, -7), "%Y-%m-%d", "%Y%m%d") }}',
+        dataset_id="relay_derived",
+        project_id="moz-fx-data-shared-prod",
+        owner="[email protected]",
+        email=["[email protected]", "[email protected]"],
+        date_partition_parameter=None,
+        depends_on_past=False,
+        parameters=["date:DATE:{{macros.ds_add(ds, -7)}}"],
+    )
+
+    relay_derived__subscription_events__v1 = bigquery_etl_query(
+        task_id="relay_derived__subscription_events__v1",
+        destination_table='subscription_events_v1${{ macros.ds_format(macros.ds_add(ds, -8), "%Y-%m-%d", "%Y%m%d") }}',
+        dataset_id="relay_derived",
+        project_id="moz-fx-data-shared-prod",
+        owner="[email protected]",
+        email=["[email protected]", "[email protected]"],
+        date_partition_parameter=None,
+        depends_on_past=False,
+        parameters=["date:DATE:{{macros.ds_add(ds, -8)}}"],
+    )
+
+    relay_derived__subscriptions__v1 = bigquery_etl_query(
+        task_id="relay_derived__subscriptions__v1",
+        destination_table="subscriptions_v1",
+        dataset_id="relay_derived",
+        project_id="moz-fx-data-shared-prod",
+        owner="[email protected]",
+        email=["[email protected]", "[email protected]"],
+        date_partition_parameter=None,
+        depends_on_past=False,
+        task_concurrency=1,
+    )
+
     stripe_external__itemized_payout_reconciliation__v5 = gke_command(
         task_id="stripe_external__itemized_payout_reconciliation__v5",
         command=[
@@ -766,3 +814,25 @@
     )
 
     mozilla_vpn_derived__users__v1.set_upstream(mozilla_vpn_external__users__v1)
+
+    relay_derived__active_subscription_ids__v1.set_upstream(
+        relay_derived__subscriptions__v1
+    )
+
+    relay_derived__active_subscriptions__v1.set_upstream(
+        relay_derived__active_subscription_ids__v1
+    )
+
+    relay_derived__active_subscriptions__v1.set_upstream(
+        relay_derived__subscriptions__v1
+    )
+
+    relay_derived__subscription_events__v1.set_upstream(
+        relay_derived__active_subscription_ids__v1
+    )
+
+    relay_derived__subscription_events__v1.set_upstream(
+        relay_derived__subscriptions__v1
+    )
+
+    relay_derived__subscriptions__v1.set_upstream(fivetran_stripe_sync_wait)

sean-rose · 2023-01-11T01:00:14Z

sql/moz-fx-data-shared-prod/relay_derived/subscription_events_live/view.sql

+  event_type,
+  granular_event_type,
+  plan_id,
+  country,


It turns out this also needed to group by country_name. I've submitted #3495 to fix it.

sean-rose · 2023-01-11T01:03:19Z

I've backfilled the tables created by this PR:

moz-fx-data-shared-prod.relay_derived.active_subscription_ids_v1
moz-fx-data-shared-prod.relay_derived.active_subscriptions_v1
moz-fx-data-shared-prod.relay_derived.subscription_events_v1
moz-fx-data-shared-prod.relay_derived.subscriptions_v1

A non-aggregated column was accidentally omitted from the `GROUP BY` clause in #3482.

YeonjooSmith added 2 commits December 29, 2022 21:48

relay dataset and data tables are created

304986e

added country_name columns to all tables

1e4c057

YeonjooSmith requested a review from sean-rose December 30, 2022 03:28

YeonjooSmith changed the title ~~Yeonjoo_relay_saasboard_mvp~~ (DS-2284, DS-2520)Creating Relay derived tables and views for Relay SaaSboard Dec 30, 2022

reformat query files for circle ci error fixing

2390dcf

Merge branch 'main' into yeonjoo_relay_saasboard_mvp

f2ead37

sean-rose requested changes Dec 30, 2022

View reviewed changes

Schema edited and dry run list updated

79fc3de

YeonjooSmith and others added 3 commits December 30, 2022 21:31

removing status column in multiple places

45f2e08

matching date_partition_offset setting with the table

c6b1de7

Co-authored-by: Sean Rose <[email protected]>

keep the consistency of friendly_name

de35a8d

Co-authored-by: Sean Rose <[email protected]>

removing unnecessary comment

23591c7

subscriptions query updated

c44c7e9

sean-rose requested changes Jan 4, 2023

View reviewed changes

YeonjooSmith and others added 2 commits January 6, 2023 10:46

renaming a CTE

e874d33

Co-authored-by: Sean Rose <[email protected]>

subscriptions_v1 schema editted

8c38cb0

Merge branch 'main' into yeonjoo_relay_saasboard_mvp

1fa369d

add "relay-phones" to be included for filtering

2804833

YeonjooSmith requested a review from sean-rose January 10, 2023 21:19

sean-rose approved these changes Jan 10, 2023

View reviewed changes

Merge branch 'main' into yeonjoo_relay_saasboard_mvp

0bb0196

YeonjooSmith enabled auto-merge (squash) January 10, 2023 22:54

YeonjooSmith merged commit 93df055 into main Jan 10, 2023

YeonjooSmith deleted the yeonjoo_relay_saasboard_mvp branch January 10, 2023 22:54

sean-rose mentioned this pull request Jan 11, 2023

Fix relay_derived.subscription_events_live view #3495

Merged

5 tasks

sean-rose reviewed Jan 11, 2023

View reviewed changes

sean-rose added a commit that referenced this pull request Jan 11, 2023

Fix relay_derived.subscription_events_live view. (#3495)

a132d63

A non-aggregated column was accidentally omitted from the `GROUP BY` clause in #3482.

YeonjooSmith mentioned this pull request Jan 11, 2023

[DS-2284]Adding Relay tables to custom-namespace.yaml for Relay SaaSboard mozilla/lookml-generator#590

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

(DS-2284, DS-2520)Creating Relay derived tables and views for Relay SaaSboard #3482

(DS-2284, DS-2520)Creating Relay derived tables and views for Relay SaaSboard #3482

YeonjooSmith commented Dec 30, 2022 •

edited

Loading

dataops-ci-bot commented Dec 30, 2022

dataops-ci-bot commented Dec 30, 2022

dataops-ci-bot commented Dec 30, 2022

sean-rose left a comment

dataops-ci-bot commented Dec 31, 2022

dataops-ci-bot commented Dec 31, 2022

dataops-ci-bot commented Dec 31, 2022

dataops-ci-bot commented Dec 31, 2022

dataops-ci-bot commented Dec 31, 2022

YeonjooSmith commented Dec 31, 2022

dataops-ci-bot commented Dec 31, 2022

dataops-ci-bot commented Jan 6, 2023

dataops-ci-bot commented Jan 9, 2023

dataops-ci-bot commented Jan 9, 2023

sean-rose Jan 11, 2023

YeonjooSmith Jan 11, 2023

sean-rose commented Jan 11, 2023

(DS-2284, DS-2520)Creating Relay derived tables and views for Relay SaaSboard #3482

(DS-2284, DS-2520)Creating Relay derived tables and views for Relay SaaSboard #3482

Conversation

YeonjooSmith commented Dec 30, 2022 • edited Loading

dataops-ci-bot commented Dec 30, 2022

Integration report for "added country_name columns to all tables"

sql.diff

dataops-ci-bot commented Dec 30, 2022

Integration report for "reformat query files for circle ci error fixing"

sql.diff

dataops-ci-bot commented Dec 30, 2022

Integration report for "Merge branch 'main' into yeonjoo_relay_saasboard_mvp"

sql.diff

sean-rose left a comment

Choose a reason for hiding this comment

dataops-ci-bot commented Dec 31, 2022

Integration report for "Schema edited and dry run list updated"

sql.diff

dataops-ci-bot commented Dec 31, 2022

Integration report for "removing status column in multiple places"

sql.diff

dataops-ci-bot commented Dec 31, 2022

Integration report for "matching date_partition_offset setting with the table"

sql.diff

dataops-ci-bot commented Dec 31, 2022

Integration report for "keep the consistency of friendly_name"

sql.diff

dataops-ci-bot commented Dec 31, 2022

Integration report for "removing unnecessary comment"

sql.diff

YeonjooSmith commented Dec 31, 2022

dataops-ci-bot commented Dec 31, 2022

Integration report for "subscriptions query updated"

sql.diff

dataops-ci-bot commented Jan 6, 2023

Integration report for "subscriptions_v1 schema editted"

sql.diff

dataops-ci-bot commented Jan 9, 2023

Integration report for "Merge branch 'main' into yeonjoo_relay_saasboard_mvp"

sql.diff

dataops-ci-bot commented Jan 9, 2023

Integration report for "add "relay-phones" to be included for filtering"

sql.diff

sean-rose Jan 11, 2023

Choose a reason for hiding this comment

YeonjooSmith Jan 11, 2023

Choose a reason for hiding this comment

sean-rose commented Jan 11, 2023

YeonjooSmith commented Dec 30, 2022 •

edited

Loading

`sql.diff`

`sql.diff`

`sql.diff`

`sql.diff`

`sql.diff`

`sql.diff`

`sql.diff`

`sql.diff`

`sql.diff`

`sql.diff`

`sql.diff`

`sql.diff`