Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ntd_id: revise dim organiztions and enrich ntd endpoints #3710

Draft
wants to merge 4 commits into
base: main
Choose a base branch
from
+5 −10
Draft

ntd_id: revise dim organiztions and enrich ntd endpoints #3710

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
e95607a
ntd_id: revise dim organiztions and enrich ntd endpoints
charlie-costanzo Feb 18, 2025
dcc871a
utilize unnested ntd_agency_info_key as ntd_id value for dim_organiza…
charlie-costanzo Feb 21, 2025
3a7e6a0
align ntd_id naming conventions
charlie-costanzo Feb 21, 2025
6567a6d
further align ntd_id conventions, this time in safety and security ta…
charlie-costanzo Feb 21, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion warehouse/models/mart/ntd_safety_and_security/fct_major_safety_events.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ SELECT
typeofservicecd,
reportername,
customer,
ntdid,
ntd_id,
dt,
execution_ts
FROM fct_major_safety_events
8 changes: 1 addition & 7 deletions warehouse/models/mart/transit_database/dim_organizations.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,10 +4,6 @@ WITH dim AS (
SELECT * FROM {{ ref('int_transit_database__organizations_dim') }}
),

ntd_agency_to_organization AS (
SELECT * FROM {{ ref('_deprecated__ntd_agency_to_organization') }}
),

dim_organizations AS (
SELECT
key,
Expand All @@ -30,7 +26,7 @@ dim_organizations AS (
-- use same May 23, 2023 cutover date as `assessment_status` --> `public_currently_operating` in downstream models for consistency
CASE
WHEN _valid_from >= '2023-05-23' THEN raw_ntd_id
ELSE ntd_to_org.ntd_id
ELSE ntd_agency_info_key
END AS ntd_id,
ntd_id_2022,
public_currently_operating,
Expand All @@ -40,8 +36,6 @@ dim_organizations AS (
_valid_to

FROM dim
LEFT JOIN ntd_agency_to_organization ntd_to_org
ON source_record_id = ntd_to_org.organization_record_id
)

SELECT * FROM dim_organizations
2 changes: 1 addition & 1 deletion warehouse/models/staging/ntd_safety_and_security/stg_ntd__major_safety_events.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@ SELECT
{{ trim_make_empty_string_null('typeofservicecd') }} AS typeofservicecd,
{{ trim_make_empty_string_null('reportername') }} AS reportername,
SAFE_CAST(customer AS INTEGER) AS customer,
{{ trim_make_empty_string_null('CAST(ntdid AS STRING)') }} AS ntdid,
{{ trim_make_empty_string_null('CAST(ntdid AS STRING)') }} AS ntd_id,
dt,
execution_ts
FROM stg_ntd__major_safety_events
3 changes: 2 additions & 1 deletion warehouse/models/staging/transit_database/stg_transit_database__organizations.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ stg_transit_database__organizations AS (
WHEN id = 'rec2DteW2sfmBJRsH' AND itp_id = 188 THEN 187
ELSE CAST(itp_id AS INTEGER)
END AS itp_id,
unnested_ntd_records AS ntd_agency_info_key,
{{ trim_make_empty_string_null('unnested_ntd_records') }} AS ntd_agency_info_key,
hubspot_company_record_id,
alias_ as alias,
details,
Expand All @@ -46,6 +46,7 @@ stg_transit_database__organizations AS (
public_currently_operating_fixed_route = "Yes" AS public_currently_operating_fixed_route,
FROM once_daily_organizations
LEFT JOIN UNNEST(once_daily_organizations.ntd_id) as unnested_ntd_records

)

SELECT * FROM stg_transit_database__organizations
Loading