Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

NTD external tables for Complete Monthly Ridership with Adjustments and Estimates Data #3470

Merged
merged 2 commits into from
Sep 23, 2024
Merged

NTD external tables for Complete Monthly Ridership with Adjustments and Estimates Data #3470

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
6c0c1ea
Add new NTD API and XLSX buckets to Airflow configuration to fix erro…
Sep 18, 2024
a9b9c65
Create NTD external tables for XLSX Complete Monthly Ridership with A…
Sep 17, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
83 changes: 83 additions & 0 deletions ...torical__complete_monthly_ridership_with_adjustments_and_estimates__calendar_year_upt.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,83 @@
operator: operators.ExternalTable
bucket: gs://calitp-ntd-xlsx-products-clean
prefix_bucket: false
post_hook: |
SELECT *
FROM `{{ get_project_id() }}`.external_ntd__ridership.historical__complete_monthly_ridership_with_adjustments_and_estimates__calendar_year_upt
LIMIT 1;
source_objects:
- "complete_monthly_ridership_with_adjustments_and_estimates/historical/calendar_year_upt/*.jsonl.gz"
destination_project_dataset_table: "external_ntd__ridership.historical__complete_monthly_ridership_with_adjustments_and_estimates__calendar_year_upt"
source_format: NEWLINE_DELIMITED_JSON
use_bq_client: true
hive_options:
mode: AUTO
require_partition_filter: false
source_uri_prefix: "complete_monthly_ridership_with_adjustments_and_estimates/historical/calendar_year_upt/"
schema_fields:
- name: ntd_id
type: NUMERIC
- name: legacy_ntd_id
type: STRING
- name: agency
type: STRING
- name: mode_type_of_service_status
type: STRING
- name: reporter_type
type: STRING
- name: uace_cd
type: NUMERIC
- name: uza_name
type: STRING
- name: mode
type: STRING
- name: tos
type: STRING
- name: _3_mode
type: STRING
- name: _2002
type: FLOAT
- name: _2003
type: FLOAT
- name: _2004
type: FLOAT
- name: _2005
type: FLOAT
- name: _2006
type: FLOAT
- name: _2007
type: FLOAT
- name: _2008
type: FLOAT
- name: _2009
type: FLOAT
- name: _2010
type: FLOAT
- name: _2011
type: FLOAT
- name: _2012
type: FLOAT
- name: _2013
type: FLOAT
- name: _2014
type: FLOAT
- name: _2015
type: FLOAT
- name: _2016
type: FLOAT
- name: _2017
type: FLOAT
- name: _2018
type: FLOAT
- name: _2019
type: FLOAT
- name: _2020
type: FLOAT
- name: _2021
type: FLOAT
- name: _2022
type: FLOAT
- name: _2023
type: FLOAT
- name: _2024
type: FLOAT
83 changes: 83 additions & 0 deletions ...torical__complete_monthly_ridership_with_adjustments_and_estimates__calendar_year_vrm.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,83 @@
operator: operators.ExternalTable
bucket: gs://calitp-ntd-xlsx-products-clean
prefix_bucket: false
post_hook: |
SELECT *
FROM `{{ get_project_id() }}`.external_ntd__ridership.historical__complete_monthly_ridership_with_adjustments_and_estimates__calendar_year_vrm
LIMIT 1;
source_objects:
- "complete_monthly_ridership_with_adjustments_and_estimates/historical/calendar_year_vrm/*.jsonl.gz"
destination_project_dataset_table: "external_ntd__ridership.historical__complete_monthly_ridership_with_adjustments_and_estimates__calendar_year_vrm"
source_format: NEWLINE_DELIMITED_JSON
use_bq_client: true
hive_options:
mode: AUTO
require_partition_filter: false
source_uri_prefix: "complete_monthly_ridership_with_adjustments_and_estimates/historical/calendar_year_vrm/"
schema_fields:
- name: ntd_id
type: NUMERIC
- name: legacy_ntd_id
type: STRING
- name: agency
type: STRING
- name: mode_type_of_service_status
type: STRING
- name: reporter_type
type: STRING
- name: uace_cd
type: NUMERIC
- name: uza_name
type: STRING
- name: mode
type: STRING
- name: tos
type: STRING
- name: _3_mode
type: STRING
- name: _2002
type: FLOAT
- name: _2003
type: FLOAT
- name: _2004
type: FLOAT
- name: _2005
type: FLOAT
- name: _2006
type: FLOAT
- name: _2007
type: FLOAT
- name: _2008
type: FLOAT
- name: _2009
type: FLOAT
- name: _2010
type: FLOAT
- name: _2011
type: FLOAT
- name: _2012
type: FLOAT
- name: _2013
type: FLOAT
- name: _2014
type: FLOAT
- name: _2015
type: FLOAT
- name: _2016
type: FLOAT
- name: _2017
type: FLOAT
- name: _2018
type: FLOAT
- name: _2019
type: FLOAT
- name: _2020
type: FLOAT
- name: _2021
type: FLOAT
- name: _2022
type: FLOAT
- name: _2023
type: FLOAT
- name: _2024
type: FLOAT
71 changes: 71 additions & 0 deletions ...roducts/historical__complete_monthly_ridership_with_adjustments_and_estimates__master.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,71 @@
operator: operators.ExternalTable
bucket: gs://calitp-ntd-xlsx-products-clean
prefix_bucket: false
post_hook: |
SELECT *
FROM `{{ get_project_id() }}`.external_ntd__ridership.historical__complete_monthly_ridership_with_adjustments_and_estimates__master
LIMIT 1;
source_objects:
- "complete_monthly_ridership_with_adjustments_and_estimates/historical/master/*.jsonl.gz"
destination_project_dataset_table: "external_ntd__ridership.historical__complete_monthly_ridership_with_adjustments_and_estimates__master"
source_format: NEWLINE_DELIMITED_JSON
use_bq_client: true
hive_options:
mode: AUTO
require_partition_filter: false
source_uri_prefix: "complete_monthly_ridership_with_adjustments_and_estimates/historical/master/"
schema_fields:
- name: ntd_id
type: NUMERIC
- name: legacy_ntd_id
type: STRING
- name: agency
type: STRING
- name: mode
type: STRING
- name: tos
type: STRING
- name: _3_mode
type: STRING
- name: mode_type_of_service_status
type: STRING
- name: reporter_type
type: STRING
- name: organization_type
type: STRING
- name: hq_city
type: STRING
- name: hq_state
type: STRING
- name: uace_cd
type: NUMERIC
- name: uza_name
type: STRING
- name: uza_sq_miles
type: NUMERIC
- name: uza_population
type: NUMERIC
- name: service_area_population
type: NUMERIC
- name: service_area_sq_miles
type: NUMERIC
- name: last_closed_report_year
type: NUMERIC
- name: last_closed_fy_end_month
type: NUMERIC
- name: last_closed_fy_end_year
type: NUMERIC
- name: passenger_miles_fy
type: FLOAT
- name: unlinked_passenger_trips_fy
type: FLOAT
- name: avg_trip_length_fy
type: FLOAT
- name: fares_fy
type: FLOAT
- name: operating_expenses_fy
type: FLOAT
- name: avg_cost_per_trip_fy
type: FLOAT
- name: avg_fares_per_trip_fy
type: FLOAT
Loading
Loading