Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add docs for bigquery-etl and Bigeye #848

Merged
merged 1 commit into from
Sep 24, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .spelling
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,7 @@ backfilling
backfills
behaviour
Bhattacharyya
Bigconfig
BigQuery
bigquery-etl
Bing
Expand Down
1 change: 1 addition & 0 deletions src/SUMMARY.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@
- [Collections](cookbooks/data_monitoring/collections.md)
- [Issues Management](cookbooks/data_monitoring/issues_management.md)
- [Cost Considerations](cookbooks/data_monitoring/cost_considerations.md)
- [bigquery-etl and Bigeye](cookbooks/data_monitoring/bigquery_etl_integration.md)
- [Further Reading](cookbooks/data_monitoring/further_reading.md)
- [Data Modeling](cookbooks/data_modeling/index.md)
- [Where to store the data model assets](cookbooks/data_modeling/where_to_store.md)
Expand Down
25 changes: 25 additions & 0 deletions src/cookbooks/data_monitoring/bigquery_etl_integration.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
# bigquery-etl and Bigeye

Monitors can be defined alongside derived datasets in bigquery-etl. Monitoring in Bigeye for a specific table can be enabled by adding `monitoring` metadata to the `metadata.yaml` file:

```yaml
friendly_name: Some Table
monitoring:
enabled: true # Enables monitoring for the table in Bigeye and deploys freshness and volume metrics
collection: Test # An existing collection these monitors should be part of in Bigeye
```

Enabling monitoring for a table automatically deploys freshness and volume metrics for this table.

## Bigconfig

Additional and custom monitors can be defined in a [Bigconfig](https://docs.bigeye.com/docs/bigconfig#example-template) `bigconfig.yml` file that is stored in the same directory as the table query. Bigconfig allows users to deploy other pre-defined monitors, such as row counts or null checks on a table or column level.

## Deployment

To generate a `bigconfig.yml` file with the default metrics when monitoring is enabled run: `bqetl monitoring update [PATH]`.
The created file can be manually edited. For tables that do not have a `bigconfig.yml` checked into the repository, the file will get generated automatically before deployment to Bigeye. Files only need to be checked in if there are some customizations.

To manually deploy a Bigconfig file run: `bqetl monitoring deploy [PATH]`. The environment variable `BIGEYE_API_KEY` needs to be set to a valid API token that can be [created in Bigeye](https://docs.bigeye.com/docs/using-api-keys).

The deployment of Bigconfig files also runs automatically as part of the [artifact deployment process](../../concepts/pipeline/artifact_deployment.md), after tables and views have been deployed.