Skip to content

Commit

Permalink
Add docs for bigquery-etl and Bigeye
Browse files Browse the repository at this point in the history
  • Loading branch information
scholtzan committed Sep 24, 2024
1 parent 4cdad76 commit 47b313b
Show file tree
Hide file tree
Showing 3 changed files with 27 additions and 0 deletions.
1 change: 1 addition & 0 deletions .spelling
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,7 @@ backfilling
backfills
behaviour
Bhattacharyya
Bigconfig
BigQuery
bigquery-etl
Bing
Expand Down
1 change: 1 addition & 0 deletions src/SUMMARY.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@
- [Collections](cookbooks/data_monitoring/collections.md)
- [Issues Management](cookbooks/data_monitoring/issues_management.md)
- [Cost Considerations](cookbooks/data_monitoring/cost_considerations.md)
- [bigquery-etl and Bigeye](cookbooks/data_monitoring/bigquery_etl_integration.md)
- [Further Reading](cookbooks/data_monitoring/further_reading.md)
- [Data Modeling](cookbooks/data_modeling/index.md)
- [Where to store the data model assets](cookbooks/data_modeling/where_to_store.md)
Expand Down
25 changes: 25 additions & 0 deletions src/cookbooks/data_monitoring/bigquery_etl_integration.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
# bigquery-etl and Bigeye

Monitors can be defined alongside derived datasets in bigquery-etl. Monitoring in Bigeye for a specific table can be enabled by adding `monitoring` metadata to the `metadata.yaml` file:

```yaml
friendly_name: Some Table
monitoring:
enabled: true # Enables monitoring for the table in Bigeye and deploys freshness and volume metrics
collection: Test # An existing collection these monitors should be part of in Bigeye
```
Enabling monitoring for a table automatically deploys freshness and volume metrics for this table.
## Bigconfig
Additional and custom monitors can be defined in a [Bigconfig](https://docs.bigeye.com/docs/bigconfig#example-template) `bigconfig.yml` file that is stored in the same directory as the table query. Bigconfig allows users to deploy other pre-defined monitors, such as row counts or null checks on a table or column level.

## Deployment

To generate a `bigconfig.yml` file with the default metrics when monitoring is enabled run: `bqetl monitoring update [PATH]`.
The created file can be manually edited. For tables that do not have a `bigconfig.yml` checked into the repository, the file will get generated automatically before deployment to Bigeye. Files only need to be checked in if there are some customizations.

To manually deploy a Bigconfig file run: `bqetl monitoring deploy [PATH]`. The environment variable `BIGEYE_API_KEY` needs to be set to a valid API token that can be [created in Bigeye](https://docs.bigeye.com/docs/using-api-keys).

The deployment of Bigconfig files also runs automatically as part of the [artifact deployment process](../../concepts/pipeline/artifact_deployment), after tables and views have been deployed.

0 comments on commit 47b313b

Please sign in to comment.