Bigeye documentation

.spelling

@@ -482,3 +482,10 @@ modeling
 2019
 pre-calculating
 pre-aggregated
+app.bigeye.com
+autometrics
+autothresholds
+customizable
+lookback
+rescan
+unmute

src/SUMMARY.md

@@ -14,6 +14,13 @@
       - [Using the Data Catalog](cookbooks/analysis/data_catalog.md)
       - [Using the Glean Dictionary](cookbooks/analysis/glean_dictionary.md)
       - [Using the Probe Dictionary](cookbooks/analysis/probe_dictionary.md)
+    - [Data Monitoring - Intro to Bigeye](cookbooks/data_monitoring/intro.md)
+      - [Interface](cookbooks/data_monitoring/interface.md)
+      - [Deploying Metrics](cookbooks/data_monitoring/deploying_metrics.md)
+      - [Collections](cookbooks/data_monitoring/collections.md)
+      - [Issues Management](cookbooks/data_monitoring/issues_management.md)
+      - [Cost Considerations](cookbooks/data_monitoring/cost_considerations.md)
+      - [Further Reading](cookbooks/data_monitoring/further_reading.md)
     - [Data Modeling](cookbooks/data_modeling/index.md)
       - [Where to store the data model assets](cookbooks/data_modeling/where_to_store.md)
       - [Using aggregates in BigQuery and Looker](cookbooks/data_modeling/using_aggregates.md)

src/cookbooks/data_monitoring/collections.md

@@ -0,0 +1,25 @@
+# Collections in Bigeye
+
+Collections help you organize and focus on specific areas of interest, making it simpler to track and address data quality across different segments of your data landscape. This feature enhances efficiency by allowing users to monitor grouped entities in a cohesive manner.
+
+![](../../assets/Bigeye/Bigeye-Collections.png)
+
+## Creating a new collection
+
+If you don’t find a collection that meets your product or requirements, admins can create a new collection.
+
+<iframe width="560" height="315" src="https://www.youtube.com/embed/YTOOTFw5MLw?si=X3VybaWasts-sdjw" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" referrerpolicy="strict-origin-when-cross-origin" allowfullscreen></iframe>
+
+If you're not an admin, please [submit a Jira ticket](https://mozilla-hub.atlassian.net/browse/DENG-4726) with the necessary details.
+
+## Adding metrics to a collection
+
+To add metrics to a collection in Bigeye, navigate to the collection you want to update and click "Add Metrics." You can search or filter for specific metrics that align with your monitoring goals.
+
+<iframe width="560" height="315" src="https://www.youtube.com/embed/ZFRAaeX6z8w?si=ba3jYHTQNZPDi9ua" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" referrerpolicy="strict-origin-when-cross-origin" allowfullscreen></iframe>
+
+## Adding notifications to a collection
+
+One useful feature of collections is the ability to add notifications. To set this up, click the "Edit" button, then navigate to the "Notifications" tab in the modal that appears.
+
+<iframe width="560" height="315" src="https://www.youtube.com/embed/QH37mnkkuW8?si=7cMx8eJq_wgbSFX_" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" referrerpolicy="strict-origin-when-cross-origin" allowfullscreen></iframe>

src/cookbooks/data_monitoring/cost_considerations.md

@@ -0,0 +1,7 @@
+# Cost considerations
+
+The Freshness and Volume metrics, which represent Pipeline Reliability, are included in the free tier. There is no charge when these metrics are added to a table.
+
+However, if additional monitors are added to a table, it becomes a billable table, and charges will apply based on the number of billable tables.
+
+**Try to avoid Autometrics!** - On tables with many columns a large number of monitors might get deployed. This increases noise and cost. Instead, it is recommended to [choose relevant metrics from the list of available metrics](deploying_metrics.md#list-of-available-metrics) manually.

src/cookbooks/data_monitoring/deploying_metrics.md

@@ -0,0 +1,71 @@
+# Deploying Metrics
+
+To deploy metrics in Bigeye, navigate to a schema, table, or column and click "Add Monitoring."
+You can choose to add metrics via 4 options
+
+1. Freshness and Volume (Pipeline Reliability)
+2. Data Quality
+3. All metrics
+4. Custom SQL, and select the metrics you wish to deploy.
+
+![](../../assets/Bigeye/Bigeye-Deploymetrics.png)
+
+![](../../assets/Bigeye/Bigeye-Deploymetrics-2.png)
+
+Next, choose the columns to monitor, set schedules, thresholds, and filters, and confirm your selections.
+You can assign the metrics to the relevant collection, grouping related metrics for easier management and monitoring.
+Once deployed, the metrics appear under the Metrics tab on the relevant schema, table, or column page.
+We can backfill the metrics for up to 28 days in the past.
+
+![](../../assets/Bigeye/Bigeye-Deploymetrics-3.png)
+
+<iframe width="560" height="315" src="https://www.youtube.com/embed/LRhbxFMPTwI?si=yJ98NhvlmGt02eq7" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" referrerpolicy="strict-origin-when-cross-origin" allowfullscreen></iframe>
+
+For more details, please refer to the Bigeye documentation on [how to deploy metrics](https://docs.bigeye.com/docs/deploy-metrics).
+
+Watch the Bigeye tutorial on [how to use the metrics page](https://www.youtube.com/watch?v=jNzSki59AWQ)
+
+## Freshness and Volume (Pipeline Reliability)
+
+Bigeye tracks data quality by monitoring the timeliness (freshness) and completeness (volume) of your data and checks them hourly.
+Initially, it looks back 28 days, then 2 days for subsequent runs. For volume, it aggregates row counts into hourly buckets, using the same lookback periods. We have an option to select [`manual`](https://docs.bigeye.com/docs/manual-thresholds) thresholds vs [`Autothresholds`](https://docs.bigeye.com/docs/autothresholds) that learn typical patterns and alert on anomalies.
+
+Only one Freshness and one Volume metric can be deployed per table.
+**Cost Consideration** Freshness and Volume metrics are included by default for each table and are free of charge.
+
+![](../../assets/Bigeye/Bigeye-Pipelinereliability.png)
+
+![](../../assets/Bigeye/Bigeye-Pipelinereliability-2.png)
+
+Please refer to Bigeye documentation for more details on [Freshness and Volume metrics](https://docs.bigeye.com/docs/freshness-and-volume-pipeline-reliability-copy).
+
+## List of available metrics
+
+Bigeye offers a range of available metrics to monitor data quality and reliability across your data pipelines.
+These metrics cover areas such as data freshness, volume, distribution, schema changes, and anomalies. You can deploy these metrics to track key performance indicators and ensure your data meets expected standards.
+
+Please refer to the Bigeye documentation for [list of available metrics](https://docs.bigeye.com/docs/available-metrics).
+
+Watch the Bigeye tutorial on the [metrics types](https://www.youtube.com/watch?v=jNzSki59AWQ)
+
+## Autometrics
+
+Autometrics are suggested metrics that monitor anomalies in column-level data, automatically generated for all new datasets in Bigeye. They can be found under the Autometrics tab in the Catalog when viewing a source, schema, table, or column page.
+
+> **Try to avoid this option!**: On tables with many columns a large number of monitors might get deployed. This increases noise and cost. Instead, it is recommended to [choose relevant metrics from the list of available metrics](#list-of-available-metrics) manually.
+
+![](../../assets/Bigeye/Bigeye-Autometrics.png)
+
+## Custom SQL
+
+Custom rules are useful for addressing unique data quality requirements that standard metrics may not cover. Once set, these rules integrate into your monitoring workflow.
+
+## Recommendations / Best Practices to deploying metrics
+
+- It's recommended to avoid deploying Autometrics extensively, as this could result in a high signal-to-noise ratio, leading to unnecessary alerts and potential distraction.
+
+- When deploying metrics on search tables, we observed that the `median` calculation using the BigQuery function does not work as expected. Due to this limitation, it is recommended to avoid using the median metric in these scenarios to ensure accurate results.
+
+- Autothresholds are recommended for freshness and volume metrics, as they automatically adjust based on typical patterns. For other metrics, it's advisable to manually set thresholds to ensure accuracy and relevance.
+
+- It is recommended to add metrics at the view level rather than directly on tables. This ensures that even if a table becomes obsolete or is upgraded, unnecessary checks on previous versions are avoided. The only exception to this rule is for freshness and volume metrics, which must be deployed directly on tables.

src/cookbooks/data_monitoring/further_reading.md

@@ -0,0 +1,15 @@
+# Bigeye - Further reading
+
+- Kinds of [alert thresholds](https://www.youtube.com/watch?v=8DWyZuU-w1c&t=9s) that Bigeye supports
+
+- Use [Deltas](https://www.youtube.com/watch?v=la8pq7rjT-M&list=PLUmsPWeo8j4U9SpGCnAe9syilD4_jBgSI&index=9) to compare datasets and validate successful data replication, migration, or model code changes.
+
+- [Row Creation](https://www.youtube.com/watch?v=TmwyEfq1Xgo&list=PLUmsPWeo8j4U9SpGCnAe9syilD4_jBgSI&index=4) - Concept of row creation time and how helps save costs and optimize your data observability
+
+- [Creating a schedule and assigning it to a metric](https://docs.bigeye.com/docs/schedules)
+
+- Creating a [Metric Template](https://www.youtube.com/watch?v=8YVIQIbec6k&list=PLUmsPWeo8j4U9SpGCnAe9syilD4_jBgSI&index=14) in order to provide specific functionality.
+
+- [Grouped metrics](https://www.youtube.com/watch?v=mhPttWI15ro&list=PLUmsPWeo8j4U9SpGCnAe9syilD4_jBgSI&index=8) allow you to employ group by aggregation.
+
+- [Lookback window](https://docs.bigeye.com/docs/metric-window-types)

src/cookbooks/data_monitoring/interface.md

@@ -0,0 +1,53 @@
+# Bigeye Interface
+
+## Catalog
+
+The Catalog tab in the left-hand menu offers a comprehensive view of all data sources connected to Bigeye, making it simple to navigate and manage your entire data ecosystem.
+
+If you are an admin, you will have access to the "Add Source" button, allowing you to easily integrate new data sources and BI tools.
+
+![](../../assets/Bigeye/Bigeye-Catalog.png)
+
+The Bigeye catalog refreshes automatically every 24 hours to detect new datasets and schema changes. You can also manually refresh the catalog anytime by clicking 'Rescan' on the schema changes tab.
+
+For more detailed information about the Catalog, please refer to the [Catalog documentation page](https://docs.bigeye.com/docs/catalog)
+
+Watch the [Bigeye tutorial](https://www.youtube.com/watch?v=8DWyZuU-w1c&t=9s) on how to navigate Bigeye Catalog.
+
+## Workspaces
+
+Workspaces in Bigeye allow multiple teams to collaborate simultaneously, with each team managing and monitoring their own data independently.
+Each Bigeye workspace includes its own Catalog, BI tools, and ETL tools, Metrics and issues, Templates and schedules, Collections, Deltas.
+
+We are in the process of setting up user workspaces that will align with our existing data access restrictions.
+
+If you do not find a suitable workspace, please submit a [Jira ticket](https://mozilla-hub.atlassian.net/browse/DENG-4727)
+
+![](../../assets/Bigeye/Bigeye-Workspace.png)
+
+## Collections
+
+Collections in Bigeye allow you to group related metrics, making it easier to manage and monitor them together.
+
+If you don’t find a collection that meets your product or requirements, admins can create a new collection.
+If you're not an admin, please submit a [Jira ticket](https://mozilla-hub.atlassian.net/browse/DENG-4726) with the necessary details.
+
+Watch the [Bigeye tutorial](https://www.youtube.com/watch?v=4H5AM0a71bs&list=PLUmsPWeo8j4U9SpGCnAe9syilD4_jBgSI&index=8) on how to navigate Bigeye Catalog.
+
+## Issues
+
+Bigeye's Issues feature helps you track and manage data quality issues detected by the platform.
+You can assign, prioritize, and resolve issues within the platform, ensuring that your data quality remains high. Issues can be categorized and filtered to streamline the resolution process across teams.
+
+For more details, refer to the Bigeye documentation on [Issues page](https://docs.bigeye.com/docs/issues)
+
+## Dashboard
+
+Users can monitor the data quality metrics and issues in a centralized view. It highlights key features such as customizable widgets, real-time metric tracking, and the ability to visualize data health at a glance. Users can configure dashboards to focus on specific metrics or tables and receive immediate insights into their data pipelines' performance.
+
+For additional guidance on using Bigeye Dashboard, please refer to the following documentation:
+
+- [How to set filters](https://docs.bigeye.com/docs/dashboard#set-filters)
+- [Monitoring Coverage](https://docs.bigeye.com/docs/dashboard#monitoring-coverage)
+- [Tracking Data Quality](https://docs.bigeye.com/docs/dashboard#data-quality)
+- [Issue Response Metrics](https://docs.bigeye.com/docs/dashboard#issue-response)

src/cookbooks/data_monitoring/intro.md

@@ -0,0 +1,12 @@
+# Introduction to Bigeye (Data Observability Platform)
+
+Mozilla uses Bigeye as its data observability platform to ensure high data quality and reliability across its pipelines. Bigeye offers powerful features like automated anomaly detection, detailed data lineage tracking, and customizable monitoring. These capabilities allow teams to swiftly identify, diagnose, and resolve data issues, enhancing overall data integrity and operational efficiency.
+
+## Accessing Bigeye
+
+You can access Mozilla's instance of Bigeye at [app.bigeye.com](https://app.bigeye.com/).
+If you do not have the necessary access or permissions, please [submit a Jira ticket](https://mozilla-hub.atlassian.net/browse/DENG-4728).
+
+## Getting Started
+
+Watch the [Bigeye tutorial](https://www.youtube.com/watch?v=8DWyZuU-w1c&t=9s) to get an overview of the platform.

src/cookbooks/data_monitoring/issues_management.md

@@ -0,0 +1,46 @@
+# Issues management
+
+The `Issues` tab allows filtering issues by parameters like severity or date, and reviewing details such as impacted metrics and tables.
+
+![The Issues overview on BigEye, allowing to filter issues](../../assets/Bigeye/issues-tab.png)
+
+## View issue details
+
+Once we click on the issue we can view a metric chart that displays a time series visualization of the alerting metric.
+
+![](../../assets/Bigeye/Issues-2.png)
+
+The metric chart in Bigeye displays a time series visualization of the alerting metric
+
+![](../../assets/Bigeye/Issues-2.png)
+
+## Status of Issue
+
+Users can change the status to "Active," "Resolved," "Muted," or "Dismissed," depending on the issue's current state. This allows for better tracking and management of issues across data pipelines. Status updates are reflected in the timeline, providing a clear history of the issue's progression.
+
+![](../../assets/Bigeye/Issues-4.png)
+
+Use the Mute button above the timeline to mute an issue and stop being notified about subsequent issue alerts. The available mute options in the Bigeye are Unmute, 1 hour, 4 hours, 12 hours, 24 hours, or 1 week
+
+![](../../assets/Bigeye/Issues-5.png)
+
+For more details, refer to the [Bigeye documentation](https://docs.bigeye.com/docs/change-the-issue-status).
+
+## Debug
+
+Use the queries in the Debug tab to troubleshoot your issue.
+
+<iframe width="560" height="315" src="https://www.youtube.com/embed/rSWAl7f8vcc?si=YA2HgEcRIyL0FKMC" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" referrerpolicy="strict-origin-when-cross-origin" allowfullscreen></iframe>
+
+## Jira Integration
+
+The Bigeye integration with Jira enables teams to track critical data quality issues seamlessly through Jira's flexible interface. The bi-directional integration ensures that updates made in either Jira or Bigeye are synced across both systems. Once a Jira ticket is created in Bigeye, any status changes or comments are automatically reflected in both platforms, keeping all team members informed.
+
+<iframe width="560" height="315" src="https://www.youtube.com/embed/5UtAkwvjt5U?si=wnpb6fqhQMPvS6wO" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" referrerpolicy="strict-origin-when-cross-origin" allowfullscreen></iframe>
+
+## Slack Integration [Coming Soon]
+
+Bigeye enables users to take direct actions on issues on Slack messages without needing to navigate to the Bigeye web interface.
+Users can resolve, mute, or dismiss alerts directly from Slack messages, ensuring efficient workflows and quick responses to data quality issues.
+
+## Recommendation / Best practices [Coming Soon]