Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

clarifies mf timespine & creates mini guide #6836

Open
wants to merge 18 commits into
base: current
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 3 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 28 additions & 2 deletions website/docs/docs/build/metricflow-time-spine.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,22 +10,41 @@ tags: [Metrics, Semantic Layer]

It's common in analytics engineering to have a date dimension or "time spine" table as a base table for different types of time-based joins and aggregations. The structure of this table is typically a base column of daily or hourly dates, with additional columns for other time grains, like fiscal quarters, defined based on the base column. You can join other tables to the time spine on the base column to calculate metrics like revenue at a point in time, or to aggregate to a specific time grain.

MetricFlow requires you to define at least one dbt model which provides a time-spine, and then specify (in YAML) the columns to be used for time-based joins. MetricFlow will join against the time-spine model for the following types of metrics and dimensions:
To use MetricFlow with time-based metrics and dimensions, you _must_ provide a time spine table. This table serves as the foundation for time-based joins and aggregations. You can either:

- Create a time spine SQL table from scratch (check out the [example time spine tables](#example-time-spine-tables) section for examples), or
- Use an existing table in your project, like a `dim_date` table

And once you have a time spine table, you need to configure it in YAML to tell MetricFlow how to use it.


:::tip
Check out our mini guide on [how to create a time spine table](/guides/mf-time-spine) to get started!
:::

## Prerequisites
MetricFlow requires you to define at least one dbt model which provides a time-spine, and then specify (in YAML) the columns to be used for time-based joins. This means you need to:

- Define at least one [time spine table](#example-time-spine-tables) at the whichever granularity needed for your metrics (either daily or hourly). You can optionally define additional tables for coarser grains (like monthly or yearly).
- [Configure each time spine table in a YAML file](#configuring-time-spine-in-yaml) to define how MetricFlow recognizes and uses its columns.

MetricFlow will then join against the time spine model for the following types of metrics and dimensions:

- [Cumulative metrics](/docs/build/cumulative)
- [Metric offsets](/docs/build/derived#derived-metric-offset)
- [Conversion metrics](/docs/build/conversion)
- [Slowly Changing Dimensions](/docs/build/dimensions#scd-type-ii)
- [Metrics](/docs/build/metrics-overview) with the `join_to_timespine` configuration set to true

To see the generated SQL for the metric and dimension types that use time spine joins, refer to the respective documentation or add the `compile=True` flag when querying the Semantic Layer to return the compiled SQL.
To see the generated SQL for the metric and dimension types that use time spine joins, refer to the respective documentation or add the `compile=true` flag when querying the Semantic Layer to return the compiled SQL.

## Configuring time spine in YAML

Time spine models are normal dbt models with extra configurations that tell dbt and MetricFlow how to use specific columns by defining their properties. Add the [`models` key](/reference/model-properties) for the time spine in your `models/` directory. If your project already includes a calendar table or date dimension, you can configure that table as a time spine. Otherwise, review the [example time-spine tables](#example-time-spine-tables) to create one. If the relevant model file (`util/_models.yml`) doesn't exist, create it and add the configuration mentioned in the [next section](#creating-a-time-spine-table).

Some things to note when configuring time spine models:

- Make sure you already have a time spine SQL table defined in your project.
- Add the configurations under the `time_spine` key for that [model's properties](/reference/model-properties), just as you would add a description or tests.
- You only need to configure time-spine models that the Semantic Layer should recognize.
- At a minimum, define a time-spine table for a daily grain.
Expand Down Expand Up @@ -147,6 +166,13 @@ Note that if you're migrating from a `metricflow_time_spine.sql` file:

## Example time spine tables

The following examples show how to create time spine tables at different granularities:

- [Daily](#daily)
- [Daily (BigQuery)](#daily-bigquery)
- [Hourly](#hourly)


### Daily

<File name="metricflow_time_spine.sql">
Expand Down
258 changes: 258 additions & 0 deletions website/docs/guides/mf-time-spine.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,258 @@
---
title: "Quickstart with MetricFlow time spine"
id: "mf-time-spine"
level: 'Intermediate'
icon: 'guides'
tags: ['dbt Cloud','Quickstart']
hide_table_of_contents: true
recently_updated: true
---

<div style={{maxWidth: '900px'}}>

## Introduction

This guide explains how to configure a time spine using the [dbt Semantic Layer Jaffle shop example project](https://github.com/dbt-labs/jaffle-sl-template) as a reference.

### Prerequisites
Before you start, make sure you have:

- A dbt project set up. If you don't have one, follow the [Quickstart guides](/guides?tags=Quickstart) guide to help you get started.

### What is a time spine?

A [time spine](/docs/build/metricflow-time-spine) is essential for time-based joins and aggregations in MetricFlow, the engine that powers the dbt Semantic Layer. It can be created from scratch or configured using an existing table (like a `dim_date` table).

To use MetricFlow with time-based metrics and dimensions, you must provide a time spine table. This table serves as the foundation for time-based joins and aggregations. You can either:

- Create a time spine SQL table from scratch, or
- Use an existing table in your project, like a `dim_date` table

And once you have a time spine table, you need to configure it in YAML to tell MetricFlow how to use it. This guide will show you how to do both!

## Add a time spine SQL model

Let's get started by assuming you're creating a time spine table from scratch. If you have a dbt project set up already and have your own time spine table (like a `dim_date` type model), you can skip this step and go to [Use an existing dim_date table](https://docs.getdbt.com//guides/mf-time-spine#using-an-existing-dim-date-table).
mirnawong1 marked this conversation as resolved.
Show resolved Hide resolved

The time spine table is a dbt model that generates a series of dates (or timestamps) at a specific granularity. In this example, let's create a daily time spine table &mdash; `time_spine_daily.sql`.

1. Navigate to the `models/marts` directory in your dbt project.
2. Add a new SQL file named `time_spine_daily.sql` with the following content:

<File name='models/marts/time_spine_daily.sql'>

```sql
{{
config(
materialized = 'table',
schema = 'marts',
alias = 'time_spine_daily'
mirnawong1 marked this conversation as resolved.
Show resolved Hide resolved
)
}}

with

base_dates as (
{{
dbt.date_spine(
'day',
"DATE('2000-01-01')",
"DATE('2030-01-01')"
)
}}
),

final as (
select
cast(date_day as date) as date_day
from base_dates
)

select *
from final
where date_day > dateadd(year, -5, current_date()) -- Keep recent dates only
and date_day < dateadd(day, 30, current_date());
```
</File>

This generates a table of daily dates ranging from 5 years in the past to 30 days into the future.


## Add YAML configuration for the time spine

Now that you've created the SQL file, configure it in YAML so MetricFlow can recognize and use it.

1. Navigate to the `models/marts` directory.
2. Add a new YAML file named `_models.yml` with the following content:

<File name='models/marts/_models.yml'>

```yaml
models:
- name: time_spine_daily
description: A time spine with one row per day, ranging from 2000-01-01 to 2030-01-01.
mirnawong1 marked this conversation as resolved.
Show resolved Hide resolved
time_spine:
standard_granularity_column: date_day # The base column used for time joins
columns:
- name: date_day
description: The base date column for daily granularity
tests:
- not_null
- unique
mirnawong1 marked this conversation as resolved.
Show resolved Hide resolved
granularity: day
```
</File>

This time spine YAML file:
- Defines `date_day` as the base column for daily granularity.
- Adds [tests](/docs/build/tests) for uniqueness and non-null values.
- Configures `time_spine` properties so MetricFlow can use the table

### Using an existing dim_date table
mirnawong1 marked this conversation as resolved.
Show resolved Hide resolved

This optional approach reuses an existing table, saving you the effort of creating a new one. However if you created a time spine table from scratch, you can skip this section.

If your project already includes a `dim_date` or similar table, you can configure it as a time spine:

1. Locate the existing table (`dim_date`).
2. Update `_models.yml` file to configure it as a time spine:

<File name='_models.yml'>

```yaml
models:
- name: dim_date
description: An existing date dimension table used as a time spine.
time_spine:
standard_granularity_column: date_day
columns:
- name: date_day
granularity: day
```
</File>

This time spine YAML file configures the `time_spine` property so MetricFlow can use the table.

## Run and test the time spine

For the time spine table you created, let's run it and validate the output.
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@patkearns10 , i think maybe lines 133-143 dont make sense. i probably should just start with 144, query dbt sl query --metrics revenue --group-by metric_time -- does that make sense to you in terms of how to test the .sql and yaml file work?
Screenshot 2025-02-07 at 17 23 16

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

eh, I can imagine most people are building the time spine for the first time. And it will fail if they do not run that time spine model.
Why do you want to not include those lines?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i think it's because when i ran dbt run, i didn't see anything that confirmed it for me -- maybe in just the compiled code?


1. Run the time spine model to create the table:
```bash
dbt run --select time_spine_daily
```

2. Validate the output by querying the generated table:
```sql
select * from your_project_schema.time_spine_daily
```

3. Check that the table:
- Contains one row per day.
- Covers the date range you want (5 years back to 30 days forward)

4. (Optional) If you have [metrics](/docs/build/metrics) already defined in your project, you can query the table/metrics using [Semantic Layer commands](/docs/build/metricflow-commands) to validate the time spine.

Let's say you have a `revenue` metric defined. You can query the table/metrics using the following command:

```bash
dbt sl query --metrics revenue --group-by metric_time
```

This will output results similar to the following in the dbt Cloud IDE:
<Lightbox src="/img/quickstarts/dbt-cloud/validate-mf-timespine-output.jpg" title="Validate the time spine output in dbt Cloud IDE" />

5. Double check that the results are correct and returning the expected data.

## Add additional granularities (optional)
mirnawong1 marked this conversation as resolved.
Show resolved Hide resolved
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i thought we could do this based on this example but when i tested it, it only returned date_day.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

did you get this working? I would be happy to hop on a call and co-dev-test together if you can find overlapping time!


To support multiple granularities (like hourly, monthly), create additional time spine tables and configure them in YAML.

1. Add a new SQL file named `time_spine_hourly.sql` with the following content:
<File name='models/marts/time_spine_hourly.sql'>

```sql
{{
config(
materialized = 'table',
)
}}

with hours as (

{{
dbt.date_spine(
'hour',
"to_date('01/01/2000','mm/dd/yyyy')",
"to_date('01/01/2025','mm/dd/yyyy')"
)
}}

),

final as (
select cast(date_hour as timestamp) as date_hour
from hours
)

select * from final
-- filter the time spine to a specific range
where date_hour > dateadd(year, -4, current_timestamp())
and date_hour < dateadd(day, 30, current_timestamp())

```
</File>

2. Then update `_models.yml` file and add the hourly time spine (below the daily time spine config):

<File name='_models.yml'>

```yaml
models:
- name: time_spine_daily
... rest of the daily time spine config ...

- name: time_spine_hourly
description: A time spine with one row per hour, ranging from 2000-01-01 to 2030-01-01.
mirnawong1 marked this conversation as resolved.
Show resolved Hide resolved
time_spine:
standard_granularity_column: date_hour
columns:
- name: date_hour
granularity: hour
```
</File>

3. Run the model to create the table:
```bash
dbt run --select time_spine_hourly
```

4. Validate the output by querying the generated table:
```bash
dbt sl query --metrics revenue --group-by metric_time__hour
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

also the ide doesn't support metric_time__hour -- is this right?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

or should i scrap this section?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would think the IDE supports metric_time__hour, why do you say otherwise?

You could try
dbt sl list dimensions --metrics revenue

```

:::tip
Try creating a monthly time spine! Duplicate your daily time spine model, adjust it to generate one row per month, and update the YAML file to include `granularity: month`. Give it a try!
:::

## What's next
mirnawong1 marked this conversation as resolved.
Show resolved Hide resolved

<ConfettiTrigger>

Congratulations 🎉! You've set up a time spine and are ready to bring the benefits of MetricFlow and the dbt Semantic Layer to your organization. You've learned:

- How to create a time spine table or use an existing table.
- How to configure a time spine in YAML.
- How to add additional granularities to your time spine.

Here are some additional resources to help you continue your journey:

- [MetricFlow time spine](/docs/build/metricflow-time-spine)
- [dbt Semantic Layer](/docs/use-dbt-semantic-layer)
- [Build metrics](/docs/build/metrics)
- [Quickstart with dbt Semantic Layer](/guides/quickstart-dbt-semantic-layer)

</ConfettiTrigger>

</div>
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading