[CT-1138] [Feature] Multiple VARS: blocks in the project yaml - last one overwrites first

[tom-packert]

Is this your first time submitting a feature request?

• I have read the expectations for open source contributors
• I have searched the existing issues, and I could not find an existing issue for this feature
• I am requesting a straightforward extension of existing dbt functionality, rather than a Big Idea better suited to a discussion

Describe the feature

I added a VARS section to an existing project.yml file. I was adding it to the top of the file as the documentation instructed. I kept getting errors when I would compile a DBT model that referenced the VARS. It turned out there was a second VARS block at the bottom of the project yaml file and this one was overwriting the first VARS block where I had added my new variables. It took about 15 minutes to figure out what was going on, but a warning about a second VARS block might save people time.

Describe alternatives you've considered

Manually look for VARS blocks, dont assume they will be at the top.

Who will this benefit?

Any new DBT developer

Are you interested in contributing this feature?

No response

Anything else?

No response

[dbeatty10]

Thanks for sharing your experience @tom-packert!

Agreed that a clear warning (or error) would be helpful in this scenario. Something like "Duplicate key vars within dbt_project.yml".

I'm going to label this as "help_wanted" in case anyone wants to take a shot at implementing this.

Explanation

Although the YAML spec says that keys "must be unique", many parsers allow duplicates including pyyaml, which is what dbt uses currently. pyyaml has tickets open for this behavior (here and here). Since pyyaml allows the duplicates without warning or error, dbt inherits that behavior as-is and merrily rolls along.

Hints for implementation

There are examples of how to enforce uniqueness with the current version of pyyaml like this or this.

[jtcohen6]

I think this issue may in fact be a duplicate of #5114. We tried resolving it a few months ago, ran into some undesirable behaviors, and ended up reverting it rather than leave users with more-surprising errors.

I'm going to close it as such, to keep the conversation centralized over there.