[docs/rfc] RFC about environment variables #9854

mx-psi · 2024-03-26T16:45:50Z

Description: Adds an RFC about how environment variable resolution should work
Link to tracking Issue: Fixes #9515, relates to:

codecov · 2024-03-26T16:55:35Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 91.70%. Comparing base (65cdb18) to head (90db1c0).
Report is 37 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #9854      +/-   ##
==========================================
- Coverage   91.95%   91.70%   -0.25%     
==========================================
  Files         357      358       +1     
  Lines       16501    16543      +42     
==========================================
- Hits        15173    15171       -2     
- Misses       1000     1045      +45     
+ Partials      328      327       -1

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

mx-psi

Self-reviewing to start some threads about topics I want to discuss

docs/rfcs/env-vars.md

mx-psi · 2024-04-03T10:33:05Z

docs/rfcs/env-vars.md

+When an invalid identifier is found, a warning log is emitted, and the
+string is kept as-is.


I am undecided about this, I feel like we should fail in some cases but I am not sure about which. open-telemetry/opentelemetry-specification/issues/3981 has some more discussion about this

I agree with aligning with the configuration spec and I hope that ends up being failing. I think failing fast is a clearer outcome for the user.

For users that mis-typed the env var name, keeping the string verbatim could mean an unintuitive error caused by the unwanted string. If we fail immediately we can log a clear error message that will be easier to troubleshoot.

If the user needs an illegal string like ${env:this-is-illegal} they can use the escape to make it exact: $${env:this-is-illegal}.

To summarize: using the env var syntax is a request to parse an env var - if the provided env var cannot be parsed we should fail.

@TylerHelmuth The one edge case here is ${1} from open-telemetry/opentelemetry-collector-contrib/issues/9984. Should we fail in that case and force uses to use escaping syntax?

Since that is what we currently do and it seems impossible to detect all kinds of syntaxes past or future that would overlap with this, I have changed this to an error on 3f05467

@mx-psi I feel like since ${1} corresponds with the regex it should be expanded. If users need the literal ${1} can they do $${1}?

I feel like since ${1} corresponds with the regex it should be expanded

I think we are on the same page, but to be clear 1 is not a valid identifier per the 'desired behavior' section (valid identifiers need to start with underscore or ASCII alphabetic characters), so it should not be expanded, but rather raise an error.

Ah ok, then we are on the same page, I just got the outcome wrong. My opinion is the ${1} should fail and if users need that literal string they should escape: $${1}.

This prevents us from having any edge cases with our unwrap syntax while keeping all functionality available.

@mx-psi where in the spec do we need to raise this issue?

Tyler and I discussed this offline but for the benefit of anybody else keeping track of this: this is being discussed on open-telemetry/opentelemetry-specification/pull/4002

docs/rfcs/env-vars.md

mx-psi · 2024-04-03T10:36:28Z

docs/rfcs/env-vars.md

+| `0123`       | string     | 0123                                     | 83                                             | 0123                           | 0123                                  |
+| `0xdeadbeef` | string     | 0xdeadbeef                               | 3735928559                                     | 0xdeadbeef                     | 0xdeadbeef                            |
+| `"0123"`     | string     | "0123"                                   | 0123                                           | 0123                           | 0123                                  |
+| `!!str 0123` | string     | !!str 0123                               | 0123                                           | 0123                           | 0123                                  |


For 'Desired behavior, inline string field' here, the behavior of the File Configuration SDK would be to set !!str 0123. I think this is a rare edge case, but as written it would set 0123 since the AsString method would not return the original representation

For 'Desired behavior, inline string field' here, the behavior of the File Configuration SDK would be to set !!str 0123.

Can you link to where this is defined? I can't see where this is addressed this outside of declaring a suggested minimum version of the YAML spec.

I think the closest one would be the combo_string_key from https://opentelemetry.io/docs/specs/otel/configuration/file-configuration/#environment-variable-substitution

I take this to mean that the original value passed by the user on the environment variable should be preserved as is when you use expansion inline

I vote we stick with whatever the configuration sig is doing

docs/rfcs/env-vars.md

Co-authored-by: Alex Boten <[email protected]>

docs/rfcs/env-vars.md

Co-authored-by: Evan Bradley <[email protected]>

…ment

docs/rfcs/env-vars.md

Co-authored-by: Alex Boten <[email protected]>

docs/rfcs/env-vars.md

TylerHelmuth · 2024-04-05T19:53:42Z

docs/rfcs/env-vars.md

+When an invalid identifier is found, a warning log is emitted, and the
+string is kept as-is.


I agree with aligning with the configuration spec and I hope that ends up being failing. I think failing fast is a clearer outcome for the user.

For users that mis-typed the env var name, keeping the string verbatim could mean an unintuitive error caused by the unwanted string. If we fail immediately we can log a clear error message that will be easier to troubleshoot.

If the user needs an illegal string like ${env:this-is-illegal} they can use the escape to make it exact: $${env:this-is-illegal}.

To summarize: using the env var syntax is a request to parse an env var - if the provided env var cannot be parsed we should fail.

docs/rfcs/env-vars.md

Make an invalid identifier an explicit error

songy23

LGTM

mx-psi · 2024-04-16T11:35:53Z

Remaining work

Decide on [docs/rfc] RFC about environment variables #9854 (comment) by raising this on the spec and seeing if we can align on that
- Filed [configuration] Fail on unsupported identifiers opentelemetry-specification#4002
~~Decide on [docs/rfc] RFC about environment variables #9854 (comment). I am going to try and do a PoC of the confmap behavior to double check what makes more sense~~

TylerHelmuth · 2024-04-16T16:08:24Z

docs/rfcs/env-vars.md

+
+The naked syntax supported in BASH is not supported in the Collector.
+Escaping is supported by using two dollar signs. Escaping is also
+honored for unsupported identifiers like `${1}` (i.e. anything that 


Based on our discussion in this thread this sentence should be updated to clarify that ${1} is no supported.

TylerHelmuth · 2024-04-16T16:09:54Z

docs/rfcs/env-vars.md

+When an invalid identifier is found, a warning log is emitted, and the
+string is kept as-is.


@mx-psi where in the spec do we need to raise this issue?

docs/rfcs/env-vars.md

Co-authored-by: Evan Bradley <[email protected]>

codeboten · 2024-04-30T17:24:10Z

Remaining work

* [ ]  Decide on [[docs/rfc] RFC about environment variables #9854 (comment)](https://github.com/open-telemetry/opentelemetry-collector/pull/9854#discussion_r1561181980) by raising this on the spec and seeing if we can align on that
  
  * Filed [[configuration] Fail on unsupported identifiers opentelemetry-specification#4002](https://github.com/open-telemetry/opentelemetry-specification/pull/4002)

* [ ]  Decide on [[docs/rfc] RFC about environment variables #9854 (comment)](https://github.com/open-telemetry/opentelemetry-collector/pull/9854#discussion_r1549449041). I am going to try and do a PoC of the confmap behavior to double check what makes more sense

@mx-psi it looks like the spec issue will merge within the next 2 days, is the second issue still blocking this work?

mx-psi · 2024-04-30T18:44:03Z

@codeboten I haven't written a fully working PoC yet. I am fine merging this without the PoC if everybody else is okay with it (it may mean minor changes while we work on implementing this)

codeboten · 2024-04-30T21:27:10Z

Let's merge this and change it as needed, the spec change was merged 👍🏻

**Description:** Adds an RFC about how environment variable resolution should work **Link to tracking Issue:** Fixes open-telemetry#9515, relates to: - open-telemetry#8215 - open-telemetry#8565 - open-telemetry#9162 - open-telemetry#9531 - open-telemetry#9532 --------- Co-authored-by: Alex Boten <[email protected]> Co-authored-by: Evan Bradley <[email protected]>

#### Description  Adds tests of current type casting behavior when using env and file provider. #### Link to tracking issue Relates to #9854, #8565, #9532

#### Description  - Add `confmap.strictlyTypedInput` feature gate that introduces stricter type checks when resolving configuration - Make `confmap.NewRetrievedFromYAML` function public so that external providers have consistent behavior when resolving YAML - Adds `confmap.Retrieved.AsString` method to retrieve string representation for retrieved values #### Link to tracking issue Relates to #9854, updates #8565, #9532

mx-psi force-pushed the mx-psi/env-vars branch from f8c7d8b to 132e35a Compare March 26, 2024 16:46

[docs/rfc] RFC about environment variables

ac9a078

mx-psi force-pushed the mx-psi/env-vars branch from 132e35a to ac9a078 Compare March 27, 2024 12:01

mx-psi added the Skip Changelog PRs that do not require a CHANGELOG.md entry label Mar 27, 2024

Make some decisions

026321d

mx-psi marked this pull request as ready for review April 3, 2024 10:24

mx-psi requested review from a team and Aneurysm9 April 3, 2024 10:24

Merge branch 'main' into mx-psi/env-vars

5ac7e49

mx-psi requested review from evan-bradley, andrzej-stencel and swiatekm April 3, 2024 10:27

mx-psi commented Apr 3, 2024

View reviewed changes

codeboten reviewed Apr 3, 2024

View reviewed changes

docs/rfcs/env-vars.md Outdated Show resolved Hide resolved

docs/rfcs/env-vars.md Outdated Show resolved Hide resolved

Apply suggestions from code review

2b0a23c

Co-authored-by: Alex Boten <[email protected]>

evan-bradley reviewed Apr 3, 2024

View reviewed changes

docs/rfcs/env-vars.md Outdated Show resolved Hide resolved

mx-psi and others added 2 commits April 3, 2024 17:21

Apply suggestions from code review

ed704e5

Co-authored-by: Evan Bradley <[email protected]>

Add a couple of links that were in review comments to the actual docu…

1a0e9d4

…ment

ankitpatel96 approved these changes Apr 4, 2024

View reviewed changes

codeboten reviewed Apr 4, 2024

View reviewed changes

docs/rfcs/env-vars.md Outdated Show resolved Hide resolved

Apply suggestions from code review

40333d3

Co-authored-by: Alex Boten <[email protected]>

TylerHelmuth reviewed Apr 5, 2024

View reviewed changes

songy23 reviewed Apr 5, 2024

View reviewed changes

docs/rfcs/env-vars.md Show resolved Hide resolved

mx-psi added 2 commits April 8, 2024 18:11

Address feedback from @TylerHelmuth

3f05467

Make an invalid identifier an explicit error

Explicitly mention the comparison table within the body

0d45523

songy23 approved these changes Apr 8, 2024

View reviewed changes

mx-psi and others added 2 commits April 8, 2024 18:28

Clarify escaping rules

0862cc9

Merge branch 'main' into mx-psi/env-vars

b7145d2

Explicitly state why we support two syntaxes

c8307a0

andrzej-stencel approved these changes Apr 16, 2024

View reviewed changes

TylerHelmuth reviewed Apr 16, 2024

View reviewed changes

evan-bradley reviewed Apr 17, 2024

View reviewed changes

docs/rfcs/env-vars.md Outdated Show resolved Hide resolved

Apply suggestions from code review

90db1c0

Co-authored-by: Evan Bradley <[email protected]>

TylerHelmuth mentioned this pull request Apr 17, 2024

[configuration] Environment variable substitution syntax is hard to extend open-telemetry/opentelemetry-specification#3981

Closed

codeboten added this to the `go.opentelemetry.io/collector/confmap` 1.0 milestone Apr 17, 2024

codeboten merged commit fe582df into open-telemetry:main Apr 30, 2024
49 checks passed

github-actions bot modified the milestones: go.opentelemetry.io/collector/confmap 1.0, next release Apr 30, 2024

mx-psi mentioned this pull request May 15, 2024

[confmap] Unify behavior of ${env:ENV} and ${ENV} #10161

Closed

mx-psi mentioned this pull request Jun 13, 2024

[chore] Add end to end tests for type casting #10399

Merged

mx-psi mentioned this pull request Jun 13, 2024

[confmap] Add strict type validation under a feature gate #10400

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[docs/rfc] RFC about environment variables #9854

[docs/rfc] RFC about environment variables #9854

mx-psi commented Mar 26, 2024 •

edited

Loading

codecov bot commented Mar 26, 2024 •

edited

Loading

mx-psi left a comment

mx-psi Apr 3, 2024

TylerHelmuth Apr 5, 2024

mx-psi Apr 8, 2024

mx-psi Apr 8, 2024

TylerHelmuth Apr 10, 2024 •

edited

Loading

mx-psi Apr 11, 2024

TylerHelmuth Apr 11, 2024

TylerHelmuth Apr 16, 2024

mx-psi Apr 19, 2024

mx-psi Apr 3, 2024

evan-bradley Apr 17, 2024

mx-psi Apr 17, 2024

TylerHelmuth Apr 17, 2024

TylerHelmuth Apr 5, 2024

songy23 left a comment

mx-psi commented Apr 16, 2024 •

edited

Loading

TylerHelmuth Apr 16, 2024

TylerHelmuth Apr 16, 2024

codeboten commented Apr 30, 2024

Remaining work

mx-psi commented Apr 30, 2024

codeboten commented Apr 30, 2024

		When an invalid identifier is found, a warning log is emitted, and the
		string is kept as-is.

[docs/rfc] RFC about environment variables #9854

[docs/rfc] RFC about environment variables #9854

Conversation

mx-psi commented Mar 26, 2024 • edited Loading

codecov bot commented Mar 26, 2024 • edited Loading

Codecov Report

mx-psi left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

TylerHelmuth Apr 10, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

songy23 left a comment

Choose a reason for hiding this comment

mx-psi commented Apr 16, 2024 • edited Loading

Remaining work

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codeboten commented Apr 30, 2024

Remaining work

mx-psi commented Apr 30, 2024

codeboten commented Apr 30, 2024

mx-psi commented Mar 26, 2024 •

edited

Loading

codecov bot commented Mar 26, 2024 •

edited

Loading

TylerHelmuth Apr 10, 2024 •

edited

Loading

mx-psi commented Apr 16, 2024 •

edited

Loading