Generator remote write update handling of X-Scope-OrgID

[mdisibio]

What this PR does:
This PR changes the header precedence in Generator remote write, so that statically configured X-Scope-OrgID is honored. This change is necessary to unlock a use case that is currently not possible. Will do my best to explain. The desired behavior is to have a setup like this:

• 2 remote write targets with different headers and auth:
  • remote write target A:
    • The usual setup:
    • dynamically set X-Scope-OrgID header == tenant ID
  • remote write target B:
    • statically configured X-Scope-OrgID (we want to target a fixed metrics tenant)
    • Dynamically injected metrics label with the tenant ID like {__id__="<tenantid>"}

Target B doesn't work currently because the logic gives highest precedence to dynamically set X-Scope-OrgID. The statically configured header is deleted and a warning logged.

This logic was inherited and designed to protect against broken authentication, but after review it doesn't seem a compelling use case. Remote write headers can be set in two locations:

1. In the global config:

metrics_generator:
  storage:
    remote_write:
      - url: ...
        headers:
          X-Scope-OrgID: ...

2. In runtime overrides per-tenant:

overrides:
  '*':
    metrics_generator_remote_write_headers:
      X-Scope-OrgID: ...

Both locations are statically configured by a Tempo operator, and it is more likely they are intentional than not.

Which issue(s) this PR fixes:
Fixes n/a

Checklist

• Tests updated
• Documentation added
• CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

[yvrhdn]

LGTM, PR is clear 👍