[chore] Schema Processor Revamp Implementation Parent PR #35248

ankitpatel96 · 2024-09-17T16:00:42Z

Description:
This is a duplicate of #35213 - I had to reopen this because the Github Actions refused to trigger.

This is the schema processor, as originally written by @MovieStoreGuy in #11547 - refreshed and rebased. I've rewritten some parts and added significant amounts of testing to many parts - resulting in quite a few changes. I will be splitting this PR into many parts.

There are several features that need to be completed before this is ready for production use:

This only supports the Schema File Format 1.0.0 - we need to support 1.1.0
Right now only downgrades work. Upgrades need some work to enable
We should improve internal metrics and observability.

tigrannajaryan · 2024-09-18T13:48:51Z

I will be splitting this PR into many parts.

@ankitpatel96 thank you for working on this. Are you looking for reviews on this PR or we should wait until you split into smaller ones?

ankitpatel96 · 2024-09-18T13:55:37Z

I will be splitting this PR into many parts.

@ankitpatel96 thank you for working on this. Are you looking for reviews on this PR or we should wait until you split into smaller ones?

Hi @tigrannajaryan, I am definitely not trying to get a review on this entire PR haha - I am in the process of splitting it into 3-4 PRs.

Part 1 - #35214
Part 2 - #35267

The rest of the code is largely done - I'm just working on splitting it into stacked PRs that makes sense.

I am working up a mini design document so people have an idea of what they are looking at.
Will leave a comment on this PR when that's done!

jsuereth · 2024-09-19T13:09:43Z

Edit: Added link to specification

One major question -> Is this not intended to support resource attribute changes?

I was going through looking for the intricate "fun" code to deal with that scenario, but I didn't see anything, nor do I see tests associated with it.

I think it's fine for to disallow that for an initial implementation/improvement, but you may want to consider the architecture/design for when you need to apply attirbute name changes on resources in addition to signals.

Specifically -> The shuffling of Data points out of a resource and into a new one is... fun. From my SDK implementation this was much simpler, as the SDK only allows one resource. For the Collector -> I think you'll need some robust code, checks and performance tests around that.

See https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/schemas/file_format_v1.1.0.md#all-section - specifically the notion there's a resource attribute rename possibility and that all must apply to that too.

ankitpatel96 · 2024-09-19T20:49:53Z

Hi Josh,
I have implemented attribute renames for the all block and the resource block but I made a mistake and didn't realize that I had to make each ResourceLogs struct have a unique Resource within in a plog.Logs object (along with all the other signal types). I cut a ticket for it - #35305 - and I think this is definitely something we can support in the future. Thanks for bringing it up!

github-actions · 2024-10-15T05:21:14Z

This PR was marked stale due to lack of activity. It will be closed in 14 days.

github-actions · 2024-10-31T05:20:45Z

This PR was marked stale due to lack of activity. It will be closed in 14 days.

github-actions · 2024-11-22T05:21:01Z

This PR was marked stale due to lack of activity. It will be closed in 14 days.

…35267) **Description:** <Describe what has changed.> This is a slice of changes from #35248 This PR details how operators are used to build the execution pipeline for a given schemafile. Changed files from the [previous PR](#35214) are: processor/schemaprocessor/internal/changelist/changelist.go processor/schemaprocessor/internal/translation/revision_v1.go processor/schemaprocessor/internal/translation/revision_v1_test.go processor/schemaprocessor/go.mod I'm asking a maintainer if they would be willing to push a copy of the previous PR's branch to the core repo so I can switch the base of this PR to the previous PR - thus only the stacked changes would be shown. Edit: this is apparently not easily supported - so asking reviewers to just focus on the changed files listed above. Sorry about that! **Testing:** <Describe what testing was performed and which tests were added.> Unit tests --------- Co-authored-by: Pablo Baeyens <[email protected]>

…pen-telemetry#35267) **Description:** <Describe what has changed.> This is a slice of changes from open-telemetry#35248 This PR details how operators are used to build the execution pipeline for a given schemafile. Changed files from the [previous PR](open-telemetry#35214) are: processor/schemaprocessor/internal/changelist/changelist.go processor/schemaprocessor/internal/translation/revision_v1.go processor/schemaprocessor/internal/translation/revision_v1_test.go processor/schemaprocessor/go.mod I'm asking a maintainer if they would be willing to push a copy of the previous PR's branch to the core repo so I can switch the base of this PR to the previous PR - thus only the stacked changes would be shown. Edit: this is apparently not easily supported - so asking reviewers to just focus on the changed files listed above. Sorry about that! **Testing:** <Describe what testing was performed and which tests were added.> Unit tests --------- Co-authored-by: Pablo Baeyens <[email protected]>

ankitpatel96 · 2024-12-20T16:11:49Z

Hi, this is ready for review! @tigrannajaryan @MovieStoreGuy please take a look!

Much of this is @MovieStoreGuy's original work - it works to lookup and cache the schema files as well as orchestrating the rest of the code / hooking it up to the collector factory. The vast majority of this PR is tests that I've written that are pretty end-to-end.

…pen-telemetry#35267) **Description:** <Describe what has changed.> This is a slice of changes from open-telemetry#35248 This PR details how operators are used to build the execution pipeline for a given schemafile. Changed files from the [previous PR](open-telemetry#35214) are: processor/schemaprocessor/internal/changelist/changelist.go processor/schemaprocessor/internal/translation/revision_v1.go processor/schemaprocessor/internal/translation/revision_v1_test.go processor/schemaprocessor/go.mod I'm asking a maintainer if they would be willing to push a copy of the previous PR's branch to the core repo so I can switch the base of this PR to the previous PR - thus only the stacked changes would be shown. Edit: this is apparently not easily supported - so asking reviewers to just focus on the changed files listed above. Sorry about that! **Testing:** <Describe what testing was performed and which tests were added.> Unit tests --------- Co-authored-by: Pablo Baeyens <[email protected]>

tigrannajaryan · 2025-01-15T15:44:28Z

I will be splitting this PR into many parts.

@ankitpatel96 do you plan to split this into smaller bits? It is over 4K line changes, hard to review such a big PR.

ankitpatel96 · 2025-01-15T15:47:29Z

I will be splitting this PR into many parts.

@ankitpatel96 do you plan to split this into smaller bits? It is over 4K line changes, hard to review such a big PR.

I can definitely try - I will say that many many (more than half) of these changes are test schema files, and input and output OTLP payloads. I will try to split up this PR further before next week if that works for you

tigrannajaryan · 2025-01-15T16:53:17Z

I can definitely try - I will say that many many (more than half) of these changes are test schema files, and input and output OTLP payloads. I will try to split up this PR further before next week if that works for you

Please do if there is a reasonable way to do it. If you can't, that's OK, but it will likely delay my review, I will need to find a bigger chunk of time to review this PR.

tigrannajaryan · 2025-01-20T23:06:14Z

@ankitpatel96 please ping me when you have a PR to review.

ankitpatel96 · 2025-01-22T16:19:38Z

@tigrannajaryan I've pushed up 2 more parts -
#37403
#37402
The first one is pretty long looking but its really just 1 file and then a lot of testing / testing helpers. The second is much shorter. The final chunk is what will be left on this PR once the previous two are merged. Again, the second is stacked on top of the first, and the final one is stacked on top of the second.

MovieStoreGuy · 2025-01-24T01:39:55Z

I will block out some time to review this detail.

#### Description This is a slice of changes from #35248 This PR details Translation which defines the complete abstraction of schema translation file. It manages Revisions to move between two versions of a schema  #### Testing Unit tests

ankitpatel96 requested a review from MovieStoreGuy as a code owner September 17, 2024 16:00

ankitpatel96 requested a review from a team September 17, 2024 16:00

github-actions bot assigned mx-psi Sep 17, 2024

github-actions bot added the processor/schema Schema processor label Sep 17, 2024

ankitpatel96 changed the title ~~Schema Processor Revamp implementation~~ [chore] Schema Processor Revamp Implementation Parent PR Sep 17, 2024

mx-psi requested a review from tigrannajaryan September 18, 2024 08:29

ankitpatel96 mentioned this pull request Sep 19, 2024

Schema Processor - ensure unique Resources #35305

Open

ankitpatel96 requested a review from a team as a code owner September 20, 2024 20:51

ankitpatel96 force-pushed the ankit-schema-processor branch from 9c0ad9c to 1829df1 Compare September 30, 2024 20:17

ankitpatel96 mentioned this pull request Oct 9, 2024

Schema Processor Revamp [Part 1] - Operators and Migrators #35214

Closed

github-actions bot added the Stale label Oct 15, 2024

ankitpatel96 force-pushed the ankit-schema-processor branch from 1829df1 to 472cedb Compare October 15, 2024 19:26

github-actions bot removed the Stale label Oct 16, 2024

ankitpatel96 force-pushed the ankit-schema-processor branch from 472cedb to af3a4fd Compare October 16, 2024 22:01

github-actions bot added the Stale label Oct 31, 2024

ankitpatel96 force-pushed the ankit-schema-processor branch from af3a4fd to 12aeb95 Compare November 7, 2024 21:59

ankitpatel96 mentioned this pull request Nov 7, 2024

[chore] Schema Processor Revamp [Part 2] - ChangeList and Revision #35267

Merged

ankitpatel96 force-pushed the ankit-schema-processor branch from 12aeb95 to 58256c1 Compare November 7, 2024 23:03

github-actions bot removed the Stale label Nov 8, 2024

github-actions bot added the Stale label Nov 22, 2024

songy23 added never stale Issues marked with this label will be never staled and automatically removed and removed Stale labels Nov 22, 2024

ankitpatel96 force-pushed the ankit-schema-processor branch from 58256c1 to 64aedc8 Compare December 19, 2024 18:07

ankitpatel96 force-pushed the ankit-schema-processor branch 2 times, most recently from fa1989e to 2e80a66 Compare January 22, 2025 03:29

ankitpatel96 added 2 commits January 22, 2025 00:06

split out translation chunk

4063e74

split out manager provider

b846651

This was referenced Jan 22, 2025

[chore] Schema Processor Revamp [Part 3] - Translation #37403

Merged

[chore] Schema Processor Revamp [Part 4] - Manager/Provider #37402

Open

final processor chunk split out

ecf3957

ankitpatel96 force-pushed the ankit-schema-processor branch from 2e80a66 to ecf3957 Compare January 22, 2025 18:50

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[chore] Schema Processor Revamp Implementation Parent PR #35248

[chore] Schema Processor Revamp Implementation Parent PR #35248

ankitpatel96 commented Sep 17, 2024

tigrannajaryan commented Sep 18, 2024

ankitpatel96 commented Sep 18, 2024 •

edited

Loading

jsuereth commented Sep 19, 2024 •

edited

Loading

ankitpatel96 commented Sep 19, 2024

github-actions bot commented Oct 15, 2024

github-actions bot commented Oct 31, 2024

github-actions bot commented Nov 22, 2024

ankitpatel96 commented Dec 20, 2024

tigrannajaryan commented Jan 15, 2025

ankitpatel96 commented Jan 15, 2025

tigrannajaryan commented Jan 15, 2025

tigrannajaryan commented Jan 20, 2025

ankitpatel96 commented Jan 22, 2025

MovieStoreGuy commented Jan 24, 2025

[chore] Schema Processor Revamp Implementation Parent PR #35248

Are you sure you want to change the base?

[chore] Schema Processor Revamp Implementation Parent PR #35248

Conversation

ankitpatel96 commented Sep 17, 2024

tigrannajaryan commented Sep 18, 2024

ankitpatel96 commented Sep 18, 2024 • edited Loading

jsuereth commented Sep 19, 2024 • edited Loading

ankitpatel96 commented Sep 19, 2024

github-actions bot commented Oct 15, 2024

github-actions bot commented Oct 31, 2024

github-actions bot commented Nov 22, 2024

ankitpatel96 commented Dec 20, 2024

tigrannajaryan commented Jan 15, 2025

ankitpatel96 commented Jan 15, 2025

tigrannajaryan commented Jan 15, 2025

tigrannajaryan commented Jan 20, 2025

ankitpatel96 commented Jan 22, 2025

MovieStoreGuy commented Jan 24, 2025

ankitpatel96 commented Sep 18, 2024 •

edited

Loading

jsuereth commented Sep 19, 2024 •

edited

Loading