BP: WASM: Load available WASM processors #1322

hariso · 2023-12-25T19:03:28Z

Description

Adds a standalone processor registry which can load WASM processors from a directory. It also contains a partial implementation of a WASM processor (only the Specification() and Teardown() methods, as that's needed for the registry).

Related to #1158.

Depends on:

Quick checks:

I have followed the Code Guidelines.
There is no other pull request for the same update/change.
I have written unit tests.
I have made sure that the PR is of reasonable size and can be easily reviewed.

go.mod

pkg/plugin/processor/standalone/registry_test.go

pkg/plugin/processor/standalone/test/wasm_processors/simple_processor/processor.wasm

lovromazgon

I love where this is going, thanks for this @hariso! 👍

pkg/plugin/processor/standalone/test/build-test-processors.sh

.gitignore

pkg/plugin/processor/standalone/registry_test.go

pkg/plugin/processor/standalone/wasm_processor_test.go

pkg/plugin/processor/standalone/wasm_processor.go

hariso

The new changes look great to me!

pkg/plugin/processor/standalone/processor.go

pkg/plugin/processor/standalone/standalone_test.go

Co-authored-by: Maha Hajja <[email protected]>

Co-authored-by: Maha Hajja <[email protected]> --------- Co-authored-by: Lovro Mažgon <[email protected]> Co-authored-by: Maha Hajja <[email protected]>

* Refactor pkg/plugin and isolate connector plugin (#1302) * refactor pkg/plugin and isolate connector plugin * fix imports * fix tests * fix import * change registry to connectorRegistry --------- Co-authored-by: Haris Osmanagić <[email protected]> * BP: WASM: Load available WASM processors (#1322) Co-authored-by: Maha Hajja <[email protected]> --------- Co-authored-by: Lovro Mažgon <[email protected]> Co-authored-by: Maha Hajja <[email protected]> * Refactor pkg/plugin and isolate connector plugin (#1302) * refactor pkg/plugin and isolate connector plugin * fix imports * fix tests * fix import * change registry to connectorRegistry --------- Co-authored-by: Haris Osmanagić <[email protected]> * BP: WASM: Load available WASM processors (#1322) Co-authored-by: Maha Hajja <[email protected]> --------- Co-authored-by: Lovro Mažgon <[email protected]> Co-authored-by: Maha Hajja <[email protected]> * Refactor processor service, processor and node (#1354) * linter * Run processor examples, gather specifications (#1384) * update processor-sdk, use config parameter from conduit-commons * add unified diff lib * add utilities for running processor examples and collecting specifications * update diff readme * fix markdown linter error * fix liter errors * update processor-sdk * fix race condition in test * BP: Display processor example raw data as string (#1397) * display raw data in processor examples as string * ensure examples are formatted the same way in processors.json * BP: add builtin processor part#1 (#1371) * add processors field.set & field.subset.exclude * delete extra test * fix tests * add field.rename processor * add exclusion validation to field.rename processor * refactor + fix error return in Process method * add field.convert processor * add filter processor * only structured data * address reviews part1 * use paramgen, update Configure method, update tests * evaluate value for processor setField * use a slice of referenceResolvers for processors with multiple fields params * do not allow .Position to be set in setField processor * update setField test, add non existent field test * use the new Rename() method from referenceResolver * address reviews * address reviews2 * fix renameField bug + add New() method to each processor * linter fix * add processor examples * processor json * Update pkg/plugin/processor/builtin/convertField.go Co-authored-by: Lovro Mažgon <[email protected]> * address reviews * setFeild example * setFeild example * make generate * fix paramgen regex * generate * fix regex * add paramgen to tools --------- Co-authored-by: Lovro Mažgon <[email protected]> * BP: Add unwrap.debezium (#1393) * BP: Add unwrap.kafkaconnect (#1395) * BP: Add webhook.http (#1382) * BP: Add unwrap.opencdc (#1387) * BP: Add encode.avro (#1401) * updated docs, regenerate * config * first pass * downgrade lib * processing * example * fix config parsing * regenerate * fix config * regenerate, tests * linter, move to opencdc * add test certs * config refactor, tests * more tests * linter * Regenerate * fix linter * git diff * git diff * regenerate * avro examples * run schema registry in examples on port 54321 * fix linter error * rename processor * fix schema registry integration test * don't run avro examples in integration tests --------- Co-authored-by: Lovro Mažgon <[email protected]> * BP: Add custom.javascript (#1374) * add json.decode processor (#1405) * add json.decode processor * make generate * processors * address reviews * make generate * add support for decoding JSON types other than map * make generate * address review comments --------- Co-authored-by: Lovro Mažgon <[email protected]> * BP: List processor plugins in API (#1406) * update proto API definitions * fix proto linter warnings * deprecate ListPlugins * implement plugin list endpoint * processor plugin orchestrator * move plugin endpoints under connector and processor service * BP: add avro.decode (#1410) Co-authored-by: Lovro Mažgon <[email protected]> * avro.decode example, remove base64 decoding --------- Co-authored-by: Lovro Mažgon <[email protected]> * BP: Base64 decode / encode processors (#1412) * encode * base64 decode * organize imports * add encodeJson processor (#1413) * encode json processor * delete forgotten test * address reviews * linter fix * add processor examples * don't encode nil, output string --------- Co-authored-by: Lovro Mažgon <[email protected]> * Builtin processors refactoring, housekeeping (#1402) * refactor builtin processors, generate specs into separate files * move processors into folder impl * go mod tidy * add example summaries * add example summaries * fix name of avro.decode processor * add example summaries * regenerate specs * update default registry * update default registry: fix tests, linter, regenerate * dep upgrades * refactorings * move json encode processor to impl/json * rename field.subset.exclude to field.exclude * rename constructor for unwrap.debezium processor * regenerate specs * dep downgrades --------- Co-authored-by: Haris Osmanagic <[email protected]> * fix linter * BP: call the conditional execution methods in the processors (#1411) * init * simplify * fix evaluation * add support for batches in runnable processor * batch test in runnable processor * optimize case when all records are passed through in a processor * make sure remaining records are copied over * fix len check * make sure processor doesn't return more records --------- Co-authored-by: Lovro Mažgon <[email protected]> * update examples and specs * disable goconst in processors (docs are repetitive) * upgrade processor sdk * add log when loading wasm processors, more docs updates --------- Co-authored-by: Haris Osmanagić <[email protected]> Co-authored-by: Maha Hajja <[email protected]>

hariso changed the base branch from main to lovro/processor-plugin December 25, 2023 19:03

hariso changed the title ~~WASM Processors~~ [WIP] WASM Processors Dec 25, 2023

hariso changed the title ~~[WIP] WASM Processors~~ [WIP] BP/WASM: Load available WASM processors Dec 27, 2023

hariso mentioned this pull request Dec 27, 2023

BP: Pluggable processors (WASM) #1158

Closed

6 tasks

hariso commented Dec 27, 2023

View reviewed changes

go.mod Outdated Show resolved Hide resolved

hariso commented Dec 27, 2023

View reviewed changes

pkg/plugin/processor/standalone/registry_test.go Outdated Show resolved Hide resolved

hariso changed the title ~~[WIP] BP/WASM: Load available WASM processors~~ BP: WASM: Load available WASM processors Dec 27, 2023

hariso commented Dec 28, 2023

View reviewed changes

pkg/plugin/processor/standalone/test/wasm_processors/simple_processor/processor.wasm Outdated Show resolved Hide resolved

hariso marked this pull request as ready for review December 28, 2023 15:00

hariso requested a review from a team as a code owner December 28, 2023 15:00

hariso force-pushed the haris/wasm-registry branch from 6e1c360 to 4f1b058 Compare December 28, 2023 19:50

hariso mentioned this pull request Dec 29, 2023

BP: WASM: Implement Configure, Open, Process #1327

Closed

4 tasks

lovromazgon reviewed Jan 4, 2024

View reviewed changes

lovromazgon and others added 11 commits January 17, 2024 15:45

implement processor plugin and builtin registry

738fd9d

WASM: Load available WASM processors

3967a57

update processor sdk

717358a

simplify error handling

badd4be

linter

35e6235

dummy change

492f53f

generate wasm on run

915bc2a

pr feedback

0049c04

update deps

df9da8d

update processor sdk

62f9bcb

refactoring, logging, utilities, and much more

941f6d0

lovromazgon force-pushed the haris/wasm-registry branch from 843df71 to 941f6d0 Compare January 17, 2024 14:55

lovromazgon changed the base branch from lovro/processor-plugin to feature/better-processors January 17, 2024 14:55

lovromazgon mentioned this pull request Jan 17, 2024

[WIP] Processor plugin #1308

Closed

4 tasks

lovromazgon added 3 commits January 18, 2024 15:38

standalone processor registry

5f4b861

cleanup tests

fe37e42

fix linter

2931f24

lovromazgon added 4 commits January 18, 2024 16:41

make processor mock generate command cross-platform compatible

d1c2556

add method to create standalone processor

3bfbb46

go mod tidy

4bd74b4

tests

a6769d4

hariso commented Jan 25, 2024

View reviewed changes

maha-hajja approved these changes Jan 25, 2024

View reviewed changes

pkg/plugin/processor/standalone/processor.go Show resolved Hide resolved

pkg/plugin/processor/standalone/standalone_test.go Outdated Show resolved Hide resolved

Update pkg/plugin/processor/standalone/standalone_test.go

d000fbc

Co-authored-by: Maha Hajja <[email protected]>

hariso merged commit 7b3087e into feature/better-processors Jan 26, 2024
2 of 3 checks passed

hariso deleted the haris/wasm-registry branch January 26, 2024 09:54

hariso added a commit that referenced this pull request Jan 30, 2024

BP: WASM: Load available WASM processors (#1322)

6807027

Co-authored-by: Maha Hajja <[email protected]> --------- Co-authored-by: Lovro Mažgon <[email protected]> Co-authored-by: Maha Hajja <[email protected]>

lovromazgon mentioned this pull request Feb 12, 2024

[Feature] Better processors #1378

Merged

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

BP: WASM: Load available WASM processors #1322

BP: WASM: Load available WASM processors #1322

hariso commented Dec 25, 2023 •

edited

Loading

lovromazgon left a comment

hariso left a comment

BP: WASM: Load available WASM processors #1322

BP: WASM: Load available WASM processors #1322

Conversation

hariso commented Dec 25, 2023 • edited Loading

Description

Quick checks:

lovromazgon left a comment

Choose a reason for hiding this comment

hariso left a comment

Choose a reason for hiding this comment

hariso commented Dec 25, 2023 •

edited

Loading