BP: Add encode.avro #1401

hariso · 2024-03-01T19:14:59Z

Description

The schemaregistry and internal packages have been taken from the legacy processor. The only change made to those is the replacement of the record package with the opencdc package (from conduit-commons).

The PR also updates some of the generated code for other processors' configurations (due to an update in paramgen).

Part of #1290

Reference from the design doc: https://github.com/ConduitIO/conduit/blob/main/docs/design-documents/20231117-better-processors.md#encodeavro

As per a conversation in a review of this PR, the processor will be renamed to avro.encode.

Quick checks:

I have followed the Code Guidelines.
There is no other pull request for the same update/change.
I have written unit tests.
I have made sure that the PR is of reasonable size and can be easily reviewed.

…-encode-avro

pkg/plugin/processor/builtin/avro/encode.go

…-encode-avro

hariso · 2024-03-05T16:34:53Z

pkg/plugin/processor/builtin/avro_examples_test.go

@@ -28,7 +28,7 @@ import (

 //nolint:govet // a more descriptive example description
 func ExampleEncodeProcessor_autoRegister() {
-	url, cleanup := schemaregistry.ExampleSchemaRegistryURL("ExampleEncodeProcessor_autoRegister")
+	url, cleanup := schemaregistry.ExampleSchemaRegistryURL("ExampleEncodeProcessor_autoRegister", 54321)


Are example tests ever run in parallel (asking because the same port is used for the HTTP example)?

I don't think they are, even if we used the -parallel flag it would only run tests that call t.Parallel(), but since there's no such function in examples I'd expect that they are always run sequentially.

…uitIO/conduit into haris/better-processors-encode-avro

lovromazgon

LGTM!

* Refactor pkg/plugin and isolate connector plugin (#1302) * refactor pkg/plugin and isolate connector plugin * fix imports * fix tests * fix import * change registry to connectorRegistry --------- Co-authored-by: Haris Osmanagić <[email protected]> * BP: WASM: Load available WASM processors (#1322) Co-authored-by: Maha Hajja <[email protected]> --------- Co-authored-by: Lovro Mažgon <[email protected]> Co-authored-by: Maha Hajja <[email protected]> * Refactor pkg/plugin and isolate connector plugin (#1302) * refactor pkg/plugin and isolate connector plugin * fix imports * fix tests * fix import * change registry to connectorRegistry --------- Co-authored-by: Haris Osmanagić <[email protected]> * BP: WASM: Load available WASM processors (#1322) Co-authored-by: Maha Hajja <[email protected]> --------- Co-authored-by: Lovro Mažgon <[email protected]> Co-authored-by: Maha Hajja <[email protected]> * Refactor processor service, processor and node (#1354) * linter * Run processor examples, gather specifications (#1384) * update processor-sdk, use config parameter from conduit-commons * add unified diff lib * add utilities for running processor examples and collecting specifications * update diff readme * fix markdown linter error * fix liter errors * update processor-sdk * fix race condition in test * BP: Display processor example raw data as string (#1397) * display raw data in processor examples as string * ensure examples are formatted the same way in processors.json * BP: add builtin processor part#1 (#1371) * add processors field.set & field.subset.exclude * delete extra test * fix tests * add field.rename processor * add exclusion validation to field.rename processor * refactor + fix error return in Process method * add field.convert processor * add filter processor * only structured data * address reviews part1 * use paramgen, update Configure method, update tests * evaluate value for processor setField * use a slice of referenceResolvers for processors with multiple fields params * do not allow .Position to be set in setField processor * update setField test, add non existent field test * use the new Rename() method from referenceResolver * address reviews * address reviews2 * fix renameField bug + add New() method to each processor * linter fix * add processor examples * processor json * Update pkg/plugin/processor/builtin/convertField.go Co-authored-by: Lovro Mažgon <[email protected]> * address reviews * setFeild example * setFeild example * make generate * fix paramgen regex * generate * fix regex * add paramgen to tools --------- Co-authored-by: Lovro Mažgon <[email protected]> * BP: Add unwrap.debezium (#1393) * BP: Add unwrap.kafkaconnect (#1395) * BP: Add webhook.http (#1382) * BP: Add unwrap.opencdc (#1387) * BP: Add encode.avro (#1401) * updated docs, regenerate * config * first pass * downgrade lib * processing * example * fix config parsing * regenerate * fix config * regenerate, tests * linter, move to opencdc * add test certs * config refactor, tests * more tests * linter * Regenerate * fix linter * git diff * git diff * regenerate * avro examples * run schema registry in examples on port 54321 * fix linter error * rename processor * fix schema registry integration test * don't run avro examples in integration tests --------- Co-authored-by: Lovro Mažgon <[email protected]> * BP: Add custom.javascript (#1374) * add json.decode processor (#1405) * add json.decode processor * make generate * processors * address reviews * make generate * add support for decoding JSON types other than map * make generate * address review comments --------- Co-authored-by: Lovro Mažgon <[email protected]> * BP: List processor plugins in API (#1406) * update proto API definitions * fix proto linter warnings * deprecate ListPlugins * implement plugin list endpoint * processor plugin orchestrator * move plugin endpoints under connector and processor service * BP: add avro.decode (#1410) Co-authored-by: Lovro Mažgon <[email protected]> * avro.decode example, remove base64 decoding --------- Co-authored-by: Lovro Mažgon <[email protected]> * BP: Base64 decode / encode processors (#1412) * encode * base64 decode * organize imports * add encodeJson processor (#1413) * encode json processor * delete forgotten test * address reviews * linter fix * add processor examples * don't encode nil, output string --------- Co-authored-by: Lovro Mažgon <[email protected]> * Builtin processors refactoring, housekeeping (#1402) * refactor builtin processors, generate specs into separate files * move processors into folder impl * go mod tidy * add example summaries * add example summaries * fix name of avro.decode processor * add example summaries * regenerate specs * update default registry * update default registry: fix tests, linter, regenerate * dep upgrades * refactorings * move json encode processor to impl/json * rename field.subset.exclude to field.exclude * rename constructor for unwrap.debezium processor * regenerate specs * dep downgrades --------- Co-authored-by: Haris Osmanagic <[email protected]> * fix linter * BP: call the conditional execution methods in the processors (#1411) * init * simplify * fix evaluation * add support for batches in runnable processor * batch test in runnable processor * optimize case when all records are passed through in a processor * make sure remaining records are copied over * fix len check * make sure processor doesn't return more records --------- Co-authored-by: Lovro Mažgon <[email protected]> * update examples and specs * disable goconst in processors (docs are repetitive) * upgrade processor sdk * add log when loading wasm processors, more docs updates --------- Co-authored-by: Haris Osmanagić <[email protected]> Co-authored-by: Maha Hajja <[email protected]>

hariso added 17 commits March 1, 2024 10:40

updated docs, regenerate

40825c6

config

143ab9d

first pass

2420f1e

downgrade lib

0064ebf

processing

ce6d71a

example

79f96df

fix config parsing

3928e97

regenerate

c4c4b37

fix config

b14c01f

regenerate, tests

bfe2558

linter, move to opencdc

ba7aefe

add test certs

b75f4df

config refactor, tests

710edd9

more tests

d5ed11c

linter

845226a

Merge branch 'feature/better-processors' into haris/better-processors…

594b4be

…-encode-avro

Regenerate

650e4b9

hariso marked this pull request as ready for review March 5, 2024 12:43

hariso requested a review from a team as a code owner March 5, 2024 12:43

hariso added 4 commits March 5, 2024 13:48

fix linter

ec96561

git diff

bc66071

git diff

4bc5eb7

regenerate

84e47b0

lovromazgon reviewed Mar 5, 2024

View reviewed changes

pkg/plugin/processor/builtin/avro/encode.go Outdated Show resolved Hide resolved

lovromazgon added 4 commits March 5, 2024 16:43

avro examples

0a20829

Merge branch 'feature/better-processors' into haris/better-processors…

3b4b92f

…-encode-avro

run schema registry in examples on port 54321

13846d0

fix linter error

d9163fc

hariso commented Mar 5, 2024

View reviewed changes

rename processor

c11674f

hariso and others added 2 commits March 5, 2024 17:38

Merge branch 'haris/better-processors-encode-avro' of github.com:Cond…

9c82f05

…uitIO/conduit into haris/better-processors-encode-avro

fix schema registry integration test

ec4434c

lovromazgon approved these changes Mar 5, 2024

View reviewed changes

don't run avro examples in integration tests

3e43dcf

lovromazgon merged commit c8eeaac into feature/better-processors Mar 5, 2024
3 checks passed

lovromazgon deleted the haris/better-processors-encode-avro branch March 5, 2024 18:03

This was referenced Mar 6, 2024

BP: Rewrite built-in processors #1290

Closed

[Feature] Better processors #1378

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

BP: Add encode.avro #1401

BP: Add encode.avro #1401

hariso commented Mar 1, 2024 •

edited

Loading

hariso Mar 5, 2024

lovromazgon Mar 5, 2024

lovromazgon left a comment

BP: Add encode.avro #1401

BP: Add encode.avro #1401

Conversation

hariso commented Mar 1, 2024 • edited Loading

Description

Quick checks:

hariso Mar 5, 2024

Choose a reason for hiding this comment

lovromazgon Mar 5, 2024

Choose a reason for hiding this comment

lovromazgon left a comment

Choose a reason for hiding this comment

hariso commented Mar 1, 2024 •

edited

Loading