feature: Add new encoding runlength v3 using Longs instead of Integers #408

astubbs · 2022-09-07T17:49:10Z

From the javadoc:

Uses {@link Long} for it's underlying structure (in order to support output encoding option using Longs), in order to support the worst case scenarios without running out of run length, as {@link Integer#MAX_VALUE} of 2,147,483,647 (2 billion records) is not completely inconceivable, whereas {@link Long#MAX_VALUE} of 9,223,372,036,854,775,807 (~9 quintillion / 9 x 10^18) is.

Checklist

Documentation (if applicable)
Changelog

Blocked by:

fix: Improvements to encoding ranges (int vs long) #439

Bumps `testcontainers.version` from 1.17.1 to 1.17.2. Updates `testcontainers` from 1.17.1 to 1.17.2 - [Release notes](https://github.com/testcontainers/testcontainers-java/releases) - [Changelog](https://github.com/testcontainers/testcontainers-java/blob/master/CHANGELOG.md) - [Commits](testcontainers/testcontainers-java@1.17.1...1.17.2) Updates `kafka` from 1.17.1 to 1.17.2 - [Release notes](https://github.com/testcontainers/testcontainers-java/releases) - [Changelog](https://github.com/testcontainers/testcontainers-java/blob/master/CHANGELOG.md) - [Commits](testcontainers/testcontainers-java@1.17.1...1.17.2) Updates `junit-jupiter` from 1.17.1 to 1.17.2 - [Release notes](https://github.com/testcontainers/testcontainers-java/releases) - [Changelog](https://github.com/testcontainers/testcontainers-java/blob/master/CHANGELOG.md) - [Commits](testcontainers/testcontainers-java@1.17.1...1.17.2) Updates `postgresql` from 1.17.1 to 1.17.2 - [Release notes](https://github.com/testcontainers/testcontainers-java/releases) - [Changelog](https://github.com/testcontainers/testcontainers-java/blob/master/CHANGELOG.md) - [Commits](testcontainers/testcontainers-java@1.17.1...1.17.2) --- updated-dependencies: - dependency-name: org.testcontainers:testcontainers dependency-type: direct:production update-type: version-update:semver-patch - dependency-name: org.testcontainers:kafka dependency-type: direct:production update-type: version-update:semver-patch - dependency-name: org.testcontainers:junit-jupiter dependency-type: direct:production update-type: version-update:semver-patch - dependency-name: org.testcontainers:postgresql dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]>

Bumps [versions-maven-plugin](https://github.com/mojohaus/versions-maven-plugin) from 2.10.0 to 2.11.0. - [Release notes](https://github.com/mojohaus/versions-maven-plugin/releases) - [Changelog](https://github.com/mojohaus/versions-maven-plugin/blob/master/ReleaseNotes.md) - [Commits](mojohaus/versions@versions-maven-plugin-2.10.0...versions-maven-plugin-2.11.0) --- updated-dependencies: - dependency-name: org.codehaus.mojo:versions-maven-plugin dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]>

… topics

https://ossindex.sonatype.org/component/pkg:maven/com.google.guava/[email protected]?utm_source=ossindex-client&utm_medium=integration&utm_content=1.8.1

Bumps `vertx.version` from 4.3.0 to 4.3.1. Updates `vertx-web-client` from 4.3.0 to 4.3.1 Updates `vertx-junit5` from 4.3.0 to 4.3.1 --- updated-dependencies: - dependency-name: io.vertx:vertx-web-client dependency-type: direct:production update-type: version-update:semver-patch - dependency-name: io.vertx:vertx-junit5 dependency-type: direct:development update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]>

Bumps [postgresql](https://github.com/pgjdbc/pgjdbc) from 42.3.5 to 42.4.0. - [Release notes](https://github.com/pgjdbc/pgjdbc/releases) - [Changelog](https://github.com/pgjdbc/pgjdbc/blob/master/CHANGELOG.md) - [Commits](pgjdbc/pgjdbc@REL42.3.5...REL42.4.0) --- updated-dependencies: - dependency-name: org.postgresql:postgresql dependency-type: direct:development update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]>

- CVE-2020-8908: Files::createTempDir local information disclosure vulnerability #4011 - Only used transitively from tests, and is a deprecated function - google/guava#4011

# Conflicts: # pom.xml

…rs.version-1.17.2' into improvements/version-bumps

…ql-postgresql-42.4.0' into improvements/version-bumps

….mojo-versions-maven-plugin-2.11.0' into improvements/version-bumps

New issues with WireMocks dep on Jetty BOM ~v9

…that Especially given we also use dependabot. If anyone is concerned to that level, they can use the plugin in their end user applications.

…iremock fat dependency (slf4j/logback collisions) Upgrades Wiremock from 2.33.2 to 2.34.0

…o-long

# Conflicts: # parallel-consumer-core/src/main/java/io/confluent/csid/utils/MathUtils.java # parallel-consumer-core/src/main/java/io/confluent/csid/utils/Range.java # parallel-consumer-core/src/main/java/io/confluent/parallelconsumer/offsets/OffsetSimultaneousEncoder.java # parallel-consumer-core/src/main/java/io/confluent/parallelconsumer/offsets/RunLengthEncoder.java

# Conflicts: # parallel-consumer-core/src/main/java/io/confluent/csid/utils/Range.java

…s/long-encoding

…atures/long-encoding

astubbs

....

rkolesnev · 2022-10-21T13:29:29Z

parallel-consumer-core/src/main/java/io/confluent/parallelconsumer/offsets/OffsetRunLength.java

+/**
+ * @author Antony Stubbs
+ */
+// todo rename - resolve name against RunLengthEncoder


RunLengthEncodingUtils or maybe consolidate into the RunLenghtEncoder if it really is just internal implementation methods?

rkolesnev · 2022-10-21T13:32:12Z

parallel-consumer-core/src/main/java/io/confluent/parallelconsumer/offsets/OffsetRunLength.java

@@ -112,11 +113,12 @@ static HighestOffsetAndIncompletes runLengthDecodeToIncompletes(OffsetEncoding e
        // decodes incompletes


Lines 109, 110 above:
Any reason only unrolling v1 and v2 buffers but not the new v3 one?

v1ShortBuffer.rewind(); v2IntegerBuffer.rewind();

...umer-core/src/main/java/io/confluent/parallelconsumer/offsets/OffsetSimultaneousEncoder.java

rkolesnev · 2022-10-21T13:46:32Z

...llel-consumer-core/src/main/java/io/confluent/parallelconsumer/offsets/RunLengthEncoder.java

                    if (runLength != shortCastRunlength)
                        throw new RunlengthV1EncodingNotSupported(msg("Runlength too long for Short ({} cast to {})", runLength, shortCastRunlength));
                    runLengthEncodedByteBuffer.putShort(shortCastRunlength);
                }
                case v2 -> {
-                    runLengthEncodedByteBuffer.putInt(runLength);
+                    runLengthEncodedByteBuffer.putInt(Math.toIntExact(runLength));


new exception type? RunlengthV2EncodingNotSupported or align both to use ArithmeticException - although i would prefer less generic exception... either Generic EncodingNotSupported with params for encoding type (i.e. bitset / runlength) and version (v1,v2 etc)
or as you have now - set of specific exceptions...

…s/long-encoding

# Conflicts: # parallel-consumer-core/src/main/java/io/confluent/csid/utils/Range.java # parallel-consumer-core/src/main/java/io/confluent/parallelconsumer/offsets/BitSetEncoder.java # parallel-consumer-core/src/main/java/io/confluent/parallelconsumer/offsets/ByteBufferEncoder.java # parallel-consumer-core/src/main/java/io/confluent/parallelconsumer/offsets/OffsetBitSet.java # parallel-consumer-core/src/main/java/io/confluent/parallelconsumer/offsets/OffsetEncoder.java # parallel-consumer-core/src/main/java/io/confluent/parallelconsumer/offsets/OffsetSimultaneousEncoder.java # parallel-consumer-core/src/main/java/io/confluent/parallelconsumer/offsets/RunLengthEncoder.java # parallel-consumer-core/src/test/java/io/confluent/parallelconsumer/ParallelEoSStreamProcessorTest.java # parallel-consumer-core/src/test/java/io/confluent/parallelconsumer/offsets/RunLengthEncoderTest.java

what-the-diff · 2022-11-21T11:28:34Z

Range.java
- Added a new constructor for the class that takes in long as parameter instead of int, and changed all other constructors to take in long parameters too.
BitSetEncoder.java
Change the underlying data structure from Integer to Long
Add a new version of RunLengthEncoder (v3) which uses longs for it's encoding, and add tests for this

eddyv · 2023-06-15T14:40:27Z

Closing - Stale.

dependabot bot and others added 30 commits May 20, 2022 16:33

START: Try to reproduce issue confluentinc#184 - run against multiple…

ec21ad0

… topics

test compiles

1b9d2ad

failling test

51ef075

fixes test - introduce ShardKey

7e32d4d

step

0a558f6

step

9628bee

step

b1fd89e

turn off ossindex for guava 31.1

d237158

https://ossindex.sonatype.org/component/pkg:maven/com.google.guava/[email protected]?utm_source=ossindex-client&utm_medium=integration&utm_content=1.8.1

step

d8ea983

step - extract

b883ba0

step - tg demo, keep consumer open

ae21fb7

test for null keys

ad6c89e

allow truth snapshot

23e5389

add missing annotations

7fdb06a

step

03ff349

build: Exclude Guava 31.1 from OSSIndex sec scan

72ac264

- CVE-2020-8908: Files::createTempDir local information disclosure vulnerability #4011 - Only used transitively from tests, and is a deprecated function - google/guava#4011

...

f905cdb

build: Update to Truth-Generator 0.1.1 for Subject discovery

12dc7cf

Merge branch 'exclude-guava-oss-index' into bugs/issue-184-reproduce

bf37740

# Conflicts: # pom.xml

Merge remote-tracking branch 'confluent/dependabot/maven/testcontaine…

080ecd3

…rs.version-1.17.2' into improvements/version-bumps

Merge remote-tracking branch 'confluent/dependabot/maven/org.postgres…

71a48dc

…ql-postgresql-42.4.0' into improvements/version-bumps

Merge remote-tracking branch 'confluent/dependabot/maven/org.codehaus…

a087efb

….mojo-versions-maven-plugin-2.11.0' into improvements/version-bumps

build: Only run OSS Index audit on runtime dependencies, not tests

3e468a7

New issues with WireMocks dep on Jetty BOM ~v9

build: Only run OSS Index audit on runtime dependencies, not tests

10ac83a

New issues with WireMocks dep on Jetty BOM ~v9

build: Don't fail for OSSIndex scan - will set up another CI job for …

7efc8de

…that Especially given we also use dependabot. If anyone is concerned to that level, they can use the plugin in their end user applications.

build: Don't fail for OSSIndex scan - will set up another CI job for …

70a37cd

…that Especially given we also use dependabot. If anyone is concerned to that level, they can use the plugin in their end user applications.

astubbs added 12 commits October 20, 2022 22:04

Merge branch 'master' into features/long-encoding

18119dc

Merge remote-tracking branch 'origin/master' into fixes/int-to-long

02edc44

fix bitset setup

d276f86

review, cohesion, concerns, clarity,

5546e2d

build(deps-dev): Fix Slf4j setup by switching away from and bumping W…

8f0bdbb

…iremock fat dependency (slf4j/logback collisions) Upgrades Wiremock from 2.33.2 to 2.34.0

Merge remote-tracking branch 'astubbs/fixes/logging' into fixes/int-t…

b5ef682

…o-long

remove unused constructor

e65b7b8

use exact casting for sanity

7eaee4f

Change encoding interface to long as well

3272332

review

e2a3d05

Merge branch 'fixes/int-to-long' into features/long-encoding

29dd963

# Conflicts: # parallel-consumer-core/src/main/java/io/confluent/csid/utils/Range.java

astubbs requested a review from rkolesnev October 21, 2022 13:17

astubbs added 7 commits October 21, 2022 14:25

review

af8f119

review

8dc65db

Merge branch 'master' into fixes/int-to-long

ab1ba7d

Merge branch 'master' into features/long-encoding

539c28e

Merge remote-tracking branch 'astubbs/fixes/int-to-long' into feature…

8413657

…s/long-encoding

review

be40b4c

Merge remote-tracking branch 'astubbs/features/long-encoding' into fe…

62b6b85

…atures/long-encoding

astubbs commented Oct 21, 2022

View reviewed changes

review

9c7e703

rkolesnev reviewed Oct 21, 2022

View reviewed changes

astubbs added 4 commits October 21, 2022 15:00

review

49ab1ef

review

bc6d592

Merge remote-tracking branch 'astubbs/fixes/int-to-long' into feature…

e76bbb2

…s/long-encoding

eddyv closed this Jun 15, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feature: Add new encoding runlength v3 using Longs instead of Integers #408

feature: Add new encoding runlength v3 using Longs instead of Integers #408

astubbs commented Sep 7, 2022 •

edited

Loading

astubbs left a comment

rkolesnev Oct 21, 2022

rkolesnev Oct 21, 2022

rkolesnev Oct 21, 2022

what-the-diff bot commented Nov 21, 2022

eddyv commented Jun 15, 2023

		@@ -112,11 +113,12 @@ static HighestOffsetAndIncompletes runLengthDecodeToIncompletes(OffsetEncoding e
		// decodes incompletes

feature: Add new encoding runlength v3 using Longs instead of Integers #408

feature: Add new encoding runlength v3 using Longs instead of Integers #408

Conversation

astubbs commented Sep 7, 2022 • edited Loading

Checklist

astubbs left a comment

Choose a reason for hiding this comment

rkolesnev Oct 21, 2022

Choose a reason for hiding this comment

rkolesnev Oct 21, 2022

Choose a reason for hiding this comment

rkolesnev Oct 21, 2022

Choose a reason for hiding this comment

what-the-diff bot commented Nov 21, 2022

eddyv commented Jun 15, 2023

astubbs commented Sep 7, 2022 •

edited

Loading