KAFKA-7632: Support Compression Levels (KIP-390) #15516

mimaison · 2024-03-11T18:40:17Z

Based on #10826 with updates to match the recent amends we made to KIP-390.

Committer Checklist (excluded from commit message)

Verify design and implementation
Verify test coverage and CI build status
Verify documentation (including upgrade notes)

mimaison · 2024-04-10T17:43:31Z

@divijvaidya It seems you've done a bit of work around compression in the past. Can you take a look? Thanks

showuon

Thanks for the PR. Left some comments.

clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java

clients/src/main/java/org/apache/kafka/common/compress/Compression.java

clients/src/main/java/org/apache/kafka/common/compress/Lz4BlockOutputStream.java

clients/src/test/java/org/apache/kafka/common/compress/GzipCompressionTest.java

clients/src/test/java/org/apache/kafka/common/compress/Lz4CompressionTest.java

clients/src/test/java/org/apache/kafka/common/record/DefaultRecordBatchTest.java

showuon · 2024-04-25T12:31:44Z

storage/src/main/java/org/apache/kafka/storage/internals/log/LogValidator.java

            throw new UnsupportedCompressionTypeException("Produce requests to inter.broker.protocol.version < 2.1 broker " +
                "are not allowed to use ZStandard compression");

        // No in place assignment situation 1
-        boolean inPlaceAssignment = sourceCompression == targetCompression;
+        boolean inPlaceAssignment = sourceCompressionType == targetCompression.type();


So we won't do re-compression if only level is different? I didn't see this in KIP. Maybe we should add it?

The broker has no easy way of retrieving the level that the producer used when compressing the records. So if the compression codec matches, I decided to keep the compressed bytes instead of decompressing and compressing everything again as this would be wasteful, especially as the producer could have already used the same compression level.

I agree. But I think we should mention this in the KIP-390 at least.

Right, I updated the KIP.

showuon · 2024-04-25T12:33:33Z

clients/src/main/java/org/apache/kafka/common/config/TopicConfig.java

+    public static final String COMPRESSION_GZIP_LEVEL_DOC = "The compression level to use if " + COMPRESSION_TYPE_CONFIG + " is set to <code>gzip</code>.";
+    public static final String COMPRESSION_LZ4_LEVEL_CONFIG = "compression.lz4.level";
+    public static final String COMPRESSION_LZ4_LEVEL_DOC = "The compression level to use if " + COMPRESSION_TYPE_CONFIG + " is set to <code>lz4</code>.";
+    public static final String COMPRESSION_ZSTD_LEVEL_CONFIG = "compression.zstd.level";
+    public static final String COMPRESSION_ZSTD_LEVEL_DOC = "The compression level to use if " + COMPRESSION_TYPE_CONFIG + " is set to <code>zstd</code>.";


nit: Should we provide the doc link for each compression type? It's hard to know which level means what.

Do you mean a link to the compression library websites?

I was thinking we added in the config description. Or maybe added in KIP-390 is good enough.

junrao

@mimaison : Thanks for the PR. Made a pass of non-testing files. Left a few comments.

clients/src/main/java/org/apache/kafka/common/compress/GzipCompression.java

clients/src/main/java/org/apache/kafka/common/compress/Compression.java

clients/src/main/java/org/apache/kafka/common/compress/NoCompression.java

junrao · 2024-05-01T23:45:06Z

clients/src/main/java/org/apache/kafka/common/compress/Lz4Compression.java

+
+    public static final int MIN_LEVEL = 1;
+    public static final int MAX_LEVEL = 17;
+    public static final int DEFAULT_LEVEL = 9;


So, every time we update the Lz4 library, we may need to update the above values? We probably want to add a note here.

I hesitated defining these constants for this reason but these levels have not changed over 10 years [0], so hopefully this won't require a lot of maintenance.

0: https://github.com/lz4/lz4-java/blame/master/src/java/net/jpountz/lz4/LZ4Constants.java#L23-L24

clients/src/main/java/org/apache/kafka/common/compress/ZstdCompression.java

clients/src/main/java/org/apache/kafka/common/telemetry/internals/ClientTelemetryUtils.java

storage/src/main/java/org/apache/kafka/storage/internals/log/LogConfig.java

junrao

@mimaison : Thanks for the updated PR. Made a pass of all files. Added a few more comments.

clients/src/test/java/org/apache/kafka/common/compress/GzipCompressionTest.java

clients/src/test/java/org/apache/kafka/common/record/MemoryRecordsBuilderTest.java

clients/src/test/java/org/apache/kafka/common/compress/NoCompressionTest.java

core/src/test/scala/unit/kafka/server/KafkaConfigTest.scala

core/src/test/scala/unit/kafka/log/LogConfigTest.scala

core/src/test/scala/unit/kafka/log/LogValidatorTest.scala

core/src/main/scala/kafka/log/UnifiedLog.scala

junrao

@mimaison : Thanks for the updated PR. Just one more comment.

server-common/src/test/java/org/apache/kafka/server/record/BrokerCompressionTypeTest.java

mimaison · 2024-05-16T13:18:18Z

I also added a couple of new tests in LogValidatorTest to check recompression only happens if the compression codec is different between the records from the producer and the topic configuration and does not happen if only the compression levels are different.

junrao

@mimaison: Thanks for the updated PR. Left one more comment.

core/src/test/scala/unit/kafka/log/LogValidatorTest.scala

junrao

@mimaison : Thanks for the updated PR. LGTM assuming all the failed tests have been triaged.

showuon

Had another look. LGTM! Just some comments to update the KIP. Thanks.

Co-authored-by: Lee Dongjin <[email protected]>

mimaison · 2024-05-21T10:55:48Z

Thanks for the reviews!
I had to rebase again so I'll wait for the CI to complete.

mimaison · 2024-05-21T15:50:48Z

None of the test failures seem related, merging to trunk

Reviewers: Jun Rao <[email protected]>, Luke Chen <[email protected]> Co-authored-by: Lee Dongjin <[email protected]>

stanislavkozlovski · 2024-07-25T12:12:04Z

Has anybody noticed that the Linear Write test in KIP-390 is inaccurate?

It suggests that the write speed on a broker is 22GB/s. I wasn't able to find an SSD on the market in 2024 (6 years later) that supports this throughput
The reason for this, I think, is that it inaccurately sets the benchmark test to write just 8192 bytes --bytes 8192 and the size to write is 8192 itself, so the test seems to just be comparing the nanoseconds it takes for the very first write and then extrapolating further to a full second.

https://cwiki.apache.org/confluence/display/KAFKA/KIP-390%3A+Support+Compression+Level

mimaison · 2024-07-26T15:00:05Z

Yeah it looks like the numbers are not accurate. To be honest it's a bit of a strange performance test. The Produce Test benchmark should be much more representative. However I did not reproduce that benchmark. It might be worth asking @dongjinleekr if he has any data points to share.

mimaison force-pushed the kip-390 branch 3 times, most recently from df9ca6e to 2f54aac Compare March 25, 2024 08:47

mimaison force-pushed the kip-390 branch 3 times, most recently from d284cd3 to 71d84bd Compare April 10, 2024 09:04

mimaison marked this pull request as ready for review April 10, 2024 10:00

mimaison force-pushed the kip-390 branch from 71d84bd to 6619e40 Compare April 10, 2024 15:09

mimaison mentioned this pull request Apr 10, 2024

KAFKA-7632: Support Compression Level #10826

Closed

3 tasks

showuon self-assigned this Apr 11, 2024

mimaison force-pushed the kip-390 branch from 6619e40 to dd728c8 Compare April 11, 2024 08:55

showuon reviewed Apr 25, 2024

View reviewed changes

junrao reviewed May 2, 2024

View reviewed changes

mimaison force-pushed the kip-390 branch from dd728c8 to 1282e3b Compare May 14, 2024 15:25

junrao reviewed May 14, 2024

View reviewed changes

junrao reviewed May 15, 2024

View reviewed changes

server-common/src/test/java/org/apache/kafka/server/record/BrokerCompressionTypeTest.java Outdated Show resolved Hide resolved

mimaison force-pushed the kip-390 branch from bc27574 to ea6b179 Compare May 16, 2024 13:12

junrao reviewed May 17, 2024

View reviewed changes

core/src/test/scala/unit/kafka/log/LogValidatorTest.scala Outdated Show resolved Hide resolved

mimaison force-pushed the kip-390 branch from e83e893 to 2158b1a Compare May 17, 2024 18:42

junrao approved these changes May 20, 2024

View reviewed changes

showuon approved these changes May 21, 2024

View reviewed changes

mimaison and others added 6 commits May 21, 2024 12:47

KAFKA-7632: Support Compression Levels (KIP-390)

7d94377

Co-authored-by: Lee Dongjin <[email protected]>

Address feedback

658b11b

Address 2nd batch of reviews

6843d9f

Address 3rd batch of reviews

9f0be37

Add LogValidation tests

35bc027

Use MAGIC_VALUE_V2 in LogValidatorTest

9b4f539

mimaison force-pushed the kip-390 branch from 2158b1a to 9b4f539 Compare May 21, 2024 10:54

mimaison merged commit affe8da into apache:trunk May 21, 2024
1 check failed

mimaison deleted the kip-390 branch May 21, 2024 15:58

rreddy-22 pushed a commit to rreddy-22/kafka-rreddy that referenced this pull request May 24, 2024

KAFKA-7632: Support Compression Levels (KIP-390) (apache#15516)

599afdf

Reviewers: Jun Rao <[email protected]>, Luke Chen <[email protected]> Co-authored-by: Lee Dongjin <[email protected]>

chiacyu pushed a commit to chiacyu/kafka that referenced this pull request Jun 1, 2024

KAFKA-7632: Support Compression Levels (KIP-390) (apache#15516)

11cb198

Reviewers: Jun Rao <[email protected]>, Luke Chen <[email protected]> Co-authored-by: Lee Dongjin <[email protected]>

TaiJuWu pushed a commit to TaiJuWu/kafka that referenced this pull request Jun 8, 2024

KAFKA-7632: Support Compression Levels (KIP-390) (apache#15516)

7c12882

Reviewers: Jun Rao <[email protected]>, Luke Chen <[email protected]> Co-authored-by: Lee Dongjin <[email protected]>

gongxuanzhang pushed a commit to gongxuanzhang/kafka that referenced this pull request Jun 12, 2024

KAFKA-7632: Support Compression Levels (KIP-390) (apache#15516)

64c6d77

Reviewers: Jun Rao <[email protected]>, Luke Chen <[email protected]> Co-authored-by: Lee Dongjin <[email protected]>

dongjoon-hyun mentioned this pull request Jul 31, 2024

[SPARK-49064][BUILD] Upgrade Kafka to 3.8.0 apache/spark#47540

Closed

JoergSiebahn mentioned this pull request Jul 31, 2024

Fix failure that occurs in distroless image with read only file system SDA-SE/sda-dropwizard-commons#3575

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

KAFKA-7632: Support Compression Levels (KIP-390) #15516

KAFKA-7632: Support Compression Levels (KIP-390) #15516

mimaison commented Mar 11, 2024 •

edited

Loading

mimaison commented Apr 10, 2024

showuon left a comment

showuon Apr 25, 2024

mimaison May 14, 2024

showuon May 21, 2024

mimaison May 21, 2024

showuon Apr 25, 2024

mimaison May 14, 2024

showuon May 21, 2024

junrao left a comment

junrao May 1, 2024

mimaison May 14, 2024

junrao left a comment

junrao left a comment

mimaison commented May 16, 2024

junrao left a comment

junrao left a comment

showuon left a comment

mimaison commented May 21, 2024

mimaison commented May 21, 2024

stanislavkozlovski commented Jul 25, 2024

mimaison commented Jul 26, 2024

KAFKA-7632: Support Compression Levels (KIP-390) #15516

KAFKA-7632: Support Compression Levels (KIP-390) #15516

Conversation

mimaison commented Mar 11, 2024 • edited Loading

Committer Checklist (excluded from commit message)

mimaison commented Apr 10, 2024

showuon left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

junrao left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

junrao left a comment

Choose a reason for hiding this comment

junrao left a comment

Choose a reason for hiding this comment

mimaison commented May 16, 2024

junrao left a comment

Choose a reason for hiding this comment

junrao left a comment

Choose a reason for hiding this comment

showuon left a comment

Choose a reason for hiding this comment

mimaison commented May 21, 2024

mimaison commented May 21, 2024

stanislavkozlovski commented Jul 25, 2024

mimaison commented Jul 26, 2024

mimaison commented Mar 11, 2024 •

edited

Loading