8346239: Improve memory efficiency of JimageDiffGenerator #22835

jerboaa · 2024-12-19T18:14:39Z

Please review this fairly simple change to improve how the JimageDiffGenerator works. The original implementation is pretty naive and read all bytes into memory and then compared them. This improved version only reads bytes on a bound buffer into memory compares those bytes and if equal continues on to reading the next bytes (2k at most) at a time. Previously it was 2*N (where N is the file size of a file in bytes) part of the JDK. Ouch.

There is still the off-chance of reading a full file into memory when the generator detects a change, but this shouldn't happen for large files since the total diff should be around 600K as of today.

This also reverts changes from JDK-8344036 other than the /timeout change to the test, which is preserved as I think this bump is no longer needed.

Testing:

GHA
jlink tests on a fastdebug build with --with-native-debug-symbols=internal (so as to have a large libjvm.so).
Manual reproducer from the bug which fails with OOM before the patch and passes after

Thoughts?

Progress

Change must be properly reviewed (1 review required, with at least 1 Reviewer)
Change must not contain extraneous whitespace
Commit message must refer to an issue

Issue

JDK-8346239: Improve memory efficiency of JimageDiffGenerator (Bug - P4)

Reviewers

Matthias Baesken (@MBaesken - Reviewer)

Reviewing

Using git

Checkout this PR locally:
$ git fetch https://git.openjdk.org/jdk.git pull/22835/head:pull/22835
$ git checkout pull/22835

Update a local copy of the PR:
$ git checkout pull/22835
$ git pull https://git.openjdk.org/jdk.git pull/22835/head

Using Skara CLI tools

Checkout this PR locally:
$ git pr checkout 22835

View PR using the GUI difftool:
$ git pr show -t 22835

Using diff file

Download this PR as a diff file:
https://git.openjdk.org/jdk/pull/22835.diff

Using Webrev

Link to Webrev Comment

…-8311302" This reverts commit cc8bd63.

bridgekeeper · 2024-12-19T18:15:35Z

👋 Welcome back sgehwolf! A progress list of the required criteria for merging this PR into master will be added to the body of your pull request. There are additional pull request commands available for use with this pull request.

openjdk · 2024-12-19T18:16:18Z

@jerboaa This change now passes all automated pre-integration checks.

ℹ️ This project also has non-automated pre-integration requirements. Please see the file CONTRIBUTING.md for details.

After integration, the commit message for the final commit will be:

8346239: Improve memory efficiency of JimageDiffGenerator

Reviewed-by: mbaesken

You can use pull request commands such as /summary, /contributor and /issue to adjust it as needed.

At the time when this comment was updated there had been 57 new commits pushed to the master branch:

f119663: 8346573: Can't use custom default file system provider with custom system class loader
379ac34: 8346838: RISC-V: runtime/CommandLine/OptionsValidation/TestOptionsWithRanges.java crash with debug VMs
0285020: 8345676: [ubsan] ProcessImpl_md.c:561:40: runtime error: applying zero offset to null pointer on macOS aarch64
0a81676: 8346881: [ubsan] logSelection.cpp:154:24 / logSelectionList.cpp:72:94 : runtime error: applying non-zero offset 1 to null pointer
08debd3: 8346993: C2 SuperWord: refactor to make more vector nodes available in VectorNode::make
de02503: 8346832: runtime/CompressedOops/CompressedCPUSpecificClassSpaceReservation.java fails on RISC-V
27646e5: 8344148: Add an explicit compiler phase for warning generation
8d388cc: 8346457: AOT cache creation crashes with "assert(pair_at(i).match() < pair_at(i+1).match()) failed: unsorted table entries"
d723597: 8345432: (ch, fs) Replace anonymous Thread with InnocuousThread
9c393a2: 8346953: Remove unnecessary @SuppressWarnings annotations (client, Make "can write" check work on Windows #2)
... and 47 more: https://git.openjdk.org/jdk/compare/4d77dbad4e15c5392878e7bc91cd8eb7ce49a482...master

As there are no conflicts, your changes will automatically be rebased on top of these commits when integrating. If you prefer to avoid this automatic rebasing, please check the documentation for the /integrate command for further details.

➡️ To integrate this PR with the above commit message to the master branch, type /integrate in a new comment.

openjdk · 2024-12-19T18:16:45Z

@jerboaa The following label will be automatically applied to this pull request:

core-libs

When this pull request is ready to be reviewed, an "RFR" email will be sent to the corresponding mailing list. If you would like to change these labels, use the /label pull request command.

jerboaa · 2024-12-19T18:16:47Z

@MBaesken @RealFYang Could you please test this PR since you originally ran into JDK-8346239 (this bug) and JDK-8344036. Thanks!

mlbridge · 2024-12-19T18:20:08Z

Webrevs

00: Full (c4c614c5)

RealFYang · 2024-12-20T02:12:27Z

@MBaesken @RealFYang Could you please test this PR since you originally ran into JDK-8346239 (this bug) and JDK-8344036. Thanks!

Yes, I can confirm that it works on my linux-aarch64 platform. Great! Thanks!

MBaesken · 2024-12-20T07:53:07Z

Hi Severin, I added it to our build/test queue.

jerboaa · 2024-12-20T09:29:11Z

Yes, I can confirm that it works on my linux-aarch64 platform. Great! Thanks!

Thank you!

jerboaa · 2024-12-20T09:29:38Z

Hi Severin, I added it to our build/test queue.

Thanks, Matthias!

jerboaa · 2025-01-07T10:41:04Z

Thanks for the review @MBaesken!

jerboaa · 2025-01-08T10:38:31Z

/integrate

openjdk · 2025-01-08T10:39:14Z

Going to push as commit f696d9c.
Since your change was applied there have been 76 commits pushed to the master branch:

cbabc04: 8332506: SIGFPE In ObjectSynchronizer::is_async_deflation_needed()
9872421: 8347094: Inline CollectedHeap::increment_total_full_collections
b741f3f: 8343629: More MergeStore benchmark
b0c935c: 8347047: Cleanup action passed to MemorySegment::reinterpret keeps old segment alive
bcefab5: 8342468: Improve API documentation for java.lang.classfile.constantpool
40f0a39: 8343342: java/io/File/GetXSpace.java fails on Windows with CD-ROM drive
021c476: 8347148: [BACKOUT] AccessFlags can be u2 in metadata
ddb5881: 8329549: Remove FORMAT64_MODIFIER
098afc8: 8339113: AccessFlags can be u2 in metadata
e413fc6: 8347127: CTW fails to build after JDK-8334733
... and 66 more: https://git.openjdk.org/jdk/compare/4d77dbad4e15c5392878e7bc91cd8eb7ce49a482...master

Your commit was automatically rebased without conflicts.

openjdk · 2025-01-08T10:39:21Z

@jerboaa Pushed as commit f696d9c.

💡 You may see a message that your pull request was closed with unmerged commits. This can be safely ignored.

AlanBateman · 2025-01-08T10:43:46Z

@slowhog Would it be possible to do a pass over this too?

slowhog · 2025-01-08T18:51:02Z

src/jdk.jlink/share/classes/jdk/tools/jlink/internal/runtimelink/JimageDiffGenerator.java

+        try {
+            try (is1; is2) {
+                while ((bytesRead1 = is1.read(buf1)) != -1 &&
+                       (bytesRead2 = is2.read(buf2)) != -1) {


what's the point to have second try block?

I think it's easier to read to while loop on is1, then read and check is2 within the loop instead of && on both.

archiecobbs · 2025-01-08T20:29:42Z

src/jdk.jlink/share/classes/jdk/tools/jlink/internal/runtimelink/JimageDiffGenerator.java

+            try (is1; is2) {
+                while ((bytesRead1 = is1.read(buf1)) != -1 &&
+                       (bytesRead2 = is2.read(buf2)) != -1) {
+                    if (bytesRead1 != bytesRead2) {


This test seems to assume there won't be any short reads. That might be true in practice, but for defensive programming reasons it would be safer to use readNBytes() instead of read().

archiecobbs · 2025-01-08T20:31:52Z

src/jdk.jlink/share/classes/jdk/tools/jlink/internal/runtimelink/JimageDiffGenerator.java

+                    if (bytesRead1 != bytesRead2) {
+                        return false;
+                    }
+                    if (bytesRead1 == buf1.length) {


This can be simplified by using Arrays.equals(byte[],int,int,byte[],int,int) for both cases.

archiecobbs · 2025-01-08T21:29:12Z

Sorry, I didn't realize this PR was already closed. You can treat my comments as "FYI".

jerboaa · 2025-01-09T09:40:48Z

Thanks for the reviews. I'll follow up with those comments/fixes in a separate PR.

jerboaa · 2025-01-09T17:33:27Z

@slowhog @AlanBateman @archiecobbs Clean ups done in #23014 PTAL.

jerboaa added 4 commits December 19, 2024 18:51

8346239: Improve memory efficiency of JimageDiffGenerator

d6022c0

Revert "8344036: Tests tools/jlink/runtimeImage fail on AIX after JDK…

b4e68c3

…-8311302" This reverts commit cc8bd63.

Bump timeout of test

8303a87

Fixup.

c4c614c

openjdk bot added the rfr Pull request is ready for review label Dec 19, 2024

openjdk bot added the core-libs [email protected] label Dec 19, 2024

MBaesken approved these changes Dec 30, 2024

View reviewed changes

openjdk bot added the ready Pull request is ready to be integrated label Dec 30, 2024

openjdk bot added the integrated Pull request has been integrated label Jan 8, 2025

openjdk bot closed this Jan 8, 2025

openjdk bot removed ready Pull request is ready to be integrated rfr Pull request is ready for review labels Jan 8, 2025

slowhog reviewed Jan 8, 2025

View reviewed changes

archiecobbs reviewed Jan 8, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

8346239: Improve memory efficiency of JimageDiffGenerator #22835

8346239: Improve memory efficiency of JimageDiffGenerator #22835

jerboaa commented Dec 19, 2024 •

edited

Loading

bridgekeeper bot commented Dec 19, 2024

openjdk bot commented Dec 19, 2024 •

edited

Loading

openjdk bot commented Dec 19, 2024

jerboaa commented Dec 19, 2024

mlbridge bot commented Dec 19, 2024

RealFYang commented Dec 20, 2024

MBaesken commented Dec 20, 2024

jerboaa commented Dec 20, 2024

jerboaa commented Dec 20, 2024

jerboaa commented Jan 7, 2025

jerboaa commented Jan 8, 2025

openjdk bot commented Jan 8, 2025

openjdk bot commented Jan 8, 2025

AlanBateman commented Jan 8, 2025

slowhog Jan 8, 2025

archiecobbs Jan 8, 2025

archiecobbs Jan 8, 2025

archiecobbs commented Jan 8, 2025

jerboaa commented Jan 9, 2025

jerboaa commented Jan 9, 2025

8346239: Improve memory efficiency of JimageDiffGenerator #22835

8346239: Improve memory efficiency of JimageDiffGenerator #22835

Conversation

jerboaa commented Dec 19, 2024 • edited Loading

Progress

Issue

Reviewers

Reviewing

bridgekeeper bot commented Dec 19, 2024

openjdk bot commented Dec 19, 2024 • edited Loading

openjdk bot commented Dec 19, 2024

jerboaa commented Dec 19, 2024

mlbridge bot commented Dec 19, 2024

Webrevs

RealFYang commented Dec 20, 2024

MBaesken commented Dec 20, 2024

jerboaa commented Dec 20, 2024

jerboaa commented Dec 20, 2024

jerboaa commented Jan 7, 2025

jerboaa commented Jan 8, 2025

openjdk bot commented Jan 8, 2025

openjdk bot commented Jan 8, 2025

AlanBateman commented Jan 8, 2025

slowhog Jan 8, 2025

Choose a reason for hiding this comment

archiecobbs Jan 8, 2025

Choose a reason for hiding this comment

archiecobbs Jan 8, 2025

Choose a reason for hiding this comment

archiecobbs commented Jan 8, 2025

jerboaa commented Jan 9, 2025

jerboaa commented Jan 9, 2025

jerboaa commented Dec 19, 2024 •

edited

Loading

openjdk bot commented Dec 19, 2024 •

edited

Loading