Java build system enhancements #2866

yuzawa-san · 2020-01-17T21:49:41Z

Description: I cleaned up the CMake files to delegate building and testing to a Gradle build system. Gradle is manages java build/testing, dependencies, and packaging better than CMake. Additionally, all of the Java build output files are in a single output directory. I added a README which explains what JAR files and build output is generated. I relaxed the requirements in JNI loader in OnnxRuntime.java to allow for flexible loading of the library (i.e. from the os libpath or if the developer wants to load the library from some other way). If the developer wants to do that, then they would not use the onnxruntime-lib.jar. I attempted to build out an azure pipeline for the java build (based on the python build). I do not know how to test that. I would assume the maintainers would have to add it. I added spotless plugin to ensure formatting compliance. I selected a stock formatting profile (google's) and reformatted the project in a consistent manner. All changes to java files are formatting except OnnxRuntime.java. Not in this PR: I am playing around with Gradle's maven publishing mechanism.

Motivation and Context

Why is this change required? What problem does it solve?

Per my original issue below, I wanted to improve the Java packaging and maintainability. I feel these changes will eventually make way for proper maven distribution.

If it fixes an open issue, please link to the issue here.

This is relevant towards #2675

msftclas · 2020-01-17T21:49:55Z

All CLA requirements met.

snnn · 2020-01-17T22:12:09Z

I'd prefer to remove all the Java related things out of cmake. We don't have to use cmake driving the java compilation. In order to make it part of the CI, we can add one extra step in win-ci-pipeline.yml. And we may directly use maven/gradle to do so. All of our Windows build machines already have gradle installed at %GRADLE_HOME%. You may invoke it directly.

yuzawa-san · 2020-01-17T22:23:22Z

Yeah I moved all of the Java stuff out of the CMake except the piece that compiles the JNI shared object, which needs to stay. I chose the Gradle wrapper for ease of developers, since they won't have to install it and we can lock the version for compatibility purposes. The normal CI process should work just fine since the --build_java flag is in there already, so I guess we'll see shortly once this runs. The pipeline I added was a new one for packaging and it creates an azure artifacts onnxruntime4j and onnxruntime4j_gpu. My vision is that these resulting artifacts can be published to maven for versioned releases and for nightly snapshot releases, just like the python stuff is pushed to PyPi: https://dev.azure.com/onnxruntime/onnxruntime/_release?_a=releases&view=mine&definitionId=1 Regardless how we want to build with respect to CI pipelines, the output is cleaner and all in build/$OS/$CONFIG/java/build.

snnn

What's the gradle-wrapper.jar ?

java/src/main/native/ai_onnxruntime_OnnxTensor.h

yuzawa-san · 2020-01-17T22:42:45Z

The gradle wrapper jar needs to be there since it is a part of the wrapper mechanism https://docs.gradle.org/current/userguide/gradle_wrapper.html#sec:adding_wrapper see the note in that section

snnn · 2020-01-17T22:48:24Z

The gradle wrapper jar needs to be there since it is a part of the wrapper mechanism https://docs.gradle.org/current/userguide/gradle_wrapper.html#sec:adding_wrapper see the note in that section

It's auto-generated right? Then why not generated it in the build?

snnn · 2020-01-17T22:49:50Z

Can we avoid using Gradle Wrapper? I'd prefer to not download something from a third-party website during the build, because it adds extra risk.

azure-pipelines · 2020-02-18T18:06:32Z

Azure Pipelines successfully started running 1 pipeline(s).

azure-pipelines · 2020-02-18T18:06:58Z

Azure Pipelines successfully started running 8 pipeline(s).

snnn · 2020-02-18T18:14:06Z

java/src/main/java/ai/onnxruntime/OnnxRuntime.java

        }
+        System.load(tempFile.getAbsolutePath());


So, if onnxruntime.dll has any dependencies that are not in the java.library.path, then this call will fail, right?

I do not know precisely since I do not work in windows environments, but I believe the load call will load the file and whatever dependency configuration is set in the library file. Linux SO files can have relative, absolute, and rpath dependency locations. DLL's should probably have similar capabilities.

It should check the system path for dependent libraries as it's basically a wrapper around the platform's dynamic library loader.

yuzawa-san · 2020-02-18T18:19:19Z

Gradle still appears to be missing https://dev.azure.com/onnxruntime/onnxruntime/_build/results?buildId=114373&view=logs&j=8c952c3b-dbd4-5685-50d1-7e2a31304cbf&t=e14f62b2-f329-5130-d196-ee9974ccf2a3

snnn · 2020-02-18T18:20:04Z

I'm fixing it.

snnn · 2020-02-18T18:51:54Z

/azp run Linux CPU CI Pipeline

azure-pipelines · 2020-02-18T18:52:05Z

Azure Pipelines successfully started running 1 pipeline(s).

snnn · 2020-02-18T19:18:49Z

@Craigacp Do you think this PR is ready to merge?

Craigacp · 2020-02-18T19:35:55Z

@snnn Assuming all the CI pipelines pass, yes.

snnn · 2020-02-18T19:46:17Z

/azp run Linux CPU CI Pipeline,Linux GPU CI Pipeline,Linux GPU TensorRT CI Pipeline,MacOS CI Pipeline,Win CPU x86 CI Pipeline,Windows CPU CI Pipeline,Windows GPU CI Pipeline,Windows GPU TensorRT CI Pipeline

snnn · 2020-02-18T19:46:49Z

/azp run Win CPU x64 NoContribops CI Pipeline,MacOS NoContribops CI Pipeline,Linux CPU x64 NoContribops CI Pipeline

azure-pipelines · 2020-02-18T19:47:04Z

Azure Pipelines successfully started running 8 pipeline(s).

azure-pipelines · 2020-02-18T19:47:10Z

Azure Pipelines successfully started running 3 pipeline(s).

yuzawa-san · 2020-02-18T21:31:13Z

Ok cool the linux build works. The windows ci pipelines timed out because they took a long time for unknown reasons. Something is off given that most of the runs for that job take around 25 minutes. Any ideas? Sadly I do not have access to a windows environment, so any help would be appreciated.

Does the gradle daemon keep running or something weird which prevents that stage from ending? I think my last CI build hung indefinitely after those pybind jobs were built. I can see the Java builds occurring in the logs.

snnn · 2020-02-18T22:00:55Z

Something is wrong here. Windows build hangs forever.

snnn · 2020-02-18T22:01:24Z

Let me try to disable gradle daemon.

snnn · 2020-02-18T22:15:48Z

/azp run Linux CPU CI Pipeline,Linux GPU CI Pipeline,Linux GPU TensorRT CI Pipeline,MacOS CI Pipeline,Win CPU x86 CI Pipeline,Windows GPU CI Pipeline,Windows GPU TensorRT CI Pipeline

snnn · 2020-02-18T22:15:59Z

/azp run Win CPU x64 NoContribops CI Pipeline,MacOS NoContribops CI Pipeline,Linux CPU x64 NoContribops CI Pipeline

snnn · 2020-02-18T22:16:15Z

This time it should be ok.

azure-pipelines · 2020-02-18T22:16:17Z

Azure Pipelines successfully started running 3 pipeline(s).

azure-pipelines · 2020-02-18T22:16:25Z

Azure Pipelines successfully started running 7 pipeline(s).

snnn

Thank you so much!

yuzawa-san added 21 commits January 15, 2020 05:59

initial gradle work

c938f9b

update cmake config to copy more aggressively to gradle

99918f7

more cmake changes

7021e5b

Merge remote-tracking branch 'upstream/master' into java-gradle

bdc4eec

redo gradle build

707993a

trigger on java file changes

a75c799

spotless

c0f9c42

hold off on assembly

2719cd6

cleanup

3724451

remove useless jar

85047ec

use more variables, put build output in dir

5b0aab3

recursive file search, set min version

32de555

readme

bcc155a

Merge remote-tracking branch 'upstream/master' into java-gradle

16a32ac

attempt java pipelines

1563343

more pipeline progress

f2e4674

refactor loader

2018b96

clean up sample

0345320

update README

03be464

update README

dab2ac0

clean up readme

eb583a8

yuzawa-san requested a review from a team as a code owner January 17, 2020 21:49

snnn reviewed Jan 17, 2020

View reviewed changes

java/src/main/native/ai_onnxruntime_OnnxTensor.h Outdated Show resolved Hide resolved

snnn reviewed Feb 18, 2020

View reviewed changes

a

4f7f20c

snnn added 2 commits February 18, 2020 11:45

update

0629b8b

Merge remote-tracking branch 'origin/master' into java-gradle

217564b

Disable daemon

970c086

snnn approved these changes Feb 18, 2020

View reviewed changes

snnn merged commit 411b3aa into microsoft:master Feb 18, 2020

yuzawa-san mentioned this pull request Feb 19, 2020

Add and run spotless formatter for Java and Gradle #3038

Merged

yuzawa-san deleted the java-gradle branch February 19, 2020 01:22

Craigacp mentioned this pull request Mar 1, 2023

Update Gradle version #14862

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Java build system enhancements #2866

Java build system enhancements #2866

yuzawa-san commented Jan 17, 2020 •

edited

Loading

msftclas commented Jan 17, 2020 •

edited

Loading

snnn commented Jan 17, 2020

yuzawa-san commented Jan 17, 2020

snnn left a comment

yuzawa-san commented Jan 17, 2020

snnn commented Jan 17, 2020

snnn commented Jan 17, 2020

azure-pipelines bot commented Feb 18, 2020

azure-pipelines bot commented Feb 18, 2020

snnn Feb 18, 2020

yuzawa-san Feb 18, 2020

Craigacp Feb 18, 2020

yuzawa-san commented Feb 18, 2020

snnn commented Feb 18, 2020

snnn commented Feb 18, 2020

azure-pipelines bot commented Feb 18, 2020

snnn commented Feb 18, 2020

Craigacp commented Feb 18, 2020

snnn commented Feb 18, 2020

snnn commented Feb 18, 2020

azure-pipelines bot commented Feb 18, 2020

azure-pipelines bot commented Feb 18, 2020

yuzawa-san commented Feb 18, 2020

snnn commented Feb 18, 2020

snnn commented Feb 18, 2020

snnn commented Feb 18, 2020

snnn commented Feb 18, 2020

snnn commented Feb 18, 2020

azure-pipelines bot commented Feb 18, 2020

azure-pipelines bot commented Feb 18, 2020

snnn left a comment

Java build system enhancements #2866

Java build system enhancements #2866

Conversation

yuzawa-san commented Jan 17, 2020 • edited Loading

msftclas commented Jan 17, 2020 • edited Loading

snnn commented Jan 17, 2020

yuzawa-san commented Jan 17, 2020

snnn left a comment

Choose a reason for hiding this comment

yuzawa-san commented Jan 17, 2020

snnn commented Jan 17, 2020

snnn commented Jan 17, 2020

azure-pipelines bot commented Feb 18, 2020

azure-pipelines bot commented Feb 18, 2020

snnn Feb 18, 2020

Choose a reason for hiding this comment

yuzawa-san Feb 18, 2020

Choose a reason for hiding this comment

Craigacp Feb 18, 2020

Choose a reason for hiding this comment

yuzawa-san commented Feb 18, 2020

snnn commented Feb 18, 2020

snnn commented Feb 18, 2020

azure-pipelines bot commented Feb 18, 2020

snnn commented Feb 18, 2020

Craigacp commented Feb 18, 2020

snnn commented Feb 18, 2020

snnn commented Feb 18, 2020

azure-pipelines bot commented Feb 18, 2020

azure-pipelines bot commented Feb 18, 2020

yuzawa-san commented Feb 18, 2020

snnn commented Feb 18, 2020

snnn commented Feb 18, 2020

snnn commented Feb 18, 2020

snnn commented Feb 18, 2020

snnn commented Feb 18, 2020

azure-pipelines bot commented Feb 18, 2020

azure-pipelines bot commented Feb 18, 2020

snnn left a comment

Choose a reason for hiding this comment

yuzawa-san commented Jan 17, 2020 •

edited

Loading

msftclas commented Jan 17, 2020 •

edited

Loading