[Java] Support configuring CUDA and TensorRT execution providers #10697
Conversation
… as provider options need to have their shared library loaded before they can be constructed.
Craigacp
force-pushed
the
java-cuda-params
branch
from
17426c3 to
cd348da
Compare
|
Rebased after #10670. |
| } | ||
| } | ||
|
|
||
| protected final long nativeHandle; |
There was a problem hiding this comment.
There was a problem hiding this comment.
Yeah, protected in Java means it's this package + subclasses, so it is visible to everything in ai.onnxruntime.
| * | ||
| * @return The api handle. | ||
| */ | ||
| protected static long getApiHandle() { |
There was a problem hiding this comment.
Because the provider implementations are in ai.onnxruntime.providers and OnnxRuntime.ortApiHandle has package level access in ai.onnxruntime so it's not visible. This allows it to be visible to subclasses of OrtProviderOptions no matter what package they are in.
I was trying to not add too many classes to the main package, but it's rather contorted here.
| * @param apiHandle The api pointer. | ||
| * @param nativePointer The native options pointer. | ||
| */ | ||
| protected abstract void close(long apiHandle, long nativePointer); |
There was a problem hiding this comment.
I mean the same thing by nativePointer and nativeHandle, so that should be nativeHandle for consistency.
|
Is it possible to get this in for the 1.11 release? The same support for C# is slated to be in that release and it would be nice to keep parity with it. |
|
/azp run Linux CPU CI Pipeline, Linux CPU Minimal Build E2E CI Pipeline, Linux CPU x64 NoContribops CI Pipeline, Linux GPU CI Pipeline, Linux GPU TensorRT CI Pipeline, Linux Nuphar CI Pipeline, Linux OpenVINO CI Pipeline, MacOS CI Pipeline |
|
/azp run MacOS NoContribops CI Pipeline, Windows CPU CI Pipeline, Windows GPU CI Pipeline, Windows GPU TensorRT CI Pipeline, Windows WebAssembly CI Pipeline, orttraining-amd-gpu-ci-pipeline, orttraining-linux-ci-pipeline, orttraining-linux-gpu-ci-pipeline, orttraining-ortmodule-distributed |
|
Azure Pipelines successfully started running 7 pipeline(s). |
1 similar comment
|
Azure Pipelines successfully started running 7 pipeline(s). |
|
/azp run Linux CPU CI Pipeline, Linux CPU Minimal Build E2E CI Pipeline, Linux CPU x64 NoContribops CI Pipeline, Linux GPU CI Pipeline, Linux GPU TensorRT CI Pipeline, Linux Nuphar CI Pipeline, Linux OpenVINO CI Pipeline, MacOS CI Pipeline |
|
/azp run MacOS NoContribops CI Pipeline, Windows CPU CI Pipeline, Windows GPU CI Pipeline, Windows GPU TensorRT CI Pipeline, Windows WebAssembly CI Pipeline, orttraining-amd-gpu-ci-pipeline, orttraining-linux-ci-pipeline, orttraining-linux-gpu-ci-pipeline, orttraining-ortmodule-distributed |
|
Azure Pipelines successfully started running 7 pipeline(s). |
1 similar comment
|
Azure Pipelines successfully started running 7 pipeline(s). |
…rosoft#10697) Java side parts for configuring CUDA and TensorRT. Adding tests for CUDA and TensorRT. Refactoring library loading logic as provider options need to have their shared library loaded before they can be constructed.
Description:
Adds support for configuring the CUDA and TensorRT execution providers using the new V2 options classes.
There are two abstract classes,
OrtProviderOptionswhich provides the basic functionality for accessing the package private pointers used to call native methods, andStringConfigProviderOptionswhich provides specific functionality for working on the V2 style string configured options classes.OrtProviderOptionscan be used as the basis for other non-string configured provider options classes, though I've not added implementations for any of those yet. The new provider options classes live inai.onnxruntime.providersalong with the previously definedNNAPIFlagsandCoreMLFlags.It may not apply cleanly over the top of some of the other PRs I have out, so I'll rebase it as those are merged if necessary.
Motivation and Context