Exception:Only one instance of LoggingManager created with InstanceType::Default can exist at any point in time.

[doruime]

I am building an ONNX inference server, and I need multiple model loaded simultaneously. The exception raised when I loaded multiple models(when I called OrtCreateEnv and OrtCreateSession), every model has its own OrtSession and OrtEnv objects. How to fix it?

[snnn]

OrtEnv should be created only once, for each process.

[doruime]

OrtEnv should be created only once, for each process.

So I can create multiple sessions under one OrtEnv, right?

[pranavsharma]

OrtEnv should be created only once, for each process.

So I can create multiple sessions under one OrtEnv, right?

yes, that's the idea.

[keepWalking]

@pranavsharma @snnn hi, I hit this issue when try to load NO.2 model using C#, (NO.1 model is loaded by other dll which is black-box for me) I followed the following C# sample: https://github.com/microsoft/onnxruntime/blob/master/csharp/sample/Microsoft.ML.OnnxRuntime.InferenceSample/Program.cs
How to resolve this?
Thx

[snnn]

Talking with @keepWalking offline.

We're solving this problem in the upcoming release.

[KangolHsu]

same error.
maybe onnxruntime can not support load more than one model at the same time?
(I use c++ lib)

[smstec]

I'm experiencing this issue as well. Any updates?

[Karnav123]

I have the same issue.

[pranavsharma]

I have the same issue.

This has been fixed already. Did you try the latest master?

[Karnav123]

I have the same issue.

This has been fixed already. Did you try the latest master?

I am using 1.3.1. I have to use it in a product and we have taken the clearance for that version only. However, I am facing many memory leak issues for large data during Run method call on the model. If you could help then it will be great.

[pranavsharma]

@Karnav123 let me understand a few basics. Which language APIs are you using?
The Env instance needs to be created only once and reused for creating sessions for any number of models. You don't need to create the env for every single model (session). A session has a 1-1 correspondence with a model.
Regarding memory leak issues, do you've a repro?

[Karnav123]

@Karnav123 let me understand a few basics. Which language APIs are you using?

@pranavsharma

• I am using C++ APIs. And My model is a simple XGBoostClassifier model.

The Env instance needs to be created only once and reused for creating sessions for any number of models. You don't need to create the env for every single model (session). A session has a 1-1 correspondence with a model.

• I have created a Model class to keep all the model-related data in it. The "Env" is static. I am loading multiple different models in a session and keeping them in a static map to use it based on the need. However, for during prediction small datasets of size (1000 * 22, 22 is a number of features and 1000 are the number of different entities.) it works fine, but for larger data, it gives memory error while Run(...) although the Tensor creation is successful for large data also.

Regarding memory leak issues, do you've a repro?

• We are using onnxruntime as a third party in a product so I can not send you the code but the class looks like the following:

class Model
{
private:
     std::unique_ptr<Ort::Session> m_onnxModel;
     std::vector<const char*> m_inputNames;
     std::vector<const char*> m_outputNames;
     static Ort::Env env{};
}

// Create the Model

// Load the ONNX model
    Ort::SessionOptions sessionOptions;
    CreateThreadingOptions()
    static Ort::Env env_;
  
    sessionOptions.SetIntraOpNumThreads(8);
    sessionOptions.SetGraphOptimizationLevel(GraphOptimizationLevel::ORT_ENABLE_EXTENDED);
    auto sess = std::make_unique<Ort::Session>(Ort::Session(env, path, sessionOptions));
    session = std::move(sess);
// Put a model in a static map so that we need not load it every time:
static std::unordered_map<std::string, Model*> m_modelMap;

I am not able to understand why it gives memory error in Run if tensor creation of large data is successful.

[pranavsharma]

• The original issue is regarding exception in LoggingManager. I assume you're not seeing this exception. If so, can you create a fresh github issue with the memory errors you're seeing?
• Can you use ORT_LOGGING_LEVEL_DEBUG when creating the environment and paste the logs?
• What kind of memory error do you see? Is it running out of memory?

[Karnav123]

• The original issue is regarding exception in LoggingManager. I assume you're not seeing this exception. If so, can you create a fresh github issue with the memory errors you're seeing?
• Can you use ORT_LOGGING_LEVEL_DEBUG when creating the environment and paste the logs?
• What kind of memory error do you see? Is it running out of memory?

I have created an issue already related to this: #4614
The memory issue is invalid access to memory.

[jnulzl]

I have the same issue.

This has been fixed already. Did you try the latest master?

Nice!! I fixed it using the latest master. The build Instructions is :

CPU

./build.sh --use_openmp --config Release --build_shared_lib --parallel --build_dir ./build_cpu_openmp

GPU

./build.sh --use_cuda --cudnn_home /usr/local/cuda --cuda_home /usr/local/cuda --config Release --build_shared_lib --parallel --build_dir ./build_cuda

Above CPU and GPU version are all OK!

And I also tested offical pre-build version(1.4.0 CPU and 1.1.2 GPU):

I meet the same error when I using CPU pre-build version 1.4.0 CPU

I have not meet this error when I using GPU pre-build version 1.1.2 GPU

Above all tested is C++ API!! OS is Ubuntu 16.04 CUDA 10.0 cudnn 7.4.2.