Multiple Model Configurations

[yinggeh]

Add --mode-config-name option when starting Triton server. Allow users to select custom configurations other than default config.pbtxt.

core: triton-inference-server/core#348

[rmccorm4]

Is this still a TODO?

[yinggeh]

This is mainly a question from me. Should I take care of the case where configuration file name contains space? In another word, I want to confirm that our command line arguement does not accept value with space. I wasn't able to pass a value with space or \space or "xxx xxx".

[jbkyang-nvi]

Maybe we should just add a note in the configurations part to not include names with spaces in them. Then we can also remove this test.

[yinggeh]

I was able to pass file name with space to the server.
/opt/tritonserver/bin/tritonserver --model-repository=/opt/tritonserver/qa/L0_custom_model_config/models --model-config-name="h100 v100"
The server will look for config file named "h100 v100".pbtxt and preserve quotes. So I think we should tell user to not use space in file name.

[yinggeh]

Fixed

[jbkyang-nvi]

Suggested change

	model repository, it is nessecery to have models configured differently
	model repository, it is necessary to have models configured differently

[yinggeh]

This is what happens when working overnight.

[yinggeh]

Fixed

[jbkyang-nvi]

I would recommend adding the examples you listed in the TEP instead to make the configurations clearer

Example 1: --model-config-name=8gpus
.
└── model repository/
    ├── model_a/
    │   ├── configs/
    │   │   ├── 4gpus.pbtxt
    │   │   └── **8gpus.pbtxt**
    │   └── config.pbtxt
    ├── model_b/
    │   ├── configs/
    │   │   └── 4gpus.pbtxt
    │   └── **config.pbtxt**
    └── model_c/
        ├── configs/
        │   └── config.pbtxt
        └── **config.pbtxt**

Example 2: --model-config-name=config
.
└── model repository/
    ├── model_a/
    │   ├── configs/
    │   │   ├── 4gpu.pbtxt
    │   │   └── 8gpu.pbtxt
    │   └── **config.pbtxt**
    ├── model_b/
    │   ├── configs/
    │   │   └── 4gpu.pbtxt
    │   └── **config.pbtxt**
    └── model_c/
        ├── configs/
        │   └── **config.pbtxt**
        └── config.pbtxt

Example 3: --model-config-name not specified
.
└── model repository/
    ├── model_a/
    │   ├── configs/
    │   │   ├── 4gpu.pbtxt
    │   │   └── 8gpu.pbtxt
    │   └── **config.pbtxt**
    ├── model_b/
    │   ├── configs/
    │   │   └── 4gpu.pbtxt
    │   └── **config.pbtxt**
    └── model_c/
        ├── configs/
        │   └── config.pbtxt
        └── **config.pbtxt**

[yinggeh]

Fixed

[jbkyang-nvi]

Suggested change

	echo -e "\n***\n*** Test Failed\n***"
	echo -e "\n***\n*** Test Failed to get model config\n***"

[yinggeh]

Fixed

[GuanLuo]

You can iterate different backend types and shapes but that is not essential for testing this feature.. I would rather just test one model setting for simplicity and group the version testings below to be:

# check available version
for version in expected_available_versions:
    ...
for version in expected_unavailable_versions:
    ...

[GuanLuo]

Actually, I don't know if the inference check is necessary as you already check the model version readiness. With that, it just seems to be an sanity check for me.

[yinggeh]

Removed unnecessary tests