Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Skip all training opset model tests #14636

Merged
merged 3 commits into from
Feb 9, 2023
Merged

Skip all training opset model tests #14636

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
c7abe1f
Disable Inception-1-qdq model in training pipeline
baijumeswani Feb 9, 2023
cd8b8d1
Mark SqueezeNet 1.0-qdq also as broken
baijumeswani Feb 9, 2023
a25e5d0
Skip all training domain models
baijumeswani Feb 9, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
31 changes: 1 addition & 30 deletions onnxruntime/test/providers/cpu/model_tests.cc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -125,13 +125,11 @@ TEST_P(ModelTest, Run) {
return;
}

#ifndef ENABLE_TRAINING
if (model_info->HasDomain(ONNX_NAMESPACE::AI_ONNX_TRAINING_DOMAIN) ||
model_info->HasDomain(ONNX_NAMESPACE::AI_ONNX_PREVIEW_TRAINING_DOMAIN)) {
SkipTest("It has training domain");
SkipTest("it has the training domain. No pipeline should need to run these tests.");
return;
}
#endif
std::set<BrokenTest> broken_tests = {
{"slice_neg_steps",
"Type parameter (Tind) bound to different types (tensor(int64) and tensor(int32) in node ()."},
Expand Down Expand Up @@ -254,36 +252,9 @@ TEST_P(ModelTest, Run) {
{"softmax_cross_entropy_input_shape_is_NCd1d2d3d4d5_mean_weight", "type error", {"opset12"}},
{"softmax_cross_entropy_mean_weight", "type error", {"opset12"}},
{"softmax_cross_entropy_mean_no_weight_ignore_index_4d", "type error", {"opset12"}},
// models from model zoo
{"BERT-Squad-int8", "failed in training", {"opset12"}},
{"DenseNet-121-12-int8", "failed in training", {"opset12"}},
{"EfficientNet-Lite4-int8", "failed in training", {"opset11"}},
{"EfficientNet-Lite4-qdq", "failed in training", {"opset11"}},
{"Faster R-CNN R-50-FPN-int8", "failed in training", {"opset12"}},
{"Inception-1-int8", "failed in training", {"opset12"}},
{"MobileNet v2-1.0-int8", "failed in traning", {"opset12"}},
{"MobileNet v2-1.0-qdq", "failed in training", {"opset12"}},
{"ResNet50-qdq", "failed in training", {"opset12"}},
{"ResNet50_int8", "failed in training", {"opset12"}},
{"ResNet50-int8", "failed in training", {"opset12"}},
{"ShuffleNet-v2-int8", "failed in training", {"opset12"}},
{"SSD-int8", "failed in training", {"opset12"}},
{"VGG 16-int8", "failed in training", {"opset12"}},
{"YOLOv3-12-int8", "failed in training", {"opset12"}},
#endif
{"mask_rcnn_keras", "this model currently has an invalid contrib op version set to 10", {}}};

#ifdef ENABLE_TRAINING_CORE
// They only failed in orttraining-iinux-gpu-ci-pipelie with TRT8.5
if (provider_name == "cpu") {
broken_tests.insert({"ShuffleNet-v2-qdq", "failed in orttraining-linux-gpu, TRT8.5 with V100, but it's a cpu test?", {"opset12"}});
}
if (provider_name == "cuda") {
broken_tests.insert({"GoogleNet-qdq", "failed in orttraining-linux-gpu, TRT8.5 with V100.", {"opset12"}});
broken_tests.insert({"ShuffleNet-v2-qdq", "failed in orttraining-linux-gpu, TRT8.5 with V100.", {"opset12"}});
}
#endif

// Some EPs may fail to pass some specific testcases.
// For example TenosrRT EP may fail on FLOAT16 related testcases if GPU doesn't support float16.
// Instead of list all these testcases, we can use following keyword set to filter out testcases wchich contain
Expand Down