[TESTS] Make a common entry point for conformance tests #3265
Conversation
github-actions
bot
added
the
documentation
kshpv
changed the title
| @@ -293,50 +390,49 @@ def test_ptq_quantization( | |||
| capsys: pytest.CaptureFixture, | |||
| extra_columns: bool, | |||
| memory_monitor: bool, | |||
| use_avx2: Optional[bool] = None, | |||
There was a problem hiding this comment.
use_avx2 used only as fixture to set env variable, no need to pass it to pipeline
| @@ -18,7 +18,7 @@ hf/hf-internal-testing/tiny-random-GPTNeoXForCausalLM_statefull_backend_OPTIMUM: | |||
| metric_value: null | |||
| hf/hf-internal-testing/tiny-random-GPTNeoXForCausalLM_stateless_backend_OPTIMUM: | |||
| metric_value: null | |||
| xfail_reason: "Issue-161969" | |||
| exception_xfail_reason: "Issue-161969" | |||
There was a problem hiding this comment.
For exception need to know exception type and msg to set xfail only for expected error
exception_xfail_reason:
msg: "Issie-123"
class: RuntimeError
errmsg: "some error"
| @@ -174,6 +178,9 @@ def fixture_ptq_report_data(output_dir, run_benchmark_app, pytestconfig): | |||
| if not run_benchmark_app: | |||
| df = df.drop(columns=["FPS"]) | |||
|
|
|||
| df = df.drop(columns=["Num sparse activations"]) | |||
There was a problem hiding this comment.
looks like report_dat can be use one function to generate csv,
some thing like save_results(daf, droped_columns)
| num_fq_nodes: int = 0 | ||
| num_int8: int = 0 | ||
| num_int4: int = 0 | ||
| num_sparse_activations: int = 0 |
There was a problem hiding this comment.
Does it have to be defined in the base.py? I believe a test pipeline should define it's own fields here.
There was a problem hiding this comment.
I think so, it makes code common. If any test does not need any field, that's fine, only values in reference data matter
| @@ -196,6 +197,7 @@ def get_result_dict(self): | |||
| "Num FQ": self.num_compress_nodes.num_fq_nodes, | |||
| "Num int4": self.num_compress_nodes.num_int4, | |||
| "Num int8": self.num_compress_nodes.num_int8, | |||
| "Num sparse activations": self.num_compress_nodes.num_sparse_activations, | |||
There was a problem hiding this comment.
Same comment as for NumCompressNodes.num_sparse_activations
| columns_to_drop = ["Num sparse activations", "Num int4"] | ||
| yield from create_fixture_report_data(output_dir, run_benchmark_app, pytestconfig, columns_to_drop) |
There was a problem hiding this comment.
It seems like if we add another conformance job similar to the activation sparsity, we will need to update PTQ and WC report data fixtures to omit the newly added fields. Am I getting this right? If so, I believe this should not be the case, and this logic should be independent of other possible conformance pipelines.
There was a problem hiding this comment.
You are right. But I do not see any problem with omitting some columns (it is already done on the develop in the same manner), it is much easier to implement in that way than to have several different classes with different logic.
What do you think?
Changes
Reason for changes
Related tickets
Tests
WC run - https://github.com/openvinotoolkit/nncf/actions/runs/13239414168 - passed
ptq run -job/manual/job/post_training_quantization/608/ - in progress
WC CI - job/manual/job/post_training_weight_compression/317 - passed