Argon AV1 test resource patch don't match the downloaded resources path

[ndufresne]

I wanted to try the new Argon suite, but its broken, the resources and the path don't match. Using GStreamer VA AV1, I'm getting this comment:

gst-launch-1.0 --no-fault filesrc location=/home/nicolas/Sources/fluster/fluster/../resources/AV1_ARGON_VECTORS/test3534_116/argon_coveragetool_av1_base_and_extended_profiles_v2.1/profile0_core/streams/test3534_116.obu ! parsebin ! vaav1dec ! video/x-raw ! videoconvert dither=none ! video/x-raw,format=I420_10LE ! videocodectestsink -m

But the path is actually:

resources/AV1_ARGON_VECTORS/argon_coveragetool_av1_base_and_extended_profiles_v2.1/profile0_core/streams/test3534_116.obu

[ndufresne]

It seems these are also raw OBU, which won't work with GStreamer typefind. We'll need to spacial case that in the gstreamer player.

[rgonzalezfluendo]

@mcesariniflu. This patch fixed the issue. But we need a more generic solution:

diff --git a/fluster/test.py b/fluster/test.py
index 6df333f..9932c89 100644
--- a/fluster/test.py
+++ b/fluster/test.py
@@ -66,12 +66,19 @@ class Test(unittest.TestCase):
 
         output_filepath = os.path.join(self.output_dir, self.test_vector.name + ".out")
 
-        input_filepath = os.path.join(
-            self.resources_dir,
-            self.test_suite.name,
-            self.test_vector.name,
-            self.test_vector.input_file,
-        )
+        if self.test_suite.name == "AV1_ARGON_VECTORS":
+            input_filepath = os.path.join(
+                self.resources_dir,
+                self.test_suite.name,
+                self.test_vector.input_file,
+            )
+        else:
+            input_filepath = os.path.join(
+                self.resources_dir,
+                self.test_suite.name,
+                self.test_vector.name,
+                self.test_vector.input_file,
+            )
         output_filepath = normalize_path(output_filepath)
         input_filepath = normalize_path(input_filepath)

[mdimopoulos]

Thanks for the bug report and the patch. We hadn't executed the test suite and missed this part of the code 🙈. Argon vectors for now are treated as a special case because they are all included in a single zip file.
An idea for future improvement is to split the test suite into various, e.g. based on the profile.

In any case i will apply the patch for now. Will update with the fix PR

[rgonzalezfluendo]

instead of doing:

fluster/test.py
69:        if self.test_suite.name == "AV1_ARGON_VECTORS":

fluster/test_suite.py
349:            if self.name != "AV1_ARGON_VECTORS":

we can create a TestSuite.single_archive attribute.

[mdimopoulos]

As far as i see in the code we have the following alternatives

• add a new json attribute during test suite generation and at runtime read it (this is clean but will break BC and we will have to regenerate all test suites)
• create a list of single archive test suite names in test_suite.py and use a conditional to assign the correct value to TestSuite.single_archive at run time (ugly due to adding yet another list in the code, does not break BC)

Any other ideas?
Any preferences?

[rgonzalezfluendo]

I prefer fist solution:

add a new json attribute during test suite generation and at runtime read it (this is clean but will break BC and we will have to regenerate all test suites)

No BC break if the attribute is optional with default value of False.

[ndufresne]

I think we'll also need a way to differentiate other AV1 content we have to far, which can be auto-discovred (gstreamer typefind) and this version that uses RAW OBU, which can't be type finded and need a different gstreamer pipeline.

[mdimopoulos]

PR: #220