refactor *_user_test_runner.sh to run in containers

[mythi]

I'd like to be able to use these scripts in containers without patching in our end-to-end testing for Kubernetes.

Essentially for containers, the flow is that the node admin has configured the WQs, and the tests are run by passing --device /dev/dsa/wq0.0 etc.

[ramesh-thomas]

@xinzhanz can you please take a look?

[xinzhanz]

@mythi If remove cleanup, start_dsa, enable_wqs, this scripts won't work. The purpose is to provide a script which can show it runs well. I don't think it is good to configure DSA/WQs out of the runner.

[mythi]

@xinzhanz no need to remove them but make them run conditionally. one flow could be the user adds: ./dsa_user_test_runner.sh --device <dev>and if <dev> is set, no provisioning is done. Alternatively, just ./dsa_user_test_runner.sh --skip-config.

[xinzhanz]

I prefer --skip-config to skip enable/disable devices in scripts. So need to guarantee the wqs in /dev/dsa are all enabled devices for user.

[mythi]

Sounds OK. Maybe you can reuse this code from our patch:

-DSA=dsa0
+DEV=`ls /dev/dsa/ | sed -ne 's|wq\([^.]\+\)\(.*\)|dsa\1/wq\1\2|p'`
+DSA=`echo $DEV | cut -f1 -d/`

things to be aware of with containers:

1. the script cannot assume dsa0 (it could be any number), 2. only mounted devices are "visibile"

[xinzhanz]

v1-0001-accel-config-test-provice-skip-config-to-run-dsa-.patch
Could you have a try if it works for you?

[mythi]

@hj-johannes-lee can you help checking this patch as part of your work

[hj-johannes-lee]

@mythi @xinzhanz I successfully ran e2e test for dsa plugin that includes accel-config demo. And the demo is patched with what you provided. So, it works for us.!

[xinzhanz]

Great! I will make a patch for review. Thanks @hj-johannes-lee @mythi .

[mythi]

Great! I will make a patch for review. Thanks @hj-johannes-lee @mythi .

can we have the same for both IAA and DSA?

[zhangl6]

I think there is a potential issue about this implementation, use my local env. as an example,
I enabled 2 work queues,
[root@emr-bkc test]# ls /dev/iax/wq1.*
/dev/iax/wq1.1 /dev/iax/wq1.4
then define variables of DEV and IAA,
DEV=ls /dev/iax/ | sed -ne 's|wq\([^.]\+\)\(.*\)|iax\1/wq\1\2|p'
IAA=echo $DEV | cut -f1 -d/
now, the content of DEV is,
[root@emr-bkc test]# echo $DEV
iax1/wq1.1 iax1/wq1.4
then run iaa test as,
./iaa_test -w 0 -l 1048576 -o 0x44 -f 0x1 -1 0x8000 -t 5000 -v -d "$DEV"
in this case, only device iax1/wq1.1 can be run, iax1/wq1.4 is ignored.
so here need to loop in all enabled devices and work queues

In iaa_user_test_runner.sh, it's necessary to check whether an op is valid and different op has different command line parameters, so each op has its own commands. There are dozens of command lines leading by iaa_test need to be modified and each of them need to loop in all the enabled devices and work queues.

As a tool for common use, I think "--skip-config" and accepting enabled devices and work queues are not a good behave for the script iaa_user_test_runner.sh. How about you guys opinions?

[mythi]

There are dozens of command lines leading by iaa_test need to be modified and each of them need to loop in all the enabled devices and work queues.

our need is fairly simple: run by using the devices that available (in containers, can be a subset of what enabled on the host) but don't do any WQ (re-)configuration and don't assume any fixed WQ id. the tests scripts have proven to be useful because of their coverage but unfortunately requires patching to be useful in containerized environments. that's the reason for this issue.

[zhangl6]

There are dozens of command lines leading by iaa_test need to be modified and each of them need to loop in all the enabled devices and work queues.

our need is fairly simple: run by using the devices that available (in containers, can be a subset of what enabled on the host) but don't do any WQ (re-)configuration and don't assume any fixed WQ id. the tests scripts have proven to be useful because of their coverage but unfortunately requires patching to be useful in containerized environments. that's the reason for this issue.

Do you need a script that can enumerate and run on all enabled work queues? I mean, for example,
if /dev/iax/wq1.1 and /dev/iax/wq1.4 are enabled, the following 2 commands are all necessary for the operation 0x44 in the script?

1. iaa_test -w 0 -l 2097152 -o 0x44 -f 0x1 -1 0x8000 -t 5000 -v -d iax1/wq1.1
2. iaa_test -w 0 -l 2097152 -o 0x44 -f 0x1 -1 0x8000 -t 5000 -v -d iax1/wq1.4

[mythi]

Do you need a script that can enumerate and run on all enabled work queues? I mean, for example,

It's not important but as far as I can think of, this would be straightforward with a loop for all devices detected.

[ramesh-thomas]

What is the status of this issue? Is there any patch that needs to be merged?

[zhangl6]

v1-0001-accel-config-test-Add-parameter-for-specifying-de.patch
Hi @mythi ,
The patch is for selecting dev and wq in iaa_user_test_runner.sh, please help check it.
Thanks.

[mythi]

v1-0001-accel-config-test-Add-parameter-for-specifying-de.patch Hi @mythi , The patch is for selecting dev and wq in iaa_user_test_runner.sh, please help check it. Thanks.

@hj-johannes-lee would you be able to help?

[hj-johannes-lee]

@mythi Let me do today or at least by tomorrow.! :)

[hj-johannes-lee]

Sorry for late message. I was struggling with finding a system that has iaa and then with kernels.
I tested with 6.1.57 (stable-latest) kernel version and works fine with the plugin and accel-config app container after patching with the patch you provided.! Thanks for the work.!

[ramesh-thomas]

Fix added in https://github.com/intel/idxd-config/releases/tag/accel-config-v4.1.4

[mythi]

We did not test with all platforms. This is problematic still:

idxd-config/test/iaa_user_test_runner.sh

Lines 502 to 514 in 6c3261f

	if [ $((IAA_OPCODE_MASK_TRANSL_FETCH & OP_CAP0)) -ne 0 ]; then
	bind_vfio
	flag="0x1"
	echo "Testing with 'block on fault' flag ON"
	test_op_transl_fetch $IAA_OPCODE_TRANSL_FETCH $flag
	flag="0x0"
	echo "Testing with 'block on fault' flag OFF"
	test_op_transl_fetch $IAA_OPCODE_TRANSL_FETCH $flag
	unbind_vfio
	fi

[ramesh-thomas]

@zhangl6 can you please take a look?