Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Adding a total_iodepth option for librbdfio #330

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open

Adding a total_iodepth option for librbdfio #330

wants to merge 1 commit into from

Conversation

harriscr
Copy link
Contributor

@harriscr harriscr commented Mar 4, 2025

This is a continuation of PR 324

Description

Add an option to the CBT configuration yaml file called total_iodepth. The total_iodepth is then split evenly among the number of volumes used for the test. If the total_iodepth for a run does not divide evenly into the number of volumes, then any remainder will be assigned 1iodepth at a time to the volumes, starting at 0.

For (a simple) example:
For an total_iodepth of 18 and volumes_per_client of 5, the following iodepth allocations would occur:

volume iodepth
0 4
1 4
2 4
3 3
4 3

If the number of volumes specified is such that there is not enough iodepth for 1 per volume, then the number of volumes for that test will be reduced so that an iodepth of 1 per volume can be achieved.

Example:
For volumes_per_client = 5 and total_iodepth=4, the benchmark would be run with 4 volumes, each of iodepth 1

Testing

Manual testing was done for a number of scenarios, with and without using workloads The below are the output from debug statements added for the purposes of testing.

Regression testing: iodepth=32, volumes_per_client=3

CHDEBUG: fio_cmd for volume 0 is /usr/local/bin/fio --ioengine=rbd --clientname=admin --pool=cbt-librbdfio --rbdname=cbt-librbdfio-hostname -f-0 --invalidate=0 --rw=randwrite --output-format=json,normal --runtime=300 --numjobs=1 --direct=1 --bs=4096B --iodepth=32 --end_fsync=0 --norandommap --write_iops_log=/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-003/iodepth-032/randwrite/output.0 --write_bw_log=/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-003/iodepth-032/randwrite/output.0 --write_lat_log=/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-003/iodepth-032/randwrite/output.0 --log_avg_msec=101 --name=cbt-librbdfio-hostname -f-file-0 > /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-003/iodepth-032/randwrite/output.0

CHDEBUG: fio_cmd for volume 1 is /usr/local/bin/fio --ioengine=rbd --clientname=admin --pool=cbt-librbdfio --rbdname=cbt-librbdfio-hostname -f-1 --invalidate=0 --rw=randwrite --output-format=json,normal --runtime=300 --numjobs=1 --direct=1 --bs=4096B --iodepth=32 --end_fsync=0 --norandommap --write_iops_log=/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-003/iodepth-032/randwrite/output.1 --write_bw_log=/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-003/iodepth-032/randwrite/output.1 --write_lat_log=/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-003/iodepth-032/randwrite/output.1 --log_avg_msec=101 --name=cbt-librbdfio-hostname -f-file-0 > /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-003/iodepth-032/randwrite/output.1

CHDEBUG: fio_cmd for volume 2 is /usr/local/bin/fio --ioengine=rbd --clientname=admin --pool=cbt-librbdfio --rbdname=cbt-librbdfio-hostname -f-2 --invalidate=0 --rw=randwrite --output-format=json,normal --runtime=300 --numjobs=1 --direct=1 --bs=4096B --iodepth=32 --end_fsync=0 --norandommap --write_iops_log=/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-003/iodepth-032/randwrite/output.2 --write_bw_log=/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-003/iodepth-032/randwrite/output.2 --write_lat_log=/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-003/iodepth-032/randwrite/output.2 --log_avg_msec=101 --name=cbt-librbdfio-hostname -f-file-0 > /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-003/iodepth-032/randwrite/output.2

total_iodepth=32, volumes_per_client=3

CHDEBUG: fio_cmd for volume 0 is /usr/local/bin/fio --ioengine=rbd --clientname=admin --pool=cbt-librbdfio --rbdname=cbt-librbdfio-hostname -f-0 --invalidate=0 --rw=randwrite --output-format=json,normal --runtime=300 --numjobs=1 --direct=1 --bs=4096B --iodepth=11 --end_fsync=0 --norandommap --write_iops_log=/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-003/iodepth-016/randwrite/output.0 --write_bw_log=/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-003/iodepth-016/randwrite/output.0 --write_lat_log=/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-003/iodepth-016/randwrite/output.0 --log_avg_msec=101 --name=cbt-librbdfio-hostname -f-file-0 > /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-003/iodepth-016/randwrite/output.0

CHDEBUG: fio_cmd for volume 1 is /usr/local/bin/fio --ioengine=rbd --clientname=admin --pool=cbt-librbdfio --rbdname=cbt-librbdfio-hostname -f-1 --invalidate=0 --rw=randwrite --output-format=json,normal --runtime=300 --numjobs=1 --direct=1 --bs=4096B --iodepth=11 --end_fsync=0 --norandommap --write_iops_log=/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-003/iodepth-016/randwrite/output.1 --write_bw_log=/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-003/iodepth-016/randwrite/output.1 --write_lat_log=/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-003/iodepth-016/randwrite/output.1 --log_avg_msec=101 --name=cbt-librbdfio-hostname -f-file-0 > /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-003/iodepth-016/randwrite/output.1

CHDEBUG: fio_cmd for volume 2 is /usr/local/bin/fio --ioengine=rbd --clientname=admin --pool=cbt-librbdfio --rbdname=cbt-librbdfio-hostname -f-2 --invalidate=0 --rw=randwrite --output-format=json,normal --runtime=300 --numjobs=1 --direct=1 --bs=4096B --iodepth=10 --end_fsync=0 --norandommap --write_iops_log=/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-003/iodepth-016/randwrite/output.2 --write_bw_log=/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-003/iodepth-016/randwrite/output.2 --write_lat_log=/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-003/iodepth-016/randwrite/output.2 --log_avg_msec=101 --name=cbt-librbdfio-hostname -f-file-0 > /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-003/iodepth-016/randwrite/output.2

total_iodepth=2, volumes_per_client=3

11:13:28 - WARNING - cbt - The total iodepth requested: 2 is less than 1 per volume.
11:13:28 - WARNING - cbt - Number of volumes per client will be reduced from 3 to 2

CHDEBUG: fio_cmd for volume 0 is /usr/local/bin/fio --ioengine=rbd --clientname=admin --pool=cbt-librbdfio --rbdname=cbt-librbdfio-hostname -f-0 --invalidate=0 --rw=randwrite --output-format=json,normal --runtime=300 --numjobs=1 --direct=1 --bs=4096B --iodepth=1 --end_fsync=0 --norandommap --write_iops_log=/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-016/randwrite/output.0 --write_bw_log=/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-016/randwrite/output.0 --write_lat_log=/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-016/randwrite/output.0 --log_avg_msec=101 --name=cbt-librbdfio-hostname -f-file-0 > /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-016/randwrite/output.0

CHDEBUG: fio_cmd for volume 1 is /usr/local/bin/fio --ioengine=rbd --clientname=admin --pool=cbt-librbdfio --rbdname=cbt-librbdfio-hostname -f-1 --invalidate=0 --rw=randwrite --output-format=json,normal --runtime=300 --numjobs=1 --direct=1 --bs=4096B --iodepth=1 --end_fsync=0 --norandommap --write_iops_log=/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-016/randwrite/output.1 --write_bw_log=/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-016/randwrite/output.1 --write_lat_log=/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-016/randwrite/output.1 --log_avg_msec=101 --name=cbt-librbdfio-hostname -f-file-0 > /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-016/randwrite/output.1

I'll update with teuthology results once they have run

@harriscr
Copy link
Contributor Author

harriscr commented Mar 4, 2025

python unit tests:

============================= slowest 5 durations ==============================
0.01s setup tests/test_bm_kvmrbdfio.py::TestBenchmarkkvmrbdfio::test_valid_archive_dir
0.01s setup tests/test_bm_nullbench.py::TestBenchmarknullbench::test_valid_archive_dir
0.01s setup tests/test_bm_rawfio.py::TestBenchmarkrawfio::test_valid_archive_dir
0.01s setup tests/test_bm_radosbench.py::TestBenchmarkradosbench::test_valid_archive_dir
0.01s setup tests/test_bm_fio.py::TestBenchmarkfio::test_valid_archive_dir
======================== 326 passed, 3 skipped in 0.47s ========================
Finished running tests!

I haven't run black, ruff or mypy against the file to check for pep-8 compliance as the unchanged file gives too many errors currently

@harriscr
Copy link
Contributor Author

harriscr commented Mar 4, 2025

Testing with workloads:

precondition: iodepth=[2]

13:45:31 - INFO - cbt - Running rbd fio precondition test, mode randwrite
13:45:31 - WARNING - cbt - CHDEBUG: Using iodepth

13:45:31 - INFO - cbt - CHDEBUG: fio command for iodepth 2 and vol 0 is
sudo /usr/local/bin/fio --ioengine=rbd --clientname=admin --pool=rbd_replicated --rbdname=cbt-librbdfio-hostname -f-0 --invalidate=0 --rw=randwrite --output-format=json --runtime=60 --time_based --ramp_time=30 --numjobs=1 --direct=1 --bs=65536B --iodepth=2 --end_fsync=0 --norandommap --write_iops_log=/tmp/cbt/00000000/LibrbdFio/randwrite_65536/iodepth-002/numjobs-001/output.0 --write_bw_log=/tmp/cbt/00000000/LibrbdFio/randwrite_65536/iodepth-002/numjobs-001/output.0 --write_lat_log=/tmp/cbt/00000000/LibrbdFio/randwrite_65536/iodepth-002/numjobs-001/output.0 --log_avg_msec=100 --name=cbt-librbdfio-hostname -f-file-0 > /tmp/cbt/00000000/LibrbdFio/randwrite_65536/iodepth-002/numjobs-001/output.0

13:45:31 - INFO - cbt - CHDEBUG: fio command for iodepth 2 and vol 1 is
sudo /usr/local/bin/fio --ioengine=rbd --clientname=admin --pool=rbd_replicated --rbdname=cbt-librbdfio-hostname -f-1 --invalidate=0 --rw=randwrite --output-format=json --runtime=60 --time_based --ramp_time=30 --numjobs=1 --direct=1 --bs=65536B --iodepth=2 --end_fsync=0 --norandommap --write_iops_log=/tmp/cbt/00000000/LibrbdFio/randwrite_65536/iodepth-002/numjobs-001/output.1 --write_bw_log=/tmp/cbt/00000000/LibrbdFio/randwrite_65536/iodepth-002/numjobs-001/output.1 --write_lat_log=/tmp/cbt/00000000/LibrbdFio/randwrite_65536/iodepth-002/numjobs-001/output.1 --log_avg_msec=100 --name=cbt-librbdfio-hostname -f-file-0 > /tmp/cbt/00000000/LibrbdFio/randwrite_65536/iodepth-002/numjobs-001/output.1

13:45:31 - INFO - cbt - CHDEBUG: fio command for iodepth 2 and vol 7 is
sudo /usr/local/bin/fio --ioengine=rbd --clientname=admin --pool=rbd_replicated --rbdname=cbt-librbdfio-hostname -f-7 --invalidate=0 --rw=randwrite --output-format=json --runtime=60 --time_based --ramp_time=30 --numjobs=1 --direct=1 --bs=65536B --iodepth=2 --end_fsync=0 --norandommap --write_iops_log=/tmp/cbt/00000000/LibrbdFio/randwrite_65536/iodepth-002/numjobs-001/output.7 --write_bw_log=/tmp/cbt/00000000/LibrbdFio/randwrite_65536/iodepth-002/numjobs-001/output.7 --write_lat_log=/tmp/cbt/00000000/LibrbdFio/randwrite_65536/iodepth-002/numjobs-001/output.7 --log_avg_msec=100 --name=cbt-librbdfio-hostname -f-file-0 > /tmp/cbt/00000000/LibrbdFio/randwrite_65536/iodepth-002/numjobs-001/output.7

The procondition runs on all 8 vlumes with an iodepth of 2, as expected.

total_iodepth=16, volumes=8

13:45:31 - INFO - cbt - Running rbd fio seq32kwrite test, mode write
13:45:31 - WARNING - cbt - CHDEBUG: Using total_iodepth

13:45:31 - INFO - cbt - CHDEBUG: fio command for iodepth 16 and vol 0 is
sudo /usr/local/bin/fio --ioengine=rbd --clientname=admin --pool=rbd_replicated --rbdname=cbt-librbdfio-hostname -f-0 --invalidate=0 --rw=write --output-format=json --runtime=60 --time_based --ramp_time=30 --numjobs=1 --direct=1 --bs=32768B --iodepth=2 --end_fsync=0 --norandommap --write_iops_log=/tmp/cbt/00000000/LibrbdFio/write_32768/iodepth-016/numjobs-001/output.0 --write_bw_log=/tmp/cbt/00000000/LibrbdFio/write_32768/iodepth-016/numjobs-001/output.0 --write_lat_log=/tmp/cbt/00000000/LibrbdFio/write_32768/iodepth-016/numjobs-001/output.0 --log_avg_msec=100 --name=cbt-librbdfio-hostname -f-file-0 > /tmp/cbt/00000000/LibrbdFio/write_32768/iodepth-016/numjobs-001/output.0

13:45:31 - INFO - cbt - CHDEBUG: fio command for iodepth 16 and vol 1 is
sudo /usr/local/bin/fio --ioengine=rbd --clientname=admin --pool=rbd_replicated --rbdname=cbt-librbdfio-hostname -f-1 --invalidate=0 --rw=write --output-format=json --runtime=60 --time_based --ramp_time=30 --numjobs=1 --direct=1 --bs=32768B --iodepth=2 --end_fsync=0 --norandommap --write_iops_log=/tmp/cbt/00000000/LibrbdFio/write_32768/iodepth-016/numjobs-001/output.1 --write_bw_log=/tmp/cbt/00000000/LibrbdFio/write_32768/iodepth-016/numjobs-001/output.1 --write_lat_log=/tmp/cbt/00000000/LibrbdFio/write_32768/iodepth-016/numjobs-001/output.1 --log_avg_msec=100 --name=cbt-librbdfio-hostname -f-file-0 > /tmp/cbt/00000000/LibrbdFio/write_32768/iodepth-016/numjobs-001/output.1

total_iodepth=7, volumes=8

13:45:32 - WARNING - cbt - The total iodepth requested: 7 is less than 1 per volume (8)
13:45:32 - WARNING - cbt - Number of volumes per client will be reduced from 8 to 7

13:45:32 - INFO - cbt - CHDEBUG: fio command for iodepth 7 and vol 0 is
sudo /usr/local/bin/fio --ioengine=rbd --clientname=admin --pool=rbd_replicated --rbdname=cbt-librbdfio-hostname -f-0 --invalidate=0 --rw=write --output-format=json --runtime=60 --time_based --ramp_time=30 --numjobs=1 --direct=1 --bs=32768B --iodepth=1 --end_fsync=0 --norandommap --write_iops_log=/tmp/cbt/00000000/LibrbdFio/write_32768/iodepth-007/numjobs-001/output.0 --write_bw_log=/tmp/cbt/00000000/LibrbdFio/write_32768/iodepth-007/numjobs-001/output.0 --write_lat_log=/tmp/cbt/00000000/LibrbdFio/write_32768/iodepth-007/numjobs-001/output.0 --log_avg_msec=100 --name=cbt-librbdfio-hostname -f-file-0 > /tmp/cbt/00000000/LibrbdFio/write_32768/iodepth-007/numjobs-001/output.0

@harriscr
Copy link
Contributor Author

harriscr commented Mar 4, 2025

Farther testing:

Using the following yaml (workload section below:

workloads:
      precondition:
        jobname: 'precond1rw'
        mode: 'randwrite'
        precond: True
        op_size: 65536
        numjobs: [ 1 ]
        iodepth: [ 2 ]
        monitor: False # whether to run the monitors along the test

      seq32kwrite:
        jobname: 'seqwrite'
        mode: 'write'
        op_size: 32768
        numjobs: [ 1 ]
        total_iodepth: [ 1, 2, 24, 36]
        pre_workload_script: "/home/harriscr/test.sh"
      seq32kread:
        jobname: 'seqread'
        mode: 'read'
        op_size: 32768
        numjobs: [ 1 ]
        iodepth: [ 4 ]

gives (extra debug added to show values used:

10:36:37 - INFO     - cbt      - Running rbd fio precondition test, mode randwrite
10:36:37 - INFO     - cbt      - CHDEBUG: staring on job 1
10:36:37 - INFO     - cbt      - CHDEBUG: iodepth_key is iodepth
10:36:37 - INFO     - cbt      - CHDEBUG: iodepth per volume is: {0: 2, 1: 2, 2: 2, 3: 2, 4: 2, 5: 2, 6: 2, 7: 2}

10:42:17 - INFO     - cbt      - CHDEBUG: fio command is sudo /usr/local/bin/fio --ioengine=rbd --clientname=admin --pool=rbd_replicated --rbdname=cbt-librbdfio-`hostname -f`-0 --invalidate=0 --rw=randwrite --output-format=json --runtime=90 --time_based --ramp_time=30 --numjobs=1 --direct=1 --bs=65536B --iodepth=2 --end_fsync=0 --norandommap --write_iops_log=/tmp/cbt/00000000/LibrbdFio/randwrite_65536/iodepth-002/numjobs-001/output.0 --write_bw_log=/tmp/cbt/00000000/LibrbdFio/randwrite_65536/iodepth-002/numjobs-001/output.0 --write_lat_log=/tmp/cbt/00000000/LibrbdFio/randwrite_65536/iodepth-002/numjobs-001/output.0 --log_avg_msec=100 --name=cbt-librbdfio-`hostname -f`-file-0  > /tmp/cbt/00000000/LibrbdFio/randwrite_65536/iodepth-002/numjobs-001/output.0

10:42:17 - INFO     - cbt      - CHDEBUG: fio command is sudo /usr/local/bin/fio --ioengine=rbd --clientname=admin --pool=rbd_replicated --rbdname=cbt-librbdfio-`hostname -f`-1 --invalidate=0 --rw=randwrite --output-format=json --runtime=90 --time_based --ramp_time=30 --numjobs=1 --direct=1 --bs=65536B --iodepth=2 --end_fsync=0 --norandommap --write_iops_log=/tmp/cbt/00000000/LibrbdFio/randwrite_65536/iodepth-002/numjobs-001/output.1 --write_bw_log=/tmp/cbt/00000000/LibrbdFio/randwrite_65536/iodepth-002/numjobs-001/output.1 --write_lat_log=/tmp/cbt/00000000/LibrbdFio/randwrite_65536/iodepth-002/numjobs-001/output.1 --log_avg_msec=100 --name=cbt-librbdfio-`hostname -f`-file-0  > /tmp/cbt/00000000/LibrbdFio/randwrite_65536/iodepth-002/numjobs-001/output.1


10:36:37 - INFO     - cbt      - Running rbd fio seq32kwrite test, mode write
10:36:37 - INFO     - cbt      - CHDEBUG: staring on job 1
10:36:37 - INFO     - cbt      - CHDEBUG: iodepth_key is total_iodepth
10:36:37 - WARNING  - cbt      - The total iodepth requested: 1 is less than 1 per volume (8)
10:36:37 - WARNING  - cbt      - Number of volumes per client will be reduced from 8 to 1
10:36:37 - INFO     - cbt      - CHDEBUG: iodepth per volume is: {0: 1}

10:42:17 - INFO     - cbt      - CHDEBUG: fio command is sudo /usr/local/bin/fio --ioengine=rbd --clientname=admin --pool=rbd_replicated --rbdname=cbt-librbdfio-`hostname -f`-0 --invalidate=0 --rw=write --output-format=json --runtime=90 --time_based --ramp_time=30 --numjobs=1 --direct=1 --bs=32768B --iodepth=1 --end_fsync=0 --norandommap --write_iops_log=/tmp/cbt/00000000/LibrbdFio/write_32768/iodepth-001/numjobs-001/output.0 --write_bw_log=/tmp/cbt/00000000/LibrbdFio/write_32768/iodepth-001/numjobs-001/output.0 --write_lat_log=/tmp/cbt/00000000/LibrbdFio/write_32768/iodepth-001/numjobs-001/output.0 --log_avg_msec=100 --name=cbt-librbdfio-`hostname -f`-file-0  > /tmp/cbt/00000000/LibrbdFio/write_32768/iodepth-001/numjobs-001/output.0

10:36:37 - WARNING  - cbt      - The total iodepth requested: 2 is less than 1 per volume (8)
10:36:37 - WARNING  - cbt      - Number of volumes per client will be reduced from 8 to 2
10:36:37 - INFO     - cbt      - CHDEBUG: iodepth per volume is: {0: 1, 1: 1}

10:36:37 - INFO     - cbt      - CHDEBUG: iodepth per volume is: {0: 3, 1: 3, 2: 3, 3: 3, 4: 3, 5: 3, 6: 3, 7: 3}

10:42:17 - INFO     - cbt      - CHDEBUG: fio command is sudo /usr/local/bin/fio --ioengine=rbd --clientname=admin --pool=rbd_replicated --rbdname=cbt-librbdfio-`hostname -f`-0 --invalidate=0 --rw=write --output-format=json --runtime=90 --time_based --ramp_time=30 --numjobs=1 --direct=1 --bs=32768B --iodepth=3 --end_fsync=0 --norandommap --write_iops_log=/tmp/cbt/00000000/LibrbdFio/write_32768/iodepth-024/numjobs-001/output.0 --write_bw_log=/tmp/cbt/00000000/LibrbdFio/write_32768/iodepth-024/numjobs-001/output.0 --write_lat_log=/tmp/cbt/00000000/LibrbdFio/write_32768/iodepth-024/numjobs-001/output.0 --log_avg_msec=100 --name=cbt-librbdfio-`hostname -f`-file-0  > /tmp/cbt/00000000/LibrbdFio/write_32768/iodepth-024/numjobs-001/output.0

10:42:17 - INFO     - cbt      - CHDEBUG: fio command is sudo /usr/local/bin/fio --ioengine=rbd --clientname=admin --pool=rbd_replicated --rbdname=cbt-librbdfio-`hostname -f`-1 --invalidate=0 --rw=write --output-format=json --runtime=90 --time_based --ramp_time=30 --numjobs=1 --direct=1 --bs=32768B --iodepth=3 --end_fsync=0 --norandommap --write_iops_log=/tmp/cbt/00000000/LibrbdFio/write_32768/iodepth-024/numjobs-001/output.1 --write_bw_log=/tmp/cbt/00000000/LibrbdFio/write_32768/iodepth-024/numjobs-001/output.1 --write_lat_log=/tmp/cbt/00000000/LibrbdFio/write_32768/iodepth-024/numjobs-001/output.1 --log_avg_msec=100 --name=cbt-librbdfio-`hostname -f`-file-0  > /tmp/cbt/00000000/LibrbdFio/write_32768/iodepth-024/numjobs-001/output.1


10:36:37 - INFO     - cbt      - CHDEBUG: iodepth per volume is: {0: 5, 1: 5, 2: 5, 3: 5, 4: 4, 5: 4, 6: 4, 7: 4}

10:42:17 - INFO     - cbt      - CHDEBUG: fio command is sudo /usr/local/bin/fio --ioengine=rbd --clientname=admin --pool=rbd_replicated --rbdname=cbt-librbdfio-`hostname -f`-1 --invalidate=0 --rw=write --output-format=json --runtime=90 --time_based --ramp_time=30 --numjobs=1 --direct=1 --bs=32768B --iodepth=5 --end_fsync=0 --norandommap --write_iops_log=/tmp/cbt/00000000/LibrbdFio/write_32768/iodepth-036/numjobs-001/output.1 --write_bw_log=/tmp/cbt/00000000/LibrbdFio/write_32768/iodepth-036/numjobs-001/output.1 --write_lat_log=/tmp/cbt/00000000/LibrbdFio/write_32768/iodepth-036/numjobs-001/output.1 --log_avg_msec=100 --name=cbt-librbdfio-`hostname -f`-file-0  > /tmp/cbt/00000000/LibrbdFio/write_32768/iodepth-036/numjobs-001/output.1

10:42:17 - INFO     - cbt      - CHDEBUG: fio command is sudo /usr/local/bin/fio --ioengine=rbd --clientname=admin --pool=rbd_replicated --rbdname=cbt-librbdfio-`hostname -f`-7 --invalidate=0 --rw=write --output-format=json --runtime=90 --time_based --ramp_time=30 --numjobs=1 --direct=1 --bs=32768B --iodepth=4 --end_fsync=0 --norandommap --write_iops_log=/tmp/cbt/00000000/LibrbdFio/write_32768/iodepth-036/numjobs-001/output.7 --write_bw_log=/tmp/cbt/00000000/LibrbdFio/write_32768/iodepth-036/numjobs-001/output.7 --write_lat_log=/tmp/cbt/00000000/LibrbdFio/write_32768/iodepth-036/numjobs-001/output.7 --log_avg_msec=100 --name=cbt-librbdfio-`hostname -f`-file-0  > /tmp/cbt/00000000/LibrbdFio/write_32768/iodepth-036/numjobs-001/output.7


10:36:37 - INFO     - cbt      - Running rbd fio seq32kread test, mode read
10:36:37 - INFO     - cbt      - CHDEBUG: staring on job 1
10:36:37 - INFO     - cbt      - CHDEBUG: iodepth_key is iodepth
10:36:37 - INFO     - cbt      - CHDEBUG: iodepth per volume is: {0: 4, 1: 4, 2: 4, 3: 4, 4: 4, 5: 4, 6: 4, 7: 4}

10:42:17 - INFO     - cbt      - CHDEBUG: fio command is sudo /usr/local/bin/fio --ioengine=rbd --clientname=admin --pool=rbd_replicated --rbdname=cbt-librbdfio-`hostname -f`-0 --invalidate=0 --rw=read --output-format=json --runtime=90 --time_based --ramp_time=30 --numjobs=1 --direct=1 --bs=32768B --iodepth=4 --end_fsync=0 --norandommap --write_iops_log=/tmp/cbt/00000000/LibrbdFio/read_32768/iodepth-004/numjobs-001/output.0 --write_bw_log=/tmp/cbt/00000000/LibrbdFio/read_32768/iodepth-004/numjobs-001/output.0 --write_lat_log=/tmp/cbt/00000000/LibrbdFio/read_32768/iodepth-004/numjobs-001/output.0 --log_avg_msec=100 --name=cbt-librbdfio-`hostname -f`-file-0  > /tmp/cbt/00000000/LibrbdFio/read_32768/iodepth-004/numjobs-001/output.0

10:36:37 - INFO     - cbt      - == Workloads completed ==

Some farther testing to make sure that the iodepth value used in the results directory structure is the total iodepth value or iodepth value, depending on which one is set.

total_iodepth - no workload
11:12:27 - INFO     - cbt      - CHDEBUG: fio command is sudo /usr/local/bin/fio --ioengine=rbd --clientname=admin --pool=rbd_replicated --rbdname=cbt-librbdfio-`hostname -f`-3 --invalidate=0 --rw=write --output-format=json --runtime=90 --time_based --ramp_time=30 --numjobs=1 --direct=1 --bs=4194304B --iodepth=1 --end_fsync=0 --norandommap --write_iops_log=/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-008/iodepth-007/write/output.3 --write_bw_log=/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-008/iodepth-007/write/output.3 --write_lat_log=/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-008/iodepth-007/write/output.3 --log_avg_msec=100 --name=cbt-librbdfio-`hostname -f`-file-0  > /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-008/iodepth-007/write/output.3

11:12:27 - INFO     - cbt      - CHDEBUG: fio command is sudo /usr/local/bin/fio --ioengine=rbd --clientname=admin --pool=rbd_replicated --rbdname=cbt-librbdfio-`hostname -f`-6 --invalidate=0 --rw=write --output-format=json --runtime=90 --time_based --ramp_time=30 --numjobs=1 --direct=1 --bs=4194304B --iodepth=1 --end_fsync=0 --norandommap --write_iops_log=/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-008/iodepth-007/write/output.6 --write_bw_log=/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-008/iodepth-007/write/output.6 --write_lat_log=/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-008/iodepth-007/write/output.6 --log_avg_msec=100 --name=cbt-librbdfio-`hostname -f`-file-0  > /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-008/iodepth-007/write/output.6

11:12:32 - INFO     - cbt      - CHDEBUG: fio command is sudo /usr/local/bin/fio --ioengine=rbd --clientname=admin --pool=rbd_replicated --rbdname=cbt-librbdfio-`hostname -f`-0 --invalidate=0 --rw=write --output-format=json --runtime=90 --time_based --ramp_time=30 --numjobs=1 --direct=1 --bs=4194304B --iodepth=128 --end_fsync=0 --norandommap --write_iops_log=/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-008/iodepth-1024/write/output.0 --write_bw_log=/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-008/iodepth-1024/write/output.0 --write_lat_log=/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-008/iodepth-1024/write/output.0 --log_avg_msec=100 --name=cbt-librbdfio-`hostname -f`-file-0  > /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-008/iodepth-1024/write/output.0

11:12:32 - INFO     - cbt      - CHDEBUG: fio command is sudo /usr/local/bin/fio --ioengine=rbd --clientname=admin --pool=rbd_replicated --rbdname=cbt-librbdfio-`hostname -f`-7 --invalidate=0 --rw=write --output-format=json --runtime=90 --time_based --ramp_time=30 --numjobs=1 --direct=1 --bs=4194304B --iodepth=128 --end_fsync=0 --norandommap --write_iops_log=/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-008/iodepth-1024/write/output.7 --write_bw_log=/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-008/iodepth-1024/write/output.7 --write_lat_log=/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-008/iodepth-1024/write/output.7 --log_avg_msec=100 --name=cbt-librbdfio-`hostname -f`-file-0  > /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-008/iodepth-1024/write/output.7


iodepth - no workload
11:14:43 - INFO     - cbt      - CHDEBUG: fio command is sudo /usr/local/bin/fio --ioengine=rbd --clientname=admin --pool=rbd_replicated --rbdname=cbt-librbdfio-`hostname -f`-0 --invalidate=0 --rw=write --output-format=json --runtime=90 --time_based --ramp_time=30 --numjobs=1 --direct=1 --bs=4194304B --iodepth=7 --end_fsync=0 --norandommap --write_iops_log=/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-008/iodepth-007/write/output.0 --write_bw_log=/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-008/iodepth-007/write/output.0 --write_lat_log=/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-008/iodepth-007/write/output.0 --log_avg_msec=100 --name=cbt-librbdfio-`hostname -f`-file-0  > /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-008/iodepth-007/write/output.0

11:14:43 - INFO     - cbt      - CHDEBUG: fio command is sudo /usr/local/bin/fio --ioengine=rbd --clientname=admin --pool=rbd_replicated --rbdname=cbt-librbdfio-`hostname -f`-7 --invalidate=0 --rw=write --output-format=json --runtime=90 --time_based --ramp_time=30 --numjobs=1 --direct=1 --bs=4194304B --iodepth=7 --end_fsync=0 --norandommap --write_iops_log=/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-008/iodepth-007/write/output.7 --write_bw_log=/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-008/iodepth-007/write/output.7 --write_lat_log=/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-008/iodepth-007/write/output.7 --log_avg_msec=100 --name=cbt-librbdfio-`hostname -f`-file-0  > /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-008/iodepth-007/write/output.7


workload - total iodepth
11:16:31 - INFO     - cbt      - CHDEBUG: workload fio command is sudo /usr/local/bin/fio --ioengine=rbd --clientname=admin --pool=rbd_replicated --rbdname=cbt-librbdfio-`hostname -f`-7 --invalidate=0 --rw=write --output-format=json --runtime=90 --time_based --ramp_time=30 --numjobs=1 --direct=1 --bs=32768B --iodepth=4 --end_fsync=0 --norandommap --write_iops_log=/tmp/cbt/00000000/LibrbdFio/write_32768/iodepth-036/numjobs-001/output.7 --write_bw_log=/tmp/cbt/00000000/LibrbdFio/write_32768/iodepth-036/numjobs-001/output.7 --write_lat_log=/tmp/cbt/00000000/LibrbdFio/write_32768/iodepth-036/numjobs-001/output.7 --log_avg_msec=100 --name=cbt-librbdfio-`hostname -f`-file-0  > /tmp/cbt/00000000/LibrbdFio/write_32768/iodepth-036/numjobs-001/output.7


workload - iodepth
11:16:31 - INFO     - cbt      - CHDEBUG: workload fio command is sudo /usr/local/bin/fio --ioengine=rbd --clientname=admin --pool=rbd_replicated --rbdname=cbt-librbdfio-`hostname -f`-2 --invalidate=0 --rw=read --output-format=json --runtime=90 --time_based --ramp_time=30 --numjobs=1 --direct=1 --bs=32768B --iodepth=4 --end_fsync=0 --norandommap --write_iops_log=/tmp/cbt/00000000/LibrbdFio/read_32768/iodepth-004/numjobs-001/output.2 --write_bw_log=/tmp/cbt/00000000/LibrbdFio/read_32768/iodepth-004/numjobs-001/output.2 --write_lat_log=/tmp/cbt/00000000/LibrbdFio/read_32768/iodepth-004/numjobs-001/output.2 --log_avg_msec=100 --name=cbt-librbdfio-`hostname -f`-file-0  > /tmp/cbt/00000000/LibrbdFio/read_32768/iodepth-004/numjobs-001/output.2

which is as expected

@harriscr harriscr mentioned this pull request Mar 4, 2025
Closed
perezjosibm
perezjosibm requested changes Mar 5, 2025
View reviewed changes
Copy link
Contributor

@perezjosibm perezjosibm left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggestions added to improve readability and maintanability

benchmark/librbdfio.py
@@ -50,6 +49,13 @@ def __init__(self, archive_dir, cluster, config):
self.rate_iops = config.get('rate_iops', None)
self.fio_out_format = config.get('fio_out_format', 'json,normal')
self.data_pool = None

iodepth_key: str = self._get_iodepth_key(config.keys()) # type: ignore[arg-type]
self.iodepth: str = config.get(iodepth_key, "16")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do you prefer to get a string rather than a numeric value? How do you validate that is a valid iodepth?

benchmark/librbdfio.py
iodepth_key: str = self._get_iodepth_key(config.keys()) # type: ignore[arg-type]
self.iodepth: str = config.get(iodepth_key, "16")
self._iodepth_per_volume: dict[int, int] = self._calculate_iodepth_per_volume(
self.volumes_per_client, int(self.iodepth), iodepth_key
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What happens if self.iodepth is not a valid numeric string? Do we catch an exception somewhere? Notice how we use the config.get() method from the .yaml, with an alternative initial numeric value if it does not convert to a valid value.

benchmark/librbdfio.py

return iodepth_key

def _calculate_iodepth_per_volume(self, number_of_volumes: int, iodepth: int, iodepth_key: str) -> dict[int, int]:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Probably best to update the member attribute within the method, rather than returning a dict that is only consumed by such an attribute. Furthermore, such method can be simplified to expect only the two arguments that are not in self already.

benchmark/librbdfio.py
"""
iodepth_key: str = "iodepth"
if "total_iodepth" in configuration_keys:
iodepth_key = "total_iodepth"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You could use a dictionary to give that key instead

benchmark/librbdfio.py
@@ -234,7 +245,8 @@ def run(self):
monitoring.start(self.run_dir)
logger.info('Running rbd fio %s test.', self.mode)
ps = []
for i in range(self.volumes_per_client):
number_of_volumes: int = len(self._iodepth_per_volume.keys())
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure why to have the iodepth as string, then convert them back to numeric? Not see the value

benchmark/librbdfio.py
return self._calculate_iodepth_per_volume_from_total_iodepth(number_of_volumes, iodepth)
else:
return self._set_iodepth_for_every_volume(number_of_volumes, iodepth)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This can also be further simplified (and probably generalised) using a simple dictionary: if you have a single volume, you can have a single iodepth (which is the same as total_iodepth), otherwise calculate the iodepth per volume (which can either be an arryay, or a dict: vol->iodepth).
And that's it, no need to several cases that obscure the logic.

benchmark/librbdfio.py
if remainder > 0:
iodepth += 1
remainder -= 1
queue_depths[volume_id] = iodepth
Copy link
Contributor

@perezjosibm perezjosibm Mar 5, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, that's what I mean in the previous comment. queue_depths[] can by an attribute of this object instance (eg. self.vol_queue_depths). This idea can be generalised further: you can have a dict (or any collection object that allows iterators) to have several options per volume, besides iodepth, controlled by a single global flag. In this way you can have all volumes with the same iodepth, or distributed as per your requirement, etc.
You might also need to run update the unit test generator to pick up this new attribute though.

benchmark/librbdfio.py

return queue_depths
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd be inclined to this be an attribute (eg. self.vol_queue_depths) as suggested above, so this method is no longer a function but an update member method.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@perezjosibm perezjosibm perezjosibm requested changes

@markhpc markhpc Awaiting requested review from markhpc

@lee-j-sanders lee-j-sanders Awaiting requested review from lee-j-sanders

Assignees
No one assigned
Labels
enhancement
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@harriscr @perezjosibm