Add a callback to get_parameter_data to follow data loading

[edumur]

Hi all,

In the effort to use qcodes as the data loader of the pyplotter I added a callback to get_parameter. This is usefull to track the progress of the data download.
Since sqlite3 does not allow to keep track of the data loading progress, we compute how many sqlite requests correspond to a certain percentage of progress which is dictated by a config parameter. Then we perform x sql request instead of one, running the callback everytime.

The whole process add an overhead of ~13% on my laptop when used which is not negligible.
However, I would argue that keeping track of the download progress for a GUI is precious and is worth the overhead.

To test the download time

import qcodes as qc
from time import time

def callback(progress):
    # print(progress)
    pass

qc.initialise_or_create_database_at('test.db')

start = time()
qc.load_by_id(2).get_parameter_data('phase')
stop = time()
dt1 = stop-start
print(dt1)
start = time()
qc.load_by_id(2).get_parameter_data('phase', callback=callback)
stop = time()
dt2 = stop-start
print(dt2)

print(dt2/dt1)

I kept the change minimal hoping other people will have comments.

[edumur]

Hi all,

I am writing to keep the thread and see if anyone has comments on it.

See you.

[edumur]

Hi all,

I am back again.
The thing is that this pull request will allow pyplotter to load the data via qcodes so I am eager to see it go through ^^' .

Hoping to see someone on the thread.

[jenshnielsen]

Sorry for the delay. I have been very busy with other things. I left some comments inline.

In general I am curious about the types of datasets that you are loading and the slowdown you are seeing in loading them. Ideally I would like it not to be an issue to load a dataset that makes it worthwhile to have a progress bar. Can you say something about the typical number of rows and parameters in your datasets?

[edumur]

test.zip
Please find the test database and the test file so that you can see the difference with and without callback.

[jenshnielsen]

Thanks starting to look ready. Could you resolve the conflicts either by merging or rebasing against master, revert the changes to many and many_many and write a small changelog note. Then it should be ready to go

[edumur]

All should be done.
I had trouble incorporating Florian Blanchet changes from commit b476b79, I hope everything is alright.

When I benchmarked the merged code I saw that:

1. SQL request where 2 to 3 times faster than before the merging (maybe an effect of Florian's work).
2. The overhead increased from 20% to 50% while using the callback.

[jenshnielsen]

@edumur Looks like some mypy issues have creapt in. will you be able to have a look?

[codecov]

Codecov Report

Merging #4688 (83c828e) into master (544295d) will increase coverage by 0.06%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master    #4688      +/-   ##
==========================================
+ Coverage   68.27%   68.33%   +0.06%     
==========================================
  Files         339      339              
  Lines       32087    32123      +36     
==========================================
+ Hits        21906    21952      +46     
+ Misses      10181    10171      -10     

[edumur]

I had to add callback: Callable[[float], None] | None = None, in get_parameter_data in data_set_in_memory.
But I do not know what I did here so please someone check.

[jenshnielsen]

The types looks good. Left a few comments inline. There is now a test.db file which should not be committed

[edumur]

Hi all,

All things seem correct (I think).
I do not understand the "Lint with Darker" error and I do not know what it is checking ^^'.

[jenshnielsen]

Darker runs the black formatter on only the part of the code that you have changed. We do this because we would like to move to a more consistent formatting but not rewrite all out code in one go losing history. You can run these lints automatically by installing the pre-commit hooks defined in https://github.com/QCoDeS/Qcodes/blob/master/.pre-commit-config.yaml using the pre-commit tool. https://pre-commit.com/ For now don't worry about it and I will run it when merging

[edumur]

I really appreciate you spending time checking this.
I updated the code so that it works now.

I tested it with various database and various percentage and so far it worked 🤞.