Slow counter overhaul #555

Neverhorst · 2019-12-04T14:17:27Z

Description

Added a whole new tool chain (GUI, logic, interface, hardware) for "slow counting".

In the process I also deleted some testing/template modules that somehow made their way into master... but should not have...

Motivation and Context

The current time series counter is very limited regarding data throughput and mixed signal capabilities.
For high data rates it will crash the NI card and dependent modules.
Also the oversampling and averaging is very inconsistent.
I implemented a wholesome replacement for "slow counting" by using the new modules TimeSeriesGui, TimeSeriesReaderLogic and NIXSeriesInStreamer.
The logic and hardware interface are more versatile and should be suited for any use case where continuous data streaming from a buffered device with variable read size is needed.
Also the counter/time trace GUI offers settings to switch channels physically and/or virtually on and off.

The old counter modules are still present but loading the respective GUI will cause a deprecation warning to pop up in order to encourage users to switch to the new modules.
I also intend to include the pop-up notification as simple tool in core... but in a different PR.

How Has This Been Tested?

On Win8.1 x64 with NI USB-6343 hardware and also with new dummy module.

Types of changes

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to not work as expected)

Checklist:

My code follows the code style of this project.
I have documented my changes in the changelog (documentation/changelog.md)
My change requires a change to the documentation.
I have updated the documentation accordingly.
I have added/updated for the module the config example in the docstring of the class accordingly.
I have checked that the change does not contain obvious errors (syntax, indentation, mutable default values).
I have tested my changes using 'Load all modules' on the default dummy configuration with my changes included.
All changed Jupyter notebooks have been stripped of their output cells.

…also with proper oversampling etc.

…phs accordingly.

# Conflicts: # logic/counter_logic.py

Fixed a bug with the data_rate StatusVar

…e its way into master

Added deprecation pop-up notification to the end of CounterGui activation to encourage users to switch to time series tool.

kay-jahnke

The follwoing error appears when the timeseries counter is running and the python hangs (e.g. due to other hardware using all resources on the computer). I have to confess I do not understand it.
The error appeared while testing on the mesoscopic setup with high data acquisition rates. The hanging I provoked by sampling a long sequence on the pulser or restarting the magnet control (in its own process).

Traceback (most recent call last):
  File "C:\Software\qudi\gui\time_series\time_series_gui.py", line 461, in update_data
    self._vb.autoRange()
  File "C:\ProgramData\Anaconda3\envs\qudi\lib\site-packages\pyqtgraph\graphicsItems\ViewBox\ViewBox.py", line 592, in autoRange
    bounds = self.childrenBoundingRect(items=items)
  File "C:\ProgramData\Anaconda3\envs\qudi\lib\site-packages\pyqtgraph\graphicsItems\ViewBox\ViewBox.py", line 1375, in childrenBoundingRect
    range = self.childrenBounds(*args, **kwds)
  File "C:\ProgramData\Anaconda3\envs\qudi\lib\site-packages\pyqtgraph\graphicsItems\ViewBox\ViewBox.py", line 1285, in childrenBounds
    px, py = [v.length() if v is not None else 0 for v in self.childGroup.pixelVectors()]
  File "C:\ProgramData\Anaconda3\envs\qudi\lib\site-packages\pyqtgraph\graphicsItems\GraphicsItem.py", line 252, in pixelVectors
    dti = fn.invertQTransform(dt)
  File "C:\ProgramData\Anaconda3\envs\qudi\lib\site-packages\pyqtgraph\functions.py", line 2306, in invertQTransform
    inv = numpy.linalg.inv(arr)
  File "C:\ProgramData\Anaconda3\envs\qudi\lib\site-packages\numpy\linalg\linalg.py", line 528, in inv
    ainv = _umath_linalg.inv(a, signature=signature, extobj=extobj)
  File "C:\ProgramData\Anaconda3\envs\qudi\lib\site-packages\numpy\linalg\linalg.py", line 89, in _raise_linalgerror_singular
    raise LinAlgError("Singular matrix")
numpy.linalg.linalg.LinAlgError: Singular matrix

Neverhorst · 2020-01-30T16:28:53Z

Thank you @kay-jahnke.
I tried to reproduce this error on dummy config with 1kHz data rate, 4 times oversampling (=4kHz sampling rate) and 20s trace window.
Started sampling a large hahnecho waveform and pressed "load all modules". Everything stutters and it takes ages but in the end... no error.
Can you give a more detailed way to reproduce this error?

kay-jahnke

As no one (me included) could reproduce the error, let's just merge this PR.
I noticed there are conflicts deleting test files for testing the new interfaces, were they still in master?

alrik-durand · 2020-06-03T11:57:42Z

Hi,

Sorry for the late message ! Could I suggest some modification to the interface architecture ?
As I mentioned in the issue #586, it would be great to have some kind of guidelines for the interface design, so all the interfaces (at least the new ones) follow a similar style.
I personally have some preferences, but i would be happy with any unified style.

I can make some suggestion/change to this code if you are interested but don't have time for this !

What are your thoughts on this ?

Alrik

# Conflicts: # documentation/changelog.md # interface/first_test_interface.py # interface/second_test_interface.py

Neverhorst · 2020-06-10T15:28:26Z

Hi,

Sorry for the late message ! Could I suggest some modification to the interface architecture ?
As I mentioned in the issue #586, it would be great to have some kind of guidelines for the interface design, so all the interfaces (at least the new ones) follow a similar style.
I personally have some preferences, but i would be happy with any unified style.

I can make some suggestion/change to this code if you are interested but don't have time for this !

What are your thoughts on this ?

Alrik

Well, I see the benefit in writing down basic/general guidelines for interface classes. Feel free to propose something.
However, a universal guideline will most certainly require months of discussions. And if we are then to "unify" all existing interfaces, this will be a huge project (many people rely on the current interfaces without contributing their projects to our qudi master).
What I want to say is that your proposed project will be very long-term and we can not stop development of new modules/interfaces until then.
So, unless you have immediate concerns about/improvements for the specific data streaming interface proposed in this PR, I would like to merge this (people want to have it and wait for it).

alrik-durand · 2020-06-10T15:37:20Z

Hi,

Thank you for your reply.
Indeed this is quite a project, but I feel like it is time for the Qudi project to think about this issues, as it is now a mature and well established platform.

I see no immediate concerns, this PR seems otherwise very nice!

I'll start working on some propositions

Neverhorst added 30 commits August 28, 2019 17:10

Try new slow counter

489812b

Slow counting seems to work now at much faster rates than before and …

a37da94

…also with proper oversampling etc.

Small fixes

2af3ca6

First attempt on a simple data input stream interface.

ad972ef

New hardware module using the in streamer interface

4ce6f58

New hardware module using the in streamer interface

f33b037

In principle everything is working now for the streamer

990d686

Better frame timing and more optimized plotting

675e877

Add config for testing

d7605a4

Added "QtCore.Slot" decorators and changed naming of GUI objects

52f1a25

Added snapshot action and artwork

1de9d86

Handled setting of moving average via GUI and remove/add smoothed gra…

271ae94

…phs accordingly.

Forgot ui file.

15c474c

Tweaked tooltips

e8bfe71

Merge remote-tracking branch 'origin/master' into slow_counter_overhaul

5beee1c

# Conflicts: # logic/counter_logic.py

Adapted imports to recent core changes

00fe373

Fixed plot performance by setting pen width to 1px

028db83

WIP

c7a8a37

Implemented trace selection for view and data acquisition

975478f

Crude current value selection functionality

13ee09d

Improved cureent value display

f62c0d3

Better data buffer handling tailored to NI API

ddb2af2

Added single point read functionality

74fb6ba

Various bugfixes.

eb7aa55

Fixed data saving

d994f60

Cleanup of channel acquisition/view configuration

110a5ed

Bugfix

ae40b37

Bugfix

4690a94

bugfix

f0b1e2f

bugfix

cf6d826

Neverhorst added 15 commits September 18, 2019 13:17

Removed test.cfg and included new modules with dummy in default.cfg

4390a16

Added second y-axis to stream plot

6e81135

Added second y-axis to stream plot

3a1ffed

Better color coding of the plots

a1b8db3

Improved readability

7645481

Merge branch 'master' into slow_counter_overhaul

f7a996c

Revision in progress for new time series streamer

168402e

Added ConfigOption for antialiased time series graph.

f615f9a

Fixed a bug with the data_rate StatusVar

Merge branch 'master' into slow_counter_overhaul

54509cf

Reverted countergui.py back to master branch state.

353a02d

Removed obsolete ni_x_series_counter HW module.

108b022

Removed hardware template module from qudi tutorials that somehow mad…

ccb22c9

…e its way into master

Removed obsolete testing modules that should have never been in master

80f0d0c

Updated changelog.

cfb7e29

Added deprecation pop-up notification to the end of CounterGui activation to encourage users to switch to time series tool.

Removed test.cfg that should not be in master

d82dfe8

Neverhorst added enhancement comp:slow_counter labels Dec 4, 2019

Neverhorst requested review from drogenlied and kay-jahnke December 4, 2019 14:17

kay-jahnke requested changes Jan 24, 2020

View reviewed changes

Minor bugfixing

62b2140

kay-jahnke approved these changes May 20, 2020

View reviewed changes

Merge branch 'master' into slow_counter_overhaul

57cf7ba

# Conflicts: # documentation/changelog.md # interface/first_test_interface.py # interface/second_test_interface.py

Neverhorst merged commit fc847a9 into master Jun 11, 2020

Neverhorst deleted the slow_counter_overhaul branch June 11, 2020 12:02

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Slow counter overhaul #555

Slow counter overhaul #555

Neverhorst commented Dec 4, 2019

kay-jahnke left a comment

Neverhorst commented Jan 30, 2020

kay-jahnke left a comment

alrik-durand commented Jun 3, 2020

Neverhorst commented Jun 10, 2020

alrik-durand commented Jun 10, 2020

Slow counter overhaul #555

Slow counter overhaul #555

Conversation

Neverhorst commented Dec 4, 2019

Description

Motivation and Context

How Has This Been Tested?

Types of changes

Checklist:

kay-jahnke left a comment

Choose a reason for hiding this comment

Neverhorst commented Jan 30, 2020

kay-jahnke left a comment

Choose a reason for hiding this comment

alrik-durand commented Jun 3, 2020

Neverhorst commented Jun 10, 2020

alrik-durand commented Jun 10, 2020