Add three new ZMQ publishers for TP results / Retrieve TP results #2230
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
MichaelHuth
force-pushed
the
feature/2230-Add_zeromq_pub_filters_for_TP
branch
4 times, most recently
from
33d4401 to
716f021
Compare
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
MichaelHuth
force-pushed
the
feature/2230-Add_zeromq_pub_filters_for_TP
branch
2 times, most recently
from
96f6592 to
9c33299
Compare
This comment was marked as off-topic.
This comment was marked as off-topic.
This comment was marked as outdated.
This comment was marked as outdated.
MichaelHuth
force-pushed
the
feature/2230-Add_zeromq_pub_filters_for_TP
branch
from
ef494e7 to
e82cd88
Compare
This comment was marked as outdated.
This comment was marked as outdated.
MichaelHuth
force-pushed
the
feature/2230-Add_zeromq_pub_filters_for_TP
branch
from
e82cd88 to
82d748d
Compare
MichaelHuth
force-pushed
the
feature/2230-Add_zeromq_pub_filters_for_TP
branch
2 times, most recently
from
28dcb99 to
e86dda8
Compare
This comment was marked as outdated.
This comment was marked as outdated.
|
@t-b ok yeah it’s probably pulling it in right now as float 64. I can’t access the system right now but I will verify as soon as I can. It’s a one-liner numpy.frombuffer command so I’m sure it’s just that. |
t-b
force-pushed
the
feature/2230-Add_zeromq_pub_filters_for_TP
branch
from
fe29626 to
8bd3794
Compare
|
Fixed conflicts and linted with new ipt version. |
- extended the TPAnalysisInput structure This is a preparation commit for adding zeromq publishers that include some of that information.
- the data is published from the TP analysis thread including additional information available in the thread through the previous commit. - The additional values are also returned by the thread and collected in the async buffer as well then in TPResult and in TPStorage. - The involved waves and their respective getters were adapted with new elements that the additional data can be stored. - As most of the elements store the same information, thus a constant was introduced with a dimension label list that is used as helper for the wave creation in the getter functions.
- The four publishers publish the same json, just with a different period. There is a filter for live, 1s, 5s and 10s publishing interval. - See PUB_TPResult for JSON description - publisher is called from TP_TSAnalysis thread
This prevent misleading naming and it more fitting to the functionality the function actually implements
Added TP_GetStoredTP and TP_GetStoredTPsFromCycle that allow to get information about a TP by tpMarker or TPs by cycle id and headstage. - both functions allow to recreate the DA wave for the TPs with the flag includeDAC - the returned data includes the AD, DA data as well as metadata for each returned TP (from TPStorage). - These TP functions use the same TP utility function.
Add a message type which not only publishes the TP properties but also the raw AD data.
t-b
force-pushed
the
feature/2230-Add_zeromq_pub_filters_for_TP
branch
from
8bd3794 to
6c7cc38
Compare
This is a test suite hook, so there is no need to call the global test begin hook. We call TestBeginCommon() form TEST_BEGIN_OVERRIDE() in UTF_Basic.ipf already.
|
@ben-at-allen I've fixed the unexpected TP message from the databrowser. And also added a test to catch errors like that in the future. Thanks for the report. |
|
Thanks @t-b ! I forgot to respond here but that older issue where I was only getting half the data back from the test pulse was on my end. Everything looks good now. Thanks again. I'll pull this new update and put it onto the rig later today or Monday. Thanks. |
Since 1c43237 (PUB: Add four publishers to publish TP data, 2024-08-23) we are sending out TP messages after doing TP analysis. But we are also using TP_TSAnalysis from within SweepFormula, so this needs to be made configurable.
In the last commit we fixed and issue where the resuse of code in SweepFormula triggered a ZeroMQ message. This is unexpected and wasteful. So let's ensure that we only ever get heartbeat messages for all tests except UTF_ZeroMQPublishing.ipf. We are doing that only when expensive tests are enabled. This was never used for Basic.pxp but the builtin procedure window already has it, and the github workflow sets it as well.
t-b
force-pushed
the
feature/2230-Add_zeromq_pub_filters_for_TP
branch
from
9c40a77 to
5b0bbdd
Compare
|
@ben-at-allen You want version 5b0bbdd (tests/Basic: Ensure that we don't get unexpected ZeroMQ messages, 2025-04-11) as I just had to fix something. |
|
From https://aiephys.slack.com/archives/C07LLQEDKU0/p1744410706950439:
So I'm merging this. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Publishers added:
testpulse results live
testpulse results 1s update
testpulse results 5s update
testpulse results 10s update
The TP results from TP analysis are packed in a JSON in TP_TSAnalysis and published to ZMQ.
The time information when the next publishing is due is stored in the TUFXOP.
PUB_Publish was changed to be threadsafe and added an optional argument to control if the JSON should be released.
Close #2157