Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ensure empty stream segments are initialised #129

Open
wants to merge 11 commits into
base: main
Choose a base branch
from
+405 −101
Open

Ensure empty stream segments are initialised #129

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
1e59368
Ensure empty stream segments are initialised
jamieforth Jan 4, 2025
4dd292a
Refactor file loading tests to use skip_if_file_not_found fixture
jamieforth Jan 28, 2025
3a472ca
Refactor file loading tests to use a global dictionary of paths
jamieforth Jan 29, 2025
e0304d9
Update test file names: Remove `data_with_` prefix
jamieforth Jan 29, 2025
84339da
More systematically test file data
jamieforth Jan 29, 2025
d4946d8
Fix effective sample rate when dejitter_timestamps=False
jamieforth Jan 29, 2025
401ee45
Add skipif
cbrnr Jan 30, 2025
387e813
Cosmetics
cbrnr Jan 30, 2025
0141684
Change effective_srate when dejitter_timestamps=True
jamieforth Jan 30, 2025
372d615
Update empty_streams test
jamieforth Jan 31, 2025
e5e2f13
Format comment
jamieforth Jan 31, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
41 changes: 23 additions & 18 deletions src/pyxdf/pyxdf.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -59,6 +59,9 @@ def __init__(self, xml):
# nominal sampling interval, in seconds, for delta decompression
self.tdiff = 1.0 / self.srate if self.srate > 0 else 0.0
self.effective_srate = 0.0
# list of segments corresponding to detected time-stamp breaks
# (each a tuple of start_idx, end_idx)
self.segments = []
# pre-calc some parsing parameters for efficiency
if self.fmt != "string":
self.dtype = np.dtype(fmts[self.fmt])
Expand Down Expand Up @@ -375,13 +378,12 @@ def load_xdf(
for stream in temp.values():
if len(stream.time_stamps) > 1:
duration = stream.time_stamps[-1] - stream.time_stamps[0]
stream.effective_srate = len(stream.time_stamps) / duration
stream.effective_srate = (len(stream.time_stamps) - 1) / duration
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are you sure it wasn't correct before? Below, the - 1 is needed because it stores the start and stop indices of segments, but here we're calculating the duration. If you only have 1 sample, you will now always get an effective_srate of 0 (previously it was 1 / duration, which I think is the correct value).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you only have 1 sample, the duration is 0, so wouldn't you end up with 1/0 in that case?

In my view you can only calculate a duration with two or more samples, and if len(stream.time_stamps) > 1 assures this.

Without -1 the effective sample rate in minimal is calculated as 11.25 Hz but it should be 10 Hz, so I think the -1 is correct.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, you are right!

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK cool, I just push a change for the calculation of effective_srate when dejitter_timestamps=True as well.

This one was doing len(time_stamps) / (duration + tdiff), which is fine when time-stamps are perfect, but off otherwise.

Perfect clock

nominal_srate = 1
t = np.arange(0, 5, nominal_srate)
(t[-1] - t[0] + nominal_srate) / len(t)

=> 1.0 # Correct

nominal_srate = 1
t = np.arange(0, 5, nominal_srate)
(t[-1] - t[0]) / (len(t) - 1)

=> 1.0 # Correct

Fast clock

nominal_srate = 1
effective_srate = 1.1
t = np.arange(0, 5, effective_srate)
(t[-1] - t[0] + nominal_srate) / len(t)

=> 1.08 # Incorrect

nominal_srate = 1
effective_srate = 1.1
t = np.arange(0, 5, effective_srate)
(t[-1] - t[0]) / (len(t) - 1)

=> 1.1 # Correct

Slow clock

nominal_srate = 1
effective_srate = 0.9
t = np.arange(0, 4, effective_srate)
(t[-1] - t[0] + nominal_srate) / len(t)

=> 0.919 # Incorrect

nominal_srate = 1
effective_srate = 0.9
t = np.arange(0, 4, effective_srate)
(t[-1] - t[0]) / (len(t) - 1)

=> 0.9 # Correct

else:
stream.effective_srate = 0.0
# initialize segment list in case jitter_removal was not selected
stream.segments = []
if len(stream.time_stamps) > 0:
stream.segments.append((0, len(stream.time_series) - 1)) # inclusive
stream.segments.append((0, len(stream.time_stamps) - 1)) # inclusive

for k in streams.keys():
stream = streams[k]
Expand Down Expand Up @@ -629,8 +631,12 @@ def _jitter_removal(streams, threshold_seconds=1, threshold_samples=500):
for stream_id, stream in streams.items():
stream.effective_srate = 0 # will be recalculated if possible
nsamples = len(stream.time_stamps)
stream.segments = []
if nsamples > 0 and stream.srate > 0:
if nsamples > 0:
if stream.srate == 0:
# Initialise default segment for irregular sampling rate streams
stream.segments.append((0, nsamples - 1)) # inclusive
continue

# Identify breaks in the time_stamps
diffs = np.diff(stream.time_stamps)
b_breaks = diffs > np.max(
Expand All @@ -645,6 +651,7 @@ def _jitter_removal(streams, threshold_seconds=1, threshold_samples=500):
seg_stops = np.hstack((break_inds - 1, nsamples - 1)) # inclusive
for a, b in zip(seg_starts, seg_stops):
stream.segments.append((a, b))

# Process each segment separately
for start_ix, stop_ix in zip(seg_starts, seg_stops):
# Calculate time stamps assuming constant intervals within each segment
Expand All @@ -658,21 +665,19 @@ def _jitter_removal(streams, threshold_seconds=1, threshold_samples=500):
# Recalculate effective_srate if possible
counts = (seg_stops + 1) - seg_starts
if np.any(counts):
# Calculate range segment duration (assuming last sample duration was
# exactly 1 * stream.tdiff)
# Calculate range segment duration
durations = (
stream.time_stamps[seg_stops] + stream.tdiff
) - stream.time_stamps[seg_starts]
stream.effective_srate = np.sum(counts) / np.sum(durations)

srate, effective_srate = stream.srate, stream.effective_srate
if srate != 0 and np.abs(srate - effective_srate) / srate > 0.1:
msg = (
"Stream %d: Calculated effective sampling rate %.4f Hz is different "
"from specified rate %.4f Hz."
)
logger.warning(msg, stream_id, effective_srate, srate)
stream.time_stamps[seg_stops] - stream.time_stamps[seg_starts]
)
stream.effective_srate = np.sum(counts - 1) / np.sum(durations)

srate, effective_srate = stream.srate, stream.effective_srate
if np.abs(srate - effective_srate) / srate > 0.1:
msg = (
"Stream %d: Calculated effective sampling rate %.4f Hz is different "
"from specified rate %.4f Hz."
)
logger.warning(msg, stream_id, effective_srate, srate)
return streams


Expand Down
Loading