Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Tests for birdnet_analyzer.analyze module #527

Draft
wants to merge 8 commits into
base: main
Choose a base branch
from
Draft

Tests for birdnet_analyzer.analyze module #527

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
99a7d5a
test: sketch of unit and end-to-end tests
kenshih Dec 14, 2024
49dd20b
main tests
kenshih Dec 14, 2024
a6100c1
add 3rd case of analyze
kenshih Dec 15, 2024
13e339f
add more helpful comments and consitent naming
kenshih Dec 15, 2024
8703779
make how-to more simple until more than 1 test is added
kenshih Dec 15, 2024
a3a100b
rename to more simple names
kenshih Dec 15, 2024
0eece4a
update python3 to general alias
kenshih Dec 21, 2024
5035c9b
update test to assume 'python' not 'python3'
kenshih Dec 21, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions birdnet_analyzer/analyze.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -819,3 +819,7 @@ def __call__(self, parser, args, values, option_string=None):
# python3 analyze.py --i example/ --o example/ --slist example/ --min_conf 0.5 --threads 4
# python3 analyze.py --i example/soundscape.wav --o example/soundscape.BirdNET.selection.table.txt --slist example/species_list.txt --threads 8
# python3 analyze.py --i example/ --o example/ --lat 42.5 --lon -76.45 --week 4 --sensitivity 1.0 --rtype table --locale de
# Alternately, from root of this repository run:
# `pytest tests/test_analyze__main.py`
# which run these 3 examples as a smoke test.
# Then, try your own variations.
4 changes: 3 additions & 1 deletion requirements.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,4 +8,6 @@ bottle
requests
keras-tuner
# below, to support birdnet_analyzer.gui
matplotlib===3.9.3
matplotlib===3.9.3
# below, for running tests when developing birdnet-analyzer itself
pytest==8.3.4
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I originally added this as requirements-dev.txt, but then thought to keep-it-simple by leaving it here

Empty file added tests/__init__.py
View file
Open in desktop
Empty file.
7 changes: 7 additions & 0 deletions tests/resources/SNAPSHOT.analyze__main.case1.expected.table.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
Selection View Channel Begin Time (s) End Time (s) Low Freq (Hz) High Freq (Hz) Common Name Species Code Confidence Begin Path File Offset (s)
1 Spectrogram 1 1 0 3.0 0 15000 Black-capped Chickadee bkcchi 0.8141 birdnet_analyzer/example/soundscape.wav 0
2 Spectrogram 1 1 9.0 12.0 0 15000 House Finch houfin 0.6394 birdnet_analyzer/example/soundscape.wav 9.0
3 Spectrogram 1 1 42.0 45.0 0 15000 Dark-eyed Junco daejun 0.7375 birdnet_analyzer/example/soundscape.wav 42.0
4 Spectrogram 1 1 54.0 57.0 0 15000 House Finch houfin 0.6071 birdnet_analyzer/example/soundscape.wav 54.0
5 Spectrogram 1 1 60.0 63.0 0 15000 Dark-eyed Junco daejun 0.5550 birdnet_analyzer/example/soundscape.wav 60.0
6 Spectrogram 1 1 72.0 75.0 0 15000 House Finch houfin 0.5638 birdnet_analyzer/example/soundscape.wav 72.0
30 changes: 30 additions & 0 deletions tests/resources/SNAPSHOT.analyze__main.case2.expected.table.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
Selection View Channel Begin Time (s) End Time (s) Low Freq (Hz) High Freq (Hz) Common Name Species Code Confidence Begin Path File Offset (s)
1 Spectrogram 1 1 0 3.0 0 15000 Black-capped Chickadee bkcchi 0.8141 birdnet_analyzer/example/soundscape.wav 0
2 Spectrogram 1 1 3.0 6.0 0 15000 Black-capped Chickadee bkcchi 0.3083 birdnet_analyzer/example/soundscape.wav 3.0
3 Spectrogram 1 1 6.0 9.0 0 15000 Tufted Titmouse tuftit 0.1864 birdnet_analyzer/example/soundscape.wav 6.0
4 Spectrogram 1 1 9.0 12.0 0 15000 House Finch houfin 0.6394 birdnet_analyzer/example/soundscape.wav 9.0
5 Spectrogram 1 1 18.0 21.0 0 15000 Blue Jay blujay 0.4353 birdnet_analyzer/example/soundscape.wav 18.0
6 Spectrogram 1 1 21.0 24.0 0 15000 Blue Jay blujay 0.3291 birdnet_analyzer/example/soundscape.wav 21.0
7 Spectrogram 1 1 21.0 24.0 0 15000 House Finch houfin 0.1867 birdnet_analyzer/example/soundscape.wav 21.0
8 Spectrogram 1 1 24.0 27.0 0 15000 Blue Jay blujay 0.1749 birdnet_analyzer/example/soundscape.wav 24.0
9 Spectrogram 1 1 27.0 30.0 0 15000 Dark-eyed Junco daejun 0.2187 birdnet_analyzer/example/soundscape.wav 27.0
10 Spectrogram 1 1 33.0 36.0 0 15000 Dark-eyed Junco daejun 0.4591 birdnet_analyzer/example/soundscape.wav 33.0
11 Spectrogram 1 1 36.0 39.0 0 15000 Dark-eyed Junco daejun 0.3537 birdnet_analyzer/example/soundscape.wav 36.0
12 Spectrogram 1 1 39.0 42.0 0 15000 House Finch houfin 0.2533 birdnet_analyzer/example/soundscape.wav 39.0
13 Spectrogram 1 1 42.0 45.0 0 15000 Dark-eyed Junco daejun 0.7375 birdnet_analyzer/example/soundscape.wav 42.0
14 Spectrogram 1 1 48.0 51.0 0 15000 Blue Jay blujay 0.1293 birdnet_analyzer/example/soundscape.wav 48.0
15 Spectrogram 1 1 51.0 54.0 0 15000 House Finch houfin 0.2564 birdnet_analyzer/example/soundscape.wav 51.0
16 Spectrogram 1 1 54.0 57.0 0 15000 House Finch houfin 0.6071 birdnet_analyzer/example/soundscape.wav 54.0
17 Spectrogram 1 1 57.0 60.0 0 15000 House Finch houfin 0.1615 birdnet_analyzer/example/soundscape.wav 57.0
18 Spectrogram 1 1 60.0 63.0 0 15000 Dark-eyed Junco daejun 0.5550 birdnet_analyzer/example/soundscape.wav 60.0
19 Spectrogram 1 1 69.0 72.0 0 15000 House Finch houfin 0.2915 birdnet_analyzer/example/soundscape.wav 69.0
20 Spectrogram 1 1 72.0 75.0 0 15000 House Finch houfin 0.5638 birdnet_analyzer/example/soundscape.wav 72.0
21 Spectrogram 1 1 78.0 81.0 0 15000 House Finch houfin 0.2561 birdnet_analyzer/example/soundscape.wav 78.0
22 Spectrogram 1 1 84.0 87.0 0 15000 House Finch houfin 0.2801 birdnet_analyzer/example/soundscape.wav 84.0
23 Spectrogram 1 1 90.0 93.0 0 15000 American Goldfinch amegfi 0.4190 birdnet_analyzer/example/soundscape.wav 90.0
24 Spectrogram 1 1 93.0 96.0 0 15000 House Finch houfin 0.3168 birdnet_analyzer/example/soundscape.wav 93.0
25 Spectrogram 1 1 96.0 99.0 0 15000 American Goldfinch amegfi 0.4765 birdnet_analyzer/example/soundscape.wav 96.0
26 Spectrogram 1 1 99.0 102.0 0 15000 House Finch houfin 0.1574 birdnet_analyzer/example/soundscape.wav 99.0
27 Spectrogram 1 1 102.0 105.0 0 15000 House Finch houfin 0.4459 birdnet_analyzer/example/soundscape.wav 102.0
28 Spectrogram 1 1 105.0 108.0 0 15000 Black-capped Chickadee bkcchi 0.1790 birdnet_analyzer/example/soundscape.wav 105.0
29 Spectrogram 1 1 117.0 120.0 0 15000 American Goldfinch amegfi 0.3924 birdnet_analyzer/example/soundscape.wav 117.0
30 changes: 30 additions & 0 deletions tests/resources/SNAPSHOT.analyze__main.case3.expected.table.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
Selection View Channel Begin Time (s) End Time (s) Low Freq (Hz) High Freq (Hz) Common Name Species Code Confidence Begin Path File Offset (s)
1 Spectrogram 1 1 0 3.0 0 15000 Schwarzkopfmeise bkcchi 0.8141 birdnet_analyzer/example/soundscape.wav 0
2 Spectrogram 1 1 3.0 6.0 0 15000 Schwarzkopfmeise bkcchi 0.3083 birdnet_analyzer/example/soundscape.wav 3.0
3 Spectrogram 1 1 6.0 9.0 0 15000 Grauhäubchenmeise tuftit 0.1864 birdnet_analyzer/example/soundscape.wav 6.0
4 Spectrogram 1 1 9.0 12.0 0 15000 Hausgimpel houfin 0.6394 birdnet_analyzer/example/soundscape.wav 9.0
5 Spectrogram 1 1 18.0 21.0 0 15000 Blauhäher blujay 0.4353 birdnet_analyzer/example/soundscape.wav 18.0
6 Spectrogram 1 1 21.0 24.0 0 15000 Blauhäher blujay 0.3291 birdnet_analyzer/example/soundscape.wav 21.0
7 Spectrogram 1 1 21.0 24.0 0 15000 Hausgimpel houfin 0.1867 birdnet_analyzer/example/soundscape.wav 21.0
8 Spectrogram 1 1 24.0 27.0 0 15000 Blauhäher blujay 0.1749 birdnet_analyzer/example/soundscape.wav 24.0
9 Spectrogram 1 1 27.0 30.0 0 15000 Winterammer daejun 0.2187 birdnet_analyzer/example/soundscape.wav 27.0
10 Spectrogram 1 1 33.0 36.0 0 15000 Winterammer daejun 0.4591 birdnet_analyzer/example/soundscape.wav 33.0
11 Spectrogram 1 1 36.0 39.0 0 15000 Winterammer daejun 0.3537 birdnet_analyzer/example/soundscape.wav 36.0
12 Spectrogram 1 1 39.0 42.0 0 15000 Hausgimpel houfin 0.2533 birdnet_analyzer/example/soundscape.wav 39.0
13 Spectrogram 1 1 42.0 45.0 0 15000 Winterammer daejun 0.7375 birdnet_analyzer/example/soundscape.wav 42.0
14 Spectrogram 1 1 48.0 51.0 0 15000 Blauhäher blujay 0.1293 birdnet_analyzer/example/soundscape.wav 48.0
15 Spectrogram 1 1 51.0 54.0 0 15000 Hausgimpel houfin 0.2564 birdnet_analyzer/example/soundscape.wav 51.0
16 Spectrogram 1 1 54.0 57.0 0 15000 Hausgimpel houfin 0.6071 birdnet_analyzer/example/soundscape.wav 54.0
17 Spectrogram 1 1 57.0 60.0 0 15000 Hausgimpel houfin 0.1615 birdnet_analyzer/example/soundscape.wav 57.0
18 Spectrogram 1 1 60.0 63.0 0 15000 Winterammer daejun 0.5550 birdnet_analyzer/example/soundscape.wav 60.0
19 Spectrogram 1 1 69.0 72.0 0 15000 Hausgimpel houfin 0.2915 birdnet_analyzer/example/soundscape.wav 69.0
20 Spectrogram 1 1 72.0 75.0 0 15000 Hausgimpel houfin 0.5638 birdnet_analyzer/example/soundscape.wav 72.0
21 Spectrogram 1 1 78.0 81.0 0 15000 Hausgimpel houfin 0.2561 birdnet_analyzer/example/soundscape.wav 78.0
22 Spectrogram 1 1 84.0 87.0 0 15000 Hausgimpel houfin 0.2801 birdnet_analyzer/example/soundscape.wav 84.0
23 Spectrogram 1 1 90.0 93.0 0 15000 Goldzeisig amegfi 0.4190 birdnet_analyzer/example/soundscape.wav 90.0
24 Spectrogram 1 1 93.0 96.0 0 15000 Hausgimpel houfin 0.3168 birdnet_analyzer/example/soundscape.wav 93.0
25 Spectrogram 1 1 96.0 99.0 0 15000 Goldzeisig amegfi 0.4765 birdnet_analyzer/example/soundscape.wav 96.0
26 Spectrogram 1 1 99.0 102.0 0 15000 Hausgimpel houfin 0.1574 birdnet_analyzer/example/soundscape.wav 99.0
27 Spectrogram 1 1 102.0 105.0 0 15000 Hausgimpel houfin 0.4459 birdnet_analyzer/example/soundscape.wav 102.0
28 Spectrogram 1 1 105.0 108.0 0 15000 Schwarzkopfmeise bkcchi 0.1790 birdnet_analyzer/example/soundscape.wav 105.0
29 Spectrogram 1 1 117.0 120.0 0 15000 Goldzeisig amegfi 0.3924 birdnet_analyzer/example/soundscape.wav 117.0
115 changes: 115 additions & 0 deletions tests/test_analyze__main.py
View file
Open in desktop
Copy link
Contributor Author

@kenshih kenshih Dec 15, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added postfix __main to signal to the reader that this test is not a “unit test” so much as an “end-to-end” test.

Reason: If unit tests are eventually added, it will help with the organization, since e2e tests usually take more time to run (seconds instead of milliseconds) & aren’t very precise about “what is broken” compared to unit tests, so it’s good to know which is which. I find there is inevitably a desire to separate them.

Happy to suggest a rename to test_analyze.py, if that is preferred.

Original file line number Diff line number Diff line change
@@ -0,0 +1,115 @@
# tests/test_analyze__main.py
#
# These are end-to-end tests that run module `birdnet_analyzer.analyze`
# as if calling it from the command-line.
#
# When run with certain arguments on known example input,
# an output file should be generated with expected specific format and data
# as found in tests/resources/SNAPSHOT.analyze__main.*)
#
# How to run these tests;
# 1. (Prerequisite) from root of this repository, in your virtual environment run:
# ```
# pip install -r requirements.txt
# ```
# 2. From root of this repository run: `pytest tests/test_analyze__main.py`
Copy link
Contributor Author

@kenshih kenshih Dec 15, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I originally started adding a DEVELOPMENT.adoc readme, not quite knowing where to add useful how-to/context, but then realized that was too noisy a change.

If you, the maintainers of this project, like having these and/or other kinds of tests, I’d be happy to help to add more of them and provide usable documentation in other PRs, as preferred. Happy also to add a simple github action PR to run such tests automatically, e.g. when someone opens a PR these could do a baseline check for regression in advance of your review.


import subprocess
import os

def test_analyze_case1_example_min_conf(tmp_path):
"""
Tests a command similar to:
python3 analyze.py --i example/ --o example/ --slist example/ --min_conf 0.5 --threads 4
"""
# GIVEN: analyze is called on example/ dir with 1 sound file, soundscape.wav, with a minimum confidence level of 0.5
output_dir = tmp_path
# comment the following line in in order to write a file to troubleshoot or take a new snapshot
# output_dir = 'birdnet_analyzer/example'
Copy link
Contributor Author

@kenshih kenshih Dec 15, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This comment is added, because this code intentionally uses pytest’s tmp_path mechanism, which cleans up after itself on each test so as not to clutter the developer’s environment or leave behind anything she might mistake for real (non-test) output.

As well, an appropriate time to change the SNAPSHOTs themselves would be when a new model is introduced, so specific values like confidence might be expected to change. Generating a new SNAPSHOT is as simple as uncommenting this line, reviewing it, then copying it over the old snapshot. The diff of such an update can sometimes be enlightening.

cmd = [
'python',
'-m','birdnet_analyzer.analyze',
'--i', 'birdnet_analyzer/example/',
'--o', output_dir,
'--slist', 'example/',
'--min_conf', '0.5',
'--threads', '4'
]

# WHEN: analyze is run
result = subprocess.run(cmd, capture_output=True, text=True)

# THEN: the file created by analyze matches the expected SNAPSHOT (e.g. with fewer results than default confidence threshold)
SNAPSHOT_FILE_OF_EXPECTED='tests/resources/SNAPSHOT.analyze__main.case1.expected.table.txt'
output_file = f'{output_dir}/soundscape.BirdNET.selection.table.txt'
assert result.returncode == 0, f"Command failed with error: {result.stderr}"
assert os.path.exists(output_file), f"File should exist but doesn't: {output_file}"
with open(output_file, 'r') as f1, open(SNAPSHOT_FILE_OF_EXPECTED, 'r') as f2:
assert f1.read() == f2.read()


def test_analyze_case2_soundscape(tmp_path):
"""
Tests a command similar to:
python3 analyze.py --i example/soundscape.wav --o example/soundscape.BirdNET.selection.table.txt --slist example/species_list.txt --threads 8
"""

# GIVEN: analyze is called on single file: soundscape.wav
output_dir = tmp_path
# comment the following line in in order to write a file to troubleshoot or take a new snapshot
# output_dir = 'birdnet_analyzer/example'
cmd = [
'python',
'-m','birdnet_analyzer.analyze',
'--i', 'birdnet_analyzer/example/soundscape.wav',
'--o', output_dir,
'--slist', 'example/species_list.txt',
'--threads', '8'
]

# WHEN: analyze is run
result = subprocess.run(cmd, capture_output=True, text=True)

# THEN: the file created by analyze matches the expected SNAPSHOT
SNAPSHOT_FILE_OF_EXPECTED='tests/resources/SNAPSHOT.analyze__main.case2.expected.table.txt'
output_file = f'{output_dir}/soundscape.BirdNET.selection.table.txt'
assert result.returncode == 0, f"Command failed with error: {result.stderr}"
assert os.path.exists(output_file), f"File should exist but doesn't: {output_file}"
with open(output_file, 'r') as f1, open(SNAPSHOT_FILE_OF_EXPECTED, 'r') as f2:
assert f1.read() == f2.read()


def test_analyze_case3_latlon_week4_sensitivity_rtype_de(tmp_path):
"""
Tests a command similar to:
python3 analyze.py --i example/ --o example/ --lat 42.5 --lon -76.45 --week 4 --sensitivity 1.0 --rtype table --locale de
"""

# GIVEN: analyze is called on with lat, lon, week, sensitivity, rtype, and locale set
output_dir = tmp_path
# comment the following line in in order to write a file to troubleshoot or take a new snapshot
# output_dir = 'birdnet_analyzer/example'
cmd = [
'python',
'-m','birdnet_analyzer.analyze',
'--i', 'birdnet_analyzer/example/',
'--o', output_dir,
'--slist', 'example/',
'--lat', '42.5',
'--lon', '-76.45',
'--week', '4',
'--sensitivity', '1.0',
'--rtype', 'table',
'--locale', 'de'
]

# WHEN: analyze is run
result = subprocess.run(cmd, capture_output=True, text=True)

# THEN: the file created by analyze matches the expected SNAPSHOT (e.g. in German with expected data)
SNAPSHOT_FILE_OF_EXPECTED='tests/resources/SNAPSHOT.analyze__main.case3.expected.table.txt'
output_file = f'{output_dir}/soundscape.BirdNET.selection.table.txt'
assert result.returncode == 0, f"Command failed with error: {result.stderr}"
assert os.path.exists(output_file), f"File should exist but doesn't: {output_file}"
with open(output_file, 'r') as f1, open(SNAPSHOT_FILE_OF_EXPECTED, 'r') as f2:
assert f1.read() == f2.read()