diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 02c5ef4bc..3361388fb 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -49,19 +49,7 @@ You can install it from here: https://github.com/powershell/powershell#get-power ### 2. A **.NET Core SDK** - We aim to use the latest stable version -- You can verify the version our project is using by looking in the [global.json](https://github.com/QutEcoacoustics/audio-analysis/tree/master/build/global.json) file - -Recommended choice is to run the `dotnet-install` script in the [./build/](https://github.com/QutEcoacoustics/audio-analysis/tree/master/build) folder. - -PowerShell: -```powershell -> ./build/dotnet-install.ps1 -``` - -Bash: -```shell -$ ./build/dotnet-install.sh -``` +- You can verify the version our project is using by looking in the [global.json](https://github.com/QutEcoacoustics/audio-analysis/tree/master/global.json) file Alternately, you can download a SDK from here: . Note: you want the _Build apps - SDK_ download. diff --git a/docs/basics/scripting.md b/docs/basics/scripting.md new file mode 100644 index 000000000..2706121b3 --- /dev/null +++ b/docs/basics/scripting.md @@ -0,0 +1,44 @@ +# Scripting AP.exe + +_AnalysisPrograms.exe_ works best when processing single audio files. +This has a few advantages: + +**It simplifies the code**. We don't need to know about the particular +way you store your data, or the way you want your data processed. + +**It allows for greater customization**. By building a set of composable +tools, it allows you to choose what analysis is done, and when. You want it +all? You got it. You only want the first bit done once, and the second bit done +100 times with parameter sweeps? Go for it. + +**It is easier to parallelize**. You want to use your university HPC? +Write your own distributed analysis? Or just run it in sequence? That's all +your choice. + +Despite these advantages, we recognize it is useful to have example of +how large analyses might be completed. Thus the scripts you find in this +folder contain various examples (in various languages) of composing +workflows with _AnalysisPrograms.exe_. + +**You can find a collection of example scripts here: https://github.com/QutEcoacoustics/audio-analysis/tree/master/scripts** + +## PowerShell + +You'll see a lot of scripts in that folder that are written in PowerShell. +If you're not familiar with it, you can consider it as the Windows equivalent +of the Bash shell. + +We like PowerShell because we think the syntax is more reasonable than Bash +and the enhanced support for dates, times, and regular expressions are well worth the investment. + +As of [PowerShell 6.0](https://github.com/PowerShell/PowerShell#-powershell) +the shell is cross platform!. If you're not +convinced, the scripts should be easy enough to reimplement in your favourite +language (like Bash)--and we would of course appreciate any translated +scripts sent back to us as contributed examples. + +## Any other language + +You can use any programming language to script AP.exe. + +R is a popular choice. We have a short guide for [using AP.exe with R](./using_r.md) diff --git a/docs/basics/toc.yml b/docs/basics/toc.yml index 8946c9e75..6739f0f92 100644 --- a/docs/basics/toc.yml +++ b/docs/basics/toc.yml @@ -10,7 +10,9 @@ - name: Config files href: config_files.md -- name: Scripting with R +- name: Scripting AP.exe + href: scripting.md +- name: Scripting AP.exe with R href: using_r.md - name: Dates diff --git a/scripts/README.md b/scripts/README.md index da399d8d0..aff0a88ec 100644 --- a/scripts/README.md +++ b/scripts/README.md @@ -1,62 +1,14 @@ # Scripts -_AnalysisPrograms.exe_ works best when processing single audio files. -This has a few advantages: - -**It simplifies the code**. We don't need to know about the particular -way you store your data, or the way you want your data processed. - -**It allows for greater customization**. By building a set of composable -tools, it allows you to choose what analysis done, and when. You wan't it -all? You got it. You only want the first bit done once, and the second done -100 times with parameter sweeps? Go for it. - -**It is easier to parallelize**. You want to use your university HPC? -Write your own distributed analysis? Or just run it in sequence? That's all -your choice. - -Despite these advantages, we recognize it is useful to have example of -how large analyses might be completed. Thus the scripts you find in this -folder contain various examples (in various languages) of composing -workflows with _AnalysisPrograms.exe_. +A folder of example scripts that use AP.exe. ## Contributions appreciated! -If you right a script that gets the job done for you, we'd be happy to -include it here as an example for others to use. - -## PowerShell - -You'll see a lot of scripts in this folder that are written in PowerShell. -If you're not familiar with it, you can consider it as the Windows equivalent -of the Bash shell. @atruskie like's PowerShell because in their personal -opinion, the syntax is more reasonable than Bash, and the enhanced support -for dates, times, and regular expressions are well worth the investment. - -As of [PowerShell 6.0](https://github.com/PowerShell/PowerShell#-powershell) -the shell is cross platform and well worth investigating. If you're not -convinced, the scripts should be easy enough to reimplement in your favourite -language (like Bash)--and we would of course appreciate any translated -scripts sent back to us as contributed examples. - -## Example headers +If you write a script that gets the job done for you, we'd be happy to +include it here as an example for others to use. Please: -We'd like to see each example script prefaced with a documentation header. -A suggested format: +1. fork the repository +2. Add your script to this folder +3. Send us a pull-request! -``` -# A simple loop to XXX for a folder of files. -# Jolene Blogs 2017 -# -# For each audio file found, -# - It runs does XXX -# - It then also does YYY -# - And sometimes does ZZZ -# - ... -# -# Assumptions: -# - AnalysisPrograms.exe is in current directory -# - ... -... script starts here ... -``` \ No newline at end of file diff --git a/scripts/find_ap.ps1 b/scripts/find_ap.ps1 new file mode 100644 index 000000000..1c70345fd --- /dev/null +++ b/scripts/find_ap.ps1 @@ -0,0 +1,35 @@ +param() + +$command = Get-Command AnalysisPrograms* -ErrorAction SilentlyContinue + +if ($null -ne $command) { + return $command.Path; + exit 0; +} + +if ($IsWindows) { + + $command = Get-Command C:\AP\AnalysisPrograms.exe -ErrorAction SilentlyContinue + + if ($null -ne $command) { + return $command.Path; + exit 0; + } +} + +if ($IsLinux) { + $command = Get-Command /AP/AnalysisPrograms.exe -ErrorAction SilentlyContinue + + if ($null -ne $command) { + return $command.Path; + exit 0; + } +} + +Write-Error @" +Can't find AnalysisPrograms.exe. +Please install it using the instructions from: +https://github.com/QutEcoacoustics/audio-analysis/blob/master/docs/installing.md +"@ + +exit 1; \ No newline at end of file diff --git a/scripts/generate_zooming.ps1 b/scripts/generate_zooming.ps1 index 4c8219405..177158a21 100644 --- a/scripts/generate_zooming.ps1 +++ b/scripts/generate_zooming.ps1 @@ -13,11 +13,15 @@ # - Users will customize the below variables before running the script $working_dir = "D:/Temp/zooming" -$csv = "$working_dir\all_oxley_creek_recordings.csv" +$csv = "$working_dir/all_oxley_creek_recordings.csv" $output = $working_dir -# The config files we copied from the ConfigFiles directory and customised for our purposes. -$indices_config_file = "$working_dir\Towsey.Acoustic.Zooming.yml" -$zooming_config_file = "$working_dir\SpectrogramZoomingConfig.yml" + +# find the path to analysis programs +$ap_path = (Get-Command AnalysisPrograms.exe).Path +$default_configs = Resolve-Path "$ap_path/../ConfigFiles" + +$indices_config_file = "$default_configs/Towsey.Acoustic.Zooming.yml" +$zooming_config_file = "$default_configs/SpectrogramZoomingConfig.yml" $remote_user = "ubuntnu" $remote_server = "server.example.com" @@ -25,17 +29,16 @@ $remote_path = "/home/ubuntu/data/original_audio" # helper functions -# find the path to analysis programs -$ap_path = (gcm AnalysisPrograms.exe).Path -# just checks whether a previous run was successful + +# just checks whether a previous run was successful function HasAlreadyRun($results_dir) { $log = Join-Path $results_dir "log.txt" if (Test-Path $log) { $match = Get-Content -Tail 1 $log | Select-String "ERRORLEVEL: (\d+)" return $match.Matches.Groups[1].Value -eq 0 } - + return $false } @@ -54,14 +57,14 @@ $recordings = Import-Csv $csv $results = @() foreach ($recording in $recordings) { Write-Output "Starting new recording $($recording.uuid)" - + # create a results object to store results - $result = New-Object "pscustomobject" | Select-Object Download,Indices,Images + $result = New-Object "pscustomobject" | Select-Object Download, Indices, Images # extract all needed meta data to create a path to the remote file # constructs a path that looks like: ".../data/b2/b24460cf-e25e-44c9-9034-af9b0a1ddcbe_20121019-140000Z.wav $uuid = $recording.uuid - $prefix = $uuid.Substring(0,2) + $prefix = $uuid.Substring(0, 2) $date = (Get-Date $recording.recorded_date).ToString("yyyyMMdd-HHmmssZ") $name = "$uuid`_$date.wav" $remote_path = "$remote_path/$prefix/$name" @@ -82,7 +85,7 @@ foreach ($recording in $recordings) { $instance_output = $local_path + "_results" # generate indices - if (HasAlreadyRun $instance_output) { + if (HasAlreadyRun $instance_output) { $result.Indices = "0*" Write-Output "Skipping indices generation for $uuid - already completed" } @@ -97,7 +100,7 @@ foreach ($recording in $recordings) { # generate zooming tiles $indices_dir = Join-Path $instance_output "Towsey.Acoustic" - AnalysisPrograms.exe DrawZoomingSpectrograms $indices_dir $zooming_config_file $instance_output -o "sqlite3" -z "Tile" -n | IndentOutput + AnalysisPrograms.exe DrawZoomingSpectrograms $indices_dir $zooming_config_file $instance_output -o "sqlite3" -z "Tile" -n | IndentOutput $result.Images = $LASTEXITCODE $results += $result diff --git a/scripts/indices_and_concat.ps1 b/scripts/indices_and_concat.ps1 new file mode 100644 index 000000000..400485542 --- /dev/null +++ b/scripts/indices_and_concat.ps1 @@ -0,0 +1,91 @@ +<# +.SYNOPSIS + Generates acoustic indices for multiple audio files and concatenates the output. +.DESCRIPTION + Generates acoustic indices for multiple audio files and concatenates the output. + Expects each directory to contain audio files for one location. +.PARAMETER $input_directories + A directory of audio files to process. +.PARAMETER $output_directory +.INPUTS + Optionally: input directories +.OUTPUTS + Files stored in $output_directory +.NOTES + Version: 2.0 + Author: Anthony Truskinger + Creation Date: 2020-01-30 + Purpose/Change: Updated docs links, add ap finder script + +.EXAMPLE + ./indices_and_concat.ps1 D:/Stud D://Thompson -time_zone_offset "10:00" -output_directory ./output +#> + +#requires -version 6 + +param( + [Parameter( + Position = 0, + Mandatory = $true, + ValueFromRemainingArguments = $true, + ValueFromPipeline = $true)] + [System.IO.DirectoryInfo[]]$input_directories, + + [Parameter( + Mandatory = $true)] + [System.IO.DirectoryInfo]$output_directory, + + [Parameter( + Mandatory = $true)] + [string]$time_zone_offset, + + $name_filter = "*" + +) + +# Do not continue running the script if a problem is encountered +$ErrorActionPreference = "Stop" + + +# get the path for AP.exe. When do this to resolve some nice default config files. +# TODO: remove this when the default config file feature is implemented in AP.exe +$ap_path = . "$PSScriptRoot/find_ap.ps1" +$default_configs = Resolve-Path "$ap_path/../ConfigFiles" + +foreach ($input_directory in $input_directories) { + Write-Output "Processing $input_directory" + + $current_group = $input_directory.Name + + $audio_files = Get-ChildItem -Recurse -File $input_directory -Include "*.wav" + $filtered_files = $audio_files | Where-Object { $_.Name -ilike $name_filter } + + $counter = 0; + foreach ($file in $filtered_files) { + $counter++ + Write-Output "Generating indices for $file, file $counter of $($filtered_files.Count)" + $name = $file.Name + + # for more information on how this command works, please see: + # https://ap.qut.ecoacoustics.info/technical/commands/analyze_long_recording.html + AnalysisPrograms.exe audio2csv $file "$default_configs/Towsey.Acoustic.yml" "$output_directory/$current_group/indices/$name" --no-debug --parallel + } + + Write-Output "Now concatenating files for $current_group" + + # for more information on how this command works, please see: + # https://ap.qut.ecoacoustics.info/technical/commands/concatenate_index_files.html + AnalysisPrograms.exe ConcatenateIndexFiles ` + --input-data-directory "$output_directory/$current_group/indices" ` + --output-directory "$output_directory" ` + -z $time_zone_offset ` + --file-stem-name $current_group ` + --directory-filter "*.*" ` + --index-properties-config "$default_configs/IndexPropertiesConfig.yml" ` + --false-colour-spectrogram-config "$default_configs/SpectrogramFalseColourConfig.yml" ` + --draw-images ` + --no-debug + +} + +Write-Output "Complete!" \ No newline at end of file diff --git a/scripts/indices_loop.ps1 b/scripts/indices_loop.ps1 index 13ea2c940..60196a3c0 100644 --- a/scripts/indices_loop.ps1 +++ b/scripts/indices_loop.ps1 @@ -1,38 +1,60 @@ -# A simple loop to generate indices for a folder of files. -# Karlina Indraswari 2017 -# -# For each audio file found, -# - It runs audio2csv to generate indices -# -# Assumptions: -# - AnalysisPrograms.exe is in current directory -# - You're working in Windows (though only trivial changes are required to work in Unix) -# - Users will customize the below variables before running the script - - -# Select the directory containing the files -$directory = "C:\temp\Emerald River Audio Snippets\20131227\" -# The directory to store the results -$base_output_directory = "C:\temp\indices_output" +<# +.SYNOPSIS + # A simple loop to generate indices for a folder of files. +.DESCRIPTION + Generates acoustic indices for multiple audio files. + Each recording will have it's indices save in a directory named after the input audio recording. +.PARAMETER $source_directory + A directory of audio files to process. +.PARAMETER $output_directory +.INPUTS + Optionally: input directories +.OUTPUTS + Files stored in $output_directory +.NOTES + Version: 2.0 + Author: Karlina Indraswari & Anthony Truskinger + Creation Date: 2017 + Purpose/Change: Updated docs links, add ap finder script + +.EXAMPLE + ./indices_loop.ps1 D:/Stud -time_zone_offset "10:00" -output_directory ./output +#> + +param( + [Parameter( + Position = 0, + Mandatory = $true)] + $source_directory, + [Parameter( + Position = 1, + Mandatory = $true)] + $output_directory +) + +$ap = . "$PSScriptRoot/find_ap.ps1" +$default_configs = Resolve-Path "$ap_path/../ConfigFiles" # Get a list of audio files inside the directory # (Get-ChildItem is just like ls, or dir) -$files = Get-ChildItem "$directory\*" -Include "*.mp3", "*.wav" +$files = Get-ChildItem "$source_directory\*" -Include "*.mp3", "*.wav" # iterate through each file -foreach($file in $files) { +foreach ($file in $files) { Write-Output ("Processing " + $file.FullName) # get just the name of the file $file_name = $file.Name # make a folder for results - $output_directory = "$base_output_directory\$file_name" - mkdir $output_directory + $instance_output_directory = Join-Path $output_directory $file_name + New-Item $instance_output_directory -ItemType Directory # prepare command - $command = ".\AnalysisPrograms.exe audio2csv `"$file`" `".\configFiles\Towsey.Acoustic.yml`" `"$output_directory`" -n" - + # for more information on how this command works, please see: + # https://ap.qut.ecoacoustics.info/technical/commands/analyze_long_recording.html + $command = "$ap audio2csv `"$file`" `"$default_configs\Towsey.Acoustic.yml`" `"$instance_output_directory`" -n --parallel" + # finally, execute the command Invoke-Expression $command } diff --git a/scripts/ribbon_plot.ps1 b/scripts/ribbon_plot.ps1 deleted file mode 100644 index 80d32c7c8..000000000 --- a/scripts/ribbon_plot.ps1 +++ /dev/null @@ -1,91 +0,0 @@ -# Creates a single image that shows ribbon plots stacked vertically -# Anthony Truskinger 2018 -# -# [2019] Deprecated: see DrawRibbonPlot command -# -# For each image file with the phrase "SpectralRibbon" in it's name -# found in a given directory it: -# - Groups the image based on it's type (e.g. ACI-ENT-EVN) -# - Sorts the images by date -# - Joins the spectral ribbons together to make a ribbon plot -# - Generates a date strip image -# - Then joins the date strip image to the ribbon plot -# - And saves the resulting image to the same folder -# -# Assumptions: -# - The input directories contain indices results -# - You want a grid of 1xN images (This could be paramterized in an update to the script) -# - The user will modify the $working_dirs variable before they run the script - - -$working_dirs = @( - "Y:\Results\20181010-115050\ConcatResult\TimedGaps\Sturt\2015July\Mistletoe", - "Y:\Results\20181010-115050\ConcatResult\TimedGaps\Sturt\2016Sep\Stud\CardA" -) - -$image_wildcard = "*.SpectralRibbon.png" - - -foreach ($working_dir in $working_dirs) { - Write-Output "Generating ribbon plots for $working_dir" - $all_images = Get-ChildItem -Recurse $working_dir -Include $image_wildcard - - # sort and bucket images - $image_types = @{} - - foreach ($image in $all_images) { - if ($image.FullName -match ".*(\d{8}).*__([-\w]{11})\..*") { - $date = [datetime]::ParseExact($Matches[1], "yyyyMMdd", [cultureinfo]::InvariantCulture) - $type = $Matches[2] - - $image_date = [PSCustomObject]@{ - Date = $date - File = $image - } - - if (-not $image_types[$type]) { - $image_types[$type] = @(, $image_date) - } - else { - $image_types[$type] += $image_date - } - } - else { - throw "Unexpected file name pattern encountered $image" - } - } - - foreach ($image_type in $image_types.Keys) { - - $sorted_images = $image_types[$image_type] | Sort-Object Date - - $n = $sorted_images.Count - # stack ribbons - $command = "magick montage -background '#333' -tile '1x' -gravity West -geometry +4+4 " - $command += ($sorted_images | ForEach-Object { - - " '$($_.File.FullName)' " - }) -join ' ' - $command += " '$working_dir/stacked_ribbon_plot_$image_type`_ribbons.png'" - - Write-Output "Generating ribbon plot for $image_type..." - Invoke-Expression $command - - # stack labels - $command = "magick -background '#333' -fill '#FFF' -gravity Center " - $command += ($sorted_images | ForEach-Object { - $date = $_.Date.ToString("yyyy-MM-dd") - " -size 200x40 -gravity center label:'$date'" - }) -join ' ' - $command += " -append '$working_dir/stacked_ribbon_plot_$image_type`_dates.png'" - - Write-Output "Generating y-axis for $image_type..." - Invoke-Expression $command - - # combine labels, ribbons, and then stick an axis on the bottom - $two_map = (Get-ChildItem ($sorted_images[0].File.Directory.FullName + "/*2Maps.png")).FullName - Write-Output "Joining ribbon plot, y-axis, and x-axis for $image_type..." - magick "$working_dir/stacked_ribbon_plot_$image_type`_dates.png" "$working_dir/stacked_ribbon_plot_$image_type`_ribbons.png" +append "xwd:-" | magick -gravity SouthEast "xwd:-" `( "$two_map" -crop "1440x18+0+0" -background '#333' -splice 4x0+0+0 `) -append "$working_dir/stacked_ribbon_plot_$image_type.png" - - } -} \ No newline at end of file diff --git a/scripts/spectrogram_loop.ps1 b/scripts/spectrogram_loop.ps1 new file mode 100644 index 000000000..ef3f0a9df --- /dev/null +++ b/scripts/spectrogram_loop.ps1 @@ -0,0 +1,69 @@ +# Generate standard-scale spectrograms from multiple one-minute recordings. +<# +.SYNOPSIS + Generates standard spectrograms for each minute for multiple audio files. +.DESCRIPTION + For every audio file found in a directory (searches recursively) this command + will run AP.exe to generate a set of standard spectrograms for each minute of + data. + NOTE: all spectrograms from all recordings will be saved to the same + directory! +.PARAMETER $source_directory + A directory of audio files to process. +.PARAMETER $output_directory +.INPUTS + Optionally: input directories +.OUTPUTS + Files stored in $output_directory +.NOTES + Version: 2.0 + Author: Anthony Truskinger & Michael Towsey + Creation Date: 2020-01-30 + Purpose/Change: Updated docs links, add ap finder script + +.EXAMPLE + ./spectrogrtam_loop.ps1 D:/Stud -output_directory ./output +#> + +param( + [Parameter( + Position = 0, + Mandatory = $true)] + $source_directory, + [Parameter( + Position = 1, + Mandatory = $true)] + $output_directory +) + +Write-Output "Draw standard-scale spectrograms of multiple one-minute recordings" + + +$ap = . "$PSScriptRoot/find_ap.ps1" +$default_configs = Resolve-Path "$ap_path/../ConfigFiles" + +$workshop_config = "$default_configs/Towsey.SpectrogramGenerator.yml" + +Write-Output "Using config File = $workshop_config" +Write-Output "Source Directory = $source_directory" +Write-Output "Output Directory = $output_directory" + +# Get a list of audio files inside the directory +# (Get-ChildItem is just like ls, or dir) +$recordingFiles = Get-ChildItem "$source_directory\*" -Include "*.wav" + +Write-Output "Found $($recordingFiles.Count) files" + + +# for each file found... +foreach ($file in $recordingFiles) { + Write-Output (">>>Processing: " + $file.Name) + + # Run the spectrogram generation command + # And also tell AP.exe to talk less (with --quiet)! + # for more information on how this command works, please see: + # https://ap.qut.ecoacoustics.info/technical/commands/analyze_long_recording.html + . $ap audio2sonogram $file $workshop_config $output_directory --quiet +} + +Write-Output "Finished!"