Add example scripts and documentation

CONTRIBUTING.md

@@ -49,19 +49,7 @@ You can install it from here: https://github.com/powershell/powershell#get-power
 ### 2. A **.NET Core SDK**
 
 - We aim to use the latest stable version
-- You can verify the version our project is using by looking in the [global.json](https://github.com/QutEcoacoustics/audio-analysis/tree/master/build/global.json) file
-
-Recommended choice is to run the `dotnet-install` script in the [./build/](https://github.com/QutEcoacoustics/audio-analysis/tree/master/build) folder.
-
-PowerShell:
-```powershell
-> ./build/dotnet-install.ps1
-```
-
-Bash:
-```shell
-$ ./build/dotnet-install.sh
-```
+- You can verify the version our project is using by looking in the [global.json](https://github.com/QutEcoacoustics/audio-analysis/tree/master/global.json) file
 
 Alternately, you can download a SDK from here: <https://dotnet.microsoft.com/download/dotnet-core/>. Note: you want the _Build apps - SDK_ download.
 

docs/basics/scripting.md

@@ -0,0 +1,44 @@
+# Scripting AP.exe
+
+_AnalysisPrograms.exe_ works best when processing single audio files.
+This has a few advantages:
+
+**It simplifies the code**. We don't need to know about the particular
+way you store your data, or the way you want your data processed.
+
+**It allows for greater customization**. By building a set of composable
+tools, it allows you to choose what analysis is done, and when. You want it
+all? You got it. You only want the first bit done once, and the second bit done
+100 times with parameter sweeps? Go for it.
+
+**It is easier to parallelize**. You want to use your university HPC?
+Write your own distributed analysis? Or just run it in sequence? That's all
+your choice.
+
+Despite these advantages, we recognize it is useful to have example of
+how large analyses might be completed. Thus the scripts you find in this
+folder contain various examples (in various languages) of composing
+workflows with _AnalysisPrograms.exe_.
+
+**You can find a collection of example scripts here: https://github.com/QutEcoacoustics/audio-analysis/tree/master/scripts**
+
+## PowerShell
+
+You'll see a lot of scripts in that folder that are written in PowerShell.
+If you're not familiar with it, you can consider it as the Windows equivalent
+of the Bash shell. 
+
+We like PowerShell because we think the syntax is more reasonable than Bash
+and the enhanced support for dates, times, and regular expressions are well worth the investment.
+
+As of [PowerShell 6.0](https://github.com/PowerShell/PowerShell#-powershell)
+the shell is cross platform!. If you're not
+convinced, the scripts should be easy enough to reimplement in your favourite
+language (like Bash)--and we would of course appreciate any translated
+scripts sent back to us as contributed examples.
+
+## Any other language
+
+You can use any programming language to script AP.exe.
+
+R is a popular choice. We have a short guide for [using AP.exe with R](./using_r.md)

docs/basics/toc.yml

@@ -10,7 +10,9 @@
 - name: Config files
   href: config_files.md
 
-- name: Scripting with R
+- name: Scripting AP.exe
+  href: scripting.md
+- name: Scripting AP.exe with R
   href: using_r.md
 
 - name: Dates

scripts/README.md

@@ -1,62 +1,14 @@
 # Scripts
 
-_AnalysisPrograms.exe_ works best when processing single audio files.
-This has a few advantages:
-
-**It simplifies the code**. We don't need to know about the particular 
-way you store your data, or the way you want your data processed.
-
-**It allows for greater customization**. By building a set of composable
-tools, it allows you to choose what analysis done, and when. You wan't it
-all? You got it. You only want the first bit done once, and the second done
-100 times with parameter sweeps? Go for it.
-
-**It is easier to parallelize**. You want to use your university HPC?
-Write your own distributed analysis? Or just run it in sequence? That's all
-your choice.
-
-Despite these advantages, we recognize it is useful to have example of
-how large analyses might be completed. Thus the scripts you find in this
-folder contain various examples (in various languages) of composing 
-workflows with _AnalysisPrograms.exe_.
+A folder of example scripts that use AP.exe.
 
 ## Contributions appreciated!
 
-If you right a script that gets the job done for you, we'd be happy to
-include it here as an example for others to use.
-
-## PowerShell
-
-You'll see a lot of scripts in this folder that are written in PowerShell.
-If you're not familiar with it, you can consider it as the Windows equivalent
-of the Bash shell. @atruskie like's PowerShell because in their personal
-opinion, the syntax is more reasonable than Bash, and the enhanced support
-for dates, times, and regular expressions are well worth the investment.
-
-As of [PowerShell 6.0](https://github.com/PowerShell/PowerShell#-powershell)
-the shell is cross platform and well worth investigating. If you're not
-convinced, the scripts should be easy enough to reimplement in your favourite
-language (like Bash)--and we would of course appreciate any translated
-scripts sent back to us as contributed examples.
-
-## Example headers
+If you write a script that gets the job done for you, we'd be happy to
+include it here as an example for others to use. Please:
 
-We'd like to see each example script prefaced with a documentation header.
-A suggested format:
+1. fork the repository
+2. Add your script to this folder
+3. Send us a pull-request!
 
-```
-# A simple loop to XXX for a folder of files.
-# Jolene Blogs 2017
-#
-# For each audio file found,
-# - It runs does XXX
-# - It then also does YYY
-# - And sometimes does ZZZ
-# - ...
-#
-# Assumptions:
-# - AnalysisPrograms.exe is in current directory
-# - ...
 
-... script starts here ...
-```

scripts/find_ap.ps1

@@ -0,0 +1,35 @@
+param()
+
+$command = Get-Command AnalysisPrograms* -ErrorAction SilentlyContinue
+
+if ($null -ne $command) {
+    return $command.Path;
+    exit 0;
+}
+
+if ($IsWindows) {
+
+    $command = Get-Command C:\AP\AnalysisPrograms.exe -ErrorAction SilentlyContinue
+
+    if ($null -ne $command) {
+        return $command.Path;
+        exit 0;
+    }
+}
+
+if ($IsLinux) {
+    $command = Get-Command /AP/AnalysisPrograms.exe -ErrorAction SilentlyContinue
+
+    if ($null -ne $command) {
+        return $command.Path;
+        exit 0;
+    }
+}
+
+Write-Error @"
+Can't find AnalysisPrograms.exe. 
+Please install it using the instructions from: 
+https://github.com/QutEcoacoustics/audio-analysis/blob/master/docs/installing.md
+"@
+
+exit 1;

scripts/generate_zooming.ps1

@@ -13,29 +13,32 @@
 # - Users will customize the below variables before running the script
 
 $working_dir = "D:/Temp/zooming"
-$csv = "$working_dir\all_oxley_creek_recordings.csv"
+$csv = "$working_dir/all_oxley_creek_recordings.csv"
 $output = $working_dir
-# The config files we copied from the ConfigFiles directory and customised for our purposes.
-$indices_config_file = "$working_dir\Towsey.Acoustic.Zooming.yml"
-$zooming_config_file = "$working_dir\SpectrogramZoomingConfig.yml"
+
+# find the path to analysis programs
+$ap_path = (Get-Command AnalysisPrograms.exe).Path
+$default_configs = Resolve-Path "$ap_path/../ConfigFiles"
+
+$indices_config_file = "$default_configs/Towsey.Acoustic.Zooming.yml"
+$zooming_config_file = "$default_configs/SpectrogramZoomingConfig.yml"
 
 $remote_user = "ubuntnu"
 $remote_server = "server.example.com"
 $remote_path = "/home/ubuntu/data/original_audio"
 
 # helper functions
 
-# find the path to analysis programs
-$ap_path = (gcm AnalysisPrograms.exe).Path
 
-# just checks whether a previous run was successful 
+
+# just checks whether a previous run was successful
 function HasAlreadyRun($results_dir) {
     $log = Join-Path $results_dir "log.txt"
     if (Test-Path $log) {
         $match = Get-Content -Tail 1 $log | Select-String "ERRORLEVEL: (\d+)"
         return $match.Matches.Groups[1].Value -eq 0
     }
-    
+
     return $false
 }
 
@@ -54,14 +57,14 @@ $recordings = Import-Csv $csv
 $results = @()
 foreach ($recording in $recordings) {
     Write-Output "Starting new recording $($recording.uuid)"
-    
+
     # create a results object to store results
-    $result =  New-Object "pscustomobject" | Select-Object Download,Indices,Images
+    $result = New-Object "pscustomobject" | Select-Object Download, Indices, Images
 
     # extract all needed meta data to create a path to the remote file
     # constructs a path that looks like: ".../data/b2/b24460cf-e25e-44c9-9034-af9b0a1ddcbe_20121019-140000Z.wav
     $uuid = $recording.uuid
-    $prefix = $uuid.Substring(0,2)
+    $prefix = $uuid.Substring(0, 2)
     $date = (Get-Date $recording.recorded_date).ToString("yyyyMMdd-HHmmssZ")
     $name = "$uuid`_$date.wav"
     $remote_path = "$remote_path/$prefix/$name"
@@ -82,7 +85,7 @@ foreach ($recording in $recordings) {
     $instance_output = $local_path + "_results"
     # generate indices
 
-    if (HasAlreadyRun $instance_output) {    
+    if (HasAlreadyRun $instance_output) {
         $result.Indices = "0*"
         Write-Output "Skipping indices generation for $uuid - already completed"
     }
@@ -97,7 +100,7 @@ foreach ($recording in $recordings) {
 
     # generate zooming tiles
     $indices_dir = Join-Path $instance_output "Towsey.Acoustic"
-    AnalysisPrograms.exe DrawZoomingSpectrograms $indices_dir $zooming_config_file $instance_output -o "sqlite3" -z "Tile" -n  | IndentOutput
+    AnalysisPrograms.exe DrawZoomingSpectrograms $indices_dir $zooming_config_file $instance_output -o "sqlite3" -z "Tile" -n | IndentOutput
     $result.Images = $LASTEXITCODE
     $results += $result
 

scripts/indices_and_concat.ps1

@@ -0,0 +1,91 @@
+<#
+.SYNOPSIS
+  Generates acoustic indices for multiple audio files and concatenates the output.
+.DESCRIPTION
+  Generates acoustic indices for multiple audio files and concatenates the output.
+  Expects each directory to contain audio files for one location.
+.PARAMETER $input_directories
+    A directory of audio files to process.
+.PARAMETER $output_directory
+.INPUTS
+  Optionally: input directories
+.OUTPUTS
+  Files stored in $output_directory
+.NOTES
+  Version:        2.0
+  Author:         Anthony Truskinger
+  Creation Date:  2020-01-30
+  Purpose/Change: Updated docs links, add ap finder script
+
+.EXAMPLE
+  ./indices_and_concat.ps1 D:/Stud D://Thompson -time_zone_offset "10:00" -output_directory ./output
+#>
+
+#requires -version 6
+
+param(
+    [Parameter(
+        Position = 0,
+        Mandatory = $true,
+        ValueFromRemainingArguments = $true,
+        ValueFromPipeline = $true)]
+    [System.IO.DirectoryInfo[]]$input_directories,
+
+    [Parameter(
+        Mandatory = $true)]
+    [System.IO.DirectoryInfo]$output_directory,
+
+    [Parameter(
+        Mandatory = $true)]
+    [string]$time_zone_offset,
+
+    $name_filter = "*"
+
+)
+
+# Do not continue running the script if a problem is encountered
+$ErrorActionPreference = "Stop"
+
+
+# get the path for AP.exe. When do this to resolve some nice default config files.
+# TODO: remove this when the default config file feature is implemented in AP.exe
+$ap_path = . "$PSScriptRoot/find_ap.ps1"
+$default_configs = Resolve-Path "$ap_path/../ConfigFiles"
+
+foreach ($input_directory in $input_directories) {
+    Write-Output "Processing $input_directory"
+
+    $current_group = $input_directory.Name
+
+    $audio_files = Get-ChildItem -Recurse -File $input_directory -Include "*.wav"
+    $filtered_files = $audio_files | Where-Object { $_.Name -ilike $name_filter }
+
+    $counter = 0;
+    foreach ($file in $filtered_files) {
+        $counter++
+        Write-Output "Generating indices for $file, file $counter of $($filtered_files.Count)"
+        $name = $file.Name
+
+        # for more information on how this command works, please see:
+        # https://ap.qut.ecoacoustics.info/technical/commands/analyze_long_recording.html
+        AnalysisPrograms.exe audio2csv $file "$default_configs/Towsey.Acoustic.yml" "$output_directory/$current_group/indices/$name" --no-debug --parallel
+    }
+
+    Write-Output "Now concatenating files for $current_group"
+
+    # for more information on how this command works, please see:
+    # https://ap.qut.ecoacoustics.info/technical/commands/concatenate_index_files.html
+    AnalysisPrograms.exe ConcatenateIndexFiles `
+        --input-data-directory "$output_directory/$current_group/indices" `
+        --output-directory "$output_directory" `
+        -z $time_zone_offset `
+        --file-stem-name $current_group `
+        --directory-filter "*.*" `
+        --index-properties-config "$default_configs/IndexPropertiesConfig.yml" `
+        --false-colour-spectrogram-config "$default_configs/SpectrogramFalseColourConfig.yml" `
+        --draw-images `
+        --no-debug
+
+}
+
+Write-Output "Complete!"