askap-vast · ajstewart · Oct 8, 2021 · Sep 8, 2021 · Sep 8, 2021 · Sep 9, 2021
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -8,6 +8,11 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 
 #### Added
 
+- Added new log files for arrow file creation and restore run and added to run detail page [#580](https://github.com/askap-vast/vast-pipeline/pull/580).
+- Added restore run test [#580](https://github.com/askap-vast/vast-pipeline/pull/580).
+- Added new run status of `DEL`, `Deleting` [#580](https://github.com/askap-vast/vast-pipeline/pull/580).
+- Added documentation pages on new action buttons [#580](https://github.com/askap-vast/vast-pipeline/pull/580).
+- Added UI action buttons to run-detail page to allow arrow file generation, deletion and restoration [#580](https://github.com/askap-vast/vast-pipeline/pull/580).
 - Added try-except error capture on pre-run checks to correctly assign pipeline run as failed if an error occurs [#576](https://github.com/askap-vast/vast-pipeline/pull/576).
 - Added support for ingesting Selavy catalogues in VOTable (XML) and CSV format [#565](https://github.com/askap-vast/vast-pipeline/pull/565)
 - Added new commands: `initingest` and `ingestimages` [#544](https://github.com/askap-vast/vast-pipeline/pull/544)
@@ -22,6 +27,8 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 
 #### Changed
 
+- Changed restore run command to only allow one run as input [#580](https://github.com/askap-vast/vast-pipeline/pull/580).
+- Changed existing documentation pages to reflect new buttons [#580](https://github.com/askap-vast/vast-pipeline/pull/580).
 - Moved creation of output backup files to occur before the config check [#576](https://github.com/askap-vast/vast-pipeline/pull/576).
 - Association test data updated with d2d fix [#574](https://github.com/askap-vast/vast-pipeline/pull/574).
 - Removed the timezone from the Timestamps being written to the the arrow file as this causes problems with vaex [#571](https://github.com/askap-vast/vast-pipeline/pull/571).
@@ -30,6 +37,8 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 
 #### Fixed
 
+- Fixed testing pandas equal deprecation warning [#580](https://github.com/askap-vast/vast-pipeline/pull/580).
+- Fixed restore run relations issue [#580](https://github.com/askap-vast/vast-pipeline/pull/580).
 - Fixed logic for full re-run requirement when UI run is being re-run from an error status [#576](https://github.com/askap-vast/vast-pipeline/pull/576).
 - Fixed d2d not being carried through the advanced association process [#574](https://github.com/askap-vast/vast-pipeline/pull/574).
 - Fixed old dictionary references in the documentation run config page [#572](https://github.com/askap-vast/vast-pipeline/pull/572).
@@ -41,6 +50,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 
 #### List of PRs
 
+- [#580](https://github.com/askap-vast/vast-pipeline/pull/580): feat, fix, doc: Added UI run action buttons.
 - [#576](https://github.com/askap-vast/vast-pipeline/pull/576): fix: Fixed UI re-run from errored status.
 - [#574](https://github.com/askap-vast/vast-pipeline/pull/574): fix: Fixed d2d assignment in advanced association.
 - [#572](https://github.com/askap-vast/vast-pipeline/pull/572): doc: Added required data page to documentation.

diff --git a/docs/exploringwebsite/runpages.md b/docs/exploringwebsite/runpages.md
@@ -10,15 +10,41 @@ Explanation of the table options can be found in the [DataTables section](datata
 
 ![!Pipeline Runs table.](../img/pipeline-runs.png){: loading=lazy }
 
-## Pipeline Run Detail Page
+| **Run Status**   | **Description**                                                                  |
+| ---------------- | -------------------------------------------------------------------------------- | 
+| **Completed**    | The run has successfully finished processing.                                    |
+| **Deleting**     | The pipeline run is currently being deleted.                                     | 
+| **Error**        | The run has encountered an error during processing and has stopped.              | 
+| **Initialised**  | The run has been created but not yet run.                                        | 
+| **Queued**       | The run has been sent to the scheduler for running but has not started yet.      | 
+| **Restoring**    | The pipeline run is currently being restored.                                    | 
+| **Running**      | The run is currently processing.                                                 | 
 
-This page presents all the information about the pipeline run, including options to edit the configuration file and to schedule the run for processing.
+## Pipeline Run Detail Page
 
-!!! note
-    For full details on how to process a run please refer to [this page](../using/processrun.md).
+This page presents all the information about the pipeline run, including options to edit the configuration file and to schedule the run for processing, restore the run, delete the run and generate the arrow measurement files.
 
 ![!Pipeline Run detail page.](../img/run-detail1.png){: loading=lazy }
 
+### Action Buttons
+
+![!Pipeline Run action buttons.](../img/action-buttons.png){: loading=lazy }
+
+For admins and creators of runs there are four action buttons available:
+
+* **Generate Arrow Files**  
+     A process to generate the arrow measurement files.
+     See [Generating Arrow Files](../../using/genarrow).
+* **Delete Run**  
+     Delete the pipeline run.
+     See [Deleting a Run](../../using/deleterun).
+* **Restore Run**  
+     A process to restore the run to the previous successful state.
+     See [Restoring a Run](../../using/restorerun).
+* **Add Images or Re-Process Run/Process Run**  
+     Process the pipeline run.
+     See [Processing a Run](../../using/processrun).
+
 ### Summary Cards
 The cards at the top of the page give a summary of the total numbers of:
 
@@ -73,11 +99,31 @@ The feedback may take a moment to appear as the check is performed.
 
 Users are able to read and post comments on a pipeline run using this form.
 
-### Log File
+### Log Files
+
+There are three log files available, which are present depending on the actions performed.
+
+#### Run Log File
+
+The full log file of the pipeline run process.
+Will display `N/A` if the run has not been processed.
 
 ![!Run log file.](../img/run-detail4.png){: loading=lazy }
 
-The full log file of the pipeline run is able to viewed.
+#### Restore Log File
+
+The log file of the restore run action. 
+Will display `N/A` if the action has not been performed.
+
+![!Restore log file.](../img/run-detail8.png){: loading=lazy }
+
+#### Generate Arrow Files Log File
+
+The log file of the generate arrow files action.
+Will display `N/A` if the action has not been performed.
+
+![!Generate arrow files log file.](../img/run-detail9.png){: loading=lazy }
+
 
 ### Image and Measurements Tables
 

diff --git a/docs/img/action-buttons.png b/docs/img/action-buttons.png
diff --git a/docs/img/add-images-detail.png b/docs/img/add-images-detail.png
diff --git a/docs/img/add-images-finished.png b/docs/img/add-images-finished.png
diff --git a/docs/img/arrow-files-available.png b/docs/img/arrow-files-available.png
diff --git a/docs/img/delete-run-button.png b/docs/img/delete-run-button.png
diff --git a/docs/img/delete-run-deleting-status.png b/docs/img/delete-run-deleting-status.png
diff --git a/docs/img/delete-run-modal.png b/docs/img/delete-run-modal.png
diff --git a/docs/img/delete-run-notification.png b/docs/img/delete-run-notification.png
diff --git a/docs/img/docs-example-run-detail.png b/docs/img/docs-example-run-detail.png
diff --git a/docs/img/generate-arrow-button.png b/docs/img/generate-arrow-button.png
diff --git a/docs/img/generate-arrow-files-log.png b/docs/img/generate-arrow-files-log.png
diff --git a/docs/img/generate-arrow-modal.png b/docs/img/generate-arrow-modal.png
diff --git a/docs/img/generate-arrow-notification.png b/docs/img/generate-arrow-notification.png
diff --git a/docs/img/process-run-button.png b/docs/img/process-run-button.png
diff --git a/docs/img/process-run-success.png b/docs/img/process-run-success.png
diff --git a/docs/img/restore-log.png b/docs/img/restore-log.png
diff --git a/docs/img/restore-notification.png b/docs/img/restore-notification.png
diff --git a/docs/img/restore-run-button.png b/docs/img/restore-run-button.png
diff --git a/docs/img/restore-run-modal.png b/docs/img/restore-run-modal.png
diff --git a/docs/img/run-detail.png b/docs/img/run-detail.png
diff --git a/docs/img/run-detail1.png b/docs/img/run-detail1.png
diff --git a/docs/img/run-detail4.png b/docs/img/run-detail4.png
diff --git a/docs/img/run-detail8.png b/docs/img/run-detail8.png
diff --git a/docs/img/run-detail9.png b/docs/img/run-detail9.png
diff --git a/docs/outputs/outputs.md b/docs/outputs/outputs.md
@@ -58,10 +58,8 @@ The two-epoch measurement pairs are also saved to arrow format due to the same r
 !!! note
     At the time of development `vaex` could not open parquets in an out-of-core context. This will be reviewed in the future if such functionality is added to `vaex`.
 
-!!! tip
-    The arrow files can be generated after a run has successfully completed (must be done by an administrator, refer to the admin command [`createmaeasarrow`](../adminusage/cli.md#createmeasarrow)).
-
 To enable the arrow files to be produced, the option `measurements.write_arrow_files` is required to be set to `True` in the pipeline run config.
+Alternatively, the arrow files can be generated after the completion of the run, see the [Generating Arrow Files page](../../using/genarrow) for full details.
 
 ### Image Data
 

diff --git a/docs/outputs/usingoutputs.md b/docs/outputs/usingoutputs.md
@@ -129,8 +129,8 @@ The table below shows what parameters act as keys to link data from the differen
 
 ## vast-tools
 
-[Link to the `vast-tools` repository](https://github.com/askap-vast/vast-tools){:target="_blank"}.
+[Link to the `vast-tools` documentation](https://vast-survey.org/vast-tools/){:target="_blank"}.
 
 VAST has developed a python library called `vast-tools` that makes the exploration of results from the pipeline simple and efficient, in addition to being designed to be used in a [Jupyter Notebook](https://jupyter.org) environment. 
 
-For full details, refer to the repository located on [GitHub](https://github.com/askap-vast/vast-tools){:target="_blank"} and be sure to take a look at the [example notebooks](https://github.com/askap-vast/vast-tools/tree/master/notebook-examples).
+Full details can be found in the documentation linked above, which also includes example notebooks of how to interact with the data.
diff --git a/docs/using/addtorun.md b/docs/using/addtorun.md
@@ -56,15 +56,14 @@ You can refresh the page to check the status of the run. You can confirm that th
 
 Once the processing has `Completed` the run detail page will now show the updated statistics and information of the run.
 
-![!Pipeline run successfully completed after adding images.](../img/add-images-finished.png){: loading=lazy }
+![!Pipeline run successfully completed after adding images.](../img/docs-example-run-detail.png){: loading=lazy }
 
 ## Restore Run to Pre-Add Version
 
-When images are added to a run, a backup is made of the run before proceeding which can be used to restore the run to the pre-addition version. For example, perhaps the wrong images were added or an error occured mid-addition that could not be resolved.
+When images are added to a run, a backup is made of the run before proceeding which can be used to restore the run to the pre-addition version. 
+For example, perhaps the wrong images were added or an error occurred mid-addition that could not be resolved.
 
-This command is currently unavailable to perform through the website interface but can be performed by an administrator of the pipeline via the command line interface. Please contact the adminstrator for the pipeline instance to request a run restoration.
-
-The command to perform this operation is `restorepiperun` and is described in the admin section [here](../../adminusage/cli/#restorepiperun).
+For full details see the documentation page for restoring a run [here](restorerun.md).
 
 !!! warning
     Do not add any further images if you wish to restore otherwise the backup version will be lost!
diff --git a/docs/using/deleterun.md b/docs/using/deleterun.md
@@ -1,5 +1,46 @@
 # Deleting a Run
 
-Currently a run can only be deleted by an administrator via the command line interface, and is not able to be performed using the website. Please contact the administrator if you wish to delete a run.
+This page describes how to delete a pipeline run through the website interface.
 
-Administrators can refer to the [`clearpiperun`](../adminusage/cli/#clearpiperun) command for details on how to reset a pipeline run.
+Deleting a run means that all outputs such as sources and associations are deleted from the database and the run itself is also removed.
+Images and the accompanying measurements are only removed if they were used solely by the deleted run.
+The run directory is also deleted that contains the output files and configuration files.
+
+A pipeline run can only be deleted by the creator or an administrator.
+
+!!! warning
+    As stated above, deleting a run through the website will also delete the full run directory, which includes the configuration files.
+    Please manually back up the configuration file if you think you are likely to revisit that particular run configuration in the future.
+
+!!! tip "Admin Tip"
+    Administrators can refer to the [`clearpiperun`](../adminusage/cli/#clearpiperun) command for details on how to reset a pipeline run via the command line.
+
+## Step-by-step Guide
+
+### 1. Navigate to the Run Detail Page
+
+Navigate to the detail page of the run you wish to delete.
+
+![!Pipeline run detail page.](../img/docs-example-run-detail.png){: loading=lazy }
+
+### 2. Click on the Delete Run Button
+
+Click on the `Delete Run` button at the top right of the page, to open the confirmation modal.
+
+![!Delete Run button.](../img/delete-run-button.png){: loading=lazy }
+![!Delete Run confirmation modal.](../img/delete-run-modal.png){: loading=lazy }
+
+### 3. Confirm Deletion
+
+To confirm the deletion click on the `Delete Run` button in the modal.
+Once pressed the website will direct back to the `Pipeline Runs` page and a confirmation message will appear in the top right.
+
+!!! note
+    Runs with lots of database entries may take a short time to delete.
+    Hence, the run may still appear in the pipeline runs list for a short time following the request with a status of `Deleting`.
+
+![!Delete Run notification.](../img/delete-run-notification.png){: loading=lazy }
+
+Refreshing the page will show a deleting status if the process is still running:
+
+![!Delete Run deleting status.](../img/delete-run-deleting-status.png){: loading=lazy }
diff --git a/docs/using/genarrow.md b/docs/using/genarrow.md
@@ -0,0 +1,66 @@
+# Generating Arrow Files
+
+This page describes how to generate the measurement arrow files for a pipeline run if the option in the configuration file to create them was turned off.
+
+Arrow files only be generated by the creator or an administrator.
+
+Two files are produced by the method:
+
+| File<img width=380/>  | Description |
+| ---- | ----------- |
+| `measurements.arrow` | An [Apache Arrow](https://arrow.apache.org/overview/){:target="_blank"} format file containing all the measurements associated with the pipeline run (see [Arrow Files](#arrow-files)). Extra processing is performed in the creation of this file such that source ids are already in place for the measurements. |
+| `measurement_pairs.arrow` | An [Apache Arrow](https://arrow.apache.org/overview/){:target="_blank"} format file containing all the measurement pair metrics (see [Arrow Files](#arrow-files)). |
+
+!!! tip "Arrow Files Available"
+    Users can see if arrow files are present for the run of interest by checking the respective run detail page.
+    ![!Arrow files available.](../img/arrow-files-available.png){: loading=lazy }
+
+
+!!! tip "Admin Tip"
+    The arrow files can be generated using the command line using the command [`createmaeasarrow`](../adminusage/cli.md#createmeasarrow)).
+
+## Why Create Arrow Files?
+
+Large pipeline runs (hundreds of images) mean that to read the measurements, hundreds of parquet files need to be read in, and can contain millions of rows.
+This can be slow using libraries such as pandas, and also consumes a lot of system memory.
+
+Instead, if the measurements are saved in the [Apache Arrow](https://arrow.apache.org/overview/){:target="_blank"} format, libraries such as [`vaex`](https://vaex.io){:target="_blank"} are able to open `.arrow` files in an out-of-core context so the memory footprint is hugely reduced along with the reading of the file being very fast.
+The two-epoch measurement pairs are also saved to arrow format due to the same reasons.
+
+See [Reading with vaex](../../outputs/usingoutputs#reading-with-vaex) for further details on using `vaex`.
+
+## Step-by-step Guide
+
+### 1. Navigate to the Run Detail Page
+
+Navigate to the detail page of the run you wish to generate arrow files for.
+
+![!Pipeline run detail page.](../img/docs-example-run-detail.png){: loading=lazy }
+
+### 2. Select the Generate Arrow Files Option
+
+Click the `Generate Arrow Files` option at the top-right of the page.
+
+![!Generate arrow button.](../img/generate-arrow-button.png){: loading=lazy }
+
+This will open the generate arrow files modal.
+
+![!Generate arrow modal.](../img/generate-arrow-modal.png){: loading=lazy }
+
+### 3. Submit Generate Arrow Files Request
+
+It is possible to overwrite existing arrow files by toggling the `Overwrite Current Files` option.
+
+When ready, click the `Generate Arrow Files` button on the modal to submit the generate request.
+A notification will show to indicate whether the submission was successful.
+
+![!Generate arrow files notification.](../img/generate-arrow-notification.png){: loading=lazy }
+
+### 4. Refresh and Check the Generate Arrow Files Log File
+
+It is possible to check the progress by looking at the Generate Arrow Files Log File which can be found on the run detail page.
+The log will not be refreshed automatically and instead the page needs to be manually refreshed.
+
+Once completed the arrow files will be available for use.
+
+![!Generate arrow files log file.](../img/generate-arrow-files-log.png){: loading=lazy }
diff --git a/docs/using/restorerun.md b/docs/using/restorerun.md
@@ -0,0 +1,58 @@
+# Restoring a Run
+
+This page details the process of restoring a pipeline run to the previous successful version.
+
+When images are added to a run, a backup is made of the run before proceeding which can be used to restore the run to the pre-addition version. 
+For example, perhaps the wrong images were added or an error occurred mid-addition that could not be resolved.
+
+A pipeline run can only be restored by the creator or an administrator.
+
+!!! tip "Admin Tip"
+    This process can also be launched via the command line using the `restorepiperun` command. 
+    It is described in the admin section [here](../../adminusage/cli/#restorepiperun).
+
+!!! warning "Warning: One time use"
+    This process can only be used to restore the run once. 
+    I.e. it is not possible to restore the run to an even earlier version.
+
+## Step-by-step Guide
+
+In this example, the `docs_example_run` will be restored to the state before the images were added in the [Adding Images to a Run](addtorun.md) example.
+
+### 1. Navigate to the Run Detail Page
+
+Navigate to the detail page of the run you wish to restore.
+
+![!Pipeline run detail page.](../img/docs-example-run-detail.png){: loading=lazy }
+
+### 2. Select the Restore Run Option
+
+Click the `Restore Run` option at the top-right of the page.
+
+![!Restore run button.](../img/restore-run-button.png){: loading=lazy }
+
+This will open the restore confirmation modal.
+
+![!Restore run modal.](../img/restore-run-modal.png){: loading=lazy }
+
+### 3. Check the Restore Configuration
+
+Shown in the modal is the configuration file of the previous successful run.
+This can be used to check that the images listed here are those that are expected.
+
+Debug level logging can also be turned on using the toggle button.
+
+When ready, click the `Restore Run` button on the modal to submit the restore request.
+A notification will show to indicate whether the submission was successful.
+
+![!Restore notification.](../img/restore-notification.png){: loading=lazy }
+
+### 4. Refresh and Check the Restore Log File
+
+While restoring the pipeline run will show a status of `Restoring`.
+It is possible to check the progress by looking at the Restore Log File which can be found on the run detail page.
+The log will not be refreshed automatically and instead the page needs to be manually refreshed.
+
+Upon a successful restoration the status will be changed back to `Completed`.
+
+![!Restore log file.](../img/restore-log.png){: loading=lazy }
diff --git a/docs/using/runconfig.md b/docs/using/runconfig.md
@@ -478,7 +478,8 @@ Boolean. When `True` then two `arrow` format files are produced:
 Producing these files for large runs (200+ images) is recommended for post-processing. Defaults to `False`.
 
 !!! note
-    The arrow files can optionally be produced after the run has completed by an administrator.
+    The arrow files can optionally be produced after the run has completed.
+    See the [Generating Arrow Files page](genarrow.md).
 
 **`measurements.ra_uncertainty`**
 Float. Defines an uncertainty error to the RA that will be added in quadrature to the existing source extraction error. Used to represent a systematic positional error. Unit is arcseconds. Defaults to 1.0.

diff --git a/mkdocs.yml b/mkdocs.yml
@@ -80,9 +80,11 @@ nav:
     - Run Configuration: using/runconfig.md
     - Processing a Run: using/processrun.md
     - Adding Images to a Run: using/addtorun.md
+    - Restoring a Run: using/restorerun.md
     - Deleting a Run: using/deleterun.md
+    - Generating Arrow Files: using/genarrow.md
     - Exploring the Pipeline Website:
-      - Overview: exploringwebsite/websiteoverview.md
+      - Website Overview: exploringwebsite/websiteoverview.md
       - DataTables: exploringwebsite/datatables.md
       - Pipeline Run Pages: exploringwebsite/runpages.md
       - Source Pages: exploringwebsite/sourcepages.md