Code reference documentation update

CHANGELOG.md

@@ -8,6 +8,8 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 
 #### Added
 
+- Added script to auto-generate code reference documentation pages [#480](https://github.com/askap-vast/vast-pipeline/pull/480).
+- Added code reference section to documentation [#480](https://github.com/askap-vast/vast-pipeline/pull/480).
 - Added new pages and sections to documentation [#471](https://github.com/askap-vast/vast-pipeline/pull/471)
 - Added `requirements/environment.yml` so make it easier for Miniconda users to get the non-Python dependencies [#472](https://github.com/askap-vast/vast-pipeline/pull/472).
 - Added `pyproject.toml` and `poetry.lock` [#472](https://github.com/askap-vast/vast-pipeline/pull/472).
@@ -22,6 +24,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 
 #### Changed
 
+- Changed non-google format docstrings to google format [#480](https://github.com/askap-vast/vast-pipeline/pull/480).
 - Changed some documentation layout and updated content [#471](https://github.com/askap-vast/vast-pipeline/pull/471).
 - Changed the `vaex` dependency to `vaex-arrow` [#472](https://github.com/askap-vast/vast-pipeline/pull/472).
 - Set `CREATE_MEASUREMENTS_ARROW_FILES = True` in the basic association test config [#472](https://github.com/askap-vast/vast-pipeline/pull/472).
@@ -58,6 +61,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 
 #### List of PRs
 
+- [#480](https://github.com/askap-vast/vast-pipeline/pull/480) feat: Code reference documentation update.
 - [#471](https://github.com/askap-vast/vast-pipeline/pull/471) feat: Documentation update.
 - [#472](https://github.com/askap-vast/vast-pipeline/pull/472) feat: Simplify install.
 - [#473](https://github.com/askap-vast/vast-pipeline/pull/473) fix: discard the selavy unit row before reading.

docs/faqs.md

@@ -11,7 +11,9 @@ As seen from the [Image Ingest page](design/imageingest.md), the pipeline does n
 * Convert the component output from a different source extractor to match that of the [`Selavy` component file](https://www.atnf.csiro.au/computing/software/askapsoft/sdp/docs/current/analysis/postprocessing.html#output-files){:target="_blank"}.
 
 The pipeline was also designed in a way such that other source extractor 'translators' could be plugged into the pipeline. 
-So a further option is to develop new translators such that the pipeline can read in output from other source extractors. Please open a discussion or issue on GitHub if you intend to give this a go!
+So a further option is to develop new translators such that the pipeline can read in output from other source extractors. 
+The translators can be found in [`vast_pipeline/surveys/translators.py`](reference/survey/translators.md).
+Please open a discussion or issue on GitHub if you intend to give this a go!
 
 !!! bug
     In reading the code recently I have a suspicion the FITS reading code is reliant on the `TELESCOP` FITS header being equal to `ASKAP`. 

docs/gen_doc_stubs.py

@@ -0,0 +1,32 @@
+#!/usr/bin/env python
+
+from pathlib import Path
+import mkdocs_gen_files
+
+
+# only allow certain directory trees from vast_pipeline.
+exclude_dirs = ['migrations', 'tests']
+
+# Problem files because of Django and mkdocs
+# See https://github.com/mkdocstrings/mkdocstrings/issues/141
+# __init__ is there just to avoid building these.
+# TODO: Fix Django pages for mkdocs.
+problem_files = ['serializers.py', 'views.py', '__init__.py']
+
+for path in Path("vast_pipeline").glob("**/*.py"):
+
+    if len(path.parts) > 2 and path.parts[1] in exclude_dirs:
+        continue
+
+    if path.name in problem_files:
+        continue
+
+    doc_path = Path(
+        "reference", path.relative_to("vast_pipeline")
+    ).with_suffix(".md")
+
+    with mkdocs_gen_files.open(doc_path, "w") as f:
+        ident = ".".join(path.relative_to(".").with_suffix("").parts)
+        print("::: " + ident, file=f)
+
+    mkdocs_gen_files.set_edit_path(doc_path, Path("..", path))

docs/theme/css/extra.css

@@ -13,3 +13,9 @@
   color: #4e73df;
 }
 
+/* Indentation for mkdocstrings items. */
+div.doc-contents:not(.first) {
+  padding-left: 25px;
+  border-left: 4px solid rgba(230, 230, 230);
+  margin-bottom: 80px;
+}

mkdocs.yml

@@ -88,9 +88,49 @@ nav:
     - Docs: developing/docsdev.md
     - Bechmarks: developing/profiling.md
   #   - Tips: developing/tips.md like this https://stackoverflow.com/questions/60160823/how-to-properly-parallelize-generic-code-with-numba-dask
-  # - Reference:
-  #   - Association Logic: reference/association.md
-  #   - Configuration: reference/configuration.md
+  - Code Reference:
+    - vast_pipeline:
+      - image:
+        - main.py: reference/image/main.md
+        - utils.py: reference/image/utils.md
+      - management:
+        - commands:
+          - clearpiperun.py: reference/management/commands/clearpiperun.md
+          - createmeasarrow.py: reference/management/commands/createmeasarrow.md
+          - debugrun.py: reference/management/commands/debugrun.md
+          - importsurvey.py: reference/management/commands/importsurvey.md
+          - initpiperun.py: reference/management/commands/initpiperun.md
+          - restorepiperun.py: reference/management/commands/restorepiperun.md
+          - runpipeline.py: reference/management/commands/runpipeline.md
+        - helpers.py: reference/management/helpers.md
+      - pipeline:
+        - association.py: reference/pipeline/association.md
+        - errors.py: reference/pipeline/errors.md
+        - finalise.py: reference/pipeline/finalise.md
+        - forced_extraction.py: reference/pipeline/forced_extraction.md
+        - loading.py: reference/pipeline/loading.md
+        - main.py: reference/pipeline/main.md
+        - model_generator.py: reference/pipeline/model_generator.md
+        - new_sources.py: reference/pipeline/new_sources.md
+        - utils.py: reference/pipeline/utils.md
+      - survey:
+        - catalogue.py: reference/survey/catalogue.md
+        - translators.py: reference/survey/translators.md
+      - utils:
+        - auth.py: reference/utils/auth.md
+        - unit_tags.py: reference/utils/unit_tags.md
+        - utils.py: reference/utils/utils.md
+        - view.py: reference/utils/view.md
+      - admin.py: reference/admin.md
+      - apps.py: reference/apps.md
+      - context_processors.py: reference/context_processors.md
+      - converters.py: reference/converters.md
+      - forms.py: reference/forms.md
+      - models.py: reference/models.md
+      - plots.py: reference/plots.md
+      # - serializers.py: reference/serializers.md
+      - urls.py: reference/urls.md
+      # - views.py: reference/views.md
   - Architecture:
     - Architecture Intro: architecture/intro.md
     - Database Schema: architecture/database.md
@@ -102,6 +142,10 @@ nav:
 
 plugins:
   - search
+  - gen-files:
+      scripts:
+      # - docs/gen_credits.py
+      - docs/gen_doc_stubs.py
   - minify:
       minify_html: true
       minify_js: true
@@ -114,7 +158,7 @@ plugins:
       handlers:
         python:
           rendering:
-            show_source: false
+            show_source: true
           setup_commands:
             - "import os"
             - "import django"