Merge pull request #676 from askap-vast/iss605-remove-home-counts

Removed home counts and new source count
askap-vast · Jan 25, 2023 · 94bfccf · 94bfccf
2 parents 67867c9 + bf216e9
commit 94bfccf
Show file tree

Hide file tree

Showing 12 changed files with 79 additions and 98 deletions.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -8,6 +8,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 
 #### Added
 
+- Added `n_new_sources` column to run model to store the number of new sources in a pipeline run [#676](https://github.com/askap-vast/vast-pipeline/pull/676).
 - Added `MAX_CUTOUT_IMAGES` to the pipeline settings to limit the number of postage stamps displayed on the source detail page [#658](https://github.com/askap-vast/vast-pipeline/pull/658).
 - Added run config option to skip calculating measurement pair metrics [#655](https://github.com/askap-vast/vast-pipeline/pull/655).
 - Added support for Python 3.10 [#641](https://github.com/askap-vast/vast-pipeline/pull/641).
@@ -66,6 +67,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 
 #### Fixed
 
+- Fixed link to JupyterHub [#676](https://github.com/askap-vast/vast-pipeline/pull/676).
 - Ensure Image models are not created if the catalogue ingest fails [#648](https://github.com/askap-vast/vast-pipeline/pull/648).
 - Fixed run failures caused by attempting to force fit images with empty catalogues [#653](https://github.com/askap-vast/vast-pipeline/pull/653).
 - Fixed a Bokeh regression that requires LabelSet values to be strings [#652](https://github.com/askap-vast/vast-pipeline/pull/652).
@@ -94,12 +96,15 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 
 #### Removed
 
+- Removed sky coverage plot from homepage [#676](https://github.com/askap-vast/vast-pipeline/pull/676).
+- Removed various counts from homepage [#676](https://github.com/askap-vast/vast-pipeline/pull/676).
 - Removed support for Python 3.7 [#641](https://github.com/askap-vast/vast-pipeline/pull/641).
 - Removed Measurements table page and measurements table from run detail page [#636](https://github.com/askap-vast/vast-pipeline/pull/636).
 - Removed the unique constraint on `models.Measurement.name` [#583](https://github.com/askap-vast/vast-pipeline/pull/583).
 
 #### List of PRs
 
+- [#676](https://github.com/askap-vast/vast-pipeline/pull/676): Removed home counts and new source count.
 - [#665](https://github.com/askap-vast/vast-pipeline/pull/665): Update Gr1N/setup-poetry to v7.
 - [#658](https://github.com/askap-vast/vast-pipeline/pull/658): feat: Add MAX_CUTOUT_IMAGES setting.
 - [#655](https://github.com/askap-vast/vast-pipeline/pull/655): feat: Add run config option to disable measurement pairs.

diff --git a/docs/exploringwebsite/websiteoverview.md b/docs/exploringwebsite/websiteoverview.md
@@ -10,15 +10,7 @@ For admins, refer to the following pages for details of the configuration and se
 
 ![!VAST Pipeline homepage.](../img/homepage.png){: loading=lazy }
 
-The homepage provides a summary of the data currently held in the pipeline instance that has been processed. The four cards at the top of the homepage provide total values
-for the amount of pipeline runs, images, measurements and sources that are stored in the database. 
-Clicking any of them will take you to the respective overview page for the data type (excluding measurements).
-
-!!! note
-    The totals presented on the homepage are totals for all pipeline runs combined!
-
-Also displayed is a sky region map that shows all the areas of the sky that have had successful and completed pipeline runs performed.
-
+The homepage contains a welcome message and links to popular pages of the list of pipeline runs, the source query page or the list of images.
 
 ## Navbar
 

diff --git a/docs/img/homepage.png b/docs/img/homepage.png
diff --git a/docs/img/schema.png b/docs/img/schema.png
diff --git a/docs/img/schema_pipeline.png b/docs/img/schema_pipeline.png
diff --git a/templates/base.html b/templates/base.html
@@ -105,7 +105,7 @@
           <span>Raise Issue</span>
         </a>
       </li>
-      
+
       <li class="nav-item">
         <a class="nav-link collapsed" target="_blank" href="https://github.com/askap-vast/vast-pipeline/discussions/new">
           <i class="far fa-comments"></i>
@@ -121,7 +121,7 @@
         <div id="collapseLinks" class="collapse" aria-labelledby="headingDataset" data-parent="#accordionSidebar">
           <div class="bg-white py-2 collapse-inner rounded">
             <a class="collapse-item" target="_blank" href="https://github.com/askap-vast">GitHub</a>
-            <a class="collapse-item" target="_blank" href="https://data.vast-survey.org">JupyterHub</a>
+            <a class="collapse-item" target="_blank" href="https://data.vast-survey.org/hub/">JupyterHub</a>
             <a class="collapse-item" target="_blank" href="https://vast-survey.org">Website</a>
             <a class="collapse-item" target="_blank" href="https://github.com/askap-vast/vast-project/wiki">Wiki</a>
           </div>

diff --git a/templates/index.html b/templates/index.html
@@ -33,15 +33,14 @@ <h1 class="h3 mb-4 text-gray-800">Welcome back!</h1>
   </div>
 
   <div class="row">
-    
+
     <!-- Nr Piperuns -->
-    <div class="col-xl-3 col-md-6 mb-4">
+    <div class="col-xl-4 col-md-6 mb-4">
         <div class="card border-left-success shadow h-100 py-2">
           <div class="card-body">
             <div class="row no-gutters align-items-center">
               <div class="col mr-2">
-                <div class="text-xs font-weight-bold text-success text-uppercase mb-1">Total Nr Pipeline Runs</div>
-                <div class="h5 mb-0 font-weight-bold text-gray-800">{{ totals.nr_pruns|intcomma }}</div>
+                <div class="text font-weight-bold text-success mb-1">Go to Pipeline Runs</div>
               </div>
               <div class="col-auto">
                 <i class="fas fa-bookmark fa-2x text-gray-300"></i>
@@ -52,63 +51,44 @@ <h1 class="h3 mb-4 text-gray-800">Welcome back!</h1>
         </div>
     </div>
 
-    <!-- Nr Images -->
-    <div class="col-xl-3 col-md-6 mb-4">
-        <div class="card border-left-primary shadow h-100 py-2">
-          <div class="card-body">
-            <div class="row no-gutters align-items-center">
-              <div class="col mr-2">
-                <div class="text-xs font-weight-bold text-primary text-uppercase mb-1">Total Nr Images</div>
-                <div class="h5 mb-0 font-weight-bold text-gray-800">{{ totals.nr_imgs|intcomma }}</div>
-              </div>
-              <div class="col-auto">
-                <i class="fas fa-images fa-2x text-gray-300"></i>
-              </div>
+    <!-- Nr Sources -->
+    <div class="col-xl-4 col-md-6 mb-4">
+      <div class="card border-left-warning shadow h-100 py-2">
+        <div class="card-body">
+          <div class="row no-gutters align-items-center">
+            <div class="col mr-2">
+              <div class="text font-weight-bold text-warning mb-1">Go to Source Query</div>
             </div>
-            <a href="{% url 'vast_pipeline:image_index' %}" class="stretched-link"></a>
-          </div>
-        </div>
-    </div>
-
-    <!-- Extracted Sources -->
-    <div class="col-xl-3 col-md-6 mb-4">
-        <div class="card border-left-info shadow h-100 py-2">
-          <div class="card-body">
-            <div class="row no-gutters align-items-center">
-              <div class="col mr-2">
-                <div class="text-xs font-weight-bold text-info text-uppercase mb-1">Total Nr Measurements</div>
-                <div class="h5 mb-0 font-weight-bold text-gray-800">{{ totals.nr_meas|intcomma }}</div>
-              </div>
-              <div class="col-auto">
-                <i class="fas fa-bolt fa-2x text-gray-300"></i>
-              </div>
+            <div class="col-auto">
+              <i class="fas fa-share-alt fa-2x text-gray-300"></i>
             </div>
           </div>
+          <a href="{% url 'vast_pipeline:source_query' %}" class="stretched-link"></a>
         </div>
+      </div>
     </div>
-          
-    <!-- Nr Sources -->
-    <div class="col-xl-3 col-md-6 mb-4">
-        <div class="card border-left-warning shadow h-100 py-2">
+
+    <!-- Nr Images -->
+    <div class="col-xl-4 col-md-6 mb-4">
+        <div class="card border-left-primary shadow h-100 py-2">
           <div class="card-body">
             <div class="row no-gutters align-items-center">
               <div class="col mr-2">
-                <div class="text-xs font-weight-bold text-warning text-uppercase mb-1">Total Nr Sources</div>
-                <div class="h5 mb-0 font-weight-bold text-gray-800">{{ totals.nr_srcs|intcomma }}</div>
+                <div class="text font-weight-bold text-primary mb-1">Go to Images</div>
               </div>
               <div class="col-auto">
-                <i class="fas fa-share-alt fa-2x text-gray-300"></i>
+                <i class="fas fa-images fa-2x text-gray-300"></i>
               </div>
             </div>
-            <a href="{% url 'vast_pipeline:source_query' %}" class="stretched-link"></a>
+            <a href="{% url 'vast_pipeline:image_index' %}" class="stretched-link"></a>
           </div>
         </div>
     </div>
 
   </div>
-  
+
   <div class="row">
-    <div class="col-lg-5">
+    <div class="col-lg-12">
       <!-- Change Log card -->
       <div class="card shadow mb-4">
         <div class="card-header py-3">
@@ -126,32 +106,8 @@ <h6 class="m-0 font-weight-bold text-primary">Welcome to the VAST Pipeline!</h6>
         </div>
       </div>
     </div>
-
-    <!-- Sky Regions -->
-   	<div class="col-lg-7">
-       <div class="card shadow mb-4">
-         <div class="card-header py-3">
-           <h6 class="m-0 font-weight-bold text-primary">Current Sky Regions</h6>
-         </div>
-         <div class="card-body">
-           <div id="celestial-map"></div>
-         </div>
-       </div>
-   	</div>
   </div>
-
 </div>
 <!-- /.container-fluid -->
 
 {% endblock content %}
-
-{% block custom_page_scripts %}
-
-<!-- Chart scripts -->
-{{ d3_celestial_skyregions|json_script:"skyregion-data" }}
-<script type="text/javascript" src="{% static 'vendor/d3-celestial/d3.min.js' %}"></script>
-<script type="text/javascript" src="{% static 'vendor/d3-celestial/d3.geo.projection.min.js' %}"></script>
-<script type="text/javascript" src="{% static 'vendor/d3-celestial/celestial.min.js' %}"></script>
-<script id="d3CelestialScript" type="text/javascript" src="{% static 'js/d3-celestial-script.min.js' %}" staticUrl="{{ static_url }}"></script>
-
-{% endblock custom_page_scripts %}
diff --git a/vast_pipeline/migrations/0011_run_n_new_sources.py b/vast_pipeline/migrations/0011_run_n_new_sources.py
@@ -0,0 +1,32 @@
+# Generated by Django 3.2.13 on 2023-01-25 18:13
+
+from django.db import migrations, models, transaction
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ('vast_pipeline', '0010_update_source_names_iau_compliant'),
+    ]
+
+    def fill_new_source_counts(apps, schema_editor):
+        """Loop over Run objects and count the number of new sources and save to new column."""
+        Run = apps.get_model("vast_pipeline", "Run")
+        Source = apps.get_model("vast_pipeline", "Source")
+
+        runs = Run.objects.all()
+
+        for run in runs:
+            with transaction.atomic():
+                n_new_sources = Source.objects.filter(run=run, new=True).count()
+                run.n_new_sources = n_new_sources
+                run.save()
+
+    operations = [
+        migrations.AddField(
+            model_name='run',
+            name='n_new_sources',
+            field=models.IntegerField(default=0, help_text='number of new sources in this run'),
+        ),
+        migrations.RunPython(fill_new_source_counts, atomic=False)
+    ]
diff --git a/vast_pipeline/models.py b/vast_pipeline/models.py
@@ -154,6 +154,10 @@ class Run(CommentableModel):
         default=0,
         help_text='number of forced measurements in this run'
     )
+    n_new_sources = models.IntegerField(
+        default=0,
+        help_text='number of new sources in this run'
+    )
     epoch_based = models.BooleanField(
         default=False,
         help_text=(

diff --git a/vast_pipeline/pipeline/finalise.py b/vast_pipeline/pipeline/finalise.py
@@ -5,7 +5,7 @@
 
 from astropy import units as u
 from astropy.coordinates import SkyCoord
-from typing import List, Dict
+from typing import List, Dict, Tuple
 
 from vast_pipeline.models import Run
 from vast_pipeline.utils.utils import StopWatch, optimize_floats, optimize_ints
@@ -87,7 +87,7 @@ def final_operations(
     add_mode: bool,
     done_source_ids: List[int],
     previous_parquets: Dict[str, str]
-) -> int:
+) -> Tuple[int, int]:
     """
     Performs the final operations of the pipeline:
     - Calculates the statistics for the final sources.
@@ -120,7 +120,9 @@ def final_operations(
             in the previous run in add mode.
 
     Returns:
-        The number of sources contained in the pipeline (used in the next steps
+        The number of sources contained in the pipeline run (used in the next steps
+            of main.py).
+        The number of new sources contained in the pipeline run (used in the next steps
             of main.py).
     """
     timer = StopWatch()
@@ -317,5 +319,8 @@ def final_operations(
 
     logger.info("Total final operations time: %.2f seconds", timer.reset_init())
 
+    nr_sources = srcs_df["id"].count()
+    nr_new_sources = srcs_df['new'].sum()
+
     # calculate and return total number of extracted sources
-    return srcs_df["id"].count()
+    return (nr_sources, nr_new_sources)
diff --git a/vast_pipeline/pipeline/main.py b/vast_pipeline/pipeline/main.py
@@ -290,7 +290,7 @@ def process_pipeline(self, p_run: Run) -> None:
 
         # STEP #6: finalise the df getting unique sources, calculating
         # metrics and upload data to database
-        nr_sources = final_operations(
+        nr_sources, nr_new_sources = final_operations(
             sources_df,
             p_run,
             new_sources_df,
@@ -312,6 +312,7 @@ def process_pipeline(self, p_run: Run) -> None:
             p_run.n_forced_measurements = (
                 nr_forced_measurements if self.config["source_monitoring"]["monitor"] else 0
             )
+            p_run.n_new_sources = nr_new_sources
             p_run.save()
 
         pass

diff --git a/vast_pipeline/views.py b/vast_pipeline/views.py
@@ -115,15 +115,7 @@ def Login(request):
 
 @login_required
 def Home(request):
-    totals = {}
-    totals['nr_pruns'] = Run.objects.count()
-    totals['nr_imgs'] = Image.objects.count()
-    totals['nr_srcs'] = Source.objects.count()
-    totals['nr_meas'] = Measurement.objects.count()
-
     context = {
-        'totals': totals,
-        'd3_celestial_skyregions': get_skyregions_collection(),
         'static_url': settings.STATIC_URL
     }
     return render(request, 'index.html', context)
@@ -619,17 +611,11 @@ def RunDetail(request, id):
         p_run['nr_meas'] = p_run['n_selavy_measurements']
         p_run['nr_frcd'] = p_run['n_forced_measurements']
         p_run['nr_srcs'] = p_run['n_sources']
+        p_run['new_srcs'] = p_run['n_new_sources']
     else:
         p_run['nr_meas'] = 'N/A'
         p_run['nr_frcd'] = 'N/A'
         p_run['nr_srcs'] = 'N/A'
-
-    if p_run_model.status == 'Completed':
-        p_run['new_srcs'] = Source.objects.filter(
-            run__id=p_run['id'],
-            new=True,
-        ).count()
-    else:
         p_run['new_srcs'] = 'N/A'
 
     # read run config