Topds4 (#4246)

* Removed compile and deprication warnings. Fixes #4110

* temp add of to pds 4 test

* Initial topds4 app and cmake updates to include inja (#4207)

* Removed compile and deprication warnings. Fixes #4110

* temp add of to pds 4 test

* Inintal topds4 app and cmake updates to include inja

* Delete extra headers

* Fixed inja include

Co-authored-by: ssides <[email protected]>

* Added conda-forge inja

* add template file

* Adds PVL to JSON converters (#4212)

* Pvl to JSON conversion

* Renamed file

* Added repeated elements and examples

* Topds4 real parameters (#4223)

* now has some real parameters

* removed inja files

* Add full orex pds4 template (#4233)

* now has some real parameters

* removed inja files

* full orex xml

* add full orex template

* Added callbacks and tests (#4226)

* Adds Xml2Json capability to topds4 branch (#4228)

* Inintal topds4 app and cmake updates to include inja

* Delete extra headers

* Fixed inja include

* Added basic working XMLToJson functionality

* Updated to work with many nested levels of different amounts of repeated tags

* Add support for unsupported xml edge cases

* Add tests that work to test mutliple nested repeated values conversion

* cleanup

* Update to include no-text value cases to work in repeated-value cases

* Update test

* Break up tests

* Fix typoes in comments

* Normalized topds4 and added tests (#4234)

* Removed data source args and added app log

* Added simple tests for topds4

* Moved line

* Added more template elements and callback for file size (#4236)

* now has some real parameters

* removed inja files

* full orex xml

* add full orex template

* More template elements, and file size callback

* Added Extra data source parameters to topds4 (#4239)

* Added extra parameters

* Added duplicate warnings

* Fixed warning

* Add writing a cube out to topds4 (#4241)

* Basic cubeatt conversion

* Add more useful function signature

* Removed hardcoded path

* Update based on feedback

* Add better band test

* Updated to fix failing tests and remove old testfiles

* Add copy of topds4 templates to CMakeLists.txt (#4242)

* Fixed callbacks and doc updates (#4243)

* Fixed callbacks and doc updates

* added some extra docs

* Added original xml label and tests

* Update cubeattribute to include an option to supply CubeAttributeInput directly (#4244)

* Update cubeatt to add cubeattributeinput option. Also add tests for all function signatures available

* Remove unnecessary comments

* Update to use explicit array conversion.

* Remove commented out code

* Update CHANGELOG to include topds4 addition

* Delete osirisrex.xml.tpl

Removed temporary file

* Delete 20190612T090019S776_map_L0pan.xml.full

Remove temporary file

* Delete 20190612T090019S776_map_L0pan.xml.tpl

Remove temp file

* Added mis-deleted file (#4249)

* Update XmlToJson.cpp

* Update topds4.xml

* Update topds4.xml

* Update topds4.xml

* Add setting of virtual bands to topds4 and remove unneeded header file includes

Co-authored-by: ssides <[email protected]>
Co-authored-by: Kristin Berry <[email protected]>

CHANGELOG.md

@@ -35,7 +35,9 @@ update the Unreleased link so that it compares against the latest release tag.
 - Added warning to ocams2isis about the model being out of date. [#4200](https://github.com/USGS-Astrogeology/ISIS3/issues/4200)
 - Added documentation to lronaccal and lrowaccal to describe why there are negative DNs in I/F calibrated images. [#3860](https://github.com/USGS-Astrogeology/ISIS3/issues/3860)
 - Update qview MeasureTool to add an option to calculate distances using RA/DEC and update qview to show DEC/RA rather than LAT/LON in lower-right corner [#3371](https://github.com/USGS-Astrogeology/ISIS3/issues/3371)
+- Added a new application, topds4, which generates an output PDS4 XML label and a PDS4-compliant ISIS Cube from an input Cube, a PDS4 label template, and optionally additional input XML, PVL, or JSON data. The Inja templating engine is used to render the output PDS4 label from the label template. [#4246](https://github.com/USGS-Astrogeology/ISIS3/pull/4246)
 - Updated spiceinit so that a user can specify a shape model and use the spice web service without any errors. [#1986](https://github.com/USGS-Astrogeology/ISIS3/issues/1986) 
+
 ### Fixed
 
 - Fixed lrowaccal so required SPICE files are reported instead of continuing without them. [#4038](https://github.com/USGS-Astrogeology/ISIS3/issues/4038)

environment.yml

@@ -24,6 +24,7 @@ dependencies:
   - gsl>=2.6
   - hdf5
   - icu
+  - inja
   - jama
   - jpeg==9b
   - kakadu==1

environment_gcc4.yml

@@ -26,6 +26,7 @@ dependencies:
   - gsl
   - hdf5
   - icu
+  - inja
   - jama
   - jpeg==9b
   - kakadu==1

isis/CMakeLists.txt

@@ -80,7 +80,7 @@ option(JP2KFLAG        "Whether or not to build using JPEG2000 support" OFF )
 option(pybindings      "Turn on to build Python bindings"               OFF )
 
 # if cmake install prefix is not set, and conda env is activated, use the
-# conda env as the install directory
+# conda env as the install directory.
 if(DEFINED ENV{CONDA_PREFIX} AND CMAKE_INSTALL_PREFIX STREQUAL "/usr/local")
  set(CMAKE_INSTALL_PREFIX $ENV{CONDA_PREFIX})
 endif()
@@ -293,6 +293,8 @@ find_package(Kakadu)
 find_package(Geos    3.5.0     REQUIRED)
 find_package(Armadillo         REQUIRED)
 find_package(Threads)
+find_package(inja              REQUIRED)
+
 
 
 # In this case, we specify the version numbers being searched for in the non-traditional installs.
@@ -347,6 +349,9 @@ foreach (_variableName ${_variableNames})
   endif()
 endforeach()
 
+# add target based linkages to ALLLIBS variable
+list(APPEND ALLLIBS pantor::inja)
+
 # Sometimes we add the same lib more than once (especially with LIBDIRS)
 list(REMOVE_DUPLICATES ALLLIBDIRS)
 list(REMOVE_DUPLICATES ALLLIBS)
@@ -496,6 +501,7 @@ file(MAKE_DIRECTORY ${CMAKE_BINARY_DIR}/appdata/translations
 		    ${CMAKE_BINARY_DIR}/appdata/templates/jigsaw
 		    ${CMAKE_BINARY_DIR}/appdata/templates/kernels
 	            ${CMAKE_BINARY_DIR}/appdata/templates/labels
+	            ${CMAKE_BINARY_DIR}/appdata/templates/topds4
 	            ${CMAKE_BINARY_DIR}/appdata/templates/maps
 		    ${CMAKE_BINARY_DIR}/appdata/templates/photometry
 	            ${CMAKE_BINARY_DIR}/appdata/images/icons
@@ -614,6 +620,10 @@ add_custom_target(hidtmgen ALL COMMAND ${CMAKE_COMMAND} -E copy_if_different
   ${CMAKE_SOURCE_DIR}/appdata/templates/hidtmgen/* ${CMAKE_BINARY_DIR}/appdata/templates/hidtmgen/)
 add_dependencies(isis hidtmgen)
 
+add_custom_target(topds4 ALL COMMAND ${CMAKE_COMMAND} -E copy_if_different
+  ${CMAKE_SOURCE_DIR}/appdata/templates/topds4/* ${CMAKE_BINARY_DIR}/appdata/templates/topds4)
+add_dependencies(isis topds4)
+
 # Add a custom build target to clean out everything that gets added to the source
 #  directory during the build process.
 # - Only a few things are added in order to make the tests work properly so

isis/appdata/templates/topds4/osirisrex.xml.tpl

@@ -0,0 +1,53 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<?xml-model href="https://pds.nasa.gov/pds4/pds/v1/PDS4_PDS_1700.sch" schematypens="http://purl.oclc.org/dsdl/schematron" ?>
+<?xml-model href="https://pds.nasa.gov/pds4/disp/v1/PDS4_DISP_1700.sch" schematypens="http://purl.oclc.org/dsdl/schematron" ?>
+<?xml-model href="https://pds.nasa.gov/pds4/geom/v1/PDS4_GEOM_1700_1401.sch" schematypens="http://purl.oclc.org/dsdl/schematron" ?>
+<?xml-model href="https://pds.nasa.gov/pds4/orex/v1/orex_ldd_OREX_1300.sch" schematypens="http://purl.oclc.org/dsdl/schematron" ?>
+<Product_Observational
+    xmlns="http://pds.nasa.gov/pds4/pds/v1"
+    xmlns:pds="http://pds.nasa.gov/pds4/pds/v1"
+    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+    xmlns:disp="http://pds.nasa.gov/pds4/disp/v1"
+    xmlns:geom="http://pds.nasa.gov/pds4/geom/v1"
+    xmlns:orex="http://pds.nasa.gov/pds4/mission/orex/v1"
+    xsi:schemaLocation=
+    "http://pds.nasa.gov/pds4/pds/v1 https://pds.nasa.gov/pds4/pds/v1/PDS4_PDS_1700.xsd
+    http://pds.nasa.gov/pds4/disp/v1 https://pds.nasa.gov/pds4/disp/v1/PDS4_DISP_1700.xsd
+    http://pds.nasa.gov/pds4/geom/v1 https://pds.nasa.gov/pds4/geom/v1/PDS4_GEOM_1700_1401.xsd
+    http://pds.nasa.gov/pds4/mission/orex/v1 https://pds.nasa.gov/pds4/mission/orex/v1/PDS4_OREX_1300.xsd">
+    <File_Area_Observational>
+        <File>
+            <file_name>filename</file_name>
+            <creation_date_time>dateandtime</creation_date_time>
+            <file_size unit="byte">filesize</file_size>
+        </File>
+        <Header>
+            <offset unit="byte">headeroffset</offset>
+            <object_length unit="byte">headerLength</object_length>
+            <parsing_standard_id>fileType</parsing_standard_id>
+        </Header>
+        <Array_2D_Image>
+            <local_identifier>Active Area</local_identifier>
+            <offset unit="byte">imageOffsetBytes</offset>
+            <axes>numAxes</axes>
+            <axis_index_order>Last Index Fastest</axis_index_order>
+            <description>OCAMS image 1024 by 1024 pixel active array.</description>
+            <Element_Array>
+                <data_type>bitType</data_type>
+                <unit>DN</unit>
+                <scaling_factor>1</scaling_factor>
+                <value_offset>WHAT IS THIS</value_offset>
+            </Element_Array>
+            <Axis_Array>
+                <axis_name>Line</axis_name>
+                <elements>{{MainLabel.IsisCube.Core.Dimensions.Lines.Value}}</elements>
+                <sequence_number>1</sequence_number>
+            </Axis_Array>
+            <Axis_Array>
+                <axis_name>Sample</axis_name>
+                <elements>{{MainLabel.IsisCube.Core.Dimensions.Samples.Value}}</elements>
+                <sequence_number>2</sequence_number>
+            </Axis_Array>
+        </Array_2D_Image>
+    </File_Area_Observational>
+</Product_Observational>

isis/src/base/apps/cubeatt/cubeatt.cpp

@@ -0,0 +1,64 @@
+#include "cubeatt.h"
+
+#include "ProcessByLine.h"
+
+using namespace std;
+using namespace Isis;
+
+namespace Isis {
+  void cubeattProcess(Buffer &in, Buffer &out);
+
+  void cubeatt(UserInterface &ui) {
+    Cube icube;
+    CubeAttributeInput inAtt = ui.GetInputAttribute("FROM");
+    if (inAtt.bands().size() != 0) {
+      icube.setVirtualBands(inAtt.bands());
+    }
+    icube.open(ui.GetFileName("FROM"));
+    cubeatt(&icube, ui);
+  }
+
+
+  // Doesn't allow specification of input attributes
+  void cubeatt(Cube *icube, UserInterface &ui) {
+    bool propTables = ui.GetBoolean("PROPTABLES");
+    QString outputFileName = ui.GetFileName("TO");
+    CubeAttributeOutput outputAttributes= ui.GetOutputAttribute("TO");
+    cubeatt(icube, outputFileName, outputAttributes, propTables);
+  }
+
+  // Doesn't allow specification of input attributes
+  void cubeatt(Cube *icube, QString outputCubePath, CubeAttributeOutput outAttributes, bool propTables) {
+    // We will be processing by line
+    ProcessByLine p;
+
+    // Should we propagate tables
+    p.PropagateTables(propTables);
+
+    // Setup the input and output cubes
+    p.SetInputCube(icube);
+    p.SetOutputCube(outputCubePath, outAttributes, 
+                    icube->sampleCount(), icube->lineCount(), icube->bandCount());
+
+    p.StartProcess(cubeattProcess);
+    p.EndProcess();
+  }
+
+  // Allows specification of both input and output attributes
+  void cubeatt(QString inputCubePath, CubeAttributeInput inAtt, QString outputCubePath, CubeAttributeOutput outputAttributes, bool propTables) {
+    Cube icube;
+    if (inAtt.bands().size() != 0) {
+      icube.setVirtualBands(inAtt.bands());
+    }
+    icube.open(inputCubePath);
+    cubeatt(&icube, outputCubePath, outputAttributes, propTables);
+  }
+
+  // Line processing routine
+  void cubeattProcess(Buffer &in, Buffer &out) {
+    // Loop and copy pixels in the line.
+    for(int i = 0; i < in.size(); i++) {
+      out[i] = in[i];
+    }
+  }
+}

isis/src/base/apps/cubeatt/cubeatt.h

@@ -0,0 +1,15 @@
+#ifndef cubeatt_h
+#define cubeatt_h
+
+#include "Cube.h"
+#include "CubeAttribute.h"
+#include "UserInterface.h"
+
+namespace Isis{
+  extern void cubeatt(Cube *icube, QString outputCubePath, CubeAttributeOutput outputAttributes, bool propTables=false);
+  extern void cubeatt(QString inputCubePath, CubeAttributeInput inputAttributes, QString outputCubePath, CubeAttributeOutput outputAttributes, bool propTables=false);
+  extern void cubeatt(Cube *icube, UserInterface &ui);
+  extern void cubeatt(UserInterface &ui);
+}
+
+#endif

isis/src/base/apps/cubeatt/main.cpp

@@ -1,32 +1,13 @@
 #include "Isis.h"
-#include "ProcessByLine.h"
+
+#include "cubeatt.h"
+
+#include "Application.h"
 
 using namespace std;
 using namespace Isis;
 
-void cubeatt(Buffer &in, Buffer &out);
-
 void IsisMain() {
-  // We will be processing by line
-  ProcessByLine p;
-
-  // Should we propagate tables
-  if(!Application::GetUserInterface().GetBoolean("PROPTABLES")) {
-    p.PropagateTables(false);
-  }
-
-  // Setup the input and output cubes
-  p.SetInputCube("FROM");
-  p.SetOutputCube("TO");
-
-  p.StartProcess(cubeatt);
-  p.EndProcess();
-}
-
-// Line processing routine
-void cubeatt(Buffer &in, Buffer &out) {
-  // Loop and copy pixels in the line.
-  for(int i = 0; i < in.size(); i++) {
-    out[i] = in[i];
-  }
-}
+  UserInterface &ui = Application::GetUserInterface();
+  cubeatt(ui);
+}

isis/src/base/apps/topds4/main.cpp

@@ -0,0 +1,25 @@
+#include "Isis.h"
+
+#include "topds4.h"
+
+using namespace std;
+using namespace Isis;
+
+void IsisMain() {
+  UserInterface &ui = Application::GetUserInterface();
+  Pvl appLog;
+
+  try {
+    topds4(ui, &appLog);
+  }
+  catch (...) {
+    for (auto grpIt = appLog.beginGroup(); grpIt!= appLog.endGroup(); grpIt++) {
+      Application::Log(*grpIt);
+    }
+    throw;
+  }
+
+  for (auto grpIt = appLog.beginGroup(); grpIt!= appLog.endGroup(); grpIt++) {
+    Application::Log(*grpIt);
+  }
+}