NEAR/MSI msi2isis and MsiCamera model improvements #4887

KrisBecker · 2022-04-05T18:06:47Z

NEAR/MSI support in ISIS has been improved and enhanced

Description

The NEAR/MSI Eros data has been improved via a NASA PDART proposal. This work includes improvements in remediation of hydrazine contamination and in support of new global color maps.

Modifications have been made to the msi2isis application that will ingest the new dataset that will be archived in the PDS. The new images have already been expanded to the standard size image 537x412 where lines are interpolated from the original dataset size of 537x244. Images of expanded size are accepted as long as all the expected values in the labels are valid. In addition, a new option called TRIM has been added that allows the user choose to trim the edges of the images. The remediated dataset has shown these data regions to be improved sufficiently to retain for mapping purposes. And the SCLK times are modified to be valid with used with the NEAR/MSI SCLK kernel. This modification, along with corresponding changes in the MsiCamera model, provides support for the sumspice application to apply Gaskell SUMFILE ephemeris data updates.

The MsiCamera model has been modified to use the SpacecraftClockStartCount SCLK values rather than StartTime which is in UTC. This change is required for sumspice compatibly. Also, additional NAIF keywords were added to the ISIS IAK that provides support for improved geometry. These values are:

INS-93001_SWAP_OBSERVER_TARGET = 'TRUE'
INS-93001_LIGHTTIME_CORRECTION = 'LT+S'
INS-93001_LT_SURFACE_CORRECT   = 'FALSE'

This results in a new version of the NEAR/MSI IAK. Both of these IAKs are now stored/track in the repository in the $ISISROOT/src/near/objs/MsiCamera/iak directory. The new version, msiAddendum002.ti, needs to be copied to $ISISDATA/near/kernels/iak directory with this PR and prior to testing/release.

Note the modifications to the MsiCamera model require a new version number bump since new keywords are being read from image labels (SpacecraftClockStartCount) and IAK (as shown above).

The serial number configuration file, NearMsiSerialNumber.trn, has also been updated to accomodate inclusion of an optional keyword we use for control-to-ground strategies.

All msi2isis and MsiCamera tests have been converted to the ISIS Google Test framework. As such, the msi2isis application has been converted to a callable function. All old app and unit tests have been removed. This means that all test data files must be removed as well. Of note is I had to explicitly add (unignore) the NEAR/MIS *.lbl files to .gitignore as lbl files are ignored repo-wide.

NOTE that I did not update the application test as I am unsure how to get these changes into the system. I think it only requires an update to the truth files once the tests have been rerun. Please advise/assist.

Related Issue

This PR address #4886.

Motivation and Context

We have produced a new set of NEAR/MSI Eros images where the remediation of hydrazine contamination has been improved. To support this dataset, modifications and updates to the NEAR/MSI ingestion application, msi2isis, and the camera model were needed. These modifications are included in this PR.

How Has This Been Tested?

The following shows results of the ctest suite for these changes:

(Mar282022) zion2[1374]: ctest -R Msi2isis                  
Test project /Users/kbecker/ISIS/Isis4Dev/Eros/MSI/Mar282022/ISIS3/build
      Start 1934: Msi2isis.Msi2isisTestGblIngestDefault
 1/13 Test #1934: Msi2isis.Msi2isisTestGblIngestDefault ..........   Passed    0.32 sec
      Start 1935: Msi2isis.Msi2isisTestGblIngestNoTrim
 2/13 Test #1935: Msi2isis.Msi2isisTestGblIngestNoTrim ...........   Passed    0.32 sec
      Start 1936: Msi2isis.Msi2isisTestCubicConvolution
 3/13 Test #1936: Msi2isis.Msi2isisTestCubicConvolution ..........   Passed    0.44 sec
      Start 1937: Msi2isis.Msi2isisTestNearestNeighbor
 4/13 Test #1937: Msi2isis.Msi2isisTestNearestNeighbor ...........   Passed    0.43 sec
      Start 1938: Msi2isis.Msi2isisTestBilinear
 5/13 Test #1938: Msi2isis.Msi2isisTestBilinear ..................   Passed    0.42 sec
      Start 1939: Msi2isis.Msi2isisTestAMissingLabelFile
 6/13 Test #1939: Msi2isis.Msi2isisTestAMissingLabelFile .........   Passed    0.14 sec
      Start 1940: Msi2isis.Msi2isisTestBInvalidInstrumentId
 7/13 Test #1940: Msi2isis.Msi2isisTestBInvalidInstrumentId ......   Passed    0.15 sec
      Start 1941: Msi2isis.Msi2isisTestCInvalidSize
 8/13 Test #1941: Msi2isis.Msi2isisTestCInvalidSize ..............   Passed    0.16 sec
      Start 1942: Msi2isis.Msi2isisTestDInvalidSize2
 9/13 Test #1942: Msi2isis.Msi2isisTestDInvalidSize2 .............   Passed    0.15 sec
      Start 1943: Msi2isis.Msi2isisTestEInvalidSampleDirection
10/13 Test #1943: Msi2isis.Msi2isisTestEInvalidSampleDirection ...   Passed    0.15 sec
      Start 1944: Msi2isis.Msi2isisTestFInvalidLineDirection
11/13 Test #1944: Msi2isis.Msi2isisTestFInvalidLineDirection .....   Passed    0.16 sec
      Start 1945: Msi2isis.Msi2isisTestGInvalidProjectedImage
12/13 Test #1945: Msi2isis.Msi2isisTestGInvalidProjectedImage ....   Passed    0.16 sec
      Start 1946: Msi2isis.Msi2isisTestHLabelTranslateError
13/13 Test #1946: Msi2isis.Msi2isisTestHLabelTranslateError ......   Passed    0.33 sec

100% tests passed, 0 tests failed out of 13

Total Test time (real) =   3.51 sec
(Mar282022) zion2[1375]: ctest -R NearMsiCameraTest
Test project /Users/kbecker/ISIS/Isis4Dev/Eros/MSI/Mar282022/ISIS3/build
    Start 2079: NearMsiCameraCube.NearMsiCameraTest
1/1 Test #2079: NearMsiCameraCube.NearMsiCameraTest ...   Passed    0.20 sec

100% tests passed, 0 tests failed out of 1

Total Test time (real) =   0.36 sec

Screenshots (if appropriate):

Types of changes

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Documentation change (update to the documentation; no code change)
Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

I have read and agree to abide by the Code of Conduct
I have read the CONTRIBUTING document.
My change requires a change to the documentation.
I have updated the documentation accordingly.
I have added tests to cover my changes.
All new and existing tests passed.
I have added myself to the .zenodo.json document.
I have added any user impacting changes to the CHANGELOG.md document.

Licensing

This project is mostly composed of free and unencumbered software released into the public domain, and we are unlikely to accept contributions that are not also released into the public domain. Somewhere near the top of each file should have these words:

This work is free and unencumbered software released into the public domain. In jurisdictions that recognize copyright laws, the author or authors of this software dedicate any and all copyright interest in the software to the public domain.

I dedicate any and all copyright interest in this software to the public domain. I make this dedication for the benefit of the public at large and to the detriment of my heirs and successors. I intend this dedication to be an overt act of relinquishment in perpetuity of all present and future rights to this software under copyright law.

Rename the NEAR/MSI ingestion app main.cpp to msi2isis.cpp to covert to callable function

The old tests have been replaced by new FunctionalTests

msi2isis has been modified to accept expanded datasets * Accept images that have already been expanded to correct aspect (square pixels) * New TRIM option added to optional turn off trimming of borders * Modify SCLK start and stop times to be compatible with NAIF SCLK kernel format. This provides compatiblity for the sumspice application * Save original start/stop SCLK values to new keywords OriginalSpacecraftClockStartCount/ OriginalSpacecraftClockStopCount * Convert the app to a callable function

msi2isis funtional test and data * Converted all old app tests to new functional tests * Added label and data files * msi2isis has accompanying .lbl files as input but .gitignore excludes these files. Add explicit ignores for these *.lbl files

MsiCamera model modified so that sumspice works correcty * Use SpacecraftClockStartCount rather than StartTime to determine observation times for sumspice compatibility * Add IAKs to track in repository * Add light time and observer parameters to IAK version 2 * Bump camera version in plugin file due to new keywords loaded at spiceinit time

MsiCamera Gtest written to replace old version * Add new MsiCamera tests and Fixtures

NearMsiSerialNumber.trn update to add option keyword ControlSN to support use of simulated image in control processes

msiCamera unitTest.cpp in favor of NearMsiCameraCube.NearMsiCameraTest

jessemapel · 2022-04-05T18:59:46Z

Gonna get started on reviewing this, but I'll respond to a few things up front:

Of note is I had to explicitly add (unignore) the NEAR/MIS *.lbl files to .gitignore as lbl files are ignored repo-wide.

This is intentional, we don't want binary files accidentally added to the git repo as it makes for extremely large diffs and bloats the repo.

NOTE that I did not update the application test as I am unsure how to get these changes into the system. I think it only requires an update to the truth files once the tests have been rerun. Please advise/assist.

Which application tests are you talking about? The old msi2isis app tests should be removed as part of this PR.

I have added any user impacting changes to the CHANGELOG.md document.

Be sure to add a changelog entry under UNRELEASED / ADDED

KrisBecker · 2022-04-05T19:09:53Z

Which application tests are you talking about? The old msi2isis app tests should be removed as part of this PR.

Actually this is the mission module tests in $ISISROOT/src/near.

(Mar282022) zion2[1377]: ctest -R near_module_test_msi
Test project /Users/kbecker/ISIS/Isis4Dev/Eros/MSI/Mar282022/ISIS3/build
    Start 1103: near_module_test_msi
1/1 Test #1103: near_module_test_msi .............***Failed   15.74 sec

0% tests passed, 1 tests failed out of 1

Label Time Summary:
app    =  15.74 sec*proc (1 test)

Total Test time (real) =  15.90 sec

The following tests FAILED:
	1103 - near_module_test_msi (Failed)
Errors while running CTest
Output from these tests are in: /Users/kbecker/ISIS/Isis4Dev/Eros/MSI/Mar282022/ISIS3/build/Testing/Temporary/LastTest.log
Use "--rerun-failed --output-on-failure" to re-run the failed cases verbosely.

I looked at the output and it appears these test require updating test data and I am uncertain how to make that change from my end.

jessemapel · 2022-04-05T19:32:42Z

I can get those tests updated once this PR is merged.

KrisBecker · 2022-04-05T19:37:32Z

Just an FYI, after all the ctests have been ran on my system, there are some residual files that consistently remain (or marked as untracked). These files are:

isis/tests/data/isisimport/pds4.xml
isis/tests/winnowGuilty.csv
isis/tests/winnowIgnored.csv

jessemapel

The code changes look good. I added a bunch of comments about how to improve the tests.

CHANGELOG.md

isis/src/near/apps/msi2isis/msi2isis.cpp

isis/tests/Fixtures.cpp

isis/tests/FunctionalTestsMsi2isis.cpp

isis/tests/Fixtures.cpp

isis/tests/NearMsiCameraTests.cpp

isis/tests/Fixtures.cpp

The MsiCamera requires a spiceinit cube, but .gitignore excludes .cub files. This commit addresses this issue by explicitly unignoring the Eros cube used for the test.

* Removed space between issue number and url in CHANGELOG.md msi2isis * Remove commented code * Simplify trimming. Let user decide uses for this option rather than assume it should not be used under certain conditions

isis/tests/Fixtures.cpp

jessemapel · 2022-04-07T15:42:45Z

I've moved the test cube to a label + ISD setup, it's not a huge savings here because the original cube is ~1MB and the label + ISD is ~15KB, but for larger images this is a big deal.

I'm going to get the SPICE updated in our system now.

…SIS3 into eros_msi_pdart_mods

KrisBecker · 2022-04-09T18:06:21Z

Just pushed test modifications that address your comments/concerns.

Please let me know if there is anything remaining preventing a merge.

Thanks...

isis/tests/FunctionalTestsMsi2isis.cpp

KrisBecker added 8 commits March 28, 2022 20:16

Rename NEAR/MSI main.cpp to msi2isis.cpp

55f46fe

Rename the NEAR/MSI ingestion app main.cpp to msi2isis.cpp to covert to callable function

Remove old msi2isis tests in tsts directory

40dbb94

The old tests have been replaced by new FunctionalTests

Add msi2isis functional tests and data

60f4596

msi2isis funtional test and data * Converted all old app tests to new functional tests * Added label and data files * msi2isis has accompanying .lbl files as input but .gitignore excludes these files. Add explicit ignores for these *.lbl files

Add MsiCamera gtest to replace old test

e466788

MsiCamera Gtest written to replace old version * Add new MsiCamera tests and Fixtures

Add optional keyword in serial number generation

d507576

NearMsiSerialNumber.trn update to add option keyword ControlSN to support use of simulated image in control processes

Removed MsiCamera unit test

98cf62c

msiCamera unitTest.cpp in favor of NearMsiCameraCube.NearMsiCameraTest

KrisBecker mentioned this pull request Apr 5, 2022

Document Testing Requirements #4864

Closed

jessemapel added the enhancement New feature or request label Apr 5, 2022

jessemapel self-requested a review April 5, 2022 18:55

Add updates to CHANGELOG.md

6e5df95

jessemapel reviewed Apr 5, 2022

View reviewed changes

jessemapel self-assigned this Apr 5, 2022

KrisBecker added 3 commits April 5, 2022 15:27

Add NEAR/MSI spiced cube for MsiCamera unit test

2af9fb5

The MsiCamera requires a spiceinit cube, but .gitignore excludes .cub files. This commit addresses this issue by explicitly unignoring the Eros cube used for the test.

Fix issues from review comments

276365c

* Removed space between issue number and url in CHANGELOG.md msi2isis * Remove commented code * Simplify trimming. Let user decide uses for this option rather than assume it should not be used under certain conditions

Cleaned up NearMsiCameraCube Fixture

faf11e4

jessemapel reviewed Apr 6, 2022

View reviewed changes

isis/tests/Fixtures.cpp Show resolved Hide resolved

Moved NEAR MSI test to use label and ISD

067119a

KrisBecker added 2 commits April 9, 2022 10:53

Address suggested improvements to msi2isis tests

5226277

Merge branch 'eros_msi_pdart_mods' of https://github.com/KrisBecker/I…

928c47f

…SIS3 into eros_msi_pdart_mods

jessemapel reviewed Apr 11, 2022

View reviewed changes

KrisBecker and others added 2 commits April 11, 2022 11:34

Updated tests after feedback from PR review

072e9c2

Merge branch 'dev' into eros_msi_pdart_mods

614cefd

jessemapel approved these changes Apr 11, 2022

View reviewed changes

jessemapel merged commit d7f2a99 into DOI-USGS:dev Apr 11, 2022

jessemapel mentioned this pull request May 12, 2022

Changed SRC camera and serial number to use start time instead of sta… #4910

Merged

13 tasks

KrisBecker mentioned this pull request Oct 14, 2022

Issues with googletest conversion and test writing documentation #5029

Closed

11 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

NEAR/MSI msi2isis and MsiCamera model improvements #4887

NEAR/MSI msi2isis and MsiCamera model improvements #4887

KrisBecker commented Apr 5, 2022 •

edited by jessemapel

Loading

jessemapel commented Apr 5, 2022

KrisBecker commented Apr 5, 2022

jessemapel commented Apr 5, 2022

KrisBecker commented Apr 5, 2022

jessemapel left a comment

jessemapel commented Apr 7, 2022 •

edited

Loading

KrisBecker commented Apr 9, 2022

NEAR/MSI msi2isis and MsiCamera model improvements #4887

NEAR/MSI msi2isis and MsiCamera model improvements #4887

Conversation

KrisBecker commented Apr 5, 2022 • edited by jessemapel Loading

Description

Related Issue

Motivation and Context

How Has This Been Tested?

Screenshots (if appropriate):

Types of changes

Checklist:

Licensing

jessemapel commented Apr 5, 2022

KrisBecker commented Apr 5, 2022

jessemapel commented Apr 5, 2022

KrisBecker commented Apr 5, 2022

jessemapel left a comment

Choose a reason for hiding this comment

jessemapel commented Apr 7, 2022 • edited Loading

KrisBecker commented Apr 9, 2022

KrisBecker commented Apr 5, 2022 •

edited by jessemapel

Loading

jessemapel commented Apr 7, 2022 •

edited

Loading