JP-2928: Update regression tests to use flight data - set_telescope_pointing.add_wcs edition

[tapastro]

Partially resolves JP-2928

This PR addresses old flight data in the tests that called set_telescope_pointing to add wcs keywords to uncal files. These tests were written in a time before the MAST engineering database became public, and so workarounds were generated. These are no longer necessary, as long as the tests use public data.

Tasks

• request a review from someone specific, to avoid making the maintainers review every PR
• add a build milestone, i.e. Build 11.3 (use the latest build if not sure)
• Does this PR change user-facing code / API? (if not, label with no-changelog-entry-needed)
  • write news fragment(s) in changes/: echo "changed something" > changes/<PR#>.<changetype>.rst (see below for change types)
  • update or add relevant tests
  • update relevant docstrings and / or docs/ page
  • start a regression test and include a link to the running job (click here for instructions)
    • Do truth files need to be updated ("okified")?
      • after the reviewer has approved these changes, run okify_regtests to update the truth files
• if a JIRA ticket exists, make sure it is resolved properly

news fragment change types...

• changes/<PR#>.general.rst: infrastructure or miscellaneous change
• changes/<PR#>.docs.rst
• changes/<PR#>.stpipe.rst
• changes/<PR#>.datamodels.rst
• changes/<PR#>.scripts.rst
• changes/<PR#>.fits_generator.rst
• changes/<PR#>.set_telescope_pointing.rst
• changes/<PR#>.pipeline.rst

stage 1

• changes/<PR#>.group_scale.rst
• changes/<PR#>.dq_init.rst
• changes/<PR#>.emicorr.rst
• changes/<PR#>.saturation.rst
• changes/<PR#>.ipc.rst
• changes/<PR#>.firstframe.rst
• changes/<PR#>.lastframe.rst
• changes/<PR#>.reset.rst
• changes/<PR#>.superbias.rst
• changes/<PR#>.refpix.rst
• changes/<PR#>.linearity.rst
• changes/<PR#>.rscd.rst
• changes/<PR#>.persistence.rst
• changes/<PR#>.dark_current.rst
• changes/<PR#>.charge_migration.rst
• changes/<PR#>.jump.rst
• changes/<PR#>.clean_flicker_noise.rst
• changes/<PR#>.ramp_fitting.rst
• changes/<PR#>.gain_scale.rst

stage 2

• changes/<PR#>.assign_wcs.rst
• changes/<PR#>.badpix_selfcal.rst
• changes/<PR#>.msaflagopen.rst
• changes/<PR#>.nsclean.rst
• changes/<PR#>.imprint.rst
• changes/<PR#>.background.rst
• changes/<PR#>.extract_2d.rst
• changes/<PR#>.master_background.rst
• changes/<PR#>.wavecorr.rst
• changes/<PR#>.srctype.rst
• changes/<PR#>.straylight.rst
• changes/<PR#>.wfss_contam.rst
• changes/<PR#>.flatfield.rst
• changes/<PR#>.fringe.rst
• changes/<PR#>.pathloss.rst
• changes/<PR#>.barshadow.rst
• changes/<PR#>.photom.rst
• changes/<PR#>.pixel_replace.rst
• changes/<PR#>.resample_spec.rst
• changes/<PR#>.residual_fringe.rst
• changes/<PR#>.cube_build.rst
• changes/<PR#>.extract_1d.rst
• changes/<PR#>.resample.rst

stage 3

• changes/<PR#>.assign_mtwcs.rst
• changes/<PR#>.mrs_imatch.rst
• changes/<PR#>.tweakreg.rst
• changes/<PR#>.skymatch.rst
• changes/<PR#>.exp_to_source.rst
• changes/<PR#>.outlier_detection.rst
• changes/<PR#>.tso_photometry.rst
• changes/<PR#>.stack_refs.rst
• changes/<PR#>.align_refs.rst
• changes/<PR#>.klip.rst
• changes/<PR#>.spectral_leak.rst
• changes/<PR#>.source_catalog.rst
• changes/<PR#>.combine_1d.rst
• changes/<PR#>.ami.rst

other

• changes/<PR#>.wfs_combine.rst
• changes/<PR#>.white_light.rst
• changes/<PR#>.cube_skymatch.rst
• changes/<PR#>.engdb_tools.rst
• changes/<PR#>.guider_cds.rst

[codecov]

Codecov Report

Attention: Patch coverage is 75.00000% with 1 line in your changes missing coverage. Please review.

Project coverage is 63.67%. Comparing base (8d2b64c) to head (2005079).
Report is 447 commits behind head on main.

Files with missing lines	Patch %	Lines
jwst/engdblog/engdblog.py	75.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #8887      +/-   ##
==========================================
+ Coverage   61.92%   63.67%   +1.74%     
==========================================
  Files         376      376              
  Lines       38691    38692       +1     
==========================================
+ Hits        23961    24638     +677     
+ Misses      14730    14054     -676     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[tapastro]

Started regression tests here: https://github.com/spacetelescope/RegressionTests/actions/runs/11333695649

[tapastro]

Alright, after rebasing the branch I ran a new set of regression tests here: https://plwishmaster.stsci.edu:8081/job/RT/job/JWST-Developers-Pull-Requests/1790/#showFailuresLink

They appear to match failures on nightly: https://plwishmaster.stsci.edu:8081/job/RT/job/JWST/3068/#showFailuresLink

[melanieclarke]

It looks like there is a pre-commit check and some unit tests that are failing.

[tapastro]

After uncovering the hornet's nest of jwst/lib unit test differences caused by changing the default behavior of our engdb machinery, I think I have a reasonable set of updates that now test on the existing public stores of information - the MAST EngDB and pysiaf, rather than local database files that mock the two. Some tests have been dropped, as they appeared to be useful only in a context where these services do not exist or in a world where JWST had not yet taken data.

[tapastro]

New RT: https://github.com/spacetelescope/RegressionTests/actions/runs/11406609500

EDIT: After many more test corrections, latest run here: https://github.com/spacetelescope/RegressionTests/actions/runs/11410434649

[tapastro]

After finding and removing the environment variable in RegressionTests, the new set of tests using the RT feature branch is here: https://github.com/spacetelescope/RegressionTests/actions/runs/11558936198

It shows no failures. I think this should be ready, then?

[melanieclarke]

This was so much more technical debt than I originally thought - thanks for taking it on! It will be great to have this sorted out.

LGTM, pending Jonathan's review, but one question for my information - where do the uncal files that are input to the setpointing tests come from? They appear to have only small differences from the truth files, after calling setpointing. Is that expected?

[tapastro]

From my understanding, the setpointing tests call add_wcs, which overwrites the input file with new header keywords pertaining to telescope orientation. It can be called on uncal or rate/ints files - I chose files from a keyword search on MAST to fit test criteria, but were otherwise not unique. The outputs were generated in the same manner as the test output is generated, and I think the two files should be functionally identical unless a SIAF update is made that alters pointings or reference values, i.e. CRVAL1.