AsdfConversionWarnings in multiple regression tests of jwst pipeline

[stscijgbot-jp]

Issue JP-3555 was created on JIRA by Ned Molter:

Regression tests with "-W error" turned up many instances of a warning within ASDF that reads

asdf.exceptions.AsdfConversionWarning: A ndarray subclass (<class 'astropy.io.fits.fitsrec.FITS_rec'>) was converted as a ndarray. This behavior will be removed from a future version of ASDF. See https://asdf.readthedocs.io/en/latest/asdf/config.html#convert-unknown-ndarray-subclasses" ```
This isn't specifically labeled a deprecation warning, but based on the message, that's what it is in practice.  The full traceback is found e.g. [here](https://plwishmaster.stsci.edu:8081/job/RT/job/JWST-Developers-Pull-Requests/1235/testReport/junit/jwst.regtest/test_nirspec_fs_spec2/*stable_deps__test_nirspec_fs_spec2_jw02072_o002_20221206t143745_spec2_00001_asn_json_nsclean*/).  I assume this should be handled/fixed inside stdatamodels, and not in the jwst pipeline, e.g. at this step of the traceback:
```java
File "/data1/jenkins/workspace/RT/JWST-Developers-Pull-Requests/miniconda/envs/tmp_env2/lib/python3.11/site-packages/stdatamodels/fits_support.py", line 507, in _create_asdf_hdu ```

[jdavies-st]

I've seen this as well in normal runtime. This is not confidence-inspiring, as it implies that an older file on disk written with an old version of asdf (last year) will at some point be not readable by a future version of asdf. I don't think that's the intent of the asdf library. All old files should be readable by all future versions.

It would be good to create a minimal working example of this problem.

[braingram]

spacetelescope/stdatamodels#268 addresses most of the warnings. One is from a regression test input file that contains invalid asdf (as noted in the PR).

[braingram]

To provide a bit more context. The warning:
A ndarray subclass (<class 'astropy.io.fits.fitsrec.FITS_rec'>) was converted as a ndarray. This behavior will be removed from a future version of ASDF.
occurs when a file is written that contains a FITS_rec instance in the asdf tree portion of the jwst file. stdatamodels makes many attempts to make sure this doesn't occur as there are many ways that FITS_rec will incorrectly handle data if naively converted to a ndarray. It is generally unsafe for asdf to treat a subclass identically to a parent class (one of the issues here with FITS_rec) and only (temporarily) does so for ndarray because of historical reasons. The warning describes how this behavior will be removed in an upcoming asdf version. At that point if stdatamodels fails to safely convert a FITS_rec during write we will see errors instead of warnings. This will have no impact on reading files as the tags for the converted FITS_rec data are all versions ndarray where all old versions are supported.

[stscijgbot-jp]

Comment by Ned Molter on JIRA:

Fixed by [this PR|[https://github.com/spacetelescope/stdatamodels/pull/268].] Brett Graham should we go ahead and resolve this ticket?

[braingram]

I believe there's one remaining warning in jwst/regtest/test_fgs_guider.py::test_fgs_guider[exptype_fgs_id_stack-cal] due to invalid asdf in a regtest input file (https://bytesalad.stsci.edu/ui/repos/tree/General/jwst-pipeline/dev/fgs/level1b/exptype_fgs_id_stack_uncal.fits?clearFilter=true). Opening and re-saving (with the stdatamodels interface) that should fix the issue but that has not yet been done.

[tapastro]

I can upload a fixed version.

[stscijgbot-jp]

Comment by Brett Graham on JIRA:

Thanks for updating the file Tyler Pauly!
The most recent run shows no AsdfConversionWarnings.
https://plwishmaster.stsci.edu:8081/job/RT/job/JWST/2833/consoleFull