Merge branch 'master' into JP-3574

.github/workflows/ci.yml

@@ -41,9 +41,8 @@ jobs:
       cache-path: /tmp/data/crds_cache
       cache-key: crds-${{ needs.crds_contexts.outputs.jwst }}
       envs: |
-        - linux: py39-oldestdeps-xdist-cov
+        - linux: py310-oldestdeps-xdist-cov
           pytest-results-summary: true
-        - linux: py39-xdist
         - linux: py310-xdist
         - linux: py311-xdist
           pytest-results-summary: true

.github/workflows/ci_cron.yml

@@ -22,7 +22,6 @@ jobs:
       cache-path: /tmp/crds_cache
       cache-key: crds-${{ needs.crds_context.outputs.jwst }}
       envs: |
-        - macos: py39-xdist
         - macos: py310-xdist
         - macos: py312-xdist
         - linux: py311-pyargs-xdist

CHANGES.rst

@@ -1,6 +1,30 @@
-1.13.5 (unreleased)
+1.14.1 (unreleased)
 ===================
 
+- 
+
+1.14.0 (2024-03-25)
+===================
+
+ami
+---
+
+- Replaced use of deprecated ``scipy.integrate.simps``
+  with ``scipy.integrate.simpson``. [#8320]
+
+- Overhaul of AMI processing. See documentation for full details. [#7862]
+
+- Additional optional input arguments for greater user processing flexibility.
+  See documentation for details. [#7862]
+
+- Bad pixel correction applied to data using new NRM reference file to calculate 
+  complex visibility support (M. Ireland method implemented by J. Kammerer). [#7862]
+
+- Make ``AmiAnalyze`` and ``AmiNormalize`` output conform to the OIFITS standard. [#7862]
+
+- Disable ``AmiAverage`` step. [#7862]
+
+
 associations
 ------------
 
@@ -13,6 +37,8 @@ associations
 - Update the level-3 rules for "tso3" associations so that NIRISS SOSS
   exposures with NINTS=1 are excluded. [#8359]
 
+- Removed blanket warning ignore for duplicate level 2 associations. [#8320]
+
 background
 ----------
 
@@ -27,6 +53,9 @@ charge_migration
   as DO_NOT_USE.  This group, and all subsequent groups, are then flagged as
   CHARGELOSS and DO_NOT_USE.  The four nearest pixel neighbor are then flagged
   in the same group. [#8336]
+
+- Added warning handler for expected NaN and inf clipping in the
+  ``sigma_clip`` function. [#8320]
 
 cube_build
 ----------
@@ -46,6 +75,9 @@ datamodels
 - Fixed a bug in the ``ModelContainer`` data model, due to which the ``models_grouped``
   property would return opened data models instead of file names. [#8191]
 
+- Removed ``test_all_datamodels_init`` test, which is a duplicate of a test in
+  ``stdatamodels``. [#8320]
+
 documentation
 -------------
 
@@ -68,6 +100,8 @@ documentation
 
 - Updated ``outlier_detection`` for IFU data to explain the method more clearly. [#8360]
 
+- Adds documentation on the 1-D residual fringe correction for MIRI MRS data that is done in ``extract_1d``. [#8371]
+
 
 emicorr
 -------
@@ -79,6 +113,8 @@ emicorr
   allow the user to run the step for given frequencies with an on-the-fly
   generated reference file. [#8270]
 
+- Fixed bug for finding correction data for subarray FULL. [#8375]
+
 exp_to_source
 -------------
 
@@ -111,6 +147,10 @@ extract_1d
 - Added saving the extraction aperture x/y limits for slit-like modes to
   keywords in the output header. [#8278]
 
+- Fixed deprecated conversions of single-element numpy arrays to scalar
+  in ``apply_apcorr``. [#8320]
+
+- Increased specificity of warning filters in ``atoca`` and ``soss_centroids``. [#8320]
 
 extract_2d
 ----------
@@ -138,21 +178,28 @@ general
 
 - Renamed the ``jail`` fixture with ``tmp_cwd_module``. [#8327]
 
+- Replaced deprecated ``tool.ruff.ignore`` and ``tool.ruff.per-file-ignores``
+  with ``tool.ruff.lint.ignore`` and ``tool.ruff.lint.per-file-ignores``. [#8320]
+
 jump
 ----
 
-- Removed a unit test in Jump that was moved to STCAL to decrease
+- Add parameters that control the flagging of saturated cores of snowballs in
+  the next integration. [#8303]
+
+- Removed a unit test in ``jump`` that was moved to STCAL to decrease
   the coupling of the two repos. [#8319]
 
+- To improve performance an additional parameter to the jump step was added
+  that sets the threshold number of differences above which iterative flagging
+  of one CR at a time is turned off. [#8304]
+
 lib
 ---
 
 - Updated ``set_velocity_aberration`` to use datamodels instead of `astropy.io.fits` for opening
   and manipulating input files. [#8285]
 
-lib
----
-
 - Added new function set_nans_to_donotuse in ``lib.basic_utils`` to
   check the science data array for NaN values and check if they have
   a DQ flag of DO_NOT_USE, or set it if not. [#8292]
@@ -197,6 +244,8 @@ photom
   was bypassed and came before the ``photom`` step, e.g. for NIRISS SOSS
   data in the FULL subarray. [#8225]
 
+- Removed blanket warning ignore for ``find_row``. [#8320]
+
 pipeline
 --------
 
@@ -212,6 +261,11 @@ pixel_replace
 - Fixed a bug that caused array size mismatches when the ``mingrad`` algorithm
   was applied to NIRSpec data. [#8312]
 
+ramp_fitting
+------------
+
+- Modified one runtime warning filter. [#8320]
+
 refpix
 ------
 
@@ -247,6 +301,10 @@ resample
 
 - Removed any reference to the "tophat" kernel for resample step. [#8364]
 
+- Increased specificity of several warning filters. [#8320]
+
+- Changed deprecated ``stpipe.extern.configobj`` to ``astropy.extern.configobj``. [#8320]
+
 residual_fringe
 ---------------
 
@@ -261,6 +319,11 @@ scripts
 
 - Remove ``migrate_data`` and ``move_wcs`` scripts. [#8321]
 
+stpipe
+------
+
+- Changed deprecated ``stpipe.extern.configobj`` to ``astropy.extern.configobj``. [#8320]
+
 source_catalog
 --------------
 
@@ -280,6 +343,13 @@ tweakreg
 
 - Fixed a bug that caused failures instead of warnings when no GAIA sources
   were found within the bounding box of the input image. [#8334]
+
+- Suppress AstropyUserWarnings regarding NaNs in the input data. [#8320]
+
+wfs_combine
+-----------
+
+- Fixed deprecated conversions of single-element numpy arrays to scalar. [#8320]
 
 
 1.13.4 (2024-01-25)

CITATION.cff

@@ -75,7 +75,7 @@ authors:
   given-names: "Maria"
   orcid: "https://orcid.org/0000-0003-2314-3453"
 title: "JWST Calibration Pipeline"
-version: 1.13.4
+version: 1.14.0
 doi: 10.5281/zenodo.7038885
-date-released: 2024-01-25
+date-released: 2024-03-25
 url: "https://github.com/spacetelescope/jwst"

JenkinsfileRT

@@ -106,5 +106,10 @@ bc1.test_configs = []
 bc2 = utils.copy(bc0)
 bc2.pip_reqs_files = ['requirements-dev-st.txt']
 
-utils.run([jobconfig, bc0, bc1, bc2])
+bc3 = utils.copy(bc0)
+bc3.conda_packages = [
+    "python=3.12",
+]
+
+utils.run([jobconfig, bc0, bc1, bc2, bc3])
 }  // withCredentials

README.md

@@ -9,16 +9,14 @@
 
 ![STScI Logo](docs/_static/stsci_logo.png)
 
-**JWST requires a C compiler for dependencies and is currently limited to Python 3.9, 3.10 or 3.11.**
+> [!IMPORTANT]
+> JWST requires a C compiler for dependencies and is currently limited to Python 3.10, 3.11, or 3.12.
 
-**Until Python 3.12 is supported, fresh conda environments will require setting the
-  Python version to one of the three supported versions.**
+> [!NOTE]
+> Linux and MacOS platforms are tested and supported.  Windows is not currently supported.
 
-**Linux and MacOS platforms are tested and supported.  Windows is not currently supported.**
-
-**If installing on MacOS Mojave 10.14, you must install
-  into an environment with python 3.9. Installation will fail on python 3.10 due
-  to lack of a stable build for dependency ``opencv-python``.**
+> [!WARNING]
+> Installation on MacOS Mojave 10.14 will fail due to lack of a stable build for dependency ``opencv-python``.
 
 ## Installation
 
@@ -218,6 +216,7 @@ the specified context and less than the context for the next release.
 
 | jwst tag            | DMS build | SDP_VER  | CRDS_CONTEXT | Released   | Ops Install | Notes                                         |
 |---------------------|-----------|----------|--------------|------------|-------------|-----------------------------------------------|
+| 1.14.0              | B10.2rc1  |          | 1215         | 2024-03-25 |             | First release candidate for B10.2             |
 | 1.13.4              |           |          | 1185         | 2024-01-25 |             | PyPI-only release for external users          |
 | 1.13.3              | B10.1     | 2023.4.0 | 1181         | 2024-01-05 |             | Final release candidate for B10.1             |
 | 1.13.2              | B10.1rc3  | 2023.4.0 | 1181         | 2023-12-21 |             | Third release candidate for B10.1             |

docs/getting_started/install.rst

@@ -38,15 +38,10 @@ shell.
 
 .. warning::
 
-    The jwst package requires a C compiler for dependencies and is currently
-    limited to Python 3.9, 3.10, or 3.11. Until Python 3.12 is supported, fresh
-    conda environments will require setting the Python version to <=3.11.
+    JWST requires a C compiler for dependencies and is currently limited to Python 3.10, 3.11, or 3.12.
 
 .. warning::
-
-    Users on MacOS Mojave (10.14) should limit their environment to python 3.9 -
-    there is a package dependency that currently fails to build on Mojave with
-    python>=3.10.
+    Installation on MacOS Mojave 10.14 will fail due to lack of a stable build for dependency ``opencv-python``.
 
 Installing Latest Release
 -------------------------

docs/jwst/ami_analyze/description.rst

@@ -20,7 +20,9 @@ the SUB80 subarray, in order to reduce execution time.
 
 Arguments
 ---------
-The ``ami_analyze`` step has four optional arguments:
+The ``ami_analyze`` step has several optional arguments. In most cases the 
+default arguments will be suitable but more advanced users may wish to test
+other options:
 
 :--oversample: The oversampling factor to be used in the model fit (default=3).
 
@@ -33,48 +35,89 @@ The ``ami_analyze`` step has four optional arguments:
 :--rotation_search: List of start, stop, and step values that define the list of
                     rotation search values. The default setting of '-3 3 1'
                     results in search values of [-3, -2, -1, 0, 1, 2, 3].
-
+
+:--bandpass: Synphot spectrum or suitable array to override filter/source 
+             (default=None)
+
+:--usebp: If True, exclude pixels marked DO_NOT_USE from fringe fitting 
+          (default=True)
+
+:--firstfew: If not None, process only the first few integrations (default=None)
+
+:--chooseholes: If not None, fit only certain fringes e.g. ['B4','B5','B6','C2']
+                (default=None)
+
+:--affine2d: User-defined Affine2d object (default=None)
+
+:--run_bpfix: Run Fourier bad pixel fix on cropped data (default=True)
+
 
 Inputs
 ------
 
-2D calibrated image
+3D calibrated image
 ^^^^^^^^^^^^^^^^^^^
-:Data model: `~jwst.datamodels.ImageModel`
-:File suffix: _cal
+:Data model: `~jwst.datamodels.DataModel`
+:File suffix: _calints
 
-The ``ami_analyze`` step takes a single calibrated image as input, which should be
-the "_cal" product resulting from :ref:`calwebb_image2 <calwebb_image2>` processing.
+The ``ami_analyze`` step takes a single calibrated image cube as input, which should be
+the "_calints" product resulting from :ref:`calwebb_image2 <calwebb_image2>` processing.
 Multiple exposures can be processed via use of an ASN file that is used as input
-to the :ref:`calwebb_ami3 <calwebb_ami3>` pipeline. The ``ami_analyze`` step itself does
-not accept an ASN as input.
+to the :ref:`calwebb_ami3 <calwebb_ami3>` pipeline. **Note:** The ``ami_analyze`` step will also
+accept a 2D "_cal" product but errors will not be computed in the output.
+The ``ami_analyze`` step itself does not accept an ASN as input.
 
 Outputs
 -------
 
+The ``ami_analyze`` step produces three output files. The first two (``_ami-oi.fits`` and ``_amimulti-oi.fits``) contain the interferometric observables, and the third (``_amilg.fits``) contains the data, LG model, and residuals.  These are described in more detail below.
+
+The output file name syntax is exposure-based, using the input file name as the root, with
+the addition of the association candidate ID and the "_ami-oi", "_amimulti-oi", or "amilg" product type suffix, e.g.
+"jw87600027001_02101_00002_nis_a3001_ami-oi.fits."
+
+Interferometric observables
+^^^^^^^^^^^^^^^^^^^^^^^^^^^
+:Data model: `~jwst.datamodels.AmiOIModel`
+:File suffix: _ami-oi.fits, _amimulti-oi.fits
+
+. The inteferometric observables are saved as OIFITS files, a registered FITS format
+for optical interferometry, containing the following list of extensions:
+
+1)  ``OI_ARRAY``: AMI subaperture information  
+2)  ``OI_TARGET``: target properties  
+3)  ``OI_T3``: extracted closure amplitudes, phases 
+4)  ``OI_VIS``: extracted visibility (fringe) amplitudes, phases
+5)  ``OI_VIS2``: squared visibility (fringe) amplitudes
+6)  ``OI_WAVELENGTH``: filter information
+
+For more information on the format and contents of OIFITS files, see the `OIFITS2 standard <https://doi.org/10.1051/0004-6361/201526405>`_.
+
+The _ami-oi.fits file contains tables of median observables over all integrations of the input file.  Errors 
+are computed as the sigma-clipped standard deviation over integrations.
+The _amimulti-oi.fits file contains observables for each integration, and does not contain error estimates. The
+structure is the same as the _ami-oi.fits file, but the following data columns are 2D, with the second dimension being 
+the number of integrations: "PISTONS", "PIST_ERR", "VISAMP", "VISAMPERR", "VISPHI", "VISPHIERR", "VIS2DATA", "VIS2ERR", "T3AMP", "T3AMPERR", "T3PHI", "T3PHIERR".
+
 LG model parameters
 ^^^^^^^^^^^^^^^^^^^
-:Data model: `~jwst.datamodels.AmiLgModel`
-:File suffix: _ami
+:Data model: `~jwst.datamodels.AmiLgFitModel`
+:File suffix: _amilg.fits
 
-The ``ami_analyze`` step produces a single output file, containing the
-following list of extensions:
+The _amilg.fits output file contains the cropped and cleaned data, model, and residuals (data - model) as well as 
+the parameters of the best-fit LG model. It contains the following extensions:
 
-1) ``FIT``: a 2D image of the fitted model
-2) ``RESID``: a 2D image of the fit residuals
-3) ``CLOSURE_AMP``: table of closure amplitudes
-4) ``CLOSURE_PHA``: table of closure phases
-5) ``FRINGE_AMP``: table of fringe amplitudes
-6) ``FRINGE_PHA``: table of fringe phases
-7) ``PUPIL_PHA``: table of pupil phases
-8) ``SOLNS``: table of fringe coefficients
-
-The output file name syntax is exposure-based, using the input file name as the root, with
-the addition of the association candidate ID and the "_ami" product type suffix, e.g.
-"jw87600027001_02101_00002_nis_a3001_ami.fits."
+1) ``CTRD``: a 3D image of the centered, cropped data
+2) ``N_CTRD``: a 3D image CTRD normalized by data peak
+3) ``FIT``: a 3D image of the best-fit model
+4) ``N_FIT``: a 3D image of FIT normalized by data peak
+5) ``RESID``: a 3D image of the fit residuals
+6) ``N_RESID``: a 3D image of RESID normalized by data peak
+7) ``SOLNS``: table of fringe coefficients
 
 Reference Files
 ---------------
-The ``ami_analyze`` step uses a THROUGHPUT reference file.
+The ``ami_analyze`` step uses a THROUGHPUT reference file and NRM reference file.
 
 .. include:: ../references_general/throughput_reffile.inc
+.. include:: ../references_general/nrm_reffile.inc