Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

JP-2005: Update NIS WFSS ASN rules for matching filter #5896

Merged
merged 2 commits into from
Mar 19, 2021
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions CHANGES.rst
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,10 @@ associations
- Removed PATTTYPE='None' constraint from Lv3MIRMRS association rule to
generate spec3 associations for undithered MRS observations. [#5804]

- Updated level2b WFSS rules to only consider exposures using the same
PUPIL value (cross filter) when matching direct images with grism images
in NIRISS WFSS observations. [#5896]

cube_build
----------

Expand Down
6 changes: 3 additions & 3 deletions docs/jwst/associations/level2_asn_rules.rst
Original file line number Diff line number Diff line change
@@ -1,13 +1,13 @@
.. asn-level2-rules:

Level 2 Associations: Rules
Stage 2 Associations: Rules
===========================

The following table describes exactly which exposures will go
through any type of Level 2b processing, such as Spec2Pipeline or
through any type of stage 2 processing, such as Spec2Pipeline or
Image2Pipeline.

.. list-table:: Exposure Modes for Level 2 Processing
.. list-table:: Exposure Modes for Stage 2 Processing
:widths: 20 20 20 20
:header-rows: 1

Expand Down
16 changes: 8 additions & 8 deletions docs/jwst/associations/level2_asn_technical.rst
Original file line number Diff line number Diff line change
@@ -1,14 +1,14 @@
.. _asn-level2-techspecs:

Level 2 Associations: Technical Specifications
Stage 2 Associations: Technical Specifications
==============================================

Logical Structure
-----------------

Independent of the actual format, all Level 2 associations have the
Independent of the actual format, all stage 2 associations have the
following structure. Again, the structure is defined and enforced by
the Level 2 schema
the stage 2 schema

* Top level, or meta, key/values
* List of products, each consisting of
Expand Down Expand Up @@ -103,23 +103,23 @@ constraints *optional*
^^^^^^^^^^^^^^^^^^^^

A list of products that would be produced by this association. For
Level2, each product is an exposure. Each product should have one
``science`` member, the exposure on which the Level2b processing will
stage 2, each product is an exposure. Each product should have one
``science`` member, the exposure on which the stage 2 processing will
occur.

Association products have two components:

name *optional*
The string template to be used by Level 2b processing tasks to create
The string template to be used by stage 2 processing tasks to create
the output file names. The product name, in general, is a prefix on
which the individual pipeline and step modules will append whatever
suffix information is needed.

If not specified, the Level2b processing modules will create a name
If not specified, the stage 2 processing modules will create a name
based off the name of the ``science`` member.

members *required*
This is a list of the exposures to be used by the Level 2b processing
This is a list of the exposures to be used by the stage 2 processing
tasks. This keyword is explained in detail in the next section.

``members`` Keyword
Expand Down
4 changes: 2 additions & 2 deletions docs/jwst/associations/level3_asn_rules.rst
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
.. _level3-asn-rules:

Level3 Associations: Rules
Stage3 Associations: Rules
==========================

.. _level3-asn-data-grouping:
Expand Down Expand Up @@ -51,7 +51,7 @@ follows:

- exposure

The basic unit of science data. Starting at Level1b, an exposure
The basic unit of science data. Starting at stage 1, an exposure
contains a single integrations of a single detector from a single
instrument for a single *snap*. Note that a single integration
actually is a number of readouts of the detector during the integration.
Expand Down
16 changes: 8 additions & 8 deletions docs/jwst/associations/level3_asn_technical.rst
Original file line number Diff line number Diff line change
@@ -1,14 +1,14 @@
.. _asn-level3-techspecs:

Level 3 Associations: Technical Specifications
Stage 3 Associations: Technical Specifications
==============================================

Logical Structure
-----------------

Independent of the actual format, all Level 3 associations have the
Independent of the actual format, all stage 3 associations have the
following structure. Again, the structure is defined and enforced by
the Level 3 schema
the stage 3 schema

* Top level, or meta, key/values
* List of products, each consisting of
Expand Down Expand Up @@ -71,7 +71,7 @@ program *optional*

target *optional*
Target ID for which this association refers to. DMS currently uses
the TARGETID header keyword in the Level2 exposure files, but there
the TARGETID header keyword in the stage 2 exposure files, but there
is no formal restrictions on value.

asn_type *optional*
Expand Down Expand Up @@ -107,15 +107,15 @@ constraints *optional*
Association products have two components:

name *optional*
The string template to be used by Level 3 processing tasks to create
The string template to be used by stage 3 processing tasks to create
the output file names. The product name, in general, is a prefix on
which the individual pipeline and step modules will append whatever
suffix information is needed.

If not specified, the Level3 processing modules will create a name root.
If not specified, the stage 3 processing modules will create a name root.

members *required*
This is a list of the exposures to be used by the Level 3 processing
This is a list of the exposures to be used by the stage 3 processing
tasks. This keyword is explained in detail in the next section.

``members`` Keyword
Expand All @@ -133,7 +133,7 @@ exptype *required*
* ``science`` *required*

The primary science exposures. There is usually more than one
since Level3 calibration involves combining multiple science
since stage 3 calibration involves combining multiple science
exposures. However, at least one exposure in an association needs
to be ``science``.

Expand Down
26 changes: 13 additions & 13 deletions docs/jwst/associations/technote_sdp_workflow.rst
Original file line number Diff line number Diff line change
Expand Up @@ -9,11 +9,11 @@ General Workflow

See :ref:`asn-associations-and-jwst` for an overview of how JWST uses
associations. This document describes how associations are used by the
ground processing system to execute the level 2 and level 3 pipelines
ground processing system to execute the stage 2 and stage 3 pipelines
based on.

Up to the initial calibration step ``calwebb_detector1``, the science
exposures are treated individually. However, starting at the level 2
exposures are treated individually. However, starting at the stage 2
calibration step, exposures may need other exposures in order to be
further processed. Instead of creating a single monolithic pipeline,
the workflow uses the associations to determine what pipeline should
Expand All @@ -32,7 +32,7 @@ data through the system.
.. figure:: graphics/workflow-generic.png
:scale: 75%

General workflow through level 2 and level 3 processing
General workflow through stage 2 and stage 3 processing

The figure represents the following workflow:

Expand All @@ -56,7 +56,7 @@ The figure represents the following workflow:
Wide Field Slitless Spectroscopy
================================

In most cases, the data will flow from level 2 to level 3, completing
In most cases, the data will flow from stage 2 to stage 3, completing
calibration. However, more complicated situations can be handled by
the same wait-then-execute process. One particular case is for the
Wide Field Slitless Spectrometry (WFSS) modes. The specific flow is
Expand All @@ -70,27 +70,27 @@ show in the figure below:
For WFSS data, at least two observations are made, one consisting of a
direct image of the field-of-view (FOV), and a second where the FOV is
dispersed using a grism. The direct image is first processed through
level 3. At the level 3 stage, a source catalog of objects found in
the image, and a segment map, used to record the minimum bounding
stage 3. At the stage 3 stage, a source catalog of objects found in
the image, and a segmentation map, used to record the minimum bounding
box sizes for each object, are generated. The source catalog is then used
as input to the level 2 processing of the spectral data. This extra
as input to the stage 2 processing of the spectral data. This extra
link between the two major stages is represented by the ``Segment &
Catalog`` file set, shown in red in the diagram. The level 2 association
Catalog`` file set, shown in red in the diagram. The stage 2 association
``grism_spec2_asn`` not only lists the needed countrate exposures, but
also the catalog file produced by the level 3 image
also the catalog file produced by the stage 3 image
processing. Hence, the workflow knows to wait for this file before
continuing the spectral processing.

Field Guide to File Names
=========================

The high-level distinctions between level 2, level 3, exposure-centric,
The high-level distinctions between stage 2, stage 3, exposure-centric,
and target-centric files can be determined by the following file patterns.

- Files produced by level 3 processing
- Files produced by stage 3 processing

Any file name that matches the following regex is a file that has
been produced by a level 3 pipeline::
been produced by a stage 3 pipeline::

.+[aocr][0-9]{3:4}.+

Expand All @@ -107,7 +107,7 @@ and target-centric files can be determined by the following file patterns.
jw[0-9]{5}-[aocr][0-9]{3:4}_.+

Such data is the result of the combination of data from several
exposures, usually produced by a level 3 calibration pipeline.
exposures, usually produced by a stage 3 calibration pipeline.

Note that these patterns are not intended to fully define all the
specific types of files there are. However, these are the main
Expand Down
8 changes: 7 additions & 1 deletion jwst/associations/lib/rules_level2b.py
Original file line number Diff line number Diff line change
Expand Up @@ -603,7 +603,6 @@ def add_catalog_members(self):
closest = directs[0] # If the search fails, just use the first.
try:
expspcin = int(getattr_from_list(science.item, ['expspcin'], _EMPTY)[1])
science_channel = getattr_from_list(science.item, ['channel'], _EMPTY)[1]
except KeyError:
# If exposure sequence cannot be determined, just fall through.
logger.debug('Science exposure %s has no EXPSPCIN defined.', science)
Expand All @@ -613,9 +612,16 @@ def add_catalog_members(self):
# For NIRCam, only consider direct images from the same channel
# as the grism image
if direct.item['exp_type'] == 'nrc_image':
science_channel = getattr_from_list(science.item, ['channel'], _EMPTY)[1]
direct_channel = getattr_from_list(direct.item, ['channel'], _EMPTY)[1]
if direct_channel != science_channel:
continue
# For NIRISS, only consider direct images with the same PUPIL value
if direct.item['exp_type'] == 'nis_image':
science_pupil = getattr_from_list(science.item, ['pupil'], _EMPTY)[1]
direct_pupil = getattr_from_list(direct.item, ['pupil'], _EMPTY)[1]
if direct_pupil != science_pupil:
continue
try:
direct_expspcin = int(getattr_from_list(
direct.item, ['expspcin'], _EMPTY
Expand Down