From 0460fabb0620025b24aff371e34c6bdf088e69d4 Mon Sep 17 00:00:00 2001 From: Howard Bushouse Date: Fri, 29 Mar 2024 11:25:18 -0400 Subject: [PATCH 1/4] JP-3554: Add MSA meta data file docs --- docs/jwst/assign_wcs/exp_types.rst | 100 +++++----- docs/jwst/assign_wcs/main.rst | 55 +++--- docs/jwst/assign_wcs/reference_files.rst | 8 +- docs/jwst/data_products/index.rst | 1 + docs/jwst/data_products/msa_metadata.rst | 221 +++++++++++++++++++++++ 5 files changed, 306 insertions(+), 79 deletions(-) create mode 100644 docs/jwst/data_products/msa_metadata.rst diff --git a/docs/jwst/assign_wcs/exp_types.rst b/docs/jwst/assign_wcs/exp_types.rst index 9264e1f1ff..3502e1e7e8 100644 --- a/docs/jwst/assign_wcs/exp_types.rst +++ b/docs/jwst/assign_wcs/exp_types.rst @@ -2,73 +2,75 @@ WCS reference file information per EXP_TYPE =========================================== -:FGS_IMAGE, FGS_FOCUS, FGS_SKYFLAT, FGS_INTFLAT: +**FGS_IMAGE, FGS_FOCUS, FGS_SKYFLAT, FGS_INTFLAT:** - | reftypes: *distortion* - | WCS pipeline coordinate frames: detector, v2v3, world - | Implements: reference file provided by NIRISS team + | reftypes: *distortion* + | WCS pipeline coordinate frames: detector, v2v3, world + | Implements: reference file provided by NIRISS team -:MIR_IMAGE, MIR_TACQ, MIR_LYOT, MIR4QPM, MIR_CORONCAL: +**MIR_IMAGE, MIR_TACQ, MIR_LYOT, MIR4QPM, MIR_CORONCAL:** - | reftypes: *distortion*, *filteroffset* - | WCS pipeline coordinate frames: detector, v2v3, world - | Implements: CDP6 reference data delivery, MIRI-TN-00070-ATC_Imager_distortion_CDP_Iss5.pdf + | reftypes: *distortion*, *filteroffset* + | WCS pipeline coordinate frames: detector, v2v3, world + | Implements: CDP6 reference data delivery, MIRI-TN-00070-ATC_Imager_distortion_CDP_Iss5.pdf -:MIR_LRS-FIXEDSLIT, MIR_LRS-SLITLESS: +**MIR_LRS-FIXEDSLIT, MIR_LRS-SLITLESS:** - | reftypes: *specwcs*, *distortion* - | WCS pipeline coordinate frames: detector, v2v3, world - | Implements: CDP4 reference data delivery, MIRI-TR-10020-MPI-Calibration-Data-Description_LRSPSFDistWave_v4.0.pdf + | reftypes: *specwcs*, *distortion* + | WCS pipeline coordinate frames: detector, v2v3, world + | Implements: CDP4 reference data delivery, MIRI-TR-10020-MPI-Calibration-Data-Description_LRSPSFDistWave_v4.0.pdf -:MIR_MRS: +**MIR_MRS:** - | reftypes: *distortion*, *specwcs*, *v2v3*, *wavelengthrange*, *regions* - | WCS pipeline coordinate frames: detector, miri_focal, xyan, v2v3, world - | Implements: CDP4 reference data delivery, MIRI-TN-00001-ETH_Iss1-3_Calibrationproduct_MRS_d2c.pdf + | reftypes: *distortion*, *specwcs*, *v2v3*, *wavelengthrange*, *regions* + | WCS pipeline coordinate frames: detector, miri_focal, xyan, v2v3, world + | Implements: CDP4 reference data delivery, MIRI-TN-00001-ETH_Iss1-3_Calibrationproduct_MRS_d2c.pdf -:NRC_IMAGE, NRC_TSIMAGE, NRC_FOCUS, NRC_TACONFIRM, NRC_TACQ: +**NRC_IMAGE, NRC_TSIMAGE, NRC_FOCUS, NRC_TACONFIRM, NRC_TACQ:** - | reftypes: *distortion*, *filteroffset* - | WCS pipeline coordinate frames: detector, v2v3, world - | Implements: Distortion file created from TEL team data. + | reftypes: *distortion*, *filteroffset* + | WCS pipeline coordinate frames: detector, v2v3, world + | Implements: Distortion file created from TEL team data. -:NRC_WFSS, NRC_TSGRISM: - | reftypes: *specwcs*, *distortion*, *filteroffset* - | WCS pipeline coordinate frames: grism_detector, detector, v2v3, world - | Implements: reference files provided by NIRCam team +**NRC_WFSS, NRC_TSGRISM:** -:NIS_IMAGE, NIS_TACQ, NIS_TACONFIRM, NIS_FOCUS: + | reftypes: *specwcs*, *distortion*, *filteroffset* + | WCS pipeline coordinate frames: grism_detector, detector, v2v3, world + | Implements: reference files provided by NIRCam team - | reftypes: *distortion*, *filteroffset* - | WCS pipeline coordinate frames: detector, v2v3, world - | Implements: reference file provided by NIRISS team +**NIS_IMAGE, NIS_TACQ, NIS_TACONFIRM, NIS_FOCUS:** -:NIS_WFSS: - | reftypes: *specwcs*, *distortion*, *filteroffset* - | WCS pipeline coordinate frames: grism_detector, detector, v2v3, world - | Implements: reference files provided by NIRISS team + | reftypes: *distortion*, *filteroffset* + | WCS pipeline coordinate frames: detector, v2v3, world + | Implements: reference file provided by NIRISS team -:NIS_SOSS: +**NIS_WFSS:** - | reftypes: *distortion*, *specwcs* - | WCS pipeline coordinate frames: detector, v2v3, world - | Implements: reference files provided by NIRISS team + | reftypes: *specwcs*, *distortion*, *filteroffset* + | WCS pipeline coordinate frames: grism_detector, detector, v2v3, world + | Implements: reference files provided by NIRISS team -:NRS_FIXEDSLIT, NRS_MSASPEC, NRS_LAMP, NRS_BRIGHTOBJ: +**NIS_SOSS:** - | reftypes: *fpa*, *camera*, *disperser*, *collimator*, *msa*, *wavelengthrange*, *fore*, *ote* - | WCS pipeline coordinate frames: detector, sca, bgwa, slit_frame, msa_frame, ote, v2v3, world - | Implements: CDP 3 delivery + | reftypes: *distortion*, *specwcs* + | WCS pipeline coordinate frames: detector, v2v3, world + | Implements: reference files provided by NIRISS team -:NRS_IFU: +**NRS_FIXEDSLIT, NRS_MSASPEC, NRS_LAMP, NRS_BRIGHTOBJ:** - | reftypes: *fpa*, *camera*, *disperser*, *collimator*, *msa*, *wavelengthrange*, *fore*, *ote*, - | *ifufore*, *ifuslicer*, *ifupost* - | WCS pipeline coordinate frames: detector, sca, bgwa, slit_frame, msa_frame, ote, v2v3, world - | Implements: CDP 3 delivery + | reftypes: *fpa*, *camera*, *disperser*, *collimator*, *msa*, *wavelengthrange*, *fore*, *ote* + | WCS pipeline coordinate frames: detector, sca, bgwa, slit_frame, msa_frame, ote, v2v3, world + | Implements: CDP 3 delivery -:NRS_IMAGING, NRS_MIMF, NRS_BOTA, NRS_CONFIRM, NRS_TACONFIRM, NRS_TASLIT, NRS_TACQ: +**NRS_IFU:** - | reftypes: *fpa*, *camera*, *disperser*, *collimator*, *msa*, *wavelengthrange*, *fore*, *ote* - | WCS pipeline coordinate frames: detector, sca, bgwa, slit_frame, msa_frame, ote, v2v3, world - | Implements: CDP 3 delivery + | reftypes: *fpa*, *camera*, *disperser*, *collimator*, *msa*, *wavelengthrange*, *fore*, *ote*, + | *ifufore*, *ifuslicer*, *ifupost* + | WCS pipeline coordinate frames: detector, sca, bgwa, slit_frame, msa_frame, ote, v2v3, world + | Implements: CDP 3 delivery + +**NRS_IMAGING, NRS_MIMF, NRS_BOTA, NRS_CONFIRM, NRS_TACONFIRM, NRS_TASLIT, NRS_TACQ:** + + | reftypes: *fpa*, *camera*, *disperser*, *collimator*, *msa*, *wavelengthrange*, *fore*, *ote* + | WCS pipeline coordinate frames: detector, sca, bgwa, slit_frame, msa_frame, ote, v2v3, world + | Implements: CDP 3 delivery diff --git a/docs/jwst/assign_wcs/main.rst b/docs/jwst/assign_wcs/main.rst index c8baf2b742..59a526dd2c 100644 --- a/docs/jwst/assign_wcs/main.rst +++ b/docs/jwst/assign_wcs/main.rst @@ -6,22 +6,23 @@ Description :Alias: assign_wcs -``jwst.assign_wcs`` is run in the beginning of the level 2B JWST pipeline. +The ``assign_wcs`` step is run at the beginning of the Stage 2 pipelines - :ref:`calwebb_image2 ` +and :ref:`calwebb_spec2 ` - for both imaging and spectroscopic exposures. It associates a WCS object with each science exposure. The WCS object transforms positions in the detector frame to positions in a world coordinate frame - ICRS and wavelength. In general there may be intermediate coordinate frames depending on the instrument. The WCS is saved in the ASDF extension of the FITS file. It can be accessed as an attribute of -the meta object when the fits file is opened as a data model. +the meta object when the FITS file is opened as a data model. The forward direction of the transforms is from detector to world coordinates -and the input positions are 0-based. +and the input pixel coordinates are 0-indexed. -``jwst.assign_wcs`` expects to find the basic WCS keywords in the -SCI header. Distortion and spectral models are stored in reference files in the +The ``assign_wcs`` step expects to find the basic WCS keywords in the +"SCI" extension header of the input FITS file. Distortion and spectral models are stored in reference files in the `ASDF `__ format. For each observing mode, determined by the value of ``EXP_TYPE`` in the science header, -assign_wcs retrieves reference files from CRDS and creates a pipeline of transforms from +``assign_wcs`` retrieves reference files from CRDS and creates a pipeline of transforms from input frame ``detector`` to a frame ``v2v3``. This part of the WCS pipeline may include intermediate coordinate frames. The basic WCS keywords are used to create the transform from frame ``v2v3`` to frame ``world``. @@ -29,14 +30,23 @@ the transform from frame ``v2v3`` to frame ``world``. For image display with software like DS9 that relies on specific WCS information, a SIP-based approximation to the WCS is fit. The results are FITS keywords stored in ``model.meta.wcsinfo``. This is not an exact fit, but is accurate to ~0.25 pixel and is sufficient -for display purposes. This step, which occurs for imaging modes early, is performed by default but -can be switched off, and parameters controlling the fit can also be adjusted. +for display purposes. This step, which occurs for imaging modes, is performed by default, but +can be switched off, and parameters controlling the SIP fit can also be adjusted. +For each exposure, ``assign_wcs`` uses reference files and WCS header keywords +to create the WCS object. What reference files are retrieved +from CRDS is determined based on `EXP_TYPE` and other keywords in the science file header. -``jwst.assign_wcs`` is based on `gwcs `__ and -uses `asdf `__. +The ``assign_wcs`` step can accept the single slope image that is the result of averaging +over all integrations or a 3D cube of integrations in the case of TSO exposures. +The ``assign_wcs`` step is based on `gwcs `__ and +uses `asdf `__. +.. Note:: In addition to CRDS reference files, applying ``assign_wcs`` to NIRSpec MOS + exposures depends critically on an MSA meta data file to provide information + for MOS slitlets in use and their constituent shutters. See :ref:`msa_metadata ` + for detailed information about the MSA meta data files and their contents. Basic WCS keywords and the transform from ``v2v3`` to ``world`` --------------------------------------------------------------- @@ -44,21 +54,21 @@ Basic WCS keywords and the transform from ``v2v3`` to ``world`` All JWST instruments use the following FITS header keywords to define the transform from ``v2v3`` to ``world``: -``RA_REF``, ``DEC_REF`` - a fiducial point on the sky, ICRS, [deg] +``RA_REF``, ``DEC_REF`` - a fiducial point on the sky, ICRS [deg] -``V2_REF``, ``V3_REF`` - a point in the V2V3 system which maps to ``RA_REF``, ``DEC_REF``, [arcsec] +``V2_REF``, ``V3_REF`` - a point in the V2V3 system that maps to ``RA_REF``, ``DEC_REF`` [arcsec] -``ROLL_REF`` - local roll angle associated with each aperture, [deg] +``ROLL_REF`` - local roll angle associated with each aperture [deg] ``RADESYS`` - standard coordinate system [ICRS] These quantities are used to create a 3D Euler angle rotation between the V2V3 spherical system, associated with the telescope, and a standard celestial system. -For spectroscopic data, ``jwst.assign_wcs`` populates keyword ``DISPAXIS`` +For spectroscopic data, ``assign_wcs`` populates the keyword ``DISPAXIS`` with an integer value that indicates whether the dispersion direction is oriented more nearly along the horizontal (DISPAXIS = 1) or vertical -(DISPAXIS = 2) direction. +(DISPAXIS = 2) direction in the image frame. Using the WCS interactively @@ -76,8 +86,8 @@ corresponding world coordinates. Using MIRI LRS fixed slit as an example: >>> print(ra, dec, lam) (329.97260532549336, 372.0242999250267, 5.4176100046836675) -The WFSS modes for NIRCam and NIRISS have a slightly different calling structure, -in addition to the (x, y) coordinate, they need to know other information about the +The WFSS modes for NIRCam and NIRISS have a slightly different calling structure. +In addition to the (x, y) coordinates, they need to know other information about the spectrum or source object. In the JWST backward direction (going from the sky to the detector) the WCS model also looks for the wavelength and order and returns the (x,y) location of that wavelength+order on the dispersed image and the original @@ -95,8 +105,8 @@ source pixel location, as entered, along with the order that was specified: The WCS provides access to intermediate coordinate frames -and transforms between any two frames in the WCS pipeline in forward or -backward direction. For example, for a NIRSpec fixed slits exposure, +and transforms between any two frames in the WCS pipeline in the forward or +backward directions. For example, for a NIRSpec fixed slits exposure, which has been through the extract_2d step: .. doctest-skip:: @@ -108,13 +118,6 @@ which has been through the extract_2d step: >>> msa2detector(0, 0, 2*10**-6) (5042.064255529629, 1119.8937888372516) -For each exposure, assign_wcs uses reference files and WCS header keywords -to create the WCS object. What reference files are retrieved -from CRDS is determined based on EXP_TYPE and other keywords in the science file header. - -The assign_wcs step can accept the single slope image that is the result of averaging -over all integrations or a 3D cube of integrations in the case of TSO exposures. - WCS of slitless grism exposures ------------------------------- diff --git a/docs/jwst/assign_wcs/reference_files.rst b/docs/jwst/assign_wcs/reference_files.rst index 7df12eae1e..c0d4e3f399 100644 --- a/docs/jwst/assign_wcs/reference_files.rst +++ b/docs/jwst/assign_wcs/reference_files.rst @@ -3,11 +3,11 @@ Reference Files WCS Reference files are in the Advanced Scientific Data Format (ASDF). The best way to create the file is to programmatically create the model and then save it to a file. -A tutorial on creating reference files in ASDF format is available at: +A tutorial on creating reference files in ASDF format is available +`here +`_. -https://github.com/spacetelescope/jwreftools/blob/master/docs/notebooks/referece_files_asdf.ipynb - -Transforms are 0-based. The forward direction is from detector to sky. +Transforms are 0-indexed. The forward direction is from detector to sky. Reference file types used by ``assign_wcs`` ------------------------------------------- diff --git a/docs/jwst/data_products/index.rst b/docs/jwst/data_products/index.rst index d9cc684110..c398adca7c 100644 --- a/docs/jwst/data_products/index.rst +++ b/docs/jwst/data_products/index.rst @@ -12,5 +12,6 @@ Data Products Information common_features.rst science_products.rst nonscience_products.rst + msa_metadata.rst guidestar_products.rst migrating.rst diff --git a/docs/jwst/data_products/msa_metadata.rst b/docs/jwst/data_products/msa_metadata.rst new file mode 100644 index 0000000000..63da4864f9 --- /dev/null +++ b/docs/jwst/data_products/msa_metadata.rst @@ -0,0 +1,221 @@ +.. _msa_metadata: + +MSA Meta Data File: ``msa`` +^^^^^^^^^^^^^^^^^^^^^^^^^^^ +While not containing any actual science data, the NIRSpec MSA meta data file is nonetheless +a crucial component of calibration processing for NIRSpec MOS exposures. +It contains all the slitlet, shutter, and source configuration information that's needed +by the :ref:`calwebb_spec2 ` pipeline to process a MOS exposure. +These meta data are originally generated by the +`MSA Planning Tool `_ +(MPT) in APT and passed through the ground system to populate the +FITS metafile. Details of the meta data and how it is used in calibration processing +are given below. + +File names for MSA meta data files are of the form: + + jw__msa.fits + +where + + - ppppp: program ID number + - ooo: observation number + - vvv: visit number + - cfg: MSA configuration + +such as `jw01180025001_01_msa.fits`. + +Each MOS science exposure contains a primary header keyword `MSAMETFL` that gives +the name of the MSA meta data file to be used when processing the exposure. +The values of the primary header keywords `MSAMETID` and `PATT_NUM` are also used +when retrieving relevant rows of data from the tables in the MSA meta data file. +If any of these keywords is missing from the input science exposure, processing +will not be able to proceed and an error will be raised. + +File Structure +-------------- +The overall structure of the MSA FITS file is as follows: + ++-----+---------------+----------+-----------+--------------------+ +| HDU | EXTNAME | HDU Type | Data Type | Dimensions | ++=====+===============+==========+===========+====================+ +| 0 | N/A | primary | N/A | N/A | ++-----+---------------+----------+-----------+--------------------+ +| 1 | SHUTTER_IMAGE | IMAGE | float32 | 342 x 730 | ++-----+---------------+----------+-----------+--------------------+ +| 2 | SHUTTER_INFO | BINTABLE | N/A | variable x 12 cols | ++-----+---------------+----------+-----------+--------------------+ +| 3 | SOURCE_INFO | BINTABLE | N/A | variable x 8 cols | ++-----+---------------+----------+-----------+--------------------+ + + - PRI: The primary data array is empty. The primary header contains a small amount of + meta data related to the observation and MSA configuration. + - SHUTTER_IMAGE: 2-D data array containing a map of MSA shutters that've been commanded + open for this MSA configuration. + - SHUTTER_INFO: 2-D FITS binary table containing meta data for each shutter + involved in this MSA configuration. One table row per shutter. + - SOURCE_INFO: 2-D FITS binary table containing meta data from the source catalog + used to create the MSA configuration. One table row per source. + +The structure of the ``SHUTTER_INFO`` table extension is as follows: + ++-------------------------------+-----------+----------------------+ +| Column Name | Data Type | Contents | ++===============================+===========+======================+ +| SLITLET_ID | int16 | Slitlet ID number | ++-------------------------------+-----------+----------------------+ +| MSA_METADATA_ID | int16 | MSA meta data ID | ++-------------------------------+-----------+----------------------+ +| SHUTTER_QUADRANT | int16 | MSA Quadrant number | ++-------------------------------+-----------+----------------------+ +| SHUTTER_ROW | int16 | Shutter row index | ++-------------------------------+-----------+----------------------+ +| SHUTTER_COLUMN | int16 | Shutter column index | ++-------------------------------+-----------+----------------------+ +| SOURCE_ID | int32 | Source ID number | ++-------------------------------+-----------+----------------------+ +| BACKGROUND | string | Background flag | ++-------------------------------+-----------+----------------------+ +| SHUTTER_STATE | string | Shutter state | ++-------------------------------+-----------+----------------------+ +| ESTIMATED_SOURCE_IN_SHUTTER_X | float32 | Source x position | ++-------------------------------+-----------+----------------------+ +| ESTIMATED_SOURCE_IN_SHUTTER_Y | float32 | Source y position | ++-------------------------------+-----------+----------------------+ +| DITHER_POINT_INDEX | int16 | Dither/nod index | ++-------------------------------+-----------+----------------------+ +| PRIMARY_SOURCE | string | Primary source flag | ++-------------------------------+-----------+----------------------+ + +The structure of the ``SOURCE_INFO`` table extension is as follows: + ++-------------+-----------+----------------------+ +| Column Name | Data Type | Contents | ++=============+===========+======================+ +| PROGRAM | int32 | Program ID number | ++-------------+-----------+----------------------+ +| SOURCE_ID | int32 | Source ID number | ++-------------+-----------+----------------------+ +| SOURCE_NAME | string | Source name | ++-------------+-----------+----------------------+ +| ALIAS | string | Source alias | ++-------------+-----------+----------------------+ +| RA | float64 | Source RA | ++-------------+-----------+----------------------+ +| DEC | float64 | Source Dec | ++-------------+-----------+----------------------+ +| PREIMAGE_ID | string | Pre-image ID | ++-------------+-----------+----------------------+ +| STELLARITY | float64 | Source stellarity | ++-------------+-----------+----------------------+ + + +The SHUTTER_INFO Meta Data +-------------------------- +It is the :ref:`assign_wcs ` step in the +:ref:`calwebb_spec2 ` pipeline that opens and loads all +of the necessary information from the MSA meta data file. For each slitlet that +it identifies as being used in the current exposure, a ``Slit`` object is +created and populated with all the shutter and source information needed in +subsequent processing steps. The list of ``Slit`` objects is attached to the +data model for the exposure being processed and passed along to all subsequent +calibration steps. + +For a given exposure, only table rows +where the values of `MSAMETID` and `PATT_NUM` in the science exposure match +the values of `msa_metdata_id` and `dither_point_index`, respectively, are +loaded, so that it's only dealing with meta data that apply to the science +exposure being processed. + +To better understand the ways in which those meta data are used, it's useful to +reference a hypothetical example of data within a ``SHUTTER_INFO`` table. +The table below shows the first 9 rows of a ``SHUTTER_INFO`` table for a MOS exposure +that's part of a standard 3-point nod MOS observation, in which the sources +are moved to different shutters within each slitlet from one exposure to +another in the observation. Only the data for slitlet 2 are shown. In this example, +slitlet 2 is comprised of 3 shutters. Because a 3-point nod pattern has been used, +there are 3 different sets of meta data for each slitlet (one set for each dither/nod +position) and hence a total of 9 entries (3 shutters x 3 dithers). + ++------+------+------+-----+-----+--------+-----+-------+-------+-------+------+-----+ +| Slit | Meta | | | | Src | | | X | Y | Dith | Pri | +| | | | | | | | | | | | | +| ID | ID | Quad | Row | Col | ID | Bkg | State | pos | pos | Pt | Src | ++======+======+======+=====+=====+========+=====+=======+=======+=======+======+=====+ +| 2 | 1 | 2 | 10 | 154 | 0 | Y | OPEN | NaN | NaN | 1 | N | ++------+------+------+-----+-----+--------+-----+-------+-------+-------+------+-----+ +| 2 | 1 | 2 | 10 | 155 | 42 | N | OPEN | 0.399 | 0.702 | 1 | Y | ++------+------+------+-----+-----+--------+-----+-------+-------+-------+------+-----+ +| 2 | 1 | 2 | 10 | 156 | 0 | Y | OPEN | NaN | NaN | 1 | N | ++------+------+------+-----+-----+--------+-----+-------+-------+-------+------+-----+ +| 2 | 1 | 2 | 10 | 154 | 42 | N | OPEN | 0.410 | 0.710 | 2 | Y | ++------+------+------+-----+-----+--------+-----+-------+-------+-------+------+-----+ +| 2 | 1 | 2 | 10 | 155 | 0 | Y | OPEN | NaN | NaN | 2 | N | ++------+------+------+-----+-----+--------+-----+-------+-------+-------+------+-----+ +| 2 | 1 | 2 | 10 | 156 | 0 | Y | OPEN | NaN | NaN | 2 | N | ++------+------+------+-----+-----+--------+-----+-------+-------+-------+------+-----+ +| 2 | 1 | 2 | 10 | 154 | 0 | Y | OPEN | NaN | NaN | 3 | N | ++------+------+------+-----+-----+--------+-----+-------+-------+-------+------+-----+ +| 2 | 1 | 2 | 10 | 155 | 0 | Y | OPEN | NaN | NaN | 3 | N | ++------+------+------+-----+-----+--------+-----+-------+-------+-------+------+-----+ +| 2 | 1 | 2 | 10 | 156 | 42 | N | OPEN | 0.389 | 0.718 | 3 | Y | ++------+------+------+-----+-----+--------+-----+-------+-------+-------+------+-----+ + +The values in the `slitlet_id` column show that we're only looking at table +rows for slitlet 2, all of which come from MSA configuration (`msa_metadata_id`) 1. +The shutters that make up slitlet 2 are all contained in MSA quadrant 2 and are located in +row 10 of that quadrant. The 3 shutters making up slitlet 2 span MSA columns 154, 155, and 156. +The remaining table entries in each row vary depending on whether a source is located in +a given shutter for each dither/nod position. You can see that the source is located in +shutter column 155 in dither position 1, 154 in dither position 2, and 156 in dither position 3. +The `source_id` column shows that source 42 is in those shutter positions for those dithers. +The `background` column, meanwhile, indicates whether a given shutter only has background +signal in it for each dither position. + +When there is a source in a shutter, the `estimated_source_in_shutter_x` and +`estimated_source_in_shutter_y` columns are populated with numerical values that +estimate the source location in the shutter. These are fractional position values that +run from 0,0 at the lower-left corner of a shutter to 1,1 at the upper-right corner +(hence 0.5,0.5 indicates the shutter center). These source positions are used in +calibration steps such as :ref:`wavecorr ` and :ref:`pathloss ` +to compute the appropriate corrections. + +The flags in the `primary_source` column are used to identify which shutter +within each slitlet should be considered the "primary" shutter. This is especially +important for slitlets that contain extended sources and hence the `source_id` and +`background` entries may indicate that the source is present in multiple shutters. + +When a slitlet is found that has no shutters with a primary source (i.e. no shutters +having `primary_source` = "Y"), it is classified as a background slitlet and assigned +a source ID value that's greater than the maximum source ID assigned to other slitlets +(because such slitlets all have a source ID of zero in the MSA meta data coming from +the ground system). +These background slitlets can then be used in :ref:`master background ` +subtraction. + + +The SOURCE_INFO Meta Data +------------------------- +The table below shows an example of a few rows of hypothetical source meta data, +consisting of the program ID, source ID, source name, source alias, source RA and Dec, +pre-image ID, and source stellarity. + ++------+------+-----------+-------+------------+-------------+--------------+------------+ +| | Src | Src | | | | Pre | | +| | | | | | | | | +| PID | ID | Name | Alias | RA | Dec | Image | Stellarity | ++======+======+===========+=======+============+=============+==============+============+ +| 1180 | 42 | 1180_0042 | Bob | 53.1456291 | -27.7674976 | 95065001_001 | 1.00 | ++------+------+-----------+-------+------------+-------------+--------------+------------+ +| 1180 | 1001 | 1180_1001 | Sue | 53.1435047 | -27.7689669 | 95065001_001 | 0.00 | ++------+------+-----------+-------+------------+-------------+--------------+------------+ +| 1180 | 3333 | 1180_3333 | Erin | 53.1485349 | -27.7696165 | 95065001_001 | 0.23 | ++------+------+-----------+-------+------------+-------------+--------------+------------+ +| 1180 | 9876 | 1180_9876 | Dave | 53.1461433 | -27.7617165 | 95065001_001 | 0.88 | ++------+------+-----------+-------+------------+-------------+--------------+------------+ + +For each slitlet identified as having a source assigned to it in the shutter meta data, +the source name, alias, RA, Dec, and stellarity are retreived from the `SOURCE_INFO` +table and stored with the ``Slit`` object created in the calibration software. +The stellarity values in particular will be used in the :ref:`source type ` +step to determine whether the source should be treated as point-like or extended. From f7b6e8ec6b80cf83152ac5617978dd9fca093ca8 Mon Sep 17 00:00:00 2001 From: Howard Bushouse Date: Mon, 1 Apr 2024 08:31:09 -0400 Subject: [PATCH 2/4] add change log entry --- CHANGES.rst | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/CHANGES.rst b/CHANGES.rst index e7cc7a8cc5..cbb5e3198f 100644 --- a/CHANGES.rst +++ b/CHANGES.rst @@ -1,7 +1,12 @@ 1.14.1 (unreleased) =================== -- +documentation +------------- + +- Added docs for the NIRSpec MSA metadata file to the data products area of RTD. + [#8399] + 1.14.0 (2024-03-25) =================== From 5e05582151dfc62d159baad26b473a13d08abc84 Mon Sep 17 00:00:00 2001 From: Howard Bushouse Date: Tue, 2 Apr 2024 07:54:13 -0400 Subject: [PATCH 3/4] updates from review --- docs/jwst/assign_wcs/main.rst | 16 ++- docs/jwst/data_products/msa_metadata.rst | 130 ++++++++++++++--------- 2 files changed, 89 insertions(+), 57 deletions(-) diff --git a/docs/jwst/assign_wcs/main.rst b/docs/jwst/assign_wcs/main.rst index 59a526dd2c..f2265c9a51 100644 --- a/docs/jwst/assign_wcs/main.rst +++ b/docs/jwst/assign_wcs/main.rst @@ -25,7 +25,8 @@ For each observing mode, determined by the value of ``EXP_TYPE`` in the science ``assign_wcs`` retrieves reference files from CRDS and creates a pipeline of transforms from input frame ``detector`` to a frame ``v2v3``. This part of the WCS pipeline may include intermediate coordinate frames. The basic WCS keywords are used to create -the transform from frame ``v2v3`` to frame ``world``. +the transform from frame ``v2v3`` to frame ``world``. All of this information is used to +create and populate the WCS object for the exposure. For image display with software like DS9 that relies on specific WCS information, a SIP-based approximation to the WCS is fit. The results are FITS keywords stored in @@ -33,20 +34,17 @@ approximation to the WCS is fit. The results are FITS keywords stored in for display purposes. This step, which occurs for imaging modes, is performed by default, but can be switched off, and parameters controlling the SIP fit can also be adjusted. -For each exposure, ``assign_wcs`` uses reference files and WCS header keywords -to create the WCS object. What reference files are retrieved -from CRDS is determined based on `EXP_TYPE` and other keywords in the science file header. - -The ``assign_wcs`` step can accept the single slope image that is the result of averaging -over all integrations or a 3D cube of integrations in the case of TSO exposures. +The ``assign_wcs`` step can accept either a ``rate`` product, which is the result of averaging +over all integrations in an exposure, or a ``rateints`` product, which is a 3D cube of +per-integration images. The ``assign_wcs`` step is based on `gwcs `__ and uses `asdf `__. .. Note:: In addition to CRDS reference files, applying ``assign_wcs`` to NIRSpec MOS - exposures depends critically on an MSA meta data file to provide information + exposures depends critically on an MSA metadata file to provide information for MOS slitlets in use and their constituent shutters. See :ref:`msa_metadata ` - for detailed information about the MSA meta data files and their contents. + for detailed information about the MSA metadata files and their contents. Basic WCS keywords and the transform from ``v2v3`` to ``world`` --------------------------------------------------------------- diff --git a/docs/jwst/data_products/msa_metadata.rst b/docs/jwst/data_products/msa_metadata.rst index 63da4864f9..09275f87b2 100644 --- a/docs/jwst/data_products/msa_metadata.rst +++ b/docs/jwst/data_products/msa_metadata.rst @@ -1,18 +1,18 @@ .. _msa_metadata: -MSA Meta Data File: ``msa`` -^^^^^^^^^^^^^^^^^^^^^^^^^^^ -While not containing any actual science data, the NIRSpec MSA meta data file is nonetheless +MSA Metadata File: ``msa`` +^^^^^^^^^^^^^^^^^^^^^^^^^^ +While not containing any actual science data, the NIRSpec MSA metadata file is nonetheless a crucial component of calibration processing for NIRSpec MOS exposures. It contains all the slitlet, shutter, and source configuration information that's needed by the :ref:`calwebb_spec2 ` pipeline to process a MOS exposure. -These meta data are originally generated by the +These metadata are originally generated by the `MSA Planning Tool `_ (MPT) in APT and passed through the ground system to populate the -FITS metafile. Details of the meta data and how it is used in calibration processing +FITS metafile. Details of the metadata and how it is used in calibration processing are given below. -File names for MSA meta data files are of the form: +File names for MSA metadata files are of the form: jw__msa.fits @@ -26,9 +26,9 @@ where such as `jw01180025001_01_msa.fits`. Each MOS science exposure contains a primary header keyword `MSAMETFL` that gives -the name of the MSA meta data file to be used when processing the exposure. +the name of the MSA metadata file to be used when processing the exposure. The values of the primary header keywords `MSAMETID` and `PATT_NUM` are also used -when retrieving relevant rows of data from the tables in the MSA meta data file. +when retrieving relevant rows of data from the tables in the MSA metadata file. If any of these keywords is missing from the input science exposure, processing will not be able to proceed and an error will be raised. @@ -49,14 +49,17 @@ The overall structure of the MSA FITS file is as follows: +-----+---------------+----------+-----------+--------------------+ - PRI: The primary data array is empty. The primary header contains a small amount of - meta data related to the observation and MSA configuration. + metadata related to the observation and MSA configuration. - SHUTTER_IMAGE: 2-D data array containing a map of MSA shutters that've been commanded open for this MSA configuration. - - SHUTTER_INFO: 2-D FITS binary table containing meta data for each shutter + - SHUTTER_INFO: 2-D FITS binary table containing metadata for each shutter involved in this MSA configuration. One table row per shutter. - - SOURCE_INFO: 2-D FITS binary table containing meta data from the source catalog + - SOURCE_INFO: 2-D FITS binary table containing metadata from the source catalog used to create the MSA configuration. One table row per source. + +The SHUTTER_INFO Metadata +------------------------- The structure of the ``SHUTTER_INFO`` table extension is as follows: +-------------------------------+-----------+----------------------+ @@ -64,7 +67,7 @@ The structure of the ``SHUTTER_INFO`` table extension is as follows: +===============================+===========+======================+ | SLITLET_ID | int16 | Slitlet ID number | +-------------------------------+-----------+----------------------+ -| MSA_METADATA_ID | int16 | MSA meta data ID | +| MSA_METADATA_ID | int16 | MSA metadata ID | +-------------------------------+-----------+----------------------+ | SHUTTER_QUADRANT | int16 | MSA Quadrant number | +-------------------------------+-----------+----------------------+ @@ -87,54 +90,52 @@ The structure of the ``SHUTTER_INFO`` table extension is as follows: | PRIMARY_SOURCE | string | Primary source flag | +-------------------------------+-----------+----------------------+ -The structure of the ``SOURCE_INFO`` table extension is as follows: - -+-------------+-----------+----------------------+ -| Column Name | Data Type | Contents | -+=============+===========+======================+ -| PROGRAM | int32 | Program ID number | -+-------------+-----------+----------------------+ -| SOURCE_ID | int32 | Source ID number | -+-------------+-----------+----------------------+ -| SOURCE_NAME | string | Source name | -+-------------+-----------+----------------------+ -| ALIAS | string | Source alias | -+-------------+-----------+----------------------+ -| RA | float64 | Source RA | -+-------------+-----------+----------------------+ -| DEC | float64 | Source Dec | -+-------------+-----------+----------------------+ -| PREIMAGE_ID | string | Pre-image ID | -+-------------+-----------+----------------------+ -| STELLARITY | float64 | Source stellarity | -+-------------+-----------+----------------------+ - +- SLITLET_ID: integer ID of each slitlet consisting of one or more + open shutters. +- MSA_METADATA_ID: integer ID corresponding to a particular MSA + configuration or MPT plan (typically the same for all rows in a + given metafile). +- SHUTTER_QUADRANT: integer ID of the MSA quadrant in which the open + shutter is located. +- SHUTTER_ROW: integer row index of the shutter within the MSA quadrant. +- SHUTTER_COLUMN: integrer column index of the shutter within the MSA + quadrant. +- SOURCE_ID: unique integer ID for each source in each slitlet, used + for matching to an entry in the ``SOURCE_INFO`` table. +- BACKGROUND: boolean indicating whether the shutter is open to background + only or contains a known source. +- SHUTTER_STATE: OPEN or CLOSED. Generally will always be OPEN, unless + the shutter is part of a long slit that is not contiguous. +- ESTIMATED_SOURCE_IN_SHUTTER_X/Y: the position of the source within the + shutter in relative units, where 0,0 is bottom-left, 0.5,0.5 is center, + and 1,1 is upper-right, as planned in the MPT. +- DITHER_POINT_INDEX: integer index of the nod sequence; matches to + header keyword `PATT_NUM`. +- PRIMARY_SOURCE: boolean indicating whether the shutter contains the + primary science source. -The SHUTTER_INFO Meta Data --------------------------- It is the :ref:`assign_wcs ` step in the :ref:`calwebb_spec2 ` pipeline that opens and loads all -of the necessary information from the MSA meta data file. For each slitlet that +of the necessary information from the MSA metadata file. For each slitlet that it identifies as being used in the current exposure, a ``Slit`` object is created and populated with all the shutter and source information needed in subsequent processing steps. The list of ``Slit`` objects is attached to the data model for the exposure being processed and passed along to all subsequent calibration steps. -For a given exposure, only table rows +For a given exposure, only the rows of shutter table where the values of `MSAMETID` and `PATT_NUM` in the science exposure match the values of `msa_metdata_id` and `dither_point_index`, respectively, are -loaded, so that it's only dealing with meta data that apply to the science -exposure being processed. +loaded. -To better understand the ways in which those meta data are used, it's useful to +To better understand the ways in which these metadata are used, it's useful to reference a hypothetical example of data within a ``SHUTTER_INFO`` table. The table below shows the first 9 rows of a ``SHUTTER_INFO`` table for a MOS exposure that's part of a standard 3-point nod MOS observation, in which the sources are moved to different shutters within each slitlet from one exposure to another in the observation. Only the data for slitlet 2 are shown. In this example, slitlet 2 is comprised of 3 shutters. Because a 3-point nod pattern has been used, -there are 3 different sets of meta data for each slitlet (one set for each dither/nod +there are 3 different sets of metadata for each slitlet (one set for each dither/nod position) and hence a total of 9 entries (3 shutters x 3 dithers). +------+------+------+-----+-----+--------+-----+-------+-------+-------+------+-----+ @@ -188,15 +189,48 @@ important for slitlets that contain extended sources and hence the `source_id` a When a slitlet is found that has no shutters with a primary source (i.e. no shutters having `primary_source` = "Y"), it is classified as a background slitlet and assigned a source ID value that's greater than the maximum source ID assigned to other slitlets -(because such slitlets all have a source ID of zero in the MSA meta data coming from +(because such slitlets all have a source ID of zero in the MSA metadata coming from the ground system). These background slitlets can then be used in :ref:`master background ` subtraction. -The SOURCE_INFO Meta Data -------------------------- -The table below shows an example of a few rows of hypothetical source meta data, +The SOURCE_INFO Metadata +------------------------ +The structure of the ``SOURCE_INFO`` table extension is as follows: + ++-------------+-----------+----------------------+ +| Column Name | Data Type | Contents | ++=============+===========+======================+ +| PROGRAM | int32 | Program ID number | ++-------------+-----------+----------------------+ +| SOURCE_ID | int32 | Source ID number | ++-------------+-----------+----------------------+ +| SOURCE_NAME | string | Source name | ++-------------+-----------+----------------------+ +| ALIAS | string | Source alias | ++-------------+-----------+----------------------+ +| RA | float64 | Source RA | ++-------------+-----------+----------------------+ +| DEC | float64 | Source Dec | ++-------------+-----------+----------------------+ +| PREIMAGE_ID | string | Pre-image ID | ++-------------+-----------+----------------------+ +| STELLARITY | float64 | Source stellarity | ++-------------+-----------+----------------------+ + +- PROGRAM: 5-digit JWST program number. +- SOURCE_ID: unique integer identifier for each source in MPT catalog. +- SOURCE_NAME: source name, typically constructed as the concatenation + of program number and source ID number. +- ALIAS: alternate source identifier from the MPT catalog. +- RA/Dec: catalog source coordinates, in decimal degrees. +- PREIMAGE_ID: name of a NIRCam mosaic used to determine the source + catalog, if one was used. +- STELLARITY: DAOphot-style stellarity value for the source, where 0 + is fully extended and 1 is an unresolved point source. + +The table below shows an example of a few rows of hypothetical source metadata, consisting of the program ID, source ID, source name, source alias, source RA and Dec, pre-image ID, and source stellarity. @@ -214,8 +248,8 @@ pre-image ID, and source stellarity. | 1180 | 9876 | 1180_9876 | Dave | 53.1461433 | -27.7617165 | 95065001_001 | 0.88 | +------+------+-----------+-------+------------+-------------+--------------+------------+ -For each slitlet identified as having a source assigned to it in the shutter meta data, +For each slitlet identified as having a source assigned to it in the shutter metadata, the source name, alias, RA, Dec, and stellarity are retreived from the `SOURCE_INFO` table and stored with the ``Slit`` object created in the calibration software. -The stellarity values in particular will be used in the :ref:`source type ` +The stellarity values are used in the :ref:`source type ` step to determine whether the source should be treated as point-like or extended. From 90906dc6f7c9ab6104865c3df725dce3e294f883 Mon Sep 17 00:00:00 2001 From: Howard Bushouse Date: Tue, 2 Apr 2024 12:08:58 -0400 Subject: [PATCH 4/4] Update docs/jwst/data_products/msa_metadata.rst Co-authored-by: Ned Molter --- docs/jwst/data_products/msa_metadata.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/jwst/data_products/msa_metadata.rst b/docs/jwst/data_products/msa_metadata.rst index 09275f87b2..edaa703b2f 100644 --- a/docs/jwst/data_products/msa_metadata.rst +++ b/docs/jwst/data_products/msa_metadata.rst @@ -123,7 +123,7 @@ subsequent processing steps. The list of ``Slit`` objects is attached to the data model for the exposure being processed and passed along to all subsequent calibration steps. -For a given exposure, only the rows of shutter table +For a given exposure, only the rows of the shutter table where the values of `MSAMETID` and `PATT_NUM` in the science exposure match the values of `msa_metdata_id` and `dither_point_index`, respectively, are loaded.