Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

extract1d for IFU data version 2- JP 1736 #5506

Merged
Show file tree
Hide file tree
Changes from 58 commits
Commits
Show all changes
62 commits
Select commit Hold shift + click to select a range
09f3f79
first attempt at setting up extract1d miri ref file for IFU data
jemorrison Oct 5, 2020
3f125dc
remove shape from mirmrs_extract1d datamodel
jemorrison Oct 5, 2020
5ed9130
fixed mirmrs_extract 1d data model
jemorrison Oct 5, 2020
3db8ead
using new apcorr ref file
jemorrison Oct 6, 2020
44057bc
fix radius scaling
jemorrison Oct 9, 2020
a206a6c
update to clean up comments
jemorrison Oct 9, 2020
1440140
fixed convert size units
jemorrison Oct 12, 2020
b7064d5
using new apcorr ref file
jemorrison Oct 12, 2020
0cc84aa
update description of mir mrs apcorr ref file
jemorrison Oct 14, 2020
1bc715a
Updates based on review
jemorrison Oct 14, 2020
5eecb27
flake8 fixes and comment clean up
jemorrison Oct 15, 2020
d9019f5
more flake8 fixes
jemorrison Oct 16, 2020
290c14e
new mirmrs_extract1d for asdf ref file
jemorrison Oct 23, 2020
49a2540
cleaned up ifuextract1d datamodel
jemorrison Oct 29, 2020
aaf1e28
updates
jemorrison Nov 9, 2020
29a5504
updates for using apcor ref asdf file
jemorrison Nov 6, 2020
50d4948
flake8 fixes
jemorrison Nov 6, 2020
7336a0e
fix NrsIfuApcorrModel
jemorrison Nov 6, 2020
7daf396
updates to tests
jemorrison Nov 9, 2020
bc3b291
updates to tests
jemorrison Nov 9, 2020
473a65b
fix conflicts
jemorrison Nov 9, 2020
107eb93
flake 8 datamodel extract1d_spec.py
jemorrison Nov 9, 2020
8730676
more flake fixes
jemorrison Nov 9, 2020
054a850
fix tests
jemorrison Nov 10, 2020
fa30cb4
more flake8 fixes
jemorrison Nov 10, 2020
f611712
update to test_apply_apcorr_ifu.py
jemorrison Nov 10, 2020
c4e82b2
typo fix
drlaw1558 Nov 12, 2020
dffe53d
Merge pull request #4 from drlaw1558/drlnov12
jemorrison Nov 13, 2020
066ca3f
first attempt at setting up extract1d miri ref file for IFU data
jemorrison Oct 5, 2020
182723f
remove shape from mirmrs_extract1d datamodel
jemorrison Oct 5, 2020
0cc7632
fixed mirmrs_extract 1d data model
jemorrison Oct 5, 2020
1230785
using new apcorr ref file
jemorrison Oct 6, 2020
968425a
fix radius scaling
jemorrison Oct 9, 2020
fbb73bd
update to clean up comments
jemorrison Oct 9, 2020
f6222b1
fixed convert size units
jemorrison Oct 12, 2020
0a3caff
using new apcorr ref file
jemorrison Oct 12, 2020
ca31cbc
update description of mir mrs apcorr ref file
jemorrison Oct 14, 2020
27fae8d
Updates based on review
jemorrison Oct 14, 2020
02f3771
flake8 fixes and comment clean up
jemorrison Oct 15, 2020
898ee06
more flake8 fixes
jemorrison Oct 16, 2020
33e8789
new mirmrs_extract1d for asdf ref file
jemorrison Oct 23, 2020
3d072db
cleaned up ifuextract1d datamodel
jemorrison Oct 29, 2020
e519b9a
updates
jemorrison Nov 9, 2020
ed406e0
updates for using apcor ref asdf file
jemorrison Nov 6, 2020
058757a
flake8 fixes
jemorrison Nov 6, 2020
8440768
fix NrsIfuApcorrModel
jemorrison Nov 6, 2020
e35d248
updates to tests
jemorrison Nov 9, 2020
8448d2f
updates to tests
jemorrison Nov 9, 2020
41af7e5
fix conflicts
jemorrison Nov 9, 2020
da95409
flake 8 datamodel extract1d_spec.py
jemorrison Nov 9, 2020
3cea987
more flake fixes
jemorrison Nov 9, 2020
3d48ba3
fix tests
jemorrison Nov 10, 2020
7edf50c
more flake8 fixes
jemorrison Nov 10, 2020
09fe97a
update to test_apply_apcorr_ifu.py
jemorrison Nov 10, 2020
e62a5d6
typ0 fix from D Law
jemorrison Nov 13, 2020
30c62db
Merge branch 'extract1d_datamodel_JP_1736' of https://github.com/jemo…
jemorrison Dec 2, 2020
b04466b
updated change log
jemorrison Dec 3, 2020
06e800f
cleaned up comment
jemorrison Dec 3, 2020
b3bcc1c
update from review. Changed how apcor ref file is done for NIRSpec. O…
jemorrison Dec 4, 2020
196b89a
fix typo
jemorrison Dec 4, 2020
5712155
Merge branch 'master' into jemorrison-extract1d_datamodel_JP_1736_v2
jemorrison Jan 7, 2021
c55ca7c
Update extract_1d_step.py
hbushouse Jan 7, 2021
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions CHANGES.rst
Original file line number Diff line number Diff line change
Expand Up @@ -56,6 +56,11 @@ datamodels

extract_1d
----------
- For IFU data (NIRSpec and MIRI) the extraction radius is now a varying size
based on wavelength. The apcorr correction if a function of wavelength and
radius size. Fixes a bug in units conversion for applying the apcorr correction.
The units are now correctly converted from arcseconds to pixels. Added an
new method to apply the apcorr correction for IFU data.

- Fixed bug involving the determination of source RA/Dec for resampled Slit
data. [#5353]
Expand Down
6 changes: 4 additions & 2 deletions jwst/datamodels/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@
from .apcorr import NrcImgApcorrModel, NisImgApcorrModel
from .apcorr import MirLrsApcorrModel, MirMrsApcorrModel
from .apcorr import NrcWfssApcorrModel, NisWfssApcorrModel
from .apcorr import NrsMosApcorrModel, NrsFsApcorrModel
from .apcorr import NrsMosApcorrModel, NrsIfuApcorrModel, NrsFsApcorrModel
from .asn import AsnModel
from .barshadow import BarshadowModel
from .combinedspec import CombinedSpecModel
Expand All @@ -21,6 +21,7 @@
from .drizpars import DrizParsModel
from .drizproduct import DrizProductModel
from .extract1dimage import Extract1dImageModel
from .extract1d_spec import IFUExtract1dModel
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I suppose it would throw a big wrench into things at this late stage to rename this new model class to Extract1dIFUModel just for consistency in naming with the existing Extract1dImageModel? If so, it's not a big deal.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That is fine I can change the names of models to be more consistent

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

changed

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, I think that means I'll have to change some things about my generation code as well. Are the 'datamodl' keywords still meant to be MirMrsExtract1dModel and MirMrsApcorrModel ? If so I should only need to change the names of the schema files.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes @drlaw1558 that does imply that the actual class name of the data model used for the extract1d reference file will be changing from IFUExtract1dModel to Extract1dIFUModel. That's only for the extract1d reference file model. The apcorr ref files models remain unchanged. I don't see MirMrsExtract1dModel referenced anywhere, or am I just missing it?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hm, it looks like this was the name Jane used in creating a test asdf file back in October; maybe that's now been superseded by MirMrsExtract1dModel -> IFUExtract1dModel -> Extract1dIFUModel. In this case I'll need to change both my code and the reference file to make sure the data model keyword is up to date.

It doesn't look like the update has been made in the schema for this branch yet though? @jemorrison your comment above said that it had been changed, but the relevant branch still seems to have the old names?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I just pushed over the new code. It is Extract1DIFUModel now.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, I see it. If we're now using schemas/extract1difu.schema.yaml should we remove the old schemas/mirmrs_extract1d.schema.yaml ?

from .flat import FlatModel
from .fringe import FringeModel
from .gain import GainModel
Expand Down Expand Up @@ -90,7 +91,7 @@
'AmiLgModel',
'FgsImgApcorrModel', 'MirImgApcorrModel', 'NrcImgApcorrModel', 'NisImgApcorrModel',
'MirLrsApcorrModel', 'MirMrsApcorrModel', 'NrcWfssApcorrModel', 'NisWfssApcorrModel',
'NrsMosApcorrModel', 'NrsFsApcorrModel',
'NrsMosApcorrModel', 'NrsFsApcorrModel','NrsIfuApcorrModel',
'AsnModel',
'BarshadowModel', 'CameraModel', 'CollimatorModel',
'CombinedSpecModel', 'ContrastModel', 'CubeModel',
Expand All @@ -99,6 +100,7 @@
'DrizParsModel',
'DrizProductModel',
'Extract1dImageModel',
'IFUExtract1dModel',
'FilteroffsetModel',
'FlatModel', 'NirspecFlatModel', 'NirspecQuadFlatModel',
'FOREModel', 'FPAModel',
Expand Down
38 changes: 28 additions & 10 deletions jwst/datamodels/apcorr.py
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
'NrcImgApcorrModel', 'NisImgApcorrModel',
'MirLrsApcorrModel', 'MirMrsApcorrModel',
'NrcWfssApcorrModel', 'NisWfssApcorrModel',
'NrsMosApcorrModel', 'NrsFsApcorrModel']
'NrsMosApcorrModel', 'NrsIfuApcorrModel','NrsFsApcorrModel']


class FgsImgApcorrModel(ReferenceFileModel):
Expand Down Expand Up @@ -90,14 +90,9 @@ class MirMrsApcorrModel(ReferenceFileModel):
factors associated with those modes.

- wavelength: float32 1D array
- nelem_wl: int16
- radius: float32 1D array
- apcorr: float32 1D array
- apcorr_err: float32 1D array
- inner_bkg: float32 1D array
- outer_bkg: float32 1D array
- axis_ratio: float32 1D array
- axis_pa: float32 1D array
- radius: float32 2D array
- apcorr: float32 2D array
- apcorr_err: float32 2D array

"""
schema_url = "http://stsci.edu/schemas/jwst_datamodel/mirmrs_apcorr.schema"
Expand Down Expand Up @@ -203,7 +198,7 @@ class NisWfssApcorrModel(ReferenceFileModel):

class NrsMosApcorrModel(ReferenceFileModel):
"""
A data model for NIRSpec MOS and IFU apcorr reference files.
A data model for NIRSpec MOS apcorr reference files.

Parameters
__________
Expand All @@ -227,6 +222,29 @@ class NrsMosApcorrModel(ReferenceFileModel):
schema_url = "http://stsci.edu/schemas/jwst_datamodel/nrsmos_apcorr.schema"


class NrsIfuApcorrModel(ReferenceFileModel):
"""
A data model for NIRSpec IFU apcorr reference files.

Parameters
__________
apcorr_table : numpy table
Aperture correction factors table
A table-like object containing row selection criteria made up
of instrument mode parameters and aperture correction
factors associated with those modes.

- filter: str[12]
- grating: str[15]
- wavelength: float32 1D array
- radius: float32 3D array
- apcorr: float32 3D array
- apcorr_err: float32 3D array

"""
schema_url = "http://stsci.edu/schemas/jwst_datamodel/nrsifu_apcorr.schema"


class NrsFsApcorrModel(ReferenceFileModel):
"""
A data model for NIRSpec Fixed-Slit apcorr reference files.
Expand Down
30 changes: 30 additions & 0 deletions jwst/datamodels/extract1d_spec.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
from .reference import ReferenceFileModel

__all__ = ['IFUExtract1dModel']


class IFUExtract1dModel(ReferenceFileModel):
"""
A data model for IFU MIRI and NIRSpec extract 1d reference files.

Parameters
__________
extract1d_params : numpy table
Basic extract 1D parameters
- region_type: ascii
- subtract_background: bool
- method: ascii
- subpixels: int16
extract1d_table : numpy table
extract1d parameters for varying wavelengths
A table-like object containing extract 1d parameters
based on wavelength
- wavelength: float32 1D array
- radius: float32 1D array
- inner_bkg: float32 1D array
- outer_bkg: float32 1D array
- axis_ratio: float32 1D array
- axis_pa: float32 1D array

"""
schema_url = "http://stsci.edu/schemas/jwst_datamodel/ifuextract1d.schema"
48 changes: 48 additions & 0 deletions jwst/datamodels/schemas/ifuextract1d.schema.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
%YAML 1.1
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Again in terms of file naming consistency, could this new schema file be renamed to extract1difu.schema.yaml, in keeping with the existing extract1dimage.schema.yaml file?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

changed

---
$schema: "http://stsci.edu/schemas/asdf/asdf-schema-1.0.0"
id: "http://stsci.edu/schemas/jwst_datamodel/ifuextract1d.schema"
title: Default IFU Data extract 1d data model
allOf:
- $ref: referencefile.schema
- $ref: keyword_exptype.schema
- $ref: keyword_pexptype.schema
- type: object
properties:
meta:
type: object
properties:
region_type:
type: string
subtract_background:
type: boolean
method:
type: string
subpixels:
type: integer
properties:
data:
type: object
properties:
wavelength:
datatype: float32
wavelength_units:
type: string
radius:
datatype: float32
radius_units:
type: string
inner_bkg:
datatype: float32
inner_bkg_units:
type: string
outer_bkg:
datatype: float32
outer_bkg_units:
type: string
axis_ratio:
datatype: float32
axis_pa:
datatype: float32
axis_pa_units:
type: string
53 changes: 19 additions & 34 deletions jwst/datamodels/schemas/mirmrs_apcorr.schema.yaml
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
%YAML 1.1
---
$schema: "http://stsci.edu/schemas/fits-schema/fits-schema"
$schema: "http://stsci.edu/schemas/asdf/asdf-schema-1.0.0"
id: "http://stsci.edu/schemas/jwst_datamodel/mirmrs_apcorr.schema"
title: MIRI MRS aperture correction data model
allOf:
Expand All @@ -10,36 +10,21 @@ allOf:
- type: object
properties:
apcorr_table:
title: Aperture correction factors table
fits_hdu: APCORR
datatype:
- name: wavelength
datatype: float32
ndim: 1
- name: nelem_wl
datatype: int16
- name: radius
datatype: float32
ndim: 1
- name: apcorr
datatype: float32
ndim: 1
- name: apcorr_err
datatype: float32
ndim: 1
- name: inner_bkg
datatype: float32
ndim: 1
- name: outer_bkg
datatype: float32
ndim: 1
- name: axis_ratio
datatype: float32
ndim: 1
- name: axis_pa
datatype: float32
ndim: 1
sizeunit:
title: Units for the SIZE or RADIUS column in the APCORR table
fits_hdu: APCORR
fits_keyword: SIZEUNIT
type: object
properties:
channel:
type: string
band:
type: string
wavelength:
datatype: float32
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why are some data type attributes named type while others are datatype? Does it make a difference? My inner hob goblin of foolish consistency suggests having them all the same.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It was trial and error to get this to work. This worked that is all I know

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So from trial and error I have found that when I create the asdf reference files, write them to disk and then read them in
using asdf.open and test them using asdf validate - all values that are strings that are called datatype in the schema fail in the validate. The error message says they are not arrays. So I am guessing that 'datatype' is for arrays and 'type' is for single values.

wavelength_units:
type: string
radius:
datatype: float32
radius_units:
type: string
apcorr:
datatype: float32
apcorr_err:
datatype: float32
41 changes: 41 additions & 0 deletions jwst/datamodels/schemas/mirmrs_extract1d.schema.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
%YAML 1.1
---
$schema: "http://stsci.edu/schemas/fits-schema/fits-schema"
id: "http://stsci.edu/schemas/jwst_datamodel/mirmrs_extract1d.schema"
title: Default MIRI MRS extract 1d data model
allOf:
- $ref: referencefile.schema
- type: object
properties:
extract1d_params:
title: default MIRI extract 1d parameters
fits_hdu: PARAMS
datatype:
- name: id
datatype: [ascii,10]
- name: region_type
datatype: [ascii,10]
- name: subtract_background
datatype: uint8
- name: method
datatype: [ascii,10]
- name: subpixels
datatype: int16
extract1d_table:
title: wavelength varying extraction parameters
fits_hdu: X1D
datatype:
- name: wavelength
datatype: float32
- name: nelem_wl
datatype: int16
- name: radius
datatype: float32
- name: inner_bkg
datatype: float32
- name: outer_bkg
datatype: float32
- name: axis_ratio
datatype: float32
- name: axis_pa
datatype: float32
Loading