Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

JP-1694: fix outlier detection #5601

Merged
merged 8 commits into from
Jan 14, 2021
Merged

JP-1694: fix outlier detection #5601

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
70157e4
testing
jemorrison Jan 8, 2021
90a76c5
add background to blot image to correctly set up noise image
jemorrison Jan 9, 2021
b37dfb5
Added DO_NOT_USE to NON_SCIENCE pixels after flat fielding
jemorrison Jan 13, 2021
01ed074
update change log
jemorrison Jan 13, 2021
4ac0ea2
update from review
jemorrison Jan 13, 2021
70b4b38
Merge branch 'master' into outlier_detection_JP_1694
jemorrison Jan 14, 2021
a22b26f
Merge branch 'master' into outlier_detection_JP_1694
jemorrison Jan 14, 2021
b13e8c7
Update CHANGES.rst
jemorrison Jan 14, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 10 additions & 0 deletions CHANGES.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,16 @@ datamodels

- Updated schemas for new keywords CROWDFLD, PRIDTYPE, PRIDTPTS, PATTNPTS, SMGRDPAT,
changed name of SUBPXPNS to SUBPXPTS, and new allowed values for PATTTYPE. [#5618]

flat_field
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor nit, but we like to keep the change entires in alphabetical order of step/pipeline name, so these should go below cube_build in the list.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok I think I finally have the change log correct. I just edited it on GitHub

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep, looks good to me now.

----------

- Added DO_NOT_USE to pixels flagged as NON_SCIENCE for non-nirspec data [#5601]

outlier_detection
-----------------

- Account for the background subtracted data in the blot image for determining the noise image used in flagging outliers [#5601]

set_telescope_pointing
----------------------
Expand Down
5 changes: 5 additions & 0 deletions jwst/flatfield/flat_field.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -215,6 +215,11 @@ def apply_flat_field(science, flat, inverse=False):
# Combine the science and flat DQ arrays
science.dq = np.bitwise_or(science.dq, flat_dq)

# Find all pixels in the flat that have a DQ value of NON_SCIENCE
# add the DO_NOT_USE flag to these pixels. We don't want to use these pixels
# in futher steps
flag_nonsci = np.bitwise_and(science.dq, dqflags.pixel['NON_SCIENCE']).astype(np.bool)
science.dq[flag_nonsci] = np.bitwise_or(science.dq[flag_nonsci], dqflags.pixel['DO_NOT_USE'])
Comment on lines +218 to +222
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Adding NON_SCIENCE doesn't seem to have any effect on the issue described in the ticket. I.e. in the dataset from the ticket, running calwebb_image3 starting with the _cal.fits files as they are resolves the issue with just the changes in outlier_detection, without having to rerun flat_field on the input data.

So what is the purpose of this section of code? Currently the NON_SCIENCE area in MIRI data gets set to zero in the resampled _i2d.fits output. I.e., I don't see a problem with what is happening currently.


#
# The following functions are for NIRSpec spectrographic data.
Expand Down
4 changes: 2 additions & 2 deletions jwst/outlier_detection/outlier_detection.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -406,7 +406,7 @@ def flag_cr(sci_image, blot_image, **pars):
exptime = sci_image.meta.exposure.exposure_time

sci_data = sci_image.data * exptime
blot_data = blot_image.data * exptime
blot_data = (blot_image.data + subtracted_background) * exptime
blot_deriv = abs_deriv(blot_data)

err_data = np.nan_to_num(sci_image.err)
Expand All @@ -421,7 +421,7 @@ def flag_cr(sci_image, blot_image, **pars):
#
# Model the noise and create a CR mask
diff_noise = np.abs(sci_data - blot_data)
ta = np.sqrt(np.abs(blot_data + subtracted_background) + err_data ** 2)
ta = np.sqrt(np.abs(blot_data) + err_data ** 2)
t2 = scl1 * blot_deriv + snr1 * ta
tmp1 = np.logical_not(np.greater(diff_noise, t2))

Expand Down