Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

JP-1737: Propagate error through Extract1D step #6014

Merged
merged 23 commits into from
May 20, 2021

Conversation

tapastro
Copy link
Contributor

@tapastro tapastro commented May 6, 2021

Closes #5373

Resolves JP-1737

Description

This PR propagates error/variance arrays through the various branches of Extract1D. Some existing functions were modified out of convenience when handling many new arrays. A first pass on documentation has been completed.

Working on regression test results now.

Checklist

  • Tests
  • Documentation
  • Change log
  • Milestone
  • Label(s)

@codecov
Copy link

codecov bot commented May 7, 2021

Codecov Report

Merging #6014 (39b50c7) into master (f9ad731) will decrease coverage by 1.68%.
The diff coverage is 53.62%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #6014      +/-   ##
==========================================
- Coverage   76.99%   75.31%   -1.69%     
==========================================
  Files         401      401              
  Lines       34063    35232    +1169     
==========================================
+ Hits        26228    26536     +308     
- Misses       7835     8696     +861     
Flag Coverage Δ *Carryforward flag
nightly 77.00% <70.77%> (ø) Carriedforward from f9ad731
unit 55.16% <28.22%> (-0.25%) ⬇️

*This pull request uses carry forward flags. Click here to find out more.

Impacted Files Coverage Δ
jwst/resample/gwcs_drizzle.py 79.54% <ø> (ø)
jwst/extract_1d/extract.py 49.53% <29.70%> (-16.95%) ⬇️
jwst/extract_1d/ifu.py 62.22% <69.46%> (-14.47%) ⬇️
jwst/extract_1d/extract1d.py 65.13% <79.31%> (-10.31%) ⬇️
jwst/combine_1d/combine1d.py 89.78% <100.00%> (ø)
jwst/extract_1d/apply_apcorr.py 84.90% <100.00%> (-10.86%) ⬇️
jwst/master_background/create_master_bkg.py 93.75% <100.00%> (+0.41%) ⬆️
jwst/master_background/master_background_step.py 75.36% <100.00%> (ø)
... and 3 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f9ad731...39b50c7. Read the comment docs.

Copy link
Collaborator

@hbushouse hbushouse left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wow, that's a lot of great work. Even a change log entry without being requested! Just a few minor questions for clarification here and there.

) -> Tuple[
float, float, np.ndarray, np.ndarray, np.ndarray, np.ndarray, np.ndarray
]:
def extract(self, data,
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

data is missing an np.ndarray type declaration

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like this still needs fixing (add np.ndarray declaration for data)

@@ -81,9 +93,27 @@ def extract1d(image, lambdas, disp_range,
countrate : ndarray, 1-D, float64
The extracted spectrum in units of counts / s.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Really should finally update these comments and maybe even the name of the countrate object. The references to countrate and counts/s are based on the old days when the extraction was done on data that had not been flux calibrated (flux calibration was applied after extraction). Now the normal case is to be working on flux calibrated data and hence the units are MJy/sr or Jy. Only if the user has skipped the photom step in earlier processing will the data ever be in count rate units.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was curious about that - it didn't seem to jive with the rest of the documentation around these files. I'll update it.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Still need to update these descriptions for the param list. I know it doesn't affect operation of the code, but let's not leave these lingering here for the future.

@tapastro
Copy link
Contributor Author

Lingering regression test issue has been resolved - the combine1d difference was caused by a missing meta.observation.bkgdtarg = 'EXTENDED' - user error in updating the new datamodel with only='PRIMARY'...

Copy link
Collaborator

@hbushouse hbushouse left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like just a couple minor requests for updates from the initial review still need to be implemented. Then it'll be good to go.

) -> Tuple[
float, float, np.ndarray, np.ndarray, np.ndarray, np.ndarray, np.ndarray
]:
def extract(self, data,
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like this still needs fixing (add np.ndarray declaration for data)

@@ -81,9 +93,27 @@ def extract1d(image, lambdas, disp_range,
countrate : ndarray, 1-D, float64
The extracted spectrum in units of counts / s.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Still need to update these descriptions for the param list. I know it doesn't affect operation of the code, but let's not leave these lingering here for the future.

@tapastro
Copy link
Contributor Author

Looks like just a couple minor requests for updates from the initial review still need to be implemented. Then it'll be good to go.

Cleaned these up - I did confirm that AttributeError is triggered using the test_miri_mrs_extract1d regtest.

tapastro added 2 commits May 19, 2021 10:35
…stro/jwst into jp-1737-propagate-error-x1d

hopefully this fixes the changes merge
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Spec2Pipeline LRS processing: Extract1d does not produce error column (all 0) in *x1d.fits file
3 participants