resample_spec output frame is spatially too small #5564
Comments
|
Comment by Beth Sargent on JIRA: In JP-1811, the comment by Sarah Kendrew also seems to mention this problem. |
|
Comment by Sarah Kendrew on JIRA: prioritizing this as mir_high. not critical for success of commissioning but is an important step for getting a usable science product from LRS slit observations, and not something that is easy to work around for a user. |
|
Comment by Howard Bushouse on JIRA: I've attached a screen shot of the s2d (resample_spec output) from running calwebb_spec2 on a single simulated LRS exposure, which looks OK. I believe the hard cutoff at the bottom end of the spectrum is due to the wavelength cutoff of the flux calibration. I'll try running calwebb_spec3 on a set of multiple exposures and see what happens. This was done using the latest B7.7 pre-release. |
|
Comment by Howard Bushouse on JIRA: I've finished running calwebb_spec3 on an ASN of 4 simulated exposures, using the along-slit-nod pattern, and 2 cycles of pattern (i.e. 2 exposures at each of the 2 nod positions). The calwebb_spec2 processing included background subtraction, which results in a pair of positive and negative traces in each cal product. When those 4 cal products are then combined in calwebb_spec3 we end up with one positive trace flanked on either side by negative traces, which is the expected layout given the nodded exposures. Again, this was done using the B7.7 release candidate #1. I've attached the level 3 s2d product. |
|
Comment by Howard Bushouse on JIRA: Beth Sargent or Sarah Kendrew could you try your notebook again using the B7.7 release and confirm whether or not this problem still exists? From the tests I've run it seems to be working OK, but you should confirm. |
|
Comment by Sarah Kendrew on JIRA: Hi Howard Bushouse - Beth Sargent and I have both re-tested independently in B7.7 (0.18.3 in my installation) and we both still find the problem. As you say the s2d output from spec2 looks fine. The combined output from spec3 has dimensions (387,5) instead of (387,44). I think the problem arises before resample_spec. Below is the log output starting at calspec3; as you can see the (387,5) dimension is showing up in the outlier_detection step already. 2021-02-19 14:31:58,141 - stpipe.Spec3Pipeline - INFO - Starting calwebb_spec3 ... 2021-02-19 14:31:58,408 - stpipe.Spec3Pipeline.outlier_detection - INFO - Step outlier_detection running with args (,). 2021-02-19 14:31:58,410 - stpipe.Spec3Pipeline.outlier_detection - INFO - Step outlier_detection parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': True, 'skip': False, 'suffix': 'crf', 'search_output_file': False, 'input_dir': '', 'weight_type': 'exptime', 'pixfrac': 1.0, 'kernel': 'square', 'fillval': 'INDEF', 'nlow': 0, 'nhigh': 0, 'maskpt': 0.7, 'grow': 1, 'snr': '4.0 3.0', 'scale': '0.5 0.4', 'backg': 0.0, 'save_intermediate_results': False, 'resample_data': True, 'good_bits': '~DO_NOT_USE', 'scale_detection': False, 'allowed_memory': None} 2021-02-19 14:31:58,412 - stpipe.Spec3Pipeline.outlier_detection - INFO - Performing outlier detection on 2 inputs 2021-02-19 14:31:58,519 - stpipe.Spec3Pipeline.outlier_detection - INFO - Resampling slit (387, 5) 2021-02-19 14:31:59,054 - stpipe.Spec3Pipeline.outlier_detection - INFO - Drizzling (1024, 1032) --> (387, 5) 2021-02-19 14:31:59,096 - stpipe.Spec3Pipeline.outlier_detection - INFO - Resampling slit (387, 5) 2021-02-19 14:31:59,631 - stpipe.Spec3Pipeline.outlier_detection - INFO - Drizzling (1024, 1032) --> (387, 5) 2021-02-19 14:31:59,660 - stpipe.Spec3Pipeline.outlier_detection - INFO - Generating median from 2 images 2021-02-19 14:31:59,662 - stpipe.Spec3Pipeline.outlier_detection - INFO - Blotting median... 2021-02-19 14:32:00,256 - stpipe.Spec3Pipeline.outlier_detection - INFO - Blotting (1024, 1032) <-- (387, 5) 2021-02-19 14:32:00,876 - stpipe.Spec3Pipeline.outlier_detection - INFO - Blotting (1024, 1032) <-- (387, 5) 2021-02-19 14:32:01,538 - stpipe.Spec3Pipeline.outlier_detection - INFO - Saved model in miri_lrs_slit_pt_nod2_v2_a3001_crf.fits 2021-02-19 14:32:01,846 - stpipe.Spec3Pipeline.outlier_detection - INFO - Saved model in miri_lrs_slit_pt_nod1_v2_a3001_crf.fits 2021-02-19 14:32:01,846 - stpipe.Spec3Pipeline.outlier_detection - INFO - Step outlier_detection done 2021-02-19 14:32:01,999 - stpipe.Spec3Pipeline.resample_spec - INFO - Step resample_spec running with args (,). 2021-02-19 14:32:02,000 - stpipe.Spec3Pipeline.resample_spec - INFO - Step resample_spec parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': True, 'skip': False, 'suffix': 's2d', 'search_output_file': True, 'input_dir': '', 'pixfrac': 1.0, 'kernel': 'square', 'fillval': 'INDEF', 'weight_type': 'exptime', 'pixel_scale_ratio': 1.0, 'single': False, 'blendheaders': True, 'allowed_memory': None} 2021-02-19 14:32:02,004 - stpipe.Spec3Pipeline.resample_spec - INFO - Drizpars reference file: /Users/skendrew//crds_cache/references/jwst/miri/jwst_miri_drizpars_0001.fits 2021-02-19 14:32:02,109 - stpipe.Spec3Pipeline.resample_spec - INFO - Blending metadata for combine_dithers_exposures 2021-02-19 14:32:02,603 - stpipe.Spec3Pipeline.resample_spec - INFO - Resampling slit (387, 5) 2021-02-19 14:32:03,164 - stpipe.Spec3Pipeline.resample_spec - INFO - Drizzling (1024, 1032) --> (387, 5) 2021-02-19 14:32:03,170 - stpipe.Spec3Pipeline.resample_spec - INFO - Resampling slit (387, 5) 2021-02-19 14:32:03,690 - stpipe.Spec3Pipeline.resample_spec - INFO - Drizzling (1024, 1032) --> (387, 5) 2021-02-19 14:32:03,697 - stpipe.Spec3Pipeline.resample_spec - INFO - Update S_REGION to POLYGON ICRS 73.636907547 0.000073100 73.637030041 0.000073100 73.637030041 0.000083282 73.636907547 0.000083282 2021-02-19 14:32:04,056 - stpipe.Spec3Pipeline.resample_spec - INFO - Saved model in combine_dithers_exposures_s2d.fits 2021-02-19 14:32:04,056 - stpipe.Spec3Pipeline.resample_spec - INFO - Step resample_spec done 2021 |
|
Comment by Howard Bushouse on JIRA: OK, I guess we'll need to get a copy of the data you're using in your notebook, so that we can debug this further. As mentioned above, when I run calwebb_spec3 on a set of nodded LRS FS exposures I happened to have laying around, it seemed to work OK and produce reasonable outputs. |
|
Comment by Sarah Kendrew on JIRA: I've attached my test script. The data are in artifcatory, so if you uncomment the first two calls you can retrieve them from there. The v2.3 in the filename refers to the version of MIRISim (fyi). The script does a full end to end run, using the other nod as background, and everything else just uses default settings. There's a few pdb.set_trace() calls in the script which you can just comment out. [^b77_e2e_lrs_slit.py] |
|
Comment by Howard Bushouse on JIRA: I have reproduced the problem you reported, in that both outlier_detection and resample_spec in the calwebb_spec3 pipeline want to create a combined output frame for the 2 images that's only 387x5 in size. Given the fact that when resample_spec is run during calwebb_spec2 it creates reasonable outputs for each image individually (dimensions 387x44) suggests that the overall structure of the WCS info in each image is OK. But the fact that the calwebb_spec3 steps think that the combined output frame for the 2 images can be covered by a grid that's only 387x5 suggests that perhaps the overall pointing info for each individual exposure just isn't consistent with the expected separation of ~1.9" between the two nod positions. |
|
Comment by Sarah Kendrew on JIRA: Thanks Howard Bushouse . What were you using to test this, that did produce a good result? Your explanation sounds plausible. I admit that I don't fully understand what WCS information MIRISim simulated data contain (hence also my trouble understanding how extract_1d() knew where to place the aperture). I will ask in the team. |
|
Comment by Howard Bushouse on JIRA: I was using your data files, as referenced in your testing script. Those result in reasonable looking s2d products for the individual exposures at the end of calwebb_spec2 processing (files miri_lrs_slit_pt_nod1_v2_s2d.fits and miri_lrs_slit_pt_nod2_v2_s2d.fits). They are a good size and contain the expected positive and negative traces, resulting from the background subtraction. It's in calwebb_spec3 that things still fail when attempting to create an output WCS that is the combination of the two exposure footprints. So I'm continuing to investigate the rest of the WCS info (e.g. the RA_REF/DEC_REF values) to see if there's something odd there. One thing about them is that they both straddle the zero line in RA and Dec: RA_REF = 0.00026308279776455 / [deg] Right ascension of the reference point RA_REF = -0.00026031780056776 / [deg] Right ascension of the reference point DEC_REF = -2.1860888891293E-05 / [deg] Declination of the reference point DEC_REF = 2.16444343946559E-05 / [deg] Declination of the reference point So we might actually have issues with handling WCS values that cross the zero line (at least for RA). |
|
Comment by Howard Bushouse on JIRA: Yep, we definitely have a problem somewhere in resampling that gets messed up with RA's around zero. TARG_RA=TARG_DEC=0 in these images, but when I evaluate the WCS of the individual resampled images from calwebb_spec2, the RA of the pixels in the positive spectral trace hovers around 180 deg, instead of 0 deg. If I add just 1 deg of offset to all of the important WCS keywords in the rate files, e.g. TARG_RA=1.0 (instead of 0.0) and add 1.0 to the RA_REF values shown in the previous comment, the WCS in the individual resampled images properly evaluates to ~1.0 deg on the positive spectral trace, and the calwebb_spec3 processing then creates combined resampled images with dimensions of 387x62, containing 2 negative traces flanking a single positive trace in the middle. See █████████████████████████████████████████████████████████████████████████████████████████████████ The reason everything worked fine in my test images is because the RA/Dec values were around 98 and -66, respectively, so it didn't have the problem with the zero crossing. By the way, I noticed that your test images have V3I_YANG=0, which is not correct for the LRS fixed-slit aperture. It should be V3I_YANG=4.71028. You can get a simulated image that has the RA axis exactly parallel to West/East and Dec parallel to North/South by setting ROLL_REF=-4.71028 (i.e. nulling out the slight rotation of the aperture in the JWST focal plane). |
|
Comment by Howard Bushouse on JIRA: James Davies data directory mentioned at top of ticket contains everything you need. TARG_RA=TARG_DEC=0, with RA_REF/DEC_REF straddling positive/negative values on either side of zero. WCS in resampled s2d products evaluates to RA=180 deg, instead of 0 deg, so something is wrapping the RA values by 180 deg. Adding a small positive offset to push all the RA values away from zero results in good products. For example: s2d=datamodels.open('miri_lrs_slit_pt_nod1_v2_s2d.fits') s2d.meta.wcs(13,300) [array(180.00000667), array(5.62654251e-07), array(8.12226798)] |
|
Comment by Sarah Kendrew on JIRA: OK, thanks for diagnosing this. It's probably a matter of hacking the header in MIRISim files before running the test, like we already do with the TSOVISIT keyword for TSOs. I will try to work with the developers or more experienced users to figure this out. Do you know why it did work in B7.5, but then not in B7.6 and 7.7? |
|
Comment by James Davies on JIRA: To be clear, I think this is a bug in jwst/jwst/resample/resample_spec.py Line 82 in b238c29 introduced in B7.6 where we updated the way we compute the spatial extent of the ensemble of nodded slits. Before that, we just used the spatial extent of the first slit and ignored the others. |
jdavies-st
changed the title
|
Comment by James Davies on JIRA: This has been fixed in jwst master. Assigning back to the reporter to verify it is fixed and to close the issue. |
|
Comment by Howard Bushouse on JIRA: Fixed by #5929 |
|
Comment by Sarah Kendrew on JIRA: I am testing this with v1.1.0 and still seeing the issue. Can you clarify:
|
|
Comment by James Davies on JIRA: We just released |
|
Comment by James Davies on JIRA: For this MIRI LRS dataset, the resulting output is 62 pixels wide in the cross-dispersion direction. We do see an offset in the output frame, but there's an open issue for that already. |
|
Comment by Sarah Kendrew on JIRA: Got it! I am checking this out in 1.2.0 and I can see the spatial offset. It looks like the resample step in Spec2 also introduces a 1-px spectral offset in one of the nods. Is that known? I can see it in both the 2D images and the extracted product - this was definitely not the case in 1.1.0. I have attached 2 figures of the 2D nodded spectra. One shows the cal.fits output, the other the s2d.fits output. The latter shows a 1-px shift.
!lrs_slit_jwst1.2.0_resample_offset_spec1.png! |
|
Comment by James Davies on JIRA: I believe the effect you are seeing is that the MIRI LRS fixed slit spectral trace isn't perfectly up and down along detector. There's a very small angle there. So when rectified in the resampling process, the effect is a small counter-clockwise rotation to get it aligned with the Y axis, compared to the _cal file. One can also see this rotation in the resampled version of the data using 1.1.0, but the weighting (amount of flux falling) in those "edge" pixels at the top and bottom is low for some reason. I'm not sure why, though we did define the extent of the spectral trace in a more robust way in the resampled image for 1.2.0, so I suspect that's what we're seeing. |
|
Comment by James Davies on JIRA: In the MIRI LRS distortion reference file, xcen and ycen give the trace relative to a nominal location, and that is translated into a 2D rotation of the image. Lines 292 to 299 in fda15ac I believe the angle is something like ~0.25 deg |
|
Comment by Sarah Kendrew on JIRA: so why is it different between the 2 nods? the trace is identical for both and we should not see a difference in the extracted spectrum between the 2 nods. this is going to give poor results when the products are combined and re-extracted. |
|
Comment by Sarah Kendrew on JIRA: actually I have a couple of ideas to look into. I'll report back. |
|
Comment by Sarah Kendrew on JIRA: I am looking at the code and I think one potential issue is that the rotation is being performed around the "zero point" as defined in the header of the reference file. This zero point corresponds to the slit centre position, but as the data is nodded, that's not where the spectrum is. I think the rotation should be executed around the zero-point of the nods, that way both spectra will be aligned in the same way after the resampling. (as an aside, there may be a better way of structuring this reference file and that is definitely something we can talk about for post-launch) |
|
Comment by James Davies on JIRA: Thanks Sarah. So you think something might be wrong in the current WCS transform? The computed transform in Agree that the reference file format is, um, creative. And not very self-documenting. An improvement to that would be most welcome. And btw, this is unrelated to the mechanics of I don't know if anyone on MIRI team has ever checked that the MIRI LRS transform in Anyway, if this turns out to be a problem with the WCS produced in |
|
Comment by Nadia Dencheva on JIRA: Sarah Kendrew Do you mean the rotation defined here https://github.com/spacetelescope/jwst/blob/master/jwst/assign_wcs/miri.py#L299 How is the location of the nodded spectrum defined? Are there keywords that can help define the new xcen, ycen?
|
|
Comment by Sarah Kendrew on JIRA: Nadia Dencheva yes, it looks like in the code lines just below the one you highlighted, a shift is applied to the zero points, which are extracted from the header (and correspond to the slit centre). there are no easy keywords that define the nod locations, but in the extract_1d step the code is able to locate the position of the source very effectively, and places the extraction aperture in the right place. I don't fully understand how that works, but there is clearly code that can locate the source in the slit. I think this issue may come back to the fact that we (the MIRI European consortium) had envisaged for the LRS spectra to not be rectified. I think the way we structured this reference file is basically not entirely compatible with the steps sequence of the JWST pipeline. Once we have figured out what is going on we should open a new ticket to focus specifically on the issue.
|
|
Comment by Sarah Kendrew on JIRA: And yes, James Davies we do have a test for the assign_wcs step which uses David Law's miricoord package and coordinate transforms. But this verifies that the pipeline follows the recipe that we have specified (which it does), and it could of course be that the recipe itself needs tweaking. |
|
Jumping in here since Sarah Kendrew pointed me to the ticket. A few quick clarifications: the pipeline is using the source location specified in the headers (TARG_RA, TARG_DEC) to determine where the source should be. Each frame has a full distortion solution embedded, using the RA_REF, DEC_REF, ROLL_REF, V2_REF, V3_REF keywords added by mirisim (which works these out based on the known dither offsets) in combination with the CDP reference file mapping pixels to v2, v3, lambda. The Extract1d code thus can figure out exactly what pixel it should be doing the extraction around. I don't see the issue with the spectral offset mentioned above though. If I run a dithered mirisim simulation with a narrow emission line at 8.4104 microns (the reference wavelength) and process it through det1 and spec2, the line is centered around y=292.0 (1-indexed) in both dither and undithered s2d frames. Same thing for lines at other wavelengths. Likewise, if I plot the x1d results from spec2 (individual files) and from running spec3 on the association file combining them, I get the attached lrs.png that doesn't show any suggestion of a spectral shift. (An unrelated issue is that all 3 are shifted by about 0.01 microns or 0.3 pixels from where they should be, but I vaguely recall that this is a known mirisim bug). What's the feature that's being shown in the plots above? If it's a continuum cutoff, my guess would be that it might be some kind of artifact from a bounding box aligned with detector columns/rows having a very slightly different wavelength cutoff at one end of the slit than the other as the trace is tilted very slightly wrt that box.
!lrs.png|thumbnail! |
|
Comment by Sarah Kendrew on JIRA: thanks David Law. what pipeline version are you using? I only saw it in the B7.8 release candidate, not in B7.7 or earlier versions. The B7.8rc has a fix in it for the resample step in Spec3, which I was testing - but then saw this other new feature/bug. the source I'm using in my simulation is one of our calibrators modeled as a simple blackbody in MIRISim. the sharp cut-off is a feature of MIRISim, due to the limited wavelength range of our current SRF reference file. I drew the red dashed line purely to show the offset between the traces a bit more clearly between the cal and s2d files. |
|
Comment by Howard Bushouse on JIRA: Not sure if this is relevant to what's happening here, but another thing that can affect where the signal appears to cut off in a cal (or resampled i2d) file is the wavelength cutoff of the flux cal data (i.e. the sensitivity curve in the photom ref file). The photom step is setup to zero-out pixels in the cal file that have wavelengths extending outside the range of the photom data. And due to the ever so slight tilt of the LRS spectral trace, I wonder if it's possible that the per-pixel wavelength assignments are also not exactly parallel to the image rows, such that in a given row the pixels over on the left side of the image have slightly different wavelength assignments than pixels over on the right. Hence pixels on one side or the other might get zeroed out, while pixels on the opposite side do not. |
|
Sarah Kendrew I was using the 1.2.0 pipeline from a couple of days ago (B7.8rc1) |
|
Comment by James Davies on JIRA: bq. And due to the ever so slight tilt of the LRS spectral trace, I wonder if it's possible that the per-pixel wavelength assignments are also not exactly parallel to the image rows, such that in a given row the pixels over on the left side of the image have slightly different wavelength assignments than pixels over on the right. Yeah, that's exactly the case. Wavelength is not constant across a row in the _cal files. The difference is ~0.01 microns to ~0.003 (depending on wavelength), left to right across the slit. |
|
Comment by Sarah Kendrew on JIRA: Looking more closely at the extracted spectra (I've attached a plot), it does look like they are the same, only one extends one pixel higher than the other. This is definitely new in the B7.8rc compared to B7.7. This extra "bump" is also seen in the combined spectrum. But sounds like this is a feature of the resampling process, not a bug? In that case, we can close this ticket. The only remaining issue would then be the spatial offset of the combined product, but sounds like that is already being handled. |
|
Comment by James Davies on JIRA: Yeah, I think this is OK. Let's close this, as the main issue is solved, and we have a ticket tracking the cross-dispersion offset. |
Issue JP-1829 was created on JIRA by Beth Sargent:
Originally, I was testing extract1d in Build 7.5, and I had developed notebooks for this (please see JP-1804). When I use the notebook for testing extract1d for LRS Fixed Slit as part of calwebb_spec3 in Build 7.6, I get an error. When I look into the error, I see a problem with the output of resample_spec in calwebb_spec3 for Build 7.6. The output array is all zeros and is too narrow. In Build 7.5, resample_spec seemed to work correctly.
The text was updated successfully, but these errors were encountered: