use doubles for filter values used with fft in refpix #8512
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The fft filtering (
fft_interp_norm
) used during the thefill_bad_regions
function in therefpix
step suffers from loss of precision leading to numerical differences across systems. Theaa
filter coefficients are defined asfloat32
(temp_a1
is the start of a number of variables used to constructaa
):jwst/jwst/refpix/irs2_subtract_reference.py
Line 1085 in f866557
but are used with a
double
precision fft result:jwst/jwst/refpix/irs2_subtract_reference.py
Lines 989 to 992 in f866557
For each loop iteration
aa
is cast todouble
and as the added 32 bits are not defined by the algorithm used for the other bits inaa
each iteration accumulates error in the output.This PR switches
aa
todouble
precision to avoid the loss of precision. It is expected that this will change the regression test output.To take one example,
test_nirspec_irs2_detector1[linearity]
currently passes on jenkins but fails on github actions:https://github.com/spacetelescope/RegressionTests/actions/runs/9261424291/job/25476839480#step:25:297
although this tests the output file for the linearity step the differences stem from the refpix step (run immediately before linearity for this test). Although the failure is only a single pixel the difference is above the tolerance:
Furthermore this failure matches that seen when running the same test locally with a mac m1 or on a system with RHEL 9 (jenkins is running RHEL 7).
With this PR the above test fails in the same way on jenkins, github actions, RHEL9 and on a mac m1.
See https://github.com/spacetelescope/RegressionTests/actions/runs/9273993994/job/25515454758 for the regression test run on github actions (with this PR) and https://plwishmaster.stsci.edu:8081/blue/organizations/jenkins/RT%2FJWST-Developers-Pull-Requests/detail/JWST-Developers-Pull-Requests/1480/pipeline/200 for the jenkins run (note that several
nan
vs0
failures due to #8488 are unrelated).This addresses one of several RHEL9 issues (which appear the largely overlap with github actions vs jenkins differences) revealed during testing for JP-3312.
Checklist for PR authors (skip items if you don't have permissions or they are not applicable)
CHANGES.rst
within the relevant release sectionHow to run regression tests on a PR