Change in pipeline output

[stscijgbot]

Issue JP-1024 was created by Jonathan Eisenhamer:

Follow test regressions from eisenhamer #21 show the following error:

FileNotFoundError: [Errno 2] No such file or directory: '/data1/jenkins/workspace/RT/eisenhamer-dev/clone/test_outputs/popen-gw16/test_image2pipeline10/jw00001001001_01101_00001_mirimage_i2d.fits'

The reason is that the pipeline is producing a different output. Need to investigate why.

Failed regressions:

• [jwst.tests_nightly.general.miri.test_image2pipeline_1.TestImage2Pipeline|https://plwishmaster.stsci.edu:8081/job/RT/job/eisenhamer-dev/21/testReport/junit/jwst.tests_nightly.general.miri.test_image2pipeline_1/TestImage2Pipeline/_stable_deps__test_image2pipeline1/]
• [jwst.tests_nightly.general.nircam.test_image2pipeline_2.TestImage2Pipeline|https://plwishmaster.stsci.edu:8081/job/RT/job/eisenhamer-dev/21/testReport/junit/jwst.tests_nightly.general.nircam.test_image2pipeline_2/TestImage2Pipeline/_stable_deps__test_image2pipeline2_cal/]

[stscijgbot]

Comment by Jonathan Eisenhamer: Update from official regression [#404|https://plwishmaster.stsci.edu:8081/job/RT/job/JWST/404/], after the merge of jp-799. There are now two separate conditions. The one mentioned in the description, where the pipeline is producing a different product. The second is where the product, though that same produce, is using the class name of the step in the output name, and not the value of the "name" parameter. The two types of regressions are listed below:

• Different product
  ** [test_image2pipeline1|https://plwishmaster.stsci.edu:8081/job/RT/job/JWST/404/testReport/junit/jwst.tests_nightly.general.miri.test_image2pipeline_1/TestImage2Pipeline/_stable_deps__test_image2pipeline1/]
  ** [test_image2pipeline2_cal|https://plwishmaster.stsci.edu:8081/job/RT/job/JWST/404/testReport/junit/jwst.tests_nightly.general.nircam.test_image2pipeline_2/TestImage2Pipeline/_stable_deps__test_image2pipeline2_cal/]
• Use of classname in output file name
  ** [test_miri_masterbg_mrs_dedicated|https://plwishmaster.stsci.edu:8081/job/RT/job/JWST/404/testReport/junit/jwst.tests_nightly.general.miri.test_miri_steps_single/TestMIRIMasterBackgroundMRSDedicated/_stable_deps__test_miri_masterbg_mrs_dedicated/]
  ** [test_miri_masterbg_lrs_nodded|https://plwishmaster.stsci.edu:8081/job/RT/job/JWST/404/testReport/junit/jwst.tests_nightly.general.miri.test_miri_steps_single/TestMIRIMasterBackgroundLRSNodded/_stable_deps__test_miri_masterbg_lrs_nodded/]
  ** [test_miri_masterbg_mrs_nodded|https://plwishmaster.stsci.edu:8081/job/RT/job/JWST/404/testReport/junit/jwst.tests_nightly.general.miri.test_miri_steps_single/TestMIRIMasterBackgroundMRSNodded/_stable_deps__test_miri_masterbg_mrs_nodded/]
  ** [test_miri_masterbg_lrs_dedicated|https://plwishmaster.stsci.edu:8081/job/RT/job/JWST/404/testReport/junit/jwst.tests_nightly.general.miri.test_miri_steps_single/TestMIRIMasterBackgroundLRSDedicated/_stable_deps__test_miri_masterbg_lrs_dedicated/]
  ** [test_nirspec_masterbg_nodded|https://plwishmaster.stsci.edu:8081/job/RT/job/JWST/404/testReport/junit/jwst.tests_nightly.general.nirspec.test_nirspec_steps_single/TestNIRSpecMasterBackgroundNodded/_stable_deps__test_nirspec_masterbg_nodded/]

[stscijgbot]

Comment by Jonathan Eisenhamer: [~rij] Discussing with [~jdavies], who was independently looking at the first case of different product, it is looking like the nested configs are not working properly anymore. The {{suffix}} definition is in a step config that is referred to from the pipeline config. The {{calweb_spec2}} cfg refers to the {{resample}} cfg where the {{suffix}} is being defined. So, it appears that the {{resample}} cfg is not being honored.

[stscijgbot]

Comment by James Davies: In the master background tests, I think there is a different subtle change in the new code that now fixes what was an old bug in how step result ModelContainers were written out with save_results=True.

I'm going spin off the masterbg test failures as a separate issue.

[stscijgbot]

Comment by James Davies: This bug only appears when doing Step.call(), not when running via strun. So it seems that whatever is happening (or not happening) is in the Step.call() method, and just_the_step_from_cmdline() function is doing it correctly. This may point towards a solution.