<jupyter_start><jupyter_text>JWST Pipeline Validation Testing Notebook: Spec2, fringe, MIRIInstruments Affected: MIRI Table of Contents1&nbsp;&nbsp;JWST Pipeline Validation Testing Notebook: Spec2, fringe, MIRI1.1&nbsp;&nbsp;Introduction and summary of test being run1.2&nbsp;&nbsp;Documentation1.3&nbsp;&nbsp;Data used1.3.1&nbsp;&nbsp;Create temporary directory1.3.2&nbsp;&nbsp;Set up import statements1.3.3&nbsp;&nbsp;Print pipeline version number1.3.4&nbsp;&nbsp;Read in data from artifactory1.3.5&nbsp;&nbsp;Read in input image as JWST data model1.3.6&nbsp;&nbsp;Run output of calwebb_detector1 through calwebb_spec21.3.7&nbsp;&nbsp;Create image models of rate and cal files1.3.8&nbsp;&nbsp;Get fringe reference file1.3.9&nbsp;&nbsp;Create image lists for looping1.3.10&nbsp;&nbsp;Check DQ flagging1.3.11&nbsp;&nbsp;Calculate the rate/cal image ratio1.3.12&nbsp;&nbsp;Compare fringe reference file with the rate/cal image ratio and check that they are equal1.4&nbsp;&nbsp;About this Notebook  Introduction and summary of test being runThis notebook processes an image through calwebb_spec2 and examines the output of the fringestep. The steps are as follow:1) Set up data path and directory and image file names.2) Run output of calwebb_detector1 through the fringe step in calwebb_spec2.3) Get fringe reference file from CRDS. 4) Compare the fringe reference file with the rate/cal image ratio and check that they are the same. DocumentationThe pipeline documentation can be found here: https://jwst-pipeline.readthedocs.io/en/latest/The pipeline code is available on GitHub: https://github.com/spacetelescope/jwst Data used The data used in this test is a simulated MIRI image created using MIRISim. The documentation for MIRISim can be found here: https://wiki.miricle.org/bin/view/Public/MIRISim_Public?Author: <NAME> (Adapted from Tea Temim's calwebb-image1 notebook)  Create temporary directory<jupyter_code># Create a temporary directory to hold notebook output, and change the working directory to that directory.
from tempfile import TemporaryDirectory
import os
data_dir = TemporaryDirectory()
os.chdir(data_dir.name)
print(data_dir)<jupyter_output><empty_output><jupyter_text>Set up import statements<jupyter_code>from astropy.io import fits, ascii
import pytest
import numpy as np
import numpy.ma as ma

import jwst
from jwst.pipeline import Detector1Pipeline, Image2Pipeline
from jwst.datamodels import RampModel, ImageModel, dqflags
from jwst.pipeline import calwebb_image2
from jwst.pipeline import calwebb_spec2

from ci_watson.artifactory_helpers import get_bigdata
import crds
import os
# Specify CRDS locations and pmap
os.environ['CRDS_SERVER_URL'] = 'https://jwst-crds.stsci.edu'
os.environ['TEST_BIGDATA']='https://bytesalad.stsci.edu/artifactory/'<jupyter_output><empty_output><jupyter_text>Print pipeline version number<jupyter_code>jwst.__version__<jupyter_output><empty_output><jupyter_text>Read in data from artifactory<jupyter_code>file1 = get_bigdata('jwst_validation_notebooks',
                      'validation_data',
                      'calwebb_spec2',
                      'spec2_miri_test',
                      'det_image_seq1_MIRIFUSHORT_12LONGexp1_rate.fits')
file2 = get_bigdata('jwst_validation_notebooks',
                      'validation_data',
                      'calwebb_spec2',
                      'spec2_miri_test',
                      'det_image_seq1_MIRIFUSHORT_12MEDIUMexp1_rate.fits')
file3 = get_bigdata('jwst_validation_notebooks',
                      'validation_data',
                      'calwebb_spec2',
                      'spec2_miri_test',
                      'det_image_seq1_MIRIFUSHORT_12SHORTexp1_rate.fits')
file4 = get_bigdata('jwst_validation_notebooks',
                      'validation_data',
                      'calwebb_spec2',
                      'spec2_miri_test',
                      'det_image_seq1_MIRIFULONG_34LONGexp1_rate.fits')
file5 = get_bigdata('jwst_validation_notebooks',
                      'validation_data',
                      'calwebb_spec2',
                      'spec2_miri_test',
                      'det_image_seq1_MIRIFULONG_34MEDIUMexp1_rate.fits')
file6 = get_bigdata('jwst_validation_notebooks',
                      'validation_data',
                      'calwebb_spec2',
                      'spec2_miri_test',
                      'det_image_seq1_MIRIFULONG_34SHORTexp1_rate.fits')<jupyter_output><empty_output><jupyter_text>Read in input image as JWST data model<jupyter_code>im1 = ImageModel(file1)
im2 = ImageModel(file2)
im3 = ImageModel(file3)
im4 = ImageModel(file4)
im5 = ImageModel(file5)
im6 = ImageModel(file6)<jupyter_output><empty_output><jupyter_text>Run output of calwebb_detector1 through calwebb_spec2<jupyter_code>def run_spec2(im):
    """Creates and runs an instance of the JWST spec2 pipeline. """
    
    spec2 = calwebb_spec2.Spec2Pipeline()
    spec2.bkg_subtract.skip = True
    spec2.assign_wcs.skip = False
    spec2.imprint_subtract.skip = True
    spec2.msa_flagging.skip = True
    spec2.extract_2d.skip = True
    spec2.wavecorr.skip = True
    spec2.srctype.skip = True
    spec2.straylight.skip = True
    spec2.fringe.skip = False
    spec2.pathloss.skip = True
    spec2.barshadow.skip = True
    spec2.photom.skip = True
    spec2.resample_spec.skip = True
    spec2.cube_build.skip = True
    spec2.extract_1d.skip = True
    spec2.flat_field.skip = True
    spec2.save_results = True
    
    spec2.run(im)
run_spec2(im1)
run_spec2(im2)
run_spec2(im3)
run_spec2(im4)
run_spec2(im5)
run_spec2(im6)<jupyter_output><empty_output><jupyter_text>Create image models of rate and cal files<jupyter_code># Get names of cal files
input_file1 = file1.replace('rate.fits', 'cal.fits')
input_file2 = file2.replace('rate.fits', 'cal.fits')
input_file3 = file3.replace('rate.fits', 'cal.fits')
input_file4 = file4.replace('rate.fits', 'cal.fits')
input_file5 = file5.replace('rate.fits', 'cal.fits')
input_file6 = file6.replace('rate.fits', 'cal.fits')
# create image models of rate and cal files
im_cal1 = ImageModel(input_file1)
im_cal2 = ImageModel(input_file2)
im_cal3 = ImageModel(input_file3)
im_cal4 = ImageModel(input_file4)
im_cal5 = ImageModel(input_file5)
im_cal6 = ImageModel(input_file6)

im_rate1 = ImageModel(file1)
im_rate2 = ImageModel(file2)
im_rate3 = ImageModel(file3)
im_rate4 = ImageModel(file4)
im_rate5 = ImageModel(file5)
im_rate6 = ImageModel(file6)<jupyter_output><empty_output><jupyter_text>Get fringe reference file<jupyter_code>fringereffile1 = im_cal1.meta.ref_file.fringe.name
fringereffile2 = im_cal2.meta.ref_file.fringe.name
fringereffile3 = im_cal3.meta.ref_file.fringe.name
fringereffile4 = im_cal4.meta.ref_file.fringe.name
fringereffile5 = im_cal5.meta.ref_file.fringe.name
fringereffile6 = im_cal6.meta.ref_file.fringe.name

print('fringe reference file', fringereffile1)
print('fringe reference file', fringereffile2)
print('fringe reference file', fringereffile3)
print('fringe reference file', fringereffile4)
print('fringe reference file', fringereffile5)
print('fringe reference file', fringereffile6)
# find location of file
basename1 = crds.core.config.pop_crds_uri(fringereffile1)
basename2 = crds.core.config.pop_crds_uri(fringereffile2)
basename3 = crds.core.config.pop_crds_uri(fringereffile3)
basename4 = crds.core.config.pop_crds_uri(fringereffile4)
basename5 = crds.core.config.pop_crds_uri(fringereffile5)
basename6 = crds.core.config.pop_crds_uri(fringereffile6)

path1 = crds.locate_file(basename1, "jwst")
path2 = crds.locate_file(basename2, "jwst")
path3 = crds.locate_file(basename3, "jwst")
path4 = crds.locate_file(basename4, "jwst")
path5 = crds.locate_file(basename5, "jwst")
path6 = crds.locate_file(basename6, "jwst")
# open reference file
fringe_im1 = ImageModel(path1)
fringe_im2 = ImageModel(path2)
fringe_im3 = ImageModel(path3)
fringe_im4 = ImageModel(path4)
fringe_im5 = ImageModel(path5)
fringe_im6 = ImageModel(path6)<jupyter_output><empty_output><jupyter_text>Create image lists for looping<jupyter_code>file_list = [file1, file2, file3, file4, file5, file6]
im_list = [im_cal1, im_cal2, im_cal3, im_cal4, im_cal5, im_cal6]
rate_list = [im_rate1, im_rate2, im_rate3, im_rate4, im_rate5, im_rate6]
fringe_list = [fringe_im1, fringe_im2, fringe_im3, fringe_im4, fringe_im5, fringe_im6]<jupyter_output><empty_output><jupyter_text>Check DQ flagging Any pixel flagged as NON_SCIENCE should also be flagged as DO_NOT_USE. Check if this is in place in both the input reference file and for the output science file of the calwebb_image2 pipeline. If there are no assert errors, the test below passes.<jupyter_code># Check if the output cal file is flagged properly
# return DQ arrays for masking
def check_dq_flags(im_cal, fringe_im):
    # Test that all pixels flagged with NON_SCIENCE are also flagged as DO_NOT_USE
    nonsciencearray = (im_cal.dq & dqflags.pixel['NON_SCIENCE'] > 0)
    badarray = (im_cal.dq & dqflags.pixel['DO_NOT_USE'] > 0)
    assert nonsciencearray.all() == badarray.all()

    # Test if the input reference file had the flags all set the same way
    nonsciencearray = (fringe_im.dq & dqflags.pixel['NON_SCIENCE'] > 0)
    badarray = (fringe_im.dq & dqflags.pixel['DO_NOT_USE'] > 0)
    assert nonsciencearray.all() == badarray.all()
    
    return badarray

badarrays = [check_dq_flags(im, fringe) for im,fringe in zip(im_list, fringe_list)]<jupyter_output><empty_output><jupyter_text>Calculate the rate/cal image ratio<jupyter_code>def get_ratios(im, rate, mask):
    # compute and return the ratio of the rate and cal images
    im_mask = ma.masked_array(im.data, mask=mask)
    ratio = rate.data/im_mask.data

    return ratio

ratios = [get_ratios(cal_im, rate_im, badarray) for cal_im, rate_im, badarray in zip(im_list, rate_list, badarrays)]<jupyter_output><empty_output><jupyter_text>Compare fringe reference file with the rate/cal image ratio and check that they are equalThe minimum and maximum values should be 1.0, implying all values are 1, and the images are equal.<jupyter_code>def check_fringe(ratio_im, fringe_im):
    return ratio_im/fringe_im.data

check_fringes = [check_fringe(ratio, fringe) for ratio, fringe in zip(ratios, fringe_list)]
def check_min_max(check_fringe_im, im_name):
    print('Minimum and maximum values for {} image are: {:.2e} and {:.2e}'.format(im_name, np.nanmin(check_fringe_im), np.nanmax(check_fringe_im)))
    #assert np.nanmin(check_fringe_im) == 1.0 and np.nanmax(check_fringe_im) == 1.0

for im, name in zip(check_fringes, file_list):
    im_name = name.split('/')[-1]
    check_min_max(im, im_name)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.093796992481203, 'fraction_numerical': 0.026879699248120302, 'mean_word_length': 4.708690987124464, 'pattern_counts': {'":': 0, '<': 50, '<?xml version=': 0, '>': 54, 'https://': 5, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '19145', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 3851, 'n_tokens_neox': 3730, 'n_words': 817, 'path': 'jwst_validation_notebooks/fringe/JWST_fringe_miri_test/fringe_miri_mrs_test.ipynb', 'repo_name': 'york-stsci/jwst_validation_notebooks', 'size': '10647'}