<jupyter_start><jupyter_text>JWST Pipeline Validation Testing Notebook: Calwebb_Detector1, RSCD step **Instruments Affected**: MIRITested on MIRI Simulated data Table of Contents   [Introduction](intro_ID)  [Imports](imports_ID)  [Run Pipeline Steps](pipeline_ID)  [Check Flagged Frames](flagging_ID)  [Results](results_ID)  [About This Notebook](about_ID)   IntroductionThis test is designed to test the RSCD step in the calwebb_detector1 pipeline. This step only applies to MIRI. The Reset Switch Charge Decay (RSCD) step corrects for a set of ramp effects that affect all ingrations after the first by simply flagging the first N groups as DO_NOT_USE. The number of groups to flag is read from the reference file based on readout mode and subarray. For more information on the pipeline step visit the links below. Step description: https://jwst-pipeline.readthedocs.io/en/latest/jwst/rscd/description.htmlPipeline code: https://github.com/spacetelescope/jwst/tree/master/jwst/rscdMost of the data for this test were created with the MIRI Data Simulator, and the documentation for that code can be found here: http://miri.ster.kuleuven.be/bin/view/Public/MIRISim_Public. This test can also be run on properly formatted ground test data. Calibration WG Requested Algorithm: A short description and link to the page: https://outerspace.stsci.edu/display/JWSTCC/Vanilla+Reset+Switch+Charge+Decay+Correction Defining TermsDefinition of terms or acronymns.JWST: James Webb Space TelescopeMIRI: Mid-Infrared InstrumentMIRISim: MIRI Data SimulatorRSCD: Reset Switch Charge Decay Description of testThis test is performed by reading in data of various types (subarray, readpattern, etc.) and running them through the rscd step to see whether the correct number of frames are flagged (read from the reference file). Create the data for testingThe set of data used in this particular test were created with the MIRI Data Simulator (MIRISim). Referring to the MIRISim link, you can see how to set up and run the simulator to re-create the input files if you wish. The data was run with a scene.ini file that specified what the scene should look like, with coordinates for the stars given in units of arcsecond offsets from the center of the field of view. The scene.ini file as well as the setup files simuation.ini and simulator.ini are needed to run the simulation.Once in the mirisim conda environment, the simulation is run with the command line:> mirisim simulation.ini[Top of Page](title_ID)  ImportsThe following packages will need to be imported for the scripts to work.* os for reading in CRDS server info and path information* ci_watson and get_bigdata for reading data from artifactory* jwst.datamodels for opening files as a JWST Datamodel* jwst.pipeline to run the pipeline step/module* jwst.associations to create association table* numpy for calculations* inspect and Markdown to get documentation for step being tested  * photutils for star finding and aperture photometry[Top of Page](title_ID)<jupyter_code>import os
os.environ['CRDS_SERVER_URL'] = 'https://jwst-crds.stsci.edu'
import crds

from astropy.io import fits
import numpy as np
from IPython.display import Markdown
from jwst.datamodels import RampModel, dqflags
from jwst.dq_init import DQInitStep
from jwst.rscd import RSCD_Step
from ci_watson.artifactory_helpers import get_bigdata
import inspect
from IPython.display import Markdown

# Get raw python docstring to see documentation for this step
raw = inspect.getdoc(RSCD_Step)

# To convert to markdown, you need convert line breaks from \n to <br />
markdown_text = "<br />".join(raw.split("\n"))

# Here you can format markdown as an output using the Markdown method.
Markdown("""
# RSCD_Step
---
{}
""".format(markdown_text))<jupyter_output><empty_output><jupyter_text>Run Pipeline StepsFor this test, run each file through dq_init and the rscd steps only. The output for the test is the output of the rscd step. We are testing that the specified number of frames for each file is flagged properly.[Top of Page](title_ID)<jupyter_code># Put in tests for FULL frame fast, FULL frame slow, subarray and FASTGRPAVG (should skip FASTGRPAVG) 

# read in input files

file1 = get_bigdata('jwst_validation_notebooks',
                    'validation_data',
                    'rscd',
                    'rscd_miri_test', 
                    'starfield_slow_seq1_MIRIMAGE_F1130Wexp1.fits')

file2 = get_bigdata('jwst_validation_notebooks',
                    'validation_data',
                    'rscd',
                    'rscd_miri_test', 
                    'starfield_fast_4ptdither_seq1_MIRIMAGE_F1130Wexp1.fits')

file3 = get_bigdata('jwst_validation_notebooks',
                    'validation_data',
                    'rscd',
                    'rscd_miri_test', 
                    'det_image_seq1_MIRIMAGE_F1130Wexp1_brightsky.fits')

file4 = get_bigdata('jwst_validation_notebooks',
                    'validation_data',
                    'rscd',
                    'rscd_miri_test', 
                    'starfield_fastgrpavg_seq1_MIRIMAGE_F1130Wexp1.fits')

file5 = get_bigdata('jwst_validation_notebooks',
                    'validation_data',
                    'rscd',
                    'rscd_miri_test', 
                    'jw04193001001_01101_00001_MIRIFULONG_uncal.fits')

filelist = get_bigdata('jwst_validation_notebooks',
                       'validation_data',
                       'rscd',
                       'rscd_miri_test', 
                       'rscd_testfiles.txt')

rscdoutfiles = 'rscd_processed_files.txt'

# read in list of input files
imlist1 = np.loadtxt(filelist,dtype=bytes, ndmin=1).astype(str)  # avoid the 'b at the start of a line

# open output file for ramp fit file names
f = open(rscdoutfiles,'w')

print('There are ', imlist1.size, ' images.')

# loop over list of files
for file in imlist1:
       
    # set up output file name
    if 'uncal' in file:
        base, remainder = file.split('_uncal')
    else:    
        base, remainder = file.split('.')
    # write out name of output files to text file for later use
    f.write(base+'_rscd_step.fits'+'\n')
    
    # check that header has needed keywords set
    model = RampModel(file)
    preim = DQInitStep.call(model)
    postim = RSCD_Step.call(preim, output_file=base)
    
f.close()

print('All files processed.')<jupyter_output><empty_output><jupyter_text>Check flagged framesCheck that the frames at the beginning of each integration after the first are flagged properly.FAST mode should mark 4 frames as 'DO_NOT"USE', SLOW mode marks 2 frames, and step should skip FASTGRPAVG.All subarrays are set to skip 4 frames. (These are the values currently set in the reference files as of 8/2020.)[Top of Page](title_ID)<jupyter_code># FAST mode should mark 4 frames as 'DO_NOT"USE', SLOW mode marks 2 frames, and step should skip FASTGRPAVG.
# All subarrays are set to skip 4 frames.

# read in list of input files
imlist2 = np.loadtxt(rscdoutfiles,dtype=bytes, ndmin=1).astype(str)  # avoid the 'b at the start of a line

# loop over list of files
for rscdfile in imlist2:
    # Read in RSCD output file

    rscdmodel = RampModel(rscdfile)
    
    # access groupdq extension of the file
    groupdq = rscdmodel.groupdq
    
    # Get readout pattern from science header
    readpatt = rscdmodel.meta.exposure.readpatt
    science_subarray = rscdmodel.meta.subarray.name
    
    # Get reference file used in step
    rscdreffile = rscdmodel.meta.ref_file.rscd.name
    print('RSCD reference file', rscdreffile)
    
    # parse linfile name to discard crds://
    file = rscdreffile.split("/")[2]
    
    # read in reference file
    # find location of file
    basename = crds.core.config.pop_crds_uri(rscdreffile)
    path = crds.locate_file(basename, "jwst")
 
    # open reference file
    reffile = fits.open(path)
    reftable = reffile['RSCD_GROUP_SKIP'].data
    refhead = reffile[0].header
    
    # read in table value for number to skip from reference file
    mask = ((reftable['READPATT'] == readpatt) & (reftable['SUBARRAY'] == 'FULL'))
    row=reftable[mask]
    numskiparray = row['group_skip']
    
    if numskiparray.size > 0:
        numskip = row['group_skip'][0] 
    else:
        numskip = 0
        
    # For all integrations after the first, flag numskip frames at the start of the ramp as 'DO_NOT_USE'
    nint = rscdmodel.meta.exposure.nints
 
    # Use assert statement to check that all integrations greater than the first and in cases where 
    # the number of frames to skip is not 0, check that all groups are properly flagged.
    for int in range(nint):
        if (int > 0 and numskip > 0):
            assert np.all(groupdq[int, 0:numskip-1, 400, 400] == dqflags.group['DO_NOT_USE'])
    
    # output print to screen to see output of tests
    print('Science file tested: ',rscdfile)
    print('Subarray: ', science_subarray, ' Readout pattern: ', readpatt, ' Number of frames to flag: ', numskip)
    print('Data quality flags for each frame in each integration')
    print(groupdq[:,:,400,400])
    print()

print('All files processed. If no Assertion Errors printed, all tests passed.')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07430306974726109, 'fraction_numerical': 0.010087862024080702, 'mean_word_length': 3.4476603955619876, 'pattern_counts': {'":': 0, '<': 15, '<?xml version=': 0, '>': 19, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '219950', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 2932, 'n_tokens_neox': 2778, 'n_words': 994, 'path': 'jwst_validation_notebooks/rscd/jwst_rscd_miri_test/jwst_miri_rscd_testing.ipynb', 'repo_name': 'brynickson/jwst_validation_notebooks', 'size': '9219'}