<jupyter_start><jupyter_text>How to create an example Level 1 WFI science data fileThis notebook shows how to manually create an example Level 1 WFI ASDF file that is appropriate for input into the `romancal` calibration pipeline.There are several examples shown, using randomly generated data arrays, as well as one that creates a Level 1 file using data arrays pulled from an existing FITS file.<jupyter_code># make sure that you have these packages installed in your local environment
import numpy as np
import matplotlib
from matplotlib import pylab as plt
import asdf
import astropy
from astropy.time import Time
from astropy.io import fits
import roman_datamodels
import roman_datamodels.stnode as rds

# the following allows us to grab example data from an STScI box folder
from astropy.utils.data import download_file
box_path = 'https://data.science.stsci.edu/redirect/Roman/Roman_Calibration_And_Datamodels/'<jupyter_output><empty_output><jupyter_text>Files currently available in the box folder linked above include:|Filename|exposure type|Process Level||:-|:-|:-||r0000101001001001001_01101_0001_WFI01_uncal.asdf|WFI_IMAGE|Level 1||r0000201001001001001_01101_0001_WFI01_uncal.asdf|WFI_IMAGE|Level 1||r0000201001001001002_01101_0001_WFI10_uncal.asdf|WFI_GRISM|Level 1||r0000201001001001003_01101_0001_WFI08_uncal.asdf|WFI_PRISM|Level 1||r0000101001001001001_01101_0001_WFI01_cal.asdf|WFI_IMAGE|Level 2||r0000201001001001001_01101_0001_WFI01_cal.asdf|WFI_IMAGE|Level 2||r0000201001001001002_01101_0001_WFI10_cal.asdf|WFI_GRISM|Level 2||r0000201001001001003_01101_0001_WFI08_cal.asdf|WFI_PRISM|Level 2||iacs01t4q_raw.fits|WFC3|HST example FITS|  Display versions of relevant packages for reference<jupyter_code>print('Dependency Versions\n-------------------')
print(f'asdf: {asdf.__version__}')
print(f'astropy: {astropy.__version__}')
print(f'numpy: {np.__version__}')
print(f'roman_datamodels: {roman_datamodels.__version__}')<jupyter_output><empty_output><jupyter_text>Create an empty Level 1 datamodelThe Roman data models will expect specific data and metadata, these can be seen and referenced in the datamodels and schemas:* [ScienceRawModel](https://github.com/spacetelescope/roman_datamodels/blob/main/src/roman_datamodels/datamodels.pyL290) is used as input to the science calibration pipeline, and represents a Level 1 file. With its associated [schema](https://github.com/spacetelescope/rad/blob/main/src/rad/resources/schemas/wfi_science_raw-1.0.0.yaml)* [ImageModel](https://github.com/spacetelescope/roman_datamodels/blob/main/src/roman_datamodels/datamodels.pyL286) will contain the collapsed and calibrated ramp for both imaging and spectroscopic modes, and represents a Level 2 file. With its associated [schema](https://github.com/spacetelescope/rad/blob/main/src/rad/resources/schemas/wfi_image-1.0.0.yaml)The level 1 datamodel expects that certain metadata are saved along with the data arrays. These can be seen referenced in the schema file for the Level 1 datamodel:%YAML 1.1---$schema: asdf://stsci.edu/datamodels/roman/schemas/rad_schema-1.0.0id: asdf://stsci.edu/datamodels/roman/schemas/wfi_science_raw-1.0.0title: |  The schema for WFI science data (both imaging and spectrographic)type: objectproperties:  meta:    allOf:      - $ref: common-1.0.0  <---- these are the expected metadata  data:    tag: tag:stsci.edu:asdf/core/ndarray-1.0.0    datatype: uint16    ndim: 3propertyOrder: [meta, data]flowStyle: blockrequired: [meta, data]That "common" [metadata file](https://github.com/spacetelescope/rad/blob/main/src/rad/resources/schemas/common-1.0.0.yaml) contains the list of items and their types, and can reference schemas for each item as well, such as:https://github.com/spacetelescope/rad/blob/main/src/rad/resources/schemas/exposure_type-1.0.0.yamlwhich shows these included items:%YAML 1.1---$schema: asdf://stsci.edu/datamodels/roman/schemas/rad_schema-1.0.0id: asdf://stsci.edu/datamodels/roman/schemas/exposure_type-1.0.0 Helper file to enumerate viewing modes for exposure and ref_exposuretype: stringtitle: Type of data in the exposure (viewing mode)enum:  - WFI_IMAGE  - WFI_GRISM  - WFI_PRISM  - WFI_DARK  - WFI_FLAT  - WFI_WFSC...If there is an enumerated list, like shown above, then those are the possible values for the attribute (keyword).<jupyter_code>level1 = rds.WfiScienceRaw() # level 1 files are ramps
level2 = rds.WfiImage()  # just as an example, this would be level 2 output datamodel from the calibration pipeline<jupyter_output><empty_output><jupyter_text>Make a somewhat random science data arrayRemember that Level 1 data are still in 3D ramp form<jupyter_code>science_data_frame = np.random.randint(low=0, high=65536,size=((4096,4096)), dtype=np.uint16)  # data at level 1 is stored as uint16
science_data_frame
plt.imshow(science_data_frame, origin="lower")
# make a small 3D cube
science_data_cube = np.array((science_data_frame, science_data_frame*2, science_data_frame*5))
np.shape(science_data_cube)
meta = {'telescope': 'ROMAN'}
meta['description'] = "Example level 1 image mode ramp with random values"
meta['author'] = '<NAME>'
meta['origin'] = 'STSCI'<jupyter_output><empty_output><jupyter_text>We can attempt to save this file using the minimal information that we haveValidation errors should be returned since we dont have all the required metadata populated yet...and we also have schemas to validate against<jupyter_code># attach our metadata and science data to the model
level1['meta'] = meta
level1['data'] = science_data_cube
af = asdf.AsdfFile()
af.tree = {'roman': level1}
af.write_to('level1_image_ramp.asdf')<jupyter_output><empty_output><jupyter_text>Let's add in the rest of the required informationWe'll cheat for now and pull it from the testing module for roman_datamodels.This code is an example of how to populate the objects, but should be used as a working reference only, and not the default call in production code. `roman_datamodels` is maintained at: https://github.com/spacetelescope/roman_datamodels/<jupyter_code>from roman_datamodels.testing import utils
common_meta = utils.mk_common_meta()
common_meta  # show it, notice this isn't just dictionary of simple type information, it contains objects as well
#Since we pulled the example from somewhere else, lets update some descriptive data to be appropriate
common_meta['description'] = "Example level 1 image mode ramp with random values"
common_meta['author'] = '<NAME>'
level1['meta'] = common_meta
af.tree = {'roman': level1}
af.write_to('level1_image_ramp.asdf')  # this time it should validate without issue<jupyter_output><empty_output><jupyter_text>The ASDF file that we wrote now contains the following information:ASDF 1.0.0ASDF_STANDARD 1.5.0%YAML 1.1%TAG ! tag:stsci.edu:asdf/--- !core/asdf-1.1.0asdf_library: !core/software-1.0.0 {author: The ASDF Developers, homepage: 'http://github.com/asdf-format/asdf',  name: asdf, version: 2.9.0.dev0}history:  extensions:  - !core/extension_metadata-1.0.0    extension_class: asdf.extension.BuiltinExtension    software: !core/software-1.0.0 {name: asdf, version: 2.9.0.dev0}  - !core/extension_metadata-1.0.0    extension_class: astropy.io.misc.asdf.extension.AstropyAsdfExtension    software: !core/software-1.0.0 {name: astropy, version: '4.2'}  - !core/extension_metadata-1.0.0    extension_class: asdf.extension._manifest.ManifestExtension    extension_uri: asdf://stsci.edu/datamodels/roman/extensions/datamodels-1.0    software: !core/software-1.0.0 {name: roman-datamodels, version: 0.8.1.dev1+g19040c6}roman: !  meta:    aperture: !      name: dummy value      position_angle: 30.0      pss_name: dummy value    author: <NAME>    cal_step: !      assign_wcs: INCOMPLETE      flat_field: INCOMPLETE      dark: INCOMPLETE      dq_init: INCOMPLETE      jump: INCOMPLETE      linearity: INCOMPLETE      ramp_fit: INCOMPLETE      saturation: INCOMPLETE    calibration_software_version: 9.9.9    coordinates: !      reference_frame: ICRS    crds_context_used: '222'    crds_software_version: 8.8.8    description: Example level 1 image mode ramp with random values    ephemeris: !      earth_angle: -999999      moon_angle: -999999      sun_angle: -999999      type: DEFINITIVE      time: -999999      ephemeris_reference_frame: dummy value      spatial_x: -999999      spatial_y: -999999      spatial_z: -999999      velocity_x: -999999      velocity_y: -999999      velocity_z: -999999    exposure: !      id: -999999      type: WFI_IMAGE      start_time: !time/time-1.1.0 2020-01-01T00:00:00.000      mid_time: !time/time-1.1.0 2020-01-01T00:00:00.000      end_time: !time/time-1.1.0 2020-01-01T00:00:00.000      start_time_mjd: -999999      mid_time_mjd: -999999      end_time_mjd: -999999      start_time_tdb: -999999      mid_time_tdb: -999999      end_time_tdb: -999999      ngroups: -999999      nframes: -999999      data_problem: false      sca_number: -999999      gain_factor: -999999      integration_time: -999999      elapsed_exposure_time: -999999      frame_divisor: -999999      groupgap: -999999      frame_time: -999999      group_time: -999999      exposure_time: -999999      effective_exposure_time: -999999      duration: -999999      datamode: -999999      integration_end: -999999      integration_start: -999999      nints: -999999      nresets_at_start: -999999      nsamples: -999999      sample_time: -999999      start_time_eng: dummy value    file_date: !time/time-1.1.0 2020-01-01T00:00:00.000    filename: dummy value    guidestar: !      gw_start_time: !time/time-1.1.0 2020-01-01T00:00:00.000      gw_stop_time: !time/time-1.1.0 2020-01-01T00:00:00.000      gw_id: dummy value      gs_ra: -999999      gs_dec: -999999      gs_ura: -999999      gs_udec: -999999      gs_mag: -999999      gs_umag: -999999      gw_pcs_mode: dummy value      gw_function_start_time: !time/time-1.1.0 2020-01-01T00:00:00.000      gw_function_end_time: !time/time-1.1.0 2020-01-01T00:00:00.000      data_start: -999999      data_end: -999999      gw_acq_exec_stat: dummy value      gs_ctd_x: -999999      gs_ctd_y: -999999      gs_ctd_ux: -999999      gs_ctd_uy: -999999      gs_epoch: dummy value      gs_mura: -999999      gs_mudec: -999999      gs_para: -999999      gw_window_xstart: -999999      gw_window_ystart: -999999      gw_window_xsize: -999999      gw_window_ysize: -999999    instrument: !      detector: WFI01      optical_element: F062      name: WFI    model_type: dummy value    observation: !      start_time: !time/time-1.1.0 2020-01-01T00:00:00.000      end_time: !time/time-1.1.0 2020-01-01T00:00:00.000      obs_id: dummy value      visit_id: dummy value      program: -999999      execution_plan: -999999      pass: -999999      observation: -999999      segment: -999999      visit: -999999      visit_file_group: -999999      visit_file_sequence: -999999      visit_file_activity: dummy value      exposure: -999999      template: dummy value      observation_label: dummy value      ma_table_name: dummy value      survey: N/A      activity_id: dummy value    origin: STSCI    photometry: !      conversion_microjanskys: -999999      conversion_megajanskys: -999999      pixelarea_steradians: -999999      pixelarea_arcsecsq: -999999    pointing: !      ra_v1: -999999      dec_v1: -999999      pa_v3: -999999    prd_software_version: 8.8.8    program: !      title: dummy value      pi_name: dummy value      category: dummy value      subcategory: dummy value      science_category: dummy value      continuation_id: -999999    sdf_software_version: 7.7.7    target: !      proposer_name: dummy value      catalog_name: dummy value      type: FIXED      ra: -999999      dec: -999999      ra_uncertainty: -999999      dec_uncertainty: -999999      proper_motion_ra: -999999      proper_motion_dec: -999999      proper_motion_epoch: dummy value      proposer_ra: -999999      proposer_dec: -999999      source_type_apt: POINT      source_type: POINT    telescope: ROMAN    velocity_aberration: !      ra_offset: -999999      dec_offset: -999999      scale_factor: -999999    visit: !      engineering_quality: OK      pointing_engdb_quality: CALCULATED      type: dummy value      start_time: !time/time-1.1.0 2020-01-01T00:00:00.000      end_time: !time/time-1.1.0 2020-01-01T00:00:00.000      status: dummy value      total_exposures: -999999      internal_target: false      target_of_opportunity: false    wcsinfo: !      v2_ref: -999999      v3_ref: -999999      vparity: -999999      v3yangle: -999999      ra_ref: -999999      dec_ref: -999999      roll_ref: -999999      s_region: dummy value  data: !core/ndarray-1.0.0    source: 0    datatype: uint16    byteorder: little    shape: [3, 4096, 4096]...      Read in an example data array from a FITS fileObviously the WFC3 data isn't the same expected size as the WFI data that we expect. We do expect to create level 1 files that are smaller, for example, the guide window arrays.The following is meant as a simple example showing how to read a fits file and move the data arrays over to the roman datamodel for saving.Validation on the datamodel and within the roman calibration code should take care of mismatches.<jupyter_code># Edit these lines if you dont want to download the example wfc3 data from the box location
test_file='iacs01t4q_raw.fits'
#wfc3_image = download_file(box_path + test_file)
fits_image=fits.open(test_file)
fits_image.info()
fits_image[0].header
fits_science_cube = np.array((fits_image['sci',10].data, fits_image['sci',8].data, fits_image['sci',6].data, fits_image['sci',3].data, fits_image['sci',1].data))
level1['data'] = fits_science_cube  # if you were doing this for real, you'd want to make sure the metadata was updated to reflect appropriate information for the data arrays you just used
common_meta['description'] = 'Level 1 WFI image ramp made from wfc3 data'
level1['meta'] = common_meta
af.tree = {'roman': level1}
af.write_to('level1_image_ramp_from_wfc3.asdf')
fits_image.close()<jupyter_output><empty_output><jupyter_text>We can read this data back in using the datamodel<jupyter_code>level1_image_ramp = roman_datamodels.open('level1_image_ramp_from_wfc3.asdf')
level1_image_ramp.data  # the science data array we saved is represented here
fig = plt.figure(figsize=(10, 30))
fig.add_subplot(1,5,1)
plt.imshow(level1_image_ramp.data[0], origin="lower")
fig.add_subplot(1,5,2)
plt.imshow(level1_image_ramp.data[1], origin="lower")
fig.add_subplot(1,5,3)
plt.imshow(level1_image_ramp.data[2], origin="lower")
fig.add_subplot(1,5,4)
plt.imshow(level1_image_ramp.data[3], origin="lower")
fig.add_subplot(1,5,5)
plt.imshow(level1_image_ramp.data[4], origin="lower")<jupyter_output><empty_output><jupyter_text>Open an example Roman file with reasonably populated metadata<jupyter_code># grab the example prism data file from the box folder or read it locally
roman_example_L1 = box_path + 'r0000201001001001002_01101_0001_WFI10_uncal.asdf'
filepath = download_file(roman_example_L1)
#filepath ='r0000201001001001002_01101_0001_WFI10_uncal.asdf' 
example_data=roman_datamodels.open(filepath)<jupyter_output><empty_output><jupyter_text>The object information can be returned as needed<jupyter_code>example_data.meta.exposure.type
np.shape(example_data.data)<jupyter_output><empty_output><jupyter_text>Or you can get a quick summary<jupyter_code>example_data.to_flat_dict()
fig = plt.figure(figsize=(10, 5))
fig.add_subplot(1,3,1)
plt.imshow(example_data.data[0], origin="lower")
fig.add_subplot(1,3,2)
plt.imshow(example_data.data[3], origin="lower")
fig.add_subplot(1,3,3)
plt.imshow(example_data.data[7], origin="lower")<jupyter_output><empty_output><jupyter_text>If you just downloded a file from Box, it is likely stored in a local temp folder, use the command below to save it to the current working directory if you like<jupyter_code>example_data.save('example_roman_grism_data.asdf')<jupyter_output><empty_output><jupyter_text>Once the Level 1 file runs through calibration, a Level 2 file is produced that looks something like this.Another notebook will show an example of running the Level 2 calibration pipeline on Level 1 data<jupyter_code>level_2_example = 'r0000201001001001002_01101_0001_WFI10_cal.asdf'
#download the example file as needed
#roman_example_L2 = box_path + level_2_example
#level_2_example = download_file(roman_example_L1)

level2 =roman_datamodels.open(level_2_example)
level2  #this is just to show the ImageModel return instead of the Level 1 ScienceRawModel<jupyter_output><empty_output><jupyter_text>Quickly looking at the file shows that some of the calibration steps have been run<jupyter_code>level2.to_flat_dict()
level2.meta.wcsinfo  #basic wcs information is currently avaialable in the file<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09353507565337002, 'fraction_numerical': 0.08414568506668262, 'mean_word_length': 4.5261070720423, 'pattern_counts': {'":': 0, '<': 61, '<?xml version=': 0, '>': 60, 'https://': 8, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 7, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '14', 'dir': 'jupyter-structured-clean-dedup', 'id': '102804', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 6549, 'n_tokens_neox': 5618, 'n_words': 1444, 'path': 'create_science_data/Create_example_level1_wfi_image.ipynb', 'repo_name': 'cshanahan1/romancal-notebooks', 'size': '16736'}