<jupyter_start><jupyter_text>Running the Roman pipeline Pipeline installation CRDS SetupRoman reference files are stored in the Calibration Reference Database System (CRDS). Access to it is enabled by the following environment variabes:```export CRDS_PATH=$HOME/crds_cacheexport CRDS_SERVER_URL=https://roman-crds-test.stsci.edu```**Note** that the above URL points to the TEST server which isn't publicly available but this is the way to set it up when a public server is online. The first time a reference file is accessed it is downloaded to the local cache. Subsequent uses of the file and offline access use the copy from the cache. Internally, at STScI the cache is available. For more information on CRDS consult the [CRDS User Guide](https://jwst-crds-test.stsci.edu/static/users_guide/index.html).  Install a public release of the pipelineThis section describes installation of a released version of the pipeline.Create a new conda environment, called "roman-test".(Caution Note: You need to be in zsh or bash. The new environment will inherit the current one. That means that if you are in the base conda environment and there are packages in it they will be available in the new environment unless you reinstall them there. This is generally bad because Python will also load packages from that base environment. A case, which is often a source for consfusion, is when ipython is not installed in the new environment but is available in the base one. Starting ipython owrks and loading a package may work but this package is imported from the base env and most often is a version different from what's desired.)```% conda create -n roman-test python=3.9```Activate the environment and install ipython and numpy```% conda activate roman-test% pip install jupyter% pip install numpy```Install the pipeline. The correct versions of the dependencies will be pulled in.```% pip install romancal```We are ready to run the pipeline. Install a development version of the pipelineThis section describes installation of the `main` branch of the github repository.Create a new conda environment, called "roman-dev".```% conda create -n roman-dev python=3.9```Activate the environment and install ipython and numpy```% conda activate roman-dev% pip install jupyter% pip install numpy```Install the pipeline```pip install git+https://github.com/spacetelescope/romancal``` Code developmentThere are several packages which are concurrently developed to support the Roman pipeline calibrations.Some are shared with the JWST pipeline.- rad: Roman schemashttps://github.com/spacetelescope/rad- roman_datamodels: Datamodels for Roman data productshttps://github.com/spacetelescope/roman_datamodels- stpipe: Pipeline runner (shared with jwst)https://github.com/spacetelescope/stpipe- stcal: Core algorithms used in the pipeline (shared with jwst)https://github.com/spacetelescope/stcal- romancal: The actual pipeline codehttps://github.com/spacetelescope/romancalInstalling romancal installs all its dependencies automatically. Opening a data file as a data model<jupyter_code>from astropy.utils.data import download_file

box_path = 'https://data.science.stsci.edu/redirect/Roman/Roman_Calibration_And_Datamodels/'
level1_file = box_path + "r0000101001001001001_01101_0001_WFI01_uncal.asdf"

level2_file = box_path + "r0000101001001001001_01101_0001_WFI01_cal.asdf"
from roman_datamodels import datamodels as rdm

l1_file = download_file(level1_file)

data_model = rdm.open(l1_file)
data_model<jupyter_output><empty_output><jupyter_text>This is a special data model. Science data formatting creates a Level1 (`ScienceRawModel`) file, which is the input to the pipeline. The first step in the pipeline, `dqinit` needs to convert this to a `RampModel`.   Running a step using reference files in CRDS On the command line```strun romancal.step.FlatFieldStep ```This runs the flat field step and saves the output in the current directory. The file suffix is `flatfieldstep`.The original `rate` suffix is preserved. *This is something that needs to be fixed!* In the Python interpreter<jupyter_code>from romancal.flatfield import FlatFieldStep

l2_file = download_file(level2_file)
result = FlatFieldStep.call(l2_file)<jupyter_output><empty_output><jupyter_text>In this case the output `result` is an ImageModel and can be saved to file.<jupyter_code>print(result)
result.save('my_flat_fielded_image.asdf')<jupyter_output><empty_output><jupyter_text>Running a step with local reference files On the command line```strun romancal.step.FlatFieldStep  --override_flat=myflat.asdf``` In the Python interpreter<jupyter_code>from romancal.flatfield import FlatFieldStep

result = FlatFieldStep.call(level2_file, override_flat="flat.asdf")<jupyter_output><empty_output><jupyter_text>In general, to run the pipeline from the interpreter and pass custom reference files, one needs to use an argument name `override_=`On the command line the syntax is `--override_=`To find out what reference types a step uses and their names:<jupyter_code>from romancal.step import *
print("flat_field: ", FlatFieldStep.reference_file_types)
print("dq_init: ", DQInitStep.reference_file_types)<jupyter_output><empty_output><jupyter_text>In terms of code development, a "pipeline" is a special "step", which runs other steps. This means the Exposure Level Pipeline can be run in the same way as an individual step.```strun romancal.pipeline.ExposurePipeline r0000101001001001001_01101_0001_WFI01_uncal.asdf```or using the pipeline alias, `roman_elp`,```strun roman_elp r0000101001001001001_01101_0001_WFI01_uncal.asdf```The syntax to run the pipeline in the Python interpreter is also the same as for an individual step.To list all available steps (note the list contains steps from all packages registered with stpipe):<jupyter_code>stpipe list<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.06820908311910882, 'fraction_numerical': 0.022964867180805485, 'mean_word_length': 6.108404384896468, 'pattern_counts': {'":': 0, '<': 25, '<?xml version=': 0, '>': 25, 'https://': 9, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '529819', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 1749, 'n_tokens_neox': 1605, 'n_words': 667, 'path': 'general/Running the Roman pipeline.ipynb', 'repo_name': 'cshanahan1/romancal-notebooks', 'size': '5835'}