<jupyter_start><jupyter_text>Worked Example of the Spec Pipeline on HST DataAuthor: <NAME>  Import Library<jupyter_code>#import the module
from pipeline import spec_pipeline

import matplotlib.pyplot as plt
%matplotlib inline

#import bokeh to enable interactive plots
from bokeh.plotting import figure
from bokeh.io import output_notebook, push_notebook, show

output_notebook()

#import yaml to read in the parameter file
import yaml<jupyter_output><empty_output><jupyter_text>CoRoT-1b Spectroscopic Data    First, it will be helpful to read in the desired parameterfile to refer back to. Note, you should create and edit a parameter file similar to the one displayed below.<jupyter_code>with open("corot1_firsttransit.yaml", "r") as stream:
    paramfile = yaml.safe_load(stream)
    
paramfile<jupyter_output><empty_output><jupyter_text>To analyze, the spectroscopic data of any target, read in the desired parameter file. After installing tshirt, import the module `spec_pipeline` from `pipeline`. Note that here, we define an object `spec` however, the name can vary. Calling the command `help(spec_pipeline.spec)` might be helpful to ensure the right keywords in the argument are called. For example, when utilizing the `spec_pipeline.spec` method from the Spectroscopic module, the keyword used is `paramFile`.<jupyter_code>spec = spec_pipeline.spec(paramFile='corot1_firsttransit.yaml')<jupyter_output><empty_output><jupyter_text>Before performing the extraction of the spectroscopic data from the spectroscopic images, call `showStarChoices`. `showStarChoices` is a method of the spec object in the spectroscopic module found in tshirt and the command `help(showStarChoices)` can help define the relevant keywords. This `showStarChoices` is used to view apertures and setting the keyword `showPlot=True` will display the plot within the Jupyter notebook space. The plot displayed will have the axes in terms of the location on the instrument detector (X & Y pixels). The color bar depicts the number of counts at each pixel. Lastly, there will be a red rectangular aperture enclosing the source as well as red shaded rectangular areas depicting the background subtraction areas.<jupyter_code>spec.showStarChoices(showPlot=True)<jupyter_output><empty_output><jupyter_text>After checking the apertures, call `do_extraction`. `do_extraction` is a method of the spec object in the spectroscopic module found in tshirt, and the command `help(do_extraction)` can help define the relevant keywords. Calling `do_extraction` will perform the extraction of spectroscopic data from all the spectroscopic images found in the parameter file defined. `do_extraction` will extract all spectroscopy. Within the argument of the method, set `useMultiproccessing=True` to run the code faster by using more CPUs.<jupyter_code>spec.do_extraction(useMultiprocessing=True)<jupyter_output>On 0 of 392
On 60 of 392
On 90 of 392
On 30 of 392
On 15 of 392
On 45 of 392
On 75 of 392
On 120 of 392
On 150 of 392
On 180 of 392
On 105 of 392
On 135 of 392
On 165 of 392
On 195 of 392
On 210 of 392
On 240 of 392
On 270 of 392<jupyter_text>To plot the spectrum of the source, call 'plot_one_spec' which is a method of the spec object in the spectroscopic module found in tshirt. The plot displayed will have an x-axis in terms of X pixel location on the detector and a y-axis in terms of counts(e-). In this particular data, each pixel was already converted to be terms of (DN/s), a value of flux. However, this method `plot_one_spec` converts this flux value back into counts for easier calculations of uncertainty down the line. The plot also depicts two distinct plots of an "optimal" extraction and "sum" extraction defined by a legend. The "optimal" extraction is essentially a weighted sum of the counts in each row in the spectroscopic images while the other is an unweighted sum. The "optimal" extracted spectrum chooses weights to result in the least amount of noise.<jupyter_code>spec.plot_one_spec()<jupyter_output><empty_output><jupyter_text>To plot the light curve of each visit, call `plot_wavebin_series`, a method of the spec object in the spectroscopic module found in tshirt. Calling `plot_wavebin_series` essentially takes the summation of all the pixel flux values enclosed within the source aperture (taking into account background subtraction) that the method `do_extraction` gathered prior. Within the argument of the method, set `interactive=True` to navigate the plot within the notebook space. By default, the argument `nbins=10` but, altering `nbins` controls how many wavelength-binned time series are plotted. To define more of the parameters call the command `help(plot_wavebin_series)`.Displayed on the plot, the x-axis will be time and the y-axis will be the normalized flux value.<jupyter_code>spec.plot_wavebin_series(nbins=1,savePlot=False, interactive=True)<jupyter_output><empty_output><jupyter_text>To plot the dynamic spectrum, call `plot_dynamic_spec`, a method of the spec object in the spectroscopic module found in tshirt. Calling `plot_dynamic_spec` essentially displays a plot depicting the change in the spectrum over time. Within the argument of the method, set `showPlot=True` to display the plot within the Jupyter notebook space. To define more of the parameters call the command `help(plot_dynamic_spec)`.Displayed in the image will be two plots. The plot on top displays the average spectrum overall. The 2D plot on the bottom displays how the spectrum changes over time. On the x-axis is the dispersion and on the y-axis is the time which correlates to the image number. Finally, the color bar displays the normalized flux which can identify the transit of an exoplanet. Displayed below are four visits visible as horizontal bands. The band located at the bottom, which is the first visit, on the plot is the visit featuring the transit.<jupyter_code>spec.plot_dynamic_spec(showPlot=True)<jupyter_output>WARNING: VerifyWarning: Card is too long, comment will be truncated. [astropy.io.fits.card]<jupyter_text>CoRoT-1b Batch Processing the Spectroscopic Data A batch object can iterate over any method the spectroscopic module found in tshirt. To perform batch processing, this is the general syntax for the code to run: `bspec.batch_run('method')`. Start by creating a batch parameter file from a normal spec parameter file. Turn parameters unique to each iteration into lists. For example, the `procFiles`, `nightName`, and `bkgRegionsY` (background regions) have been customized for the four visits. Next, read in the batch parameter file.<jupyter_code>with open("corot1_batch_file.yaml", "r") as stream:
    bparamfile = yaml.safe_load(stream)
    
bparamfile<jupyter_output><empty_output><jupyter_text>To analyze, the spectroscopic data of any target, read in the desired parameter file. After installing tshirt, import the module `spec_pipeline` from `pipeline`. Note that here, we define an object `bspec` however, the name can vary. Calling the command `help(spec_pipeline.batch_spec)` might be helpful to ensure the right keywords in the argument are called. For example, when utilizing the `spec_pipeline.batch_spec` method from the spectroscopic module, the keyword used is `batchFile`.<jupyter_code>bspec = spec_pipeline.batch_spec(batchFile='corot1_batch_file.yaml')<jupyter_output><empty_output><jupyter_text>Before performing the extraction of the spectroscopic data from the spectroscopic images, call `showStarChoices`. `showStarChoices` is a method of the spec object in the spectroscopic module found in tshirt and the command `help(showStarChoices)` can help define the relevant keywords. This `showStarChoices` is used to view apertures and setting the keyword `showPlot=True` will display the plot within the Jupyter notebook space. The keyword `vmax` defines the maximum value for imshow. Since this is a batch parameter file, there should be a plot displayed for each distinguished visit. These plots displayed will have the axes in terms of the location on the detection instrument (X & Y pixels). The color bar depicts the number of counts at each pixel. Lastly, there will be a red rectangular aperture enclosing the source as well as red shaded rectangular areas depicting the background subtraction areas.<jupyter_code>bspec.batch_run('showStarChoices',vmax=1000,showPlot=True)<jupyter_output>Working on showStarChoices for batch CoRoT-1 corot1_visit1 
Working on showStarChoices for batch CoRoT-1 corot1_visit2 
Working on showStarChoices for batch CoRoT-1 corot1_visit3 
Working on showStarChoices for batch CoRoT-1 corot1_visit4<jupyter_text>After checking the apertures, call `do_extraction`. `do_extraction`  is a method of the spec object in the spectroscopic module found in tshirt, and the command `help(do_extraction)` can help define the relevant keywords. Calling `do_extraction` will perform the extraction of spectroscopic data from all the spectroscopic images found in the batch parameter file defined. `do_extraction` will extract all spectroscopy. Within the argument of the method, set `useMultiproccessing=True` to run the code faster by using more CPUs.<jupyter_code>bspec.batch_run('do_extraction',useMultiprocessing=True)<jupyter_output>Working on do_extraction for batch CoRoT-1 corot1_visit1 
On 0 of 98
On 15 of 98
On 30 of 98
On 45 of 98
On 60 of 98
On 75 of 98
On 90 of 98
Working on do_extraction for batch CoRoT-1 corot1_visit2<jupyter_text>To plot the spectrum of the source for all visits, call `plot_one_spec` which is a method of the spec object in the spectroscopic module found in tshirt.The plots displayed will have an x-axis in terms of X pixel location on the detector and a y-axis in terms of counts(e-). In this particular data, each pixel was already converted to be terms of (DN/s), a value of flux. However, this method `plot_one_spec` converts this flux value back into counts for easier calculations of uncertainty down the line. The plot also depicts two distinct plots of an "optimal" extraction and "sum" extraction defined by a legend. The "optimal" extraction is essentially a weighted sum of the counts in each row in the spectroscopic images while the other is an unweighted sum. The "optimal" extracted spectrum chooses weights to result in the least amount of noise. For bright sources like this example, the sum and optimal extractions will give very similar results.<jupyter_code>bspec.batch_run('plot_one_spec')<jupyter_output>Working on plot_one_spec for batch CoRoT-1 corot1_visit1<jupyter_text>To plot the light curve of each visit separately with a batch file, call `plot_wavebin_series`, a method of the spec object in the spectroscopic module found in tshirt. Calling `plot_wavebin_series` essentially takes the summation of all the pixel flux values enclosed within the source aperture (taking into account background subtraction) that the method `do_extraction` gathered prior. Within the argument of the method, set `interactive=True` to navigate the plot within the notebook space. By default, the argument `nbins=10` but, altering `nbins` controls how many wavelength-binned time series are plotted. To define more of the parameters call the command `help(plot_wavebin_series)`.Displayed on the plot, the x-axis will be time and the y-axis will be the normalized flux value.<jupyter_code>bspec.batch_run('plot_wavebin_series', nbins=1, interactive=True)<jupyter_output>Working on plot_wavebin_series for batch CoRoT-1 corot1_visit1<jupyter_text>To plot the dynamic spectrum for each visit separately, call `plot_dynamic_spec`, a method of the spec object in the spectroscopic module found in tshirt. Calling `plot_dynamic_spec` essentially displays a plot depicting the change in the spectrum over time. Within the argument of the method, set `showPlot=True` to display the plot within the Jupyter notebook space. To define more of the parameters call the command `help(plot_dynamic_spec)`.Displayed in each image will be two plots. The plot on top displays the average spectrum overall. The 2D plot on the bottom displays how the spectrum changes over time. On the x-axis is the dispersion and on the y-axis is the time which correlates to the image number. Finally, the color bar displays the normalized flux which can identify the transit of an exoplanet. In this example for a batch file, each visit has a separate dynamic spectrum plotted.<jupyter_code>bspec.batch_run('plot_dynamic_spec', showPlot=True)<jupyter_output>Working on plot_dynamic_spec for batch CoRoT-1 corot1_visit1
{'fraction_non_alphanumeric': 0.04970830336450092, 'fraction_numerical': 0.013266203148725326, 'mean_word_length': 5.663471778487753, 'pattern_counts': {'":': 0, '<': 55, '<?xml version=': 0, '>': 55, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '15', 'dir': 'jupyter-structured-clean-dedup', 'id': '85588', 'license': 'MIT', 'n_tokens_mistral': 3583, 'n_tokens_neox': 3231, 'n_words': 1692, 'path': 'docs/spec_pipeline/CoRoT-1b_Example_Annotated_Notebook.ipynb', 'repo_name': 'kglidic/tshirt', 'size': '12519'}