<jupyter_start><jupyter_text>Create a SUBSTRIP256 Simulation<jupyter_code># Generate the simulation
tso256 = TSO(nints=1, ngrps=2, star=STAR_DATA)
tso256.simulate()<jupyter_output>Simulating New Target in New Target Simulation
Configuration: SUBSTRIP256 + CLEAR
Groups: 2, Integrations: 1

Constructing frames for chunk 1/1...
Chunk 1/1 finished: 2.512 s
Adding noise to TSO...
Generating noise for integration 1/1
NG: Starting mknoise() at DATETIME =  09:13:15.025162
NG: Initializing results cube at DATETIME =  09:13:15.508534
NG: Generating rd_noise at DATETIME =  09:13:15.525708
NG: Adding c_pink noise at DATETIME =  09:13:15.555429
NG: Adding u_pink noise at DATETIME =  09:13:15.889521
NG: Adding acn noise at DATETIME =  09:13:16.208504
NG: Adding PCA-zero "picture frame" noise at DATETIME =  09:13:16.609854
NG: Adding pedestal drift at DATETIME =  09:13:17.030746
NG: Adding dark current at DATETIME =  09:13:17.039231
Noise model finished: 2.34 s

Total time: 4.861 s<jupyter_text>Do a simple column sum extraction<jupyter_code># Get the order 1 data
trace = tso256.tso_order1_ideal
shape = trace.shape

# Reshape into 3D and NaN reference pixels
data = trace.reshape((shape[0] * shape[1], shape[2], shape[3]))
data[:, :, :4] = np.nan
data[:, :, -4:] = np.nan
data[:, -4:, :] = np.nan

# Add up counts
order1_counts = np.nansum(data, axis=1)

# Compare to input spectrum
input_spec = np.interp(tso256.avg_wave[0], STAR_DATA[0].value, STAR_DATA[1].value, right=np.nan, left=np.nan)

# Spectrum plot
spec_fig = figure(height=300)
spec_fig.line(order1_wave[10:-10], input_spec[10:-10], color='green', legend='Input')

# Residual plot
res_fig = figure(height=200)
colors = ['blue', 'cyan']
for n, group in enumerate(order1_counts):
    
    # Convert counts to flux
    order1_flux = order1_counts[n] / tso256.order1_response / (tso256.frame_time * (n + 1))

    # Plot it
    spec_fig.line(order1_wave[10:-10], order1_flux.value[10:-10], color=colors[n], legend='Group {}'.format(n + 1))

    # Mean residual (away from edges)
    residual = (order1_flux.value - input_spec) / input_spec
    res_fig.line(order1_wave[10:-10], residual[10:-10], legend='Group {}'.format(n + 1))

    # Check that the mean residual is less than 0.01
    print('Mean residual for group {}: {}'.format(n + 1, np.nanmean(residual)))

show(column(spec_fig, res_fig))
# Get the order 2 data
trace = tso256.tso_order2_ideal
shape = trace.shape

# Reshape into 3D and NaN reference pixels
data = trace.reshape((shape[0] * shape[1], shape[2], shape[3]))
data[:, :, :4] = np.nan
data[:, :, -4:] = np.nan
data[:, -4:, :] = np.nan

# Add up counts
order2_counts = np.nansum(data, axis=1)

# Compare to input spectrum
input_spec = np.interp(tso256.avg_wave[1], STAR_DATA[0].value, STAR_DATA[1].value, right=np.nan, left=np.nan)

# Spectrum plot
spec_fig = figure(height=300)
spec_fig.line(order2_wave[10:-10], input_spec[10:-10], color='green', legend='Input')

# Residual plot
res_fig = figure(height=200)
colors = ['blue', 'cyan']
for n, group in enumerate(order2_counts):
    
    # Convert counts to flux
    order2_flux = order2_counts[n] / tso256.order2_response / (tso256.frame_time * (n + 1))

    # Plot it
    spec_fig.line(order2_wave[15:-10], order2_flux.value[15:-10], color=colors[n], legend='Group {}'.format(n + 1))

    # Mean residual (away from edges)
    residual = (order2_flux.value - input_spec) / input_spec
    res_fig.line(order2_wave[15:-10], residual[15:-10], color=colors[n], legend='Group {}'.format(n + 1))

    # Check that the mean residual is less than 0.01
    print('Mean residual for group {}: {}'.format(n + 1, np.nanmean(residual[15:-10])))

show(column(spec_fig, res_fig))<jupyter_output>Mean residual for group 1: 0.000507599128308602
Mean residual for group 2: 0.0005077075202217876
{'fraction_non_alphanumeric': 0.1440588853838065, 'fraction_numerical': 0.08727655099894847, 'mean_word_length': 4.765151515151516, 'pattern_counts': {'":': 0, '<': 7, '<?xml version=': 0, '>': 7, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '2', 'dir': 'jupyter-structured-clean-dedup', 'id': '107552', 'license': 'MIT', 'n_tokens_mistral': 1655, 'n_tokens_neox': 1421, 'n_words': 365, 'path': 'notebooks/End-to-end Demo.ipynb', 'repo_name': 'spacetelescope/AWESim_SOSS', 'size': '3804'}