<jupyter_start><jupyter_text>Forecast run parsingThe Solar Forecast Arbiter is designed to analyze *non-overlapping* forecasts. The [documentation](https://solarforecastarbiter.org/definitions/) details the motivation for this choice. Briefly, this forces forecast users and providers to think carefully about what kinds of forecasts they want to analyze. Ideally, these choices will be informed by one or more decision-making processes. Forecast providers often design systems that create many overlapping forecasts. For example, new forecasts may be issued once per hour and extend for 48 hours. In some systems, the raw forecasts may also have a higher temporal resolution than the end-user application requires. Providers or end users need to parse these forecasts by parameters such as lead time before they can be analyzed. Similarly, these forecast runs must be parsed into one or more non-overlapping "forecast evaluation time series" before they can be uploaded to the Solar Forecast Arbiter. The figure below illustrates this situation. It shows three forecast runs (green) issued 1 hour apart, each with 15 minute intervals and a length of 3 hours. These forecast runs are sliced, resampled, and concatenated into two different forecast evaluation time series: a 1 hour ahead, 15 minute interval forecast (blue); and a 2 hour ahead, 1 hour interval forecast (red).![timeline](https://solarforecastarbiter.org/images/timeline_merged.svg)This tutorial with demonstrate how to prepare many forecast runs for analysis in the Solar Forecast Arbiter.<jupyter_code>import datetime
from pathlib import Path
import os
import shutil

import numpy as np
import pandas as pd<jupyter_output><empty_output><jupyter_text>Let's generate forecast data from a simulated operational system. The system will create a new forecast every hour, saving each forecast to a csv file. The forecast run attributes will be similar to those shown in the figure above:* 1 hour issue frequency* 3 hour run length* 0 lead time* 15 minute interval lengthThe forecast values will be equal to the initialization hour + the forecast hour.<jupyter_code># make a new directory to store the output. 
# a cell at the end of the notebook will delete this directory using shutil.rmtree(fx_path)
fx_path = Path('generated_forecasts')
fx_path.mkdir()
issue_start = pd.Timestamp('2020-01-01', tz='UTC')
issue_end = issue_start + pd.Timedelta('12h')
issue_freq = pd.Timedelta('1h')
closed = 'left'  # consistent with interval averages labeled by the beginning of the interval
issue_times = pd.date_range(start=issue_start, end=issue_end, freq=issue_freq, closed=closed)

lead_time = pd.Timedelta('0min')
run_length = pd.Timedelta('3h')
interval_length = pd.Timedelta('15min')

time_format = '%Y%m%dT%H%M%SZ'

for issue_time in issue_times:
    fx_start = issue_time + lead_time
    fx_end = fx_start + run_length 
    fx_index = pd.date_range(start=fx_start, end=fx_end, freq=interval_length, closed=closed, name='timestamp')
    fx_values = issue_time.hour + fx_index.hour
    fx = pd.Series(fx_values, index=fx_index, name='value')
    issue_time_str = issue_time.strftime(time_format)
    header = f'# forecast issued at {issue_time_str}\n'
    file_name = f'fx_{issue_time_str}.csv'
    with open(fx_path / file_name, 'w') as f:
        f.write(header)
        fx.to_csv(f)<jupyter_output><empty_output><jupyter_text>Here's a list of all of the files we created<jupyter_code>fx_files = sorted(fx_path.iterdir())
fx_files<jupyter_output><empty_output><jupyter_text>Now inspect a couple of the files to confirm the data is as we expected.<jupyter_code>with open(fx_files[0]) as f:
    print(f.read())
with open(fx_files[1]) as f:
    print(f.read())<jupyter_output># forecast issued at 20200101T010000Z
timestamp,value
2020-01-01 01:00:00+00:00,2
2020-01-01 01:15:00+00:00,2
2020-01-01 01:30:00+00:00,2
2020-01-01 01:45:00+00:00,2
2020-01-01 02:00:00+00:00,3
2020-01-01 02:15:00+00:00,3
2020-01-01 02:30:00+00:00,3
2020-01-01 02:45:00+00:00,3
2020-01-01 03:00:00+00:00,4
2020-01-01 03:15:00+00:00,4
2020-01-01 03:30:00+00:00,4
2020-01-01 03:45:00+00:00,4<jupyter_text>Ok, so we have a series of overlapping forecasts and we need to slice and reassemble them into something the Solar Forecast Arbiter can use. The for loop below accomplishes this for the hour ahead, 15 minute interval forecast shown in blue in the figure.<jupyter_code>lead_time = pd.Timedelta('1h')
run_length = pd.Timedelta('1h')

fx_parsed = []
for fx_run in fx_files:
    fx = pd.read_csv(fx_run, comment='#', index_col=0, parse_dates=True)
    # get issue time from filename. remove .csv suffix, then remove fx_ prefix
    issue_time = fx_run.name.split('.')[0].split('_')[1]
    issue_time = pd.Timestamp(issue_time)
    fx_start = issue_time + lead_time
    # -1ns to account for interval_label=beginning when slicing
    fx_end = fx_start + run_length - pd.Timedelta('1ns')  
    fx_sliced = fx.loc[fx_start:fx_end]
    fx_parsed.append(fx_sliced)
    
blue_fx_concat = pd.concat(fx_parsed)
blue_fx_concat<jupyter_output><empty_output><jupyter_text>Next we repeat the process for the 2 hour ahead, 1 hour interval red forecast.<jupyter_code>lead_time = pd.Timedelta('2h')
interval_length = pd.Timedelta('1h')
run_length = pd.Timedelta('1h')

fx_parsed = []
for fx_run in fx_files:
    fx = pd.read_csv(fx_run, comment='#', index_col=0, parse_dates=True)
    # get issue time from filename. remove .csv suffix, then remove fx_ prefix
    issue_time = fx_run.name.split('.')[0].split('_')[1]
    issue_time = pd.Timestamp(issue_time)
    fx_start = issue_time + lead_time
    # -1ns to account for interval_label=beginning when slicing
    fx_end = fx_start + run_length - pd.Timedelta('1ns')  
    fx_sliced = fx.loc[fx_start:fx_end].resample(interval_length).mean()
    fx_parsed.append(fx_sliced)
    
red_fx_concat = pd.concat(fx_parsed)
red_fx_concat<jupyter_output><empty_output><jupyter_text>These forecasts could be uploaded to the Solar Forecast Arbiter if the corresponding metadata exists. Users may choose to create the metadata using the [Dashboard](https://dashboard.solarforecastarbiter.org) or programmatically. See the [Data Upload and Download in Python](data_upload_download.ipynb) tutorial for an example of how to programmatically create the metadata.A more sophisticated workflow might start by formally defining the metadata in the Solar Forecast Arbiter and then use that metadata to parse the forecast runs. This is left as an exercise for the reader! Finally, let's be tidy and delete the directory and files that we created.<jupyter_code>shutil.rmtree(fx_path)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08834134615384616, 'fraction_numerical': 0.04417067307692308, 'mean_word_length': 5.298013245033113, 'pattern_counts': {'":': 0, '<': 28, '<?xml version=': 0, '>': 28, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '608147', 'license': 'MIT', 'n_tokens_mistral': 2318, 'n_tokens_neox': 2097, 'n_words': 712, 'path': 'forecast_parsing.ipynb', 'repo_name': 'lboeman/2019-Denver-Workshop', 'size': '6656'}