<jupyter_start><jupyter_text>Demonstration of Integration with Statistical Clear Sky Fitting Package8/25/20  Notebook setup and library imports<jupyter_code>import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
# PVInsight Code Imports
from solardatatools import DataHandler
from solardatatools.dataio import get_pvdaq_data<jupyter_output><empty_output><jupyter_text>Load data table from external sourceFor today's example, we're loading data from NREL's PVDAQ API, which is a publically available PV generatation data set.<jupyter_code>data_frame = get_pvdaq_data(sysid=34, year=range(2011, 2015), api_key='DEMO_KEY')[0]
dh = DataHandler(data_frame)
dh.run_pipeline(power_col='ac_power')<jupyter_output>total time: 14.22 seconds
--------------------------------
Breakdown
--------------------------------
Preprocessing              2.59s
Cleaning                   6.71s
Filtering/Summarizing      4.92s
    Data quality           0.24s
    Clear day detect       0.20s
    Clipping detect        3.58s
    Capacity change detect 0.90s<jupyter_text>SCSF is in a seperate package, but if the software is installed in your environment, you can fit a clear sky model to your data directly from the `DataHandler` class instance.<jupyter_code>dh.fit_statistical_clear_sky_model()<jupyter_output>----------------------
SCSF Problem Setup
----------------------
Matrix Size: 96 x 1461 = 140256 power measurements
Sparsity: 72.91%
37997 non-zero measurements under clear conditions
Model size: 96 x 6 + 6 x 1461 = 9342 parameters


----------------------
Algorithm Iterations
----------------------
Starting at Objective: 9.542e+09, f1: 3.822e+07, f2: 4.433e+02, f3: 7.191e+09, f4: 2.312e+09
    Minimizing left matrix<jupyter_text>The clear sky model is accessable as an attribute of the `DataHandler`<jupyter_code>dh.scsf<jupyter_output><empty_output><jupyter_text>The `SCSF` attribute is also a class, with its own plotting functions.<jupyter_code>dh.scsf.plot_measured_clear_matrices(units='W');<jupyter_output><empty_output><jupyter_text>The `DataHandler` instance can access the data in the `SCSF` instance as well, as shown below.<jupyter_code>dh.plot_daily_signals(start_day=150, show_legend=True);<jupyter_output><empty_output><jupyter_text>The fit model can be used for other tasks as well. Below, we use it to find clear *time periods* within the data set.<jupyter_code>dh.find_clear_times()
dh.plot_daily_signals(start_day=150, boolean_mask=dh.boolean_masks.clear_times, mask_label='clear times',
                      show_legend=True);<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.130451563103049, 'fraction_numerical': 0.04399845619451949, 'mean_word_length': 3.844859813084112, 'pattern_counts': {'":': 0, '<': 27, '<?xml version=': 0, '>': 27, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '651517', 'license': 'BSD-2-Clause', 'n_tokens_mistral': 902, 'n_tokens_neox': 806, 'n_words': 232, 'path': 'notebooks/Statistical_Clear_Sky_Fitting_Integration.ipynb', 'repo_name': 'catzzz/solar-data-tools', 'size': '2591'}