<jupyter_start><jupyter_text>STEP 02: Data exploration, Solar panel production Version 001In this notebook, the data exploration of the **solar panel production** is covered. There are three data sources that are being explored:1. Weather: Information on temperature, cloud coverage, precipitation, etc at the location using the DarkSkyAPI2. Sun position: Information on the position of the sun in the sky at the location using pysolar.solar3. Solar panel production: Information on the amount of energy produced by the solar panels at the location<jupyter_code>import datetime
import numpy as np
import pandas as pd
import requests
import re
import json
import os
from dateutil import tz
import matplotlib.pyplot as plt
import seaborn as sns
%matplotlib inline
sns.set()<jupyter_output><empty_output><jupyter_text>Load data<jupyter_code>path_to_files = './data/production/'
data = pd.read_csv(f'{path_to_files}/solar_production.data', index_col=0) # make it .data so that it is not csv
data['Time'] = pd.to_datetime(data['Time'])
data = data.set_index('Time')
data.head()<jupyter_output><empty_output><jupyter_text>Describe data<jupyter_code>data.info()
data.describe()<jupyter_output><empty_output><jupyter_text>Production is in Watt.  Visualize<jupyter_code>fig = plt.figure(figsize=(10,6))
plt.plot(data['Production'])
plt.title('Production, full period')
plt.show()
fig = plt.figure(figsize=(10,6))
plt.plot(data.loc['2017-11-01':'2017-11-02','Production'])
plt.title('Production, 48 hours')
plt.show()<jupyter_output><empty_output><jupyter_text>Check that it is 96 entries per day, which is every 15 minutes for 24 hours<jupyter_code>data_agg = data.reset_index()
data_agg['Day'] = data_agg['Time'].map(lambda x: f'{x.year}-{x.month}-{x.day}')
data_agg.groupby('Day').agg(['count']).min()
data_agg.groupby('Day').agg(['count']).max()<jupyter_output><empty_output><jupyter_text>Aggregate by day<jupyter_code># production data
path_to_files = './data/production/'
data_prod = pd.read_csv(f'{path_to_files}/solar_production.data', index_col=0) # make it .data so that it is not csv
data_prod['Time'] = pd.to_datetime(data_prod['Time'])
data_prod = data_prod.set_index('Time')

data_prod['Day'] = data_prod.index.map(lambda x: f'{x.year}-{x.month}-{x.day}')
data_prod = data_prod.groupby('Day')['Production'].agg(['sum'])
data_prod.columns = ['Production_sum']

data_prod =  data_prod.reset_index()
data_prod['Day'] = pd.to_datetime(data_prod['Day'])
data_prod = data_prod.sort_values(by=['Day'])
data_prod = data_prod.set_index('Day')
data_prod.head()
fig = plt.figure(figsize=(10,6))
plt.plot(data_prod['Production_sum'])
plt.title('Production_sum, full period')
plt.show()
fig = plt.figure(figsize=(10,6))
plt.plot(data_prod.loc['2017-11-01':'2017-11-10','Production_sum'])
plt.title('Production_sum, 48 hours')
plt.show()
plt.boxplot(data_prod['Production_sum'])
plt.show()
plt.hist(data_prod['Production_sum'], bins=3)
plt.show()
data_prod.describe()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.14964847673250753, 'fraction_numerical': 0.02176096417810512, 'mean_word_length': 7.635838150289017, 'pattern_counts': {'":': 0, '<': 25, '<?xml version=': 0, '>': 25, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '398104', 'license': 'MIT', 'n_tokens_mistral': 1106, 'n_tokens_neox': 1033, 'n_words': 229, 'path': 'forecasting/02.data_exp.solar_panel_production.001.ipynb', 'repo_name': 'kromerh/solarAnalytics', 'size': '2987'}