<jupyter_start><jupyter_text>Tesla Energy Time Series Data Challenge   > 💁‍ Author: <NAME> > 📧 Email: <EMAIL>    > 📌 GitHub Link: https://github.com/vickymei/tesla_energy_project  Summary This project focuses on solving the problem of identifying malfunctioning Energy Production Sites by 1. Built a ETL pipeline to collect signal data through 2 Energy Realtime Data API Endpoints2. Collected 29 hours historical data 3. Provided a Plotly Dash dashboard for visualizing data from 42 sites4. Proposed solutions for more in-depth Anomaly Detection This Notebook will explain this project in the following order: Part 1. ETL Pipeline Building Part 2. Data Processing & Plotly VisualizationPart 3. Analysis and Insights on Malfunctioning SitesPart 4. More Anomaly Detection Thoughts  Part 1. ETL Pipeline Building  This section aims at building data pipeline and collecting signal data using pipelinel.  1. There will be one json file generated every minute. After running the script for roughly 29 hours, there are 29 hours * 60 = 1740 files in data/ directory2. Json files are named by timestamp of generation time3. Try Except is utilized for possible api call failure ![jsons.JPG](attachment:jsons.JPG) ```pythondef get_sites_from_api():    """Return list of all existing sites."""    load_dotenv()    json_data = requests.get(        "https://te-data-test.herokuapp.com/api/sites?token="+os.environ.get("api-token")).json()    sites = json_data["sites"]    return sitesdef get_signals_from_api(site):    """Return site signal data."""    load_dotenv()    json_data = requests.get(        "https://te-data-test.herokuapp.com/api/signals?token="+os.environ.get("api-token")+f"&site={site}").json()    return json_data``` ```pythondef main():    """Write signal data to local file system in json format every minute. """    sites = get_sites_from_api()    site_data = []    timestamp_now = datetime.now()    timestamp_file_name = f"signal-{timestamp_now.year}-{timestamp_now.month}-{timestamp_now.day}-{timestamp_now.hour}-{timestamp_now.minute}-{timestamp_now.second}.json"    for site in sites:        try:            site_data.append(get_signals_from_api(site))        except:            error_data = {                'signals': {'SITE_SM_batteryInstPower': None, 'SITE_SM_siteInstPower': None, 'SITE_SM_solarInstPower': None},                'site': site, 'timestamp': None}            site_data.append(error_data)        with open(f"data/{timestamp_file_name}", 'w') as f:            json.dump(site_data, f)    return 'success'``` Note: By setting sleeping time and calling api at a relatively low frequency (per minute), I didn't notice data loss due to API issues during data collection.   Part 2. Data Processing & Plotly Visualization   Data Processing<jupyter_code>import json
import pandas as pd
import os
import missingno as msno
import seaborn as sns
import warnings
import numpy as np
import pandas as pd
from pandas.core.common import SettingWithCopyWarning
warnings.simplefilter(action="ignore", category=SettingWithCopyWarning)
%matplotlib inline
pd.set_option('display.max_rows', 20)
# Read all json files and print statements about number of records
path_to_json = 'data/'
json_files = [f'data/{pos_json}' for pos_json in os.listdir(path_to_json) if pos_json.endswith('.json')]
print(f"There are {len(json_files)} json files in total.")
js = []
for fn in json_files:
    with open(fn, 'r') as f:
        js = js + json.load(f)
print(f"There are {len(js)} signal from 42 sites in total.")
# Transform dataframe for later analysis 
df = pd.DataFrame(js)
df.head(5)
# Transform dataframe for later analysis 
df_main = pd.concat([df.drop(['signals'], axis=1), df['signals'].apply(pd.Series)], axis=1)
df_main = df_main[['site', 'timestamp', 'SITE_SM_solarInstPower']]
df_main['timestamp'] = pd.to_datetime(df_main['timestamp'])
df_main.head(5)
# Put all sites' name into a list 
sites =['134a3fa6', '8d9fed87', '5688ed10', '2b33a48d', '07333ad0', '38c8ae33',
         'adc42b19', 'e9ba8cec', 'e12c2148', '4b78aae6', 'e724ca65', '135433c1',
         '90606897', '02ebf5c7', 'c8eb2d3d', '2b98cbdd', '39146e59', '55af2f9b',
         '28731623', '3193e230', 'e6bcf7cf', '7da0acb7', 'c18b6195', '20abb173',
         'f34b386a', 'f7f9ac09', '5fc96249', '82c74b9e', 'b255f7ad', '61bff705',
         '619fd2b9', '260f359a', '4faff963', '499a251d', 'dfc6fdf5', '64e1616f',
         '93c8a2c1', 'eec02ec5', '90791ae9', '49b6c0dd', 'd0926969', '7435e9d3']
# Ignore the minor 1 or 2 sec difference for all sites signal from one api call, using timestamp of site 134a3fa6
df_timestamp = df_main[df_main['site']=="134a3fa6"].reset_index(drop=True)[['timestamp']]

# We are investigating solar production signals, so variable 'SITE_SM_solarInstPower' is the target variable  
for site in sites:
    df_site = df_main[df_main['site']==site].reset_index(drop=True)
    df_timestamp.loc[:, site] = df_site['SITE_SM_solarInstPower'].tolist()

# timestamp dataframe is saved in csv format for plotly dashboard building    
df_timestamp.to_csv('timestamp.csv')
df_timestamp.head(5)<jupyter_output><empty_output><jupyter_text>Plotly Visualization  The screenshot below is a simple plotly dashboard to visualize all sites signals. You could select different site code to see data coming from any of the 42 sites. In order to run the app on your local machine, you need to run `python viz.py` in your terminal and visit http://127.0.0.1:8050/ in your web browser. ![2.JPG](attachment:2.JPG) >The Code snippet below creates the simple dashboard above.  ```pythonapp = dash.Dash(__name__)sites = ['134a3fa6', '8d9fed87', '5688ed10', '2b33a48d', '07333ad0', '38c8ae33',         'adc42b19', 'e9ba8cec', 'e12c2148', '4b78aae6', 'e724ca65', '135433c1',         '90606897', '02ebf5c7', 'c8eb2d3d', '2b98cbdd', '39146e59', '55af2f9b',         '28731623', '3193e230', 'e6bcf7cf', '7da0acb7', 'c18b6195', '20abb173',         'f34b386a', 'f7f9ac09', '5fc96249', '82c74b9e', 'b255f7ad', '61bff705',         '619fd2b9', '260f359a', '4faff963', '499a251d', 'dfc6fdf5', '64e1616f',         '93c8a2c1', 'eec02ec5', '90791ae9', '49b6c0dd', 'd0926969', '7435e9d3']app.layout = dash.html.Div([    dash.html.H4('Tesla Energy - Solar Power Production Daily Monitor'),    dash.dcc.Graph(id="time-series-chart"),    dash.html.P("Select Site Code: "),    dash.dcc.Dropdown(        id="ticker",        options=sites,        value="134a3fa6",        clearable=False,    ),])@app.callback(    dash.Output("time-series-chart", "figure"),    dash.Input("ticker", "value"))def display_time_series(ticker):    df1 = pd.read_csv('timestamp.csv')    fig = px.bar(df1, x='timestamp', y=ticker)    return fig```  Part 3. Analysis and Insights on Malfunctioning Sites  Sites return No Signals (Missing Values)<jupyter_code>df_missing = df_timestamp.drop('timestamp', axis=1)
msno.matrix(df_missing, color=(1, 0.38, 0.27))<jupyter_output><empty_output><jupyter_text>Sites return negative signals (Solar Production should always be positive)<jupyter_code>df_negative = df_missing.fillna(0)
df_negative[df_negative < 0] = np.nan
msno.matrix(df_negative, color=(0.27, 0.52, 1.0))<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1288069293098631, 'fraction_numerical': 0.07069013690975133, 'mean_word_length': 4.782714054927302, 'pattern_counts': {'":': 0, '<': 16, '<?xml version=': 0, '>': 19, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 1, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '286697', 'license': 'Apache-2.0', 'n_tokens_mistral': 2692, 'n_tokens_neox': 2416, 'n_words': 677, 'path': 'analysis.ipynb', 'repo_name': 'vickymei/tesla_energy_project', 'size': '7173'}