<jupyter_start><jupyter_text>AWS IoT Analytics | Building Energy Consumption PredictionApplying model trained from the training job to automatically predict the building energy consumption and Output Dataset to IoT Analytics Dataset.     Set-up: Import Required Notebook Libraries<jupyter_code>#This notebook uses holidays package

try:
    import holidays
    import lightgbm as lgb
except:
    !pip install holidays
    import holidays
    !pip install lightgbm
    import lightgbm as lgb
import pandas as pd
import numpy as np
import boto3
import os
import sys
from sklearn.preprocessing import LabelEncoder
from sklearn.model_selection import KFold
import datetime
import gc
from sklearn.metrics import mean_squared_error

from sklearn.base import BaseEstimator, TransformerMixin
from sklearn.preprocessing import OneHotEncoder, StandardScaler
from sklearn.impute import SimpleImputer
from sklearn.pipeline import FeatureUnion, Pipeline
import warnings

warnings.filterwarnings("ignore", message="numpy.dtype size changed")
warnings.filterwarnings("ignore", message="numpy.ufunc size changed")
warnings.simplefilter(action='ignore', category=FutureWarning)<jupyter_output><empty_output><jupyter_text>Step 1  |  Load Meter-reading and weather Data from IoTAnalytics<jupyter_code># Before actually loading the data we need to set up an IoT Analytics client for accessing datasets.
# create IoT Analytics client
client = boto3.client('iotanalytics')

dataset = "jh_iot_analytics_data_set"
dataset_url = client.get_dataset_content(datasetName = dataset)['entries'][0]['dataURI']<jupyter_output><empty_output><jupyter_text>Now we can get the data location (URL) for the given dataset and start working with the data (In order to need to perform get_dataset_content, you need to grant iot analytics corresponding IAM permission):<jupyter_code># start working with the data
test_df = pd.read_csv(dataset_url,parse_dates=True)
if test_df.empty:
    raise Exception('No data found')
    
# start working with the data
drop_col = ['meter_reading','__dt']
test_df.drop(drop_col, axis=1, inplace=True) # removes unnecessary columns
test_df['timestamp'] = pd.to_datetime(test_df['timestamp'] / 1000., unit='s')
weather_dataset = "iot_analytics_data_set"
weather_dataset_url = client.get_dataset_content(datasetName = weather_dataset)['entries'][0]['dataURI']

# start working with the data
weather_test_df = pd.read_csv(weather_dataset_url,parse_dates=True)
if weather_test_df.empty:
    raise Exception('No data found')
    
# start working with the data
drop_col = ['__dt']
weather_test_df.drop(drop_col, axis=1, inplace=True) # removes unnecessary columns<jupyter_output><empty_output><jupyter_text>Step 2  |  Feature Engineering   (1) WeathertransformerAdded missing time-series data by finding start_date-end_dateThen fill in missed data invweather data, temperature, cloud coverage, due_temperature, sea_level, wind_direction, wind_speed, precip_depth<jupyter_code>from weathertranformer import WeatherTranformer
weather_test_df = WeatherTranformer(True).fit_transform(weather_test_df)<jupyter_output><empty_output><jupyter_text>(2) Smoothing FilterSmooth air and dew temperature<jupyter_code>weather_test_df.dropna(subset=['air_temperature'],inplace=True)
from SGFilter import SGFilterTranformer
weather_test_df = SGFilterTranformer(True).fit_transform(weather_test_df)<jupyter_output><empty_output><jupyter_text>(3) Rolling WindowCalculate min max std within a time window of 24<jupyter_code>from Rollwindow import RollwinTranformer
weather_test_df = RollwinTranformer(True,24).fit_transform(weather_test_df)<jupyter_output><empty_output><jupyter_text>(4) Get Building related metadata from S3 bucket and Merge all datasets together<jupyter_code>bucket = "check-ride-data-explore"
file_name = "input/ashrae-energy-prediction/building_metadata.csv"

s3 = boto3.client('s3') 
# 's3' is a key word. create connection to S3 using default config and all buckets within S3

obj = s3.get_object(Bucket= bucket, Key= file_name) 
# get object and file (key) from bucket

building_df = pd.read_csv(obj['Body']) # 'Body' is a key word

test_df = test_df.merge(building_df, left_on='building_id',right_on='building_id',how='left')

weather_test_df['timestamp'] = pd.to_datetime(weather_test_df['timestamp'])
test_df = test_df.merge(weather_test_df,how='left',left_on=['site_id','timestamp'],right_on=['site_id','timestamp'])<jupyter_output><empty_output><jupyter_text>(5) Numerical FeaturesFeature transform for Numerical Features<jupyter_code>from NumericalEng import NumericalTransformer
test_df = NumericalTransformer(True, True, True, True).fit_transform(test_df)
test_time_df = test_df<jupyter_output><empty_output><jupyter_text>(6) Holidays FeaturesAdd one feature to state if that day is public holiday or not<jupyter_code>from HolidayFea import HolidayTranformer
test_df = HolidayTranformer(True).fit_transform(test_df)<jupyter_output><empty_output><jupyter_text>(7) One hot Encoding for Primary useOne hot encoding for categorical feature<jupyter_code>from LabelEncode import CategoricalTransformer
test_df.dropna(subset=['building_id'],inplace=True)
test_df = CategoricalTransformer().fit_transform(test_df)<jupyter_output><empty_output><jupyter_text>Step 3  |  Generate Prediction  Setup test_features and load model for prediction<jupyter_code>import joblib

gbm_pickle = joblib.load('lgb.pkl')
results = gbm_pickle.predict(test_df, num_iteration=gbm_pickle.best_iteration) 
pred_df = pd.DataFrame({'pred':results})
pred_df.loc[pred_df.pred < 0, 'pred'] = 0

test_final_df = pd.concat([test_time_df,pred_df], axis=1, sort=False)<jupyter_output><empty_output><jupyter_text>Step 4  |  Create Dataset to IoT Dataset<jupyter_code>from io import StringIO
from datetime import datetime
bucket='check-ride-data-explore'
csv_key='prediction_result.csv'
prefix = 'energy_prediction' + datetime.now().strftime('%Y-%m-%d') + "/"


s3 = boto3.resource('s3')
csv_buffer = StringIO()
test_final_df.to_csv(csv_buffer)
s3.Object(bucket, prefix + csv_key).put(Body = csv_buffer.getvalue())<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09129650876905425, 'fraction_numerical': 0.006556302245533519, 'mean_word_length': 8.04, 'pattern_counts': {'":': 0, '<': 50, '<?xml version=': 0, '>': 49, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '455257', 'license': 'MIT-0', 'n_tokens_mistral': 1991, 'n_tokens_neox': 1915, 'n_words': 470, 'path': 'sagemaker-notebook/jh_check_ride_iot_inference.ipynb', 'repo_name': 'Julia-Bobo-Hu/IoTAnalytics-Realtime-Ingestion-Inference', 'size': '6101'}