<jupyter_start><jupyter_text>Step 1: Basic Exploratory Data Analysis<jupyter_code>import pandas as pd
import matplotlib.pyplot as plt
import numpy as np
%matplotlib inline<jupyter_output><empty_output><jupyter_text>Basic per device metrics and Data Quality<jupyter_code>df = pd.read_csv("../data/device_failure.csv")
df.loc[:,'date'] = pd.to_datetime(df['date'])
df.head()
df.isnull().describe()
df["date"].describe()<jupyter_output><empty_output><jupyter_text>Good. At least no null values<jupyter_code>total_obs = df.shape[0]
unique_dev = df.device.unique().shape[0]
print ("Total Observations        : %s"%total_obs)
print ("Unique Devices            : %s"%unique_dev)
print ("Unique devices that failed: %d"%df[df['failure'] == 1].device.unique().size)
print ("Avg number of observations per device: %f"%(float(total_obs)/unique_dev))<jupyter_output>Total Observations        : 124494
Unique Devices            : 1168
Unique devices that failed: 106
Avg number of observations per device: 106.587329<jupyter_text>Distribution of number of readings for devices NumDevices/day that were observed --> points to systemic ingestion/sampling problem<jupyter_code>df.groupby(["device"]).count()["attribute1"].hist(bins=20)
plt.xlabel("Num of days (#number of readings)")
plt.ylabel("Num of devices")
df.groupby(["date"]).count()["device"].plot.line()
plt.ylabel("#devices")<jupyter_output><empty_output><jupyter_text>See if there are gaps in dates for sensor data. See how bad it is<jupyter_code>df_date_min = df.groupby("device").min()["date"]
df_date_max = df.groupby("device").max()["date"]
df_date_count = df.groupby("device").count()["date"]
df2 = pd.concat([df_date_min, df_date_max, df_date_count], axis=1)
df2.columns = ["start", "end", "xcount"]
df2.loc[:,"diff_dates"] = (df2.end - df2.start).dt.days+1
df2.loc[:,'deltas'] = df2.diff_dates - df2.xcount
df2.loc[:,'missing'] = df2.deltas.map(lambda x: x != 0)
df2.groupby("missing").count()["start"]<jupyter_output><empty_output><jupyter_text>At least majority of cases, readings are continuous For missing data, how many days are we missing?<jupyter_code>missing = df2[df2["missing"] == True]
missing.loc[:,"miss_pct"] = (missing["deltas"]/missing["diff_dates"])*100
missing.miss_pct.describe()
missing.miss_pct.hist()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.161822125813449, 'fraction_numerical': 0.020824295010845987, 'mean_word_length': 6.843537414965986, 'pattern_counts': {'":': 0, '<': 24, '<?xml version=': 0, '>': 25, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '296779', 'license': 'Apache-2.0', 'n_tokens_mistral': 841, 'n_tokens_neox': 802, 'n_words': 167, 'path': 'eda/Step1_EDA.ipynb', 'repo_name': 'viswanath-puttagunta/Amazon_2017_submit', 'size': '2305'}