<jupyter_start><jupyter_text>Viewing the time series in a plot<jupyter_code>dataPath = 'Data/2016-2019 (All data)/2016-2019_sensorData.csv'
airQualityData=pd.read_csv(dataPath, header=15,sep=';').rename(columns={'Start':'Start','Slut':'Stop'})
airQualityData.rename(columns = lambda x: re.sub('NOX.*','NOX',x), inplace = True)
airQualityData.rename(columns = lambda x: re.sub('PM10.*','PM10',x), inplace = True)
airQualityData.rename(columns = lambda x: re.sub('PM2.5.*','PM2_5',x), inplace = True)
airQualityData.rename(columns = lambda x: re.sub('NO2.*','NO2',x), inplace = True)
airQualityData.head(1)

airQualityData['Start'] = pd.to_datetime(airQualityData['Start']) #converts the start column into a proper datatime type in order to plot<jupyter_output><empty_output><jupyter_text>View percentage of available data<jupyter_code>total = airQualityData.isnull().sum().sort_values(ascending = False)
total = total[airQualityData.isnull().sum().sort_values(ascending = False) != 0]
percent =  100 - total / len(airQualityData) * 100
percent[airQualityData.isnull().sum().sort_values(ascending = False) != 0].plot.bar(figsize = (20,10))<jupyter_output><empty_output><jupyter_text>1- Autocorrelation<jupyter_code>resampled=airQualityData.groupby(['Start']).mean().resample('1m').mean().ffill()#resampling into 4 hours with mean taken of the 4 hours values
fig , ax =  plt.subplots(figsize=(20,15))
sns.lineplot(data=resampled['PM10'],ax=ax)
from statsmodels.graphics.tsaplots import plot_acf
from statsmodels.graphics.tsaplots import plot_pacf
plot_pacf(resampled['PM10'])
airQualityData.groupby(['Start']).mean().resample('4h').mean()<jupyter_output><empty_output><jupyter_text>2-Correlation<jupyter_code>processed_df=airQualityData.drop(columns=['Start','Stop','Station Name','Classification','Longitude','Latitude','National Station Code'])<jupyter_output><empty_output><jupyter_text>With outliers and missing data<jupyter_code>fig , ax =plt.subplots(figsize=(10,10))
sns.heatmap(processed_df.corr(), annot=True)<jupyter_output><empty_output><jupyter_text>--------------------------------------------------------------------------------------------- no need for what is below pandas corr() doesn't get affected by NaN With outliers and no missing data for black carbon<jupyter_code>fig , ax =plt.subplots(figsize=(10,10))
sns.heatmap(processed_df[processed_df['Black Carbon'].notnull()].corr(), annot=True)<jupyter_output><empty_output><jupyter_text>With outliers and no missing data for O3<jupyter_code>fig , ax =plt.subplots(figsize=(10,10))
sns.heatmap(processed_df[processed_df['O3'].notnull()& processed_df['PM10'].notnull()].corr(), annot=True)<jupyter_output><empty_output><jupyter_text>With outliers and no missing data for CO<jupyter_code>fig , ax =plt.subplots(figsize=(10,10))
sns.heatmap(processed_df[processed_df['CO'].notnull() & processed_df['PM10'].notnull()].corr(), annot=True)<jupyter_output><empty_output><jupyter_text>---------------------------------------------------------------------------------------------  Outliers removed<jupyter_code>Q1 = processed_df.quantile(0.25)
Q3=  processed_df.quantile(0.75)
IQR= Q3-Q1
print(IQR)
processed_df_Outlier= processed_df[~((processed_df < (Q1 - 1.5 * IQR)) |(processed_df > (Q3 + 1.5 * IQR))).any(axis=1)]
fig , ax =plt.subplots(figsize=(10,10))
sns.heatmap(processed_df_Outlier.corr(), annot=True)<jupyter_output>NO2             26.400
NOX as NO2      54.100
PM2.5            4.900
CO               0.150
Black Carbon     0.682
O3              31.200
PM10            14.400
dtype: float64
{'fraction_non_alphanumeric': 0.202079820123665, 'fraction_numerical': 0.03906689151208544, 'mean_word_length': 5.063032367972743, 'pattern_counts': {'":': 0, '<': 37, '<?xml version=': 0, '>': 37, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '289720', 'license': 'MIT', 'n_tokens_mistral': 1352, 'n_tokens_neox': 1202, 'n_words': 199, 'path': 'feature-selection-all-data.ipynb', 'repo_name': 'jadali17/Air-Quality-Prediction', 'size': '3558'}