<jupyter_start><jupyter_text>Exploratory Data AnalysisThe goal is to try and gain insights from the data prior to modeling. Recall that in the first step assumptions about the data were to be identified and explored; this is a good time to more deeply investigate these assumptions. Human experts can be of particular use in this step, answering questions about correlations which may not be obvious to the machine learning practitioner. Studying features and their characteristics is done here, as is general visualization of features and their values (think of how much easier it is, for example, to quickly identify outliers by box plot than by numerical interrogation). Documenting the findings of your exploration for later use is good practice.<jupyter_code>import pandas as pd
data = pd.read_csv('housing.csv')
data.head()<jupyter_output><empty_output><jupyter_text>Estimates of Locationan estimate on where most of the data is located (i.e. central tendencies) Meanaverage value Medianthe middle value when the data is sorted (50th percentile)<jupyter_code>from scipy.stats import trim_mean 

total_rooms = data['total_rooms']

mean = total_rooms.mean() #average value
trimmed_mean = trim_mean(total_rooms, 0.1) #10% of the values dropped from both ends
median = total_rooms.median()

print(f'[Total Rooms] \nMean: {mean} \nTrimmed Mean(10%): {trimmed_mean} \nMedian: {median}')<jupyter_output>[Total Rooms] 
Mean: 2635.7630813953488 
Trimmed Mean(10%): 2294.5578367248063 
Median: 2127.0<jupyter_text>Estimates of Variabilitymeasures how tightly clustered os pread out the data is Deviation Variance Range Percentile Inter-quartile Rangethe difference between the 75th percentile and 25th percentile<jupyter_code>from statsmodels import robust

standard_deviation = total_rooms.std()
median_absolute_deviation = robust.scale.mad(total_rooms)
quantile = total_rooms.quantile([0.05, 0.25, 0.5, 0.75, 0.95])

print(f'[Total Rooms] \nSTD: {standard_deviation} \nMAD: {median_absolute_deviation} \nQuantile: \n{quantile}')<jupyter_output>[Total Rooms] 
STD: 2181.615251582787 
MAD: 1181.6339681489646 
Quantile: 
0.05     620.95
0.25    1447.75
0.50    2127.00
0.75    3148.00
0.95    6213.20
Name: total_rooms, dtype: float64<jupyter_text>Correlation<jupyter_code>data.corr()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07748917748917748, 'fraction_numerical': 0.0632034632034632, 'mean_word_length': 5.857566765578635, 'pattern_counts': {'":': 0, '<': 15, '<?xml version=': 0, '>': 15, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '555870', 'license': 'CC0-1.0', 'n_tokens_mistral': 831, 'n_tokens_neox': 710, 'n_words': 253, 'path': '01_End2End_ML/01_Data_Exploration.ipynb', 'repo_name': 'data-science-tsb/machine-learning-notes', 'size': '2310'}