<jupyter_start><jupyter_text>Load and visualize data  Imports<jupyter_code>import os
import numpy as np
import sklearn as skl
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
# better visualization for long outputs
from IPython.core.display import HTML
display(HTML("<style>pre { white-space: pre !important; }</style>"))<jupyter_output><empty_output><jupyter_text>Load data<jupyter_code>data_path = '/home/lorenzo/skl-repo/0_data/california_housing.csv'
df = pd.read_csv(data_path)
df.head()<jupyter_output><empty_output><jupyter_text>Insights on the data<jupyter_code>df.describe()
df.info()
df['ocean_proximity'].value_counts()
df.columns
df.isna().sum()
df.dtypes<jupyter_output><empty_output><jupyter_text>Change dtype of columns<jupyter_code>df = df.astype({'total_rooms': 'int32',
                'population': 'int32',
                'households': 'int32',
                'ocean_proximity': 'category'})
df.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 20640 entries, 0 to 20639
Data columns (total 10 columns):
 #   Column              Non-Null Count  Dtype   
---  ------              --------------  -----   
 0   longitude           20640 non-null  float64 
 1   latitude            20640 non-null  float64 
 2   housing_median_age  20640 non-null  float64 
 3   total_rooms         20640 non-null  int32   
 4   total_bedrooms      20433 non-null  float64 
 5   population          20640 non-null  int32   
 6   households          20640 non-null  int32   
 7   median_income       20640 non-null  float64 
 8   median_house_value  20640 non-null  float64 
 9   ocean_proximity     20640 non-null  category
dtypes: category(1), float64(6), int32(3)
memory usage: 1.2 MB<jupyter_text>Visualization<jupyter_code>%matplotlib inline
df.hist(bins=50, figsize=(15,10))
df.plot(kind='scatter', x='longitude', y='latitude')
df.plot(kind='scatter', x='longitude', y='latitude', alpha=0.15)<jupyter_output><empty_output><jupyter_text>We can make a multidimensional plot leveraging color and other features:* x-axis for longitude* y-axis for latitude* circle radius for population* color for house prices<jupyter_code>df.plot(kind='scatter', x='longitude', y='latitude',
        s=df['population']/100, label='population',
        c='median_house_value', cmap=plt.get_cmap('jet'), colorbar=True,
         alpha=0.3, figsize = (10,7))
plt.legend()<jupyter_output><empty_output><jupyter_text>Correlations<jupyter_code>corr_matrix = df.corr()
corr_matrix
plt.figure(figsize=(10,8))
ax = sns.heatmap(
    corr_matrix, 
    vmin=-1, vmax=1, center=0,
    cmap=sns.diverging_palette(20, 220, n=200),
    square=True
)
ax.set_xticklabels(
    ax.get_xticklabels(),
    rotation=45,
    horizontalalignment='right'
)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12567132116004295, 'fraction_numerical': 0.05012531328320802, 'mean_word_length': 3.9892857142857143, 'pattern_counts': {'":': 0, '<': 31, '<?xml version=': 0, '>': 31, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '613574', 'license': 'Apache-2.0', 'n_tokens_mistral': 1088, 'n_tokens_neox': 973, 'n_words': 202, 'path': '1_load_visualize_data/load_visualize_data.ipynb', 'repo_name': 'Lorenzo-Giardi/skl-repo', 'size': '2793'}