<jupyter_start><jupyter_text>Data Loading  Loading the california housing dataset CSV```We collected information on the variables using all the block groups in California from the 1990 Census. In this sample a block group on average includes 1425.5 individuals living in a geographically co mpact area. Naturally, the geographical area included varies inversely with the population density. W e computed distances among the centroids of each block group as measured in latitude and longitude. W e excluded all the block groups reporting zero entries for the independent and dependent variables. T he final data contained 20,640 observations on 9 variables. The dependent variable is ln(median house value).```<jupyter_code>housing = pd.read_csv("data/housing.csv")
housing.head()
housing.shape
housing.ocean_proximity.value_counts()
continuous_dependent = housing.columns[:-2]
housing[continuous_dependent].hist();
housing[continuous_dependent].hist(bins="auto")
plt.tight_layout()
housing.population.sort_values().tail()<jupyter_output><empty_output><jupyter_text>ExerciseCome up with a way to visualize the population data which can help us inspect the outliers.<jupyter_code># your solution here
# my solution
# . . .<jupyter_output><empty_output><jupyter_text>ExerciseWe can do better!<jupyter_code># your solution here
# my solution
# . . .<jupyter_output><empty_output><jupyter_text>ExerciseCreate a 2 x 3 subplot grid showing housing_median_age, total_rooms, total_bedrooms, population, households and median_incomeeach as a hexbin.  Dealing with missing values a little bit<jupyter_code>housing.isnull().sum()
housing_nonull = housing.dropna().copy()<jupyter_output><empty_output><jupyter_text>scatter matrix / pair plot<jupyter_code>pd.plotting.scatter_matrix(housing_nonull.iloc[:, 2:-2]);
pd.plotting.scatter_matrix(housing_nonull.iloc[:, 2:-2], c=housing.ocean_proximity);
# error
unique_proximity, ocean_proximity_int = np.unique(housing_nonull.ocean_proximity, return_inverse=True)
ocean_proximity_int
unique_proximity
pd.plotting.scatter_matrix(housing_nonull.iloc[:, 2:-2], c=ocean_proximity_int);
plt.matshow(np.arange(4).reshape(1, 4))
plt.xticks(range(4), unique_proximity)
import seaborn.apionly as sns
sns.pairplot(housing_nonull.iloc[:, 2:], hue='ocean_proximity')<jupyter_output><empty_output><jupyter_text>Exercise- Confirm the coloring of the scatter matrix makes sense by plotting latitude vs longitude colored by ocean distance.- Are the two outliers the same for all the plots?<jupyter_code># solution here<jupyter_output><empty_output><jupyter_text>Exploring the target (dependent variable)<jupyter_code>housing_nonull.plot('median_income', 'median_house_value', kind='scatter', alpha=.1)<jupyter_output><empty_output><jupyter_text>ExerciseDo a scatter plot of all the continuous dependent variables against the median house value.<jupyter_code># solution<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09162663006177076, 'fraction_numerical': 0.01029512697323267, 'mean_word_length': 7.833333333333333, 'pattern_counts': {'":': 0, '<': 33, '<?xml version=': 0, '>': 33, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '318013', 'license': 'MIT', 'n_tokens_mistral': 956, 'n_tokens_neox': 897, 'n_words': 250, 'path': '03 - Exploration with Pandas, Matplotlib and Seaborn.ipynb', 'repo_name': 'lbias/jupyter-ml', 'size': '2914'}