<jupyter_start><jupyter_text>Storytelling Data Visualization LabIn this lab you'll use a dataset called `housing_prices.csv` which contains the sales data of houses. The dataset and descriptions of the columns are available from [Kaggle](https://www.kaggle.com/c/house-prices-advanced-regression-techniques/data). For your convenience, you can review the descriptions of the data columns from [here](data-description.txt).Pretend you are a data analyst at an investment company where the board decided to make investments in real estates. Your boss asked you to analyze this housing sales dataset and present to the investment managers on **what features of houses are strong indicators of the final sale price**. You need to present your findings in intuitive ways so that the investment managers understand where your conclusions come from. You will use the appropriate data visualization graphs to tell your stories.  Challenge 1 - Understanding the DatasetAfter receiving the data and clarifying your objectives with your boss, you will first try to understand the dataset. This allows you to decide how you will start your research in the next step. First, import the basic libraries and the dataset.<jupyter_code>import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns

df = pd.read_csv('housing_prices.csv')<jupyter_output><empty_output><jupyter_text>As a routine before analyzing a dataset, print the first few rows of the dataset<jupyter_code>df.head()<jupyter_output><empty_output><jupyter_text>You find the dataset has 81 columns which are a lot.  Since the column `Id` is meaningless in our data visualization work, let's drop it<jupyter_code># your code here<jupyter_output><empty_output><jupyter_text>You care about missing values. If a column has too many missing values, it is not reliable to use it to predict sales price. In the cell below, calculate the percentage of missing values for each column. Make a table containing the column name and the percentage of missing values. Print the columns where more than 20% of values are missing. An example of your output looks like:![Missing Values](missing-values.png)[This reference](https://stackoverflow.com/questions/51070985/find-out-the-percentage-of-missing-values-in-each-column-in-the-given-dataset) can help you make the missing values table.<jupyter_code># your code here<jupyter_output><empty_output><jupyter_text>Drop the columns you find that have more than 20% missing values.After dropping, check the shape of your dataframes. You should have 75 columns now.<jupyter_code># your code here<jupyter_output><empty_output><jupyter_text>Since you're asked to analyze sale prices, first let's see if the sale prices (column `SalePrice`) has a normal distribution. This is important because normally distributed data can be better represented with mathematical models. In the cell below, use the propriate graph to visualize the shape of distribution of the sale prices. Then explain what you find from the graph about data distribution.<jupyter_code># your code here

# your comment here<jupyter_output><empty_output><jupyter_text>Bonus Challenge 1 - Adjust Data DistributionIf you used the correct method in the previous step, you should have found the data distribution is skewed to the left. In order to improve your data visualization in the next steps, you can opt to adjust the `SalePrice` column by applying a mathematical function to the values. The goal is to produce a bell-shape normal distribution after applying the mathematical function to the sale price.*This technique is optional in data visualization but you'll find it useful in your future machine learning analysis.* In the cell below, adjust the `SalePrice` column so that the data are normally distributed.Try applying various mathematical functions such as square root, power, and log to the `SalePrice` column. Visualize the distribution of the adjusted data until you find a function that makes the data normally distributed. **Create a new column called `SalePriceAdjusted` to store the adjusted sale price.**[This reference](https://trainingdatascience.com/workshops/histograms-and-skewed-data/) shows you examples on how to adjust skewed data.<jupyter_code># your code here<jupyter_output><empty_output><jupyter_text>Challenge 2 - Exploring Data with Common SenseNow that we have a general understanding of the dataset, we start exploring the data with common sense by means of data visualization. Yes, in data analysis and even machine learning you are often required to use common sense. You use your common sense to make a scientific guess (i.e. hypothesis) then use data analytics methods to test your hypothesis.This dataset is about housing sales. According to common sense, housing prices depend on the following factors:* **Size of the house** (`GrLivArea`, `LotArea`, and `GarageArea`).* **Number of rooms** (`BedroomAbvGr`, `KitchenAbvGr`, `FullBath`, `HalfBath`, `BsmtFullBath`, `BsmtHalfBath`).* **How long the house has been built or remodeled** (`YearBuilt` and `YearRemodAdd`).* **Neighborhood of the house** (`Neighborhood`). In this challenge, use the appropriate graph type to visualize the relationships between `SalePrice` (or `SalePriceAdjusted`) and the fields above. Note that:* Transform certain columns in order to visualize the data properly based on common sense. For example:    * Visualizing how the number of half bathrooms affected the sale price probably does not make sense. You can create a new column to calculate the total number of bathrooms/rooms then visualize with the calculated number.    * `YearBuilt` and `YearRemodAdd` are year numbers not the age of the house. You can create two new columns for how long the house has been built or remodeled then visualize with the calculated columns.* Make comments to explain your thinking process.<jupyter_code># your code here
# add cells as needed<jupyter_output><empty_output><jupyter_text>Bonus Challenge 2 - Exploring Data with Correlation HeatmapNow you have explored data visualizations with certain fields based on common sense. In the dataset there are many other fields that you are not sure whether they are important factors for the sale price. What is the best way to explore those fields without investigating them individually?Making scatter matrix is not an option here because there are too many fields which makes it extremely time consuming to create scatter matrix. One option you have is to create a heatmap. Heatmaps are much less expensive to create than scatter matrixes. You can use heatmaps to visualize the pairwise correlations between each two variables.Here is a [reference](https://seaborn.pydata.org/examples/many_pairwise_correlations.html) you can use to learn how to creat the pairwise correlation heatmap. Your heatmap should look like below:![Corr Heatmap](heatmap.png)<jupyter_code># your code here<jupyter_output><empty_output><jupyter_text>In your heatmap, you can easily identify the highly correlated (either positively or negatively) variables by looking for the grids with darker colors.  In the cell below, summarize what variables are highly correlated to the sale price?<jupyter_code># your comment here<jupyter_output><empty_output><jupyter_text>Challenge 3 - Present Your StoriesNow based on your findings from the explorations, summarize and present your stories. Present the top 5 factors that affect the sale price.Use the following format to present each factor:1. A title line about the factor.1. No more than 3 sentences to describe the relationship between the factor and the sale price.1. Support your point with the appropriate graph.<jupyter_code># your responses here
# add cells as needed<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.053140719721398166, 'fraction_numerical': 0.003353540564942603, 'mean_word_length': 5.499580888516346, 'pattern_counts': {'":': 0, '<': 45, '<?xml version=': 0, '>': 45, 'https://': 4, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '664046', 'license': 'MIT', 'n_tokens_mistral': 1933, 'n_tokens_neox': 1841, 'n_words': 1042, 'path': 'module-2/Storytelling-Data-Viz/your-code/main.ipynb', 'repo_name': 'Skye-FY/daft-miami-0120-labs', 'size': '7753'}