<jupyter_start><jupyter_text>House Prices Analysis  Objective of the study is to perform EDA for the variables which will help later to build model later to predict the area which might get affected.   Calling libraries<jupyter_code>import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
import numpy as np
from scipy.stats import norm
from sklearn.preprocessing import StandardScaler
from scipy import stats
import warnings
warnings.filterwarnings('ignore')
%matplotlib inline<jupyter_output><empty_output><jupyter_text>Reading in data<jupyter_code>df_train = pd.read_csv('house_train.csv')
df_train.columns
df_train.info()
df_train.head()<jupyter_output><empty_output><jupyter_text>Looking at the number of rows and columns using shape function¶<jupyter_code>df_train.shape
df_train['SalePrice'].describe()<jupyter_output><empty_output><jupyter_text>Histogram for Saleprice<jupyter_code>sns.distplot(df_train['SalePrice'],bins=100);<jupyter_output><empty_output><jupyter_text>skewness and kurtosis of Saleprice<jupyter_code>Skewness - Measure of relative symmetry.The larger its absolute value the more asymmetric the distribution.

1. Zero indicates symmetry. 

2. Positive values indicate a long right tail

3. Negative values indicate a long left tail.<jupyter_output><empty_output><jupyter_text>print("Skewness: %f" % df_train['SalePrice'].skew())<jupyter_code>Kurtosis - Measure of the relative peakedness of a distribution.

1. K = 3 indicates a normal “bellshaped” distribution.

2. K < 3 indicates a flatter than a normal distribution with shorter tails.

3. K > 3 indicates a more peaked than a normal distribution with longer tails.<jupyter_output><empty_output><jupyter_text>print("Kurtosis: %f" % df_train['SalePrice'].kurt())<jupyter_code># Let us do some bivariate analysis to see which variables might be good for predicting sales price

# box plot overallqual/saleprice<jupyter_output><empty_output><jupyter_text>var var = 'OverallQual'data = pd.concat([df_train['SalePrice'], df_train['OverallQual']], axis=1)f, ax = plt.subplots(figsize=(8, 6))fig = sns.boxplot(x='OverallQual', y="SalePrice", data=data)fig.axis(ymin=0, ymax=800000); var = 'YearBuilt'data = pd.concat([df_train['SalePrice'], df_train[var]], axis=1)f, ax = plt.subplots(figsize=(16, 8))fig = sns.boxplot(x=var, y="SalePrice", data=data)fig.axis(ymin=0, ymax=800000);plt.xticks(rotation=90);<jupyter_code># correlation matrix<jupyter_output><empty_output><jupyter_text>corrmat = df_train.corr()f, ax = plt.subplots(figsize=(12, 9))sns.heatmap(corrmat, vmax=.8, square=True); Let us make this a bit betterscatterplotsns.set()cols = ['SalePrice', 'OverallQual', 'GrLivArea', 'GarageCars', 'TotalBsmtSF', 'FullBath', 'YearBuilt']sns.pairplot(df_train[cols], size = 2.5)plt.show(); corr=df_train.corr()["SalePrice"]corr<jupyter_code># Plotting correlations<jupyter_output><empty_output><jupyter_text>num_feat=df_train.columns[df_train.dtypes!=object]num_feat=num_feat[1:-1] labels = []values = []for col in num_feat:    labels.append(col)    values.append(np.corrcoef(df_train[col].values, df_train.SalePrice.values)[0,1])    ind = np.arange(len(labels))width = 0.9fig, ax = plt.subplots(figsize=(12,40))rects = ax.barh(ind, np.array(values), color='red')ax.set_yticks(ind+((width)/2.))ax.set_yticklabels(labels, rotation='horizontal')ax.set_xlabel("Correlation coefficient")ax.set_title("Correlation Coefficients w.r.t Sale Price");<jupyter_code># Some data transformations
# histogram and normal probability plot<jupyter_output><empty_output><jupyter_text>sns.distplot(df_train['SalePrice'], fit=norm);fig = plt.figure()res = stats.probplot(df_train['SalePrice'], pldata:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZsAAAEWCAYAAACwtjr+AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALE<KEY>)<jupyter_code># Applying log transformation<jupyter_output><empty_output><jupyter_text>df_train['SalePrice'] = np.log(df_train['SalePrice']) We apply lg transtransformed histogram and normal probability plotsns.distplot(df_train['SalePrice'], fit=norm);fig = plt.figure()res = stats.probplot(df_train['SalePrice'], plot=plt)<jupyter_code># Histogram and normal probability plot<jupyter_output><empty_output><jupyter_text>sns.distplot(df_train['GrLivArea'], fit=norm);fig = plt.figure()res = stats.probplot(df_train['GrLivArea'], plot=plt) df_train['GrLivArea'] = np.log(df_train['GrLivArea'])transformed histogram and normal probability plotsns.distplot(df_train['GrLivArea'], fit=norm);fig = plt.figure()res = stats.probplot(df_train['GrLivArea'], plot=plt)<jupyter_code># Histogram and normal probability plot<jupyter_output><empty_output><jupyter_text>sns.distplot(df_train['TotalBsmtSF'], fit=norm);fig = plt.figure()res = stats.probplot(df_train['TotalBsmtSF'], plot=plt)<jupyter_code># How about missing values<jupyter_output><empty_output><jupyter_text>null_columns=df_train.columns[df_train.isnull().any()]df_train[null_columns].isnull().sum()<jupyter_code># Some multivariate analysis<jupyter_output><empty_output><jupyter_text>df_train['LotFrontage'].corr(df_train['LotArea']) df_train['SqrtLotArea']=np.sqrt(df_train['LotArea'])df_train['LotFrontage'].corr(df_train['SqrtLotArea']) sns.jointplot(df_train['LotFrontage'],df_train['SqrtLotArea'],color='gold'); plt.scatter(df_train["MasVnrArea"],df_train["SalePrice"])plt.title("MasVnrArea Vs SalePrice ")plt.ylabel("SalePrice")plt.xlabel("Mas Vnr Area in sq feet");<jupyter_code># Basement variable<jupyter_output><empty_output><jupyter_text>plt.scatter(df_train["TotalBsmtSF"],df_train["SalePrice"])plt.title("TotalBsmtSF Vs SalePrice ")plt.ylabel("SalePrice")plt.xlabel("TotalBsmtSF");<jupyter_code># Treatment of outliers
# there are few outliers in total basement area lets remove them<jupyter_output><empty_output><jupyter_text>upperlimit = np.percentile(df_train.TotalBsmtSF.values, 99.5)df_train['TotalBsmtSF'].ix[df_train['TotalBsmtSF']>upperlimit] = upperlimitplt.scatter(df_train.TotalBsmtSF, df_train["SalePrice"].values,color='orange')plt.title("TotalBsmtSF Vs SalePrice ")plt.ylabel("SalePrice")plt.xlabel("Total Basement in sq feet");<jupyter_code># Dealing with categorical variables<jupyter_output><empty_output><jupyter_text>sns.factorplot("Fireplaces","SalePrice",data=df_train,hue="FireplaceQu");<jupyter_code># In above, prices increase with quality and number of fireplaces<jupyter_output><empty_output><jupyter_text>Does amount of 1st floor area help in sales price prediction<jupyter_code>plt.scatter(df_train["1stFlrSF"],df_train.SalePrice, color='red')
plt.title("Sale Price wrt 1st floor")
plt.ylabel('Sale Price (in dollars)')
plt.xlabel("1st Floor in square feet");<jupyter_output><empty_output><jupyter_text>Create a new variables - sales price per sq ft<jupyter_code>df_train['SalePriceSF'] = df_train['SalePrice']/df_train['GrLivArea']
plt.hist(df_train['SalePriceSF'], bins=15,color="gold")
plt.title("Sale Price per Square Foot")
plt.ylabel('Number of Sales')
plt.xlabel('Price per square feet');
from sklearn.linear_model import LinearRegression
import scipy, scipy.stats
import statsmodels.formula.api as sm<jupyter_output><empty_output><jupyter_text>Let us now do a very small prediction<jupyter_code>result = sm.OLS( df_train['SalePrice'], df_train['TotalBsmtSF'] ).fit()
result.summary()<jupyter_output><empty_output><jupyter_text>In class lab WAP : Perform EDA on the given data set    DataSet Name: ForestFire Dataset  This dataset contains 517rows and 13 columns. These are the observations collected from the northeast region of Portugal, with the aim of predicting the burned area (or size) of forest fires.  perform an exploratory data analysis (EDA) in such a way that the data is well prepared to build a suitable model which can predict the burned area of forest fires. Also, solve the below questions.<jupyter_code>1.Find head of the data set with first 10 observations 
2.Check for the missing data?
3.Convert to factors of all categorical variables?
4.Visualize the data Distribution of integer or numerical variables using plots?
5.Create Box plot of integer or numerical variables?
# Take home assignment***

Based on the previous dataset "House Prices", solve the below questions:
1. There are nominal and Ordinal data in the set, divide the total variables into two categories.
2. create a Box Plot of “Area” variable
3. check for the outlier?
4. Detect the outliers if there are any?
5. create box plot on data set where area is greater than zero ?
6. Create a bar plot for Area Vs Month/Days ?
7. Create a scatter plot for required variables?
8. Correlation Plot between all the quantitative fields?<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12141945958483126, 'fraction_numerical': 0.009741389307665545, 'mean_word_length': 8.635754189944134, 'pattern_counts': {'":': 0, '<': 95, '<?xml version=': 0, '>': 96, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '23', 'dir': 'jupyter-structured-clean-dedup', 'id': '435382', 'license': 'MIT', 'n_tokens_mistral': 3040, 'n_tokens_neox': 2898, 'n_words': 662, 'path': 'Practice_Notebooks/Day5.ipynb', 'repo_name': 'Aujasvi-Moudgil/EDA-Miscellaneous-Data', 'size': '24435'}