I was going through a solution of the Housing prices competition on Kaggle (Human Analog's Kernel on House Prices: Advance Regression Techniques) and came across this part:# Transform the skewed numeric features by taking log(feature + 1).# This will make the features more normal.from scipy.stats import skewskewed = train_df_munged[numeric_features].apply(lambda x: skew(x.dropna().astype(float)))skewed = skewed[skewed > 0.75]skewed = skewed.indextrain_df_munged[skewed] = np.log1p(train_df_munged[skewed])test_df_munged[skewed] = np.log1p(test_df_munged[skewed])I am not sure of what is the need for converting a skewed distribution into a normal distribution. Please, can someone explain in detail: Why is this being done here? or How is this helpful?How is this different from feature-scaling?Is this a necessary step for feature-engineering? What is likely to happen if I skip this step?
Why do we convert skewed data into a normal distribution
regression;feature extraction;feature engineering;kaggle;feature scaling
You might want to interpret your coefficients. That is, to be able to say things like if I increase my variable $X_1$ by 1, then, on average and all else being equal, $Y$ should increase by $\beta_1$.For your coefficients to be interpretable, linear regression assumes a bunch of things.One of these things is no multicollinearity. That is, your $X$ variables should not be correlated against each other.Another is Homoscedasticity. The errors your model commits should have the same variance, i.e. you should ensure the linear regression does not make small errors for low values of $X$ and big errors for higher values of $X$. In other words, the difference between what you predict $\hat Y$ and the true values $Y$ should be constant. You can ensure that by making sure that $Y$ follows a Gaussian distribution. (The proof is highly mathematical.)Depending on your data, you may be able to make it Gaussian. Typical transformations are taking the inverse, the logarithm or square roots. Many others exist of course, it all depends on your data. You have to look at your data, and then do a histogram or run a normality test, such as the Shapiro-Wilk test.These are all techniques to build an unbiased estimator. I don't think it has anything to do with convergence as others have said (sometimes you may also want to normalize your data, but that is a different topic).Following the linear regression assumptions is important if you want to either interpret the coefficients or if you want to use statistical tests in your model. Otherwise, forget about it.Applying the logarithm or normalizing your data, is also important because linear regression optimization algorithms typically minimize $\|\hat y - y\|^2$, so if you have some big $y$ outliers, your estimator is going to be VERY concerned about minimizing those, since it is concerned about the squared error, not absolute error. Normalizing your data is important in those case and this is why scikit-learn has a normalize option in the LinearRegression constructor.