<jupyter_start><jupyter_text>Regression formulaLooks like you're good for the linearity assumption, and additionally, the distributions for height and weight look reasonable (and even pretty normal!). Now, let's run the regression. Statsmodels allows users to fit statistical models using R-style formulas. The formula framework is quite powerful and for simple regression it is written using a ~ as Y ~ X.The formula gives instruction for a general structure for a regression call. For a statsmodels ols calls, you'll need a Pandas dataframe with column names that you will add to your formula.<jupyter_code>f = 'weight~height'<jupyter_output><empty_output><jupyter_text>You can we pass the formula with variable names to ols along with fit() to fit a linear model to given variables.<jupyter_code>model = ols(formula=f, data=df).fit()<jupyter_output><empty_output><jupyter_text>Now, you can go ahead and inspect your fitted model in many ways. First, let's get a summary of what the model contains using model.summary()<jupyter_code>model.summary()<jupyter_output><empty_output><jupyter_text>Warnings:[1] Standard Errors assume that the covariance matrix of the errors is correctly specified.Wow, that's a lot of information. Statsmodels performs a ton of tests and calculates measures to identify goodness of fit.You can find the R-Squared, which is 0.95 i.e. the data are very linearly relatedYou can also look at the coefficients of the model for intercept and slope (next to "height")Kurtosis and Skew values are shown hereA lot of significance testing is being done hereHere is a brief description of these measures:The left part of the first table gives some specifics on the data and the model:Dep. Variable: Singular. Which variable is the point of interest of the modelModel: Technique used, an abbreviated version of Method (see methods for more).Method: The loss function optimized in the parameter selection process. Least Squares since it picks the parameters that reduce the training error. This is also known as Mean Square Error [MSE].No. Observations: The number of observations used by the model, or size of the training data.Degrees of Freedom Residuals: Degrees of freedom of the residuals, which is the number of observations – number of parameters. Intercept is a parameter. The purpose of Degrees of Freedom is to reflect the impact of descriptive/summarizing statistics in the model, which in regression is the coefficient. Since the observations must "live up" to these parameters, they only have so many free observations, and the rest must be reserved to "live up" to the parameters' prophecy. This internal mechanism ensures that there are enough observations to match the parameters.Degrees of Freedom Model: The number of parameters in the model (not including the constant/intercept term if present)Covariance Type: Robust regression methods are designed to be not overly affected by violations of assumptions by the underlying data-generating process. Since this model is Ordinary Least Squares, it is non-robust and therefore highly sensitive to outliers.The right part of the first table shows the goodness of fit:R-squared: The coefficient of determination, the Sum Squares of Regression divided by Total Sum Squares. This translates to the percent of variance explained by the model. The remaining percentage represents the variance explained by error, the E term, the part that model and predictors fail to grasp.Adj. R-squared: Version of the R-Squared that penalizes additional independent variables.F-statistic: A measure of how significant the fit is. The mean squared error of the model divided by the mean squared error of the residuals. Feeds into the calculation of the P-Value.Prob (F-statistic) or P-Value: The probability that a sample like this would yield the above statistic, and whether the model's verdict on the null hypothesis will consistently represent the population. Does not measure effect magnitude, instead measures the integrity and consistency of this test on this group of data.Log-likelihood: The log of the likelihood function.AIC: The Akaike Information Criterion. Adjusts the log-likelihood based on the number of observations and the complexity of the model. Penalizes the model selection metrics when more independent variables are added.BIC: The Bayesian Information Criterion. Similar to the AIC, but has a higher penalty for models with more parameters. Penalizes the model selection metrics when more independent variables are added.The second table shows the coefficient report:coef: The estimated value of the coefficient. By how much the model multiplies the independent value by.std err: The basic standard error of the estimate of the coefficient. Average distance deviation of the points from the model, which offers a unit relevant way to gauge model accuracy.t: The t-statistic value. This is a measure of how statistically significant the coefficient is.P > |t|: P-value that the null-hypothesis that the coefficient = 0 is true. If it is less than the confidence level, often 0.05, it indicates that there is a statistically significant relationship between the term and the response.[95.0% Conf. Interval]: The lower and upper values of the 95% confidence interval. Specific range of the possible coefficient values.The third table shows information about the residuals, autocorrelation, and multicollinearity:Skewness: A measure of the symmetry of the data about the mean. Normally-distributed errors should be symmetrically distributed about the mean (equal amounts above and below the line). The normal distribution has 0 skew.Kurtosis: A measure of the shape of the distribution. Compares the amount of data close to the mean with those far away from the mean (in the tails), so model "peakiness". The normal distribution has a Kurtosis of 3, and the greater the number, the more the curve peaks.Omnibus D’Angostino’s test: Provides a combined statistical test for the presence of skewness and kurtosis.Prob(Omnibus): The above statistic turned into a probabilityJarque-Bera: A different test of the skewness and kurtosisProb (JB): The above statistic turned into a probabilityDurbin-Watson: A test for the presence of autocorrelation (that the errors are not independent), which is often important in time-series analysisCond. No: A test for multicollinearity (if in a fit with multiple parameters, the parameters are related to each other).The interpretation of some of these measures will be explained in the next lessons. For others, you'll get a better insight into them in the lessons on statistics. Visualize error terms>You can also plot some visualizations to check the regression assumptions with respect to the error terms. You'll use sm.graphics.plot_regress_exog() for some built-in visualization capabilities of statsmodels. Here is how to do it:<jupyter_code>fig = plt.figure(figsize=(15,8))
fig = sm.graphics.plot_regress_exog(model, "height", fig=fig)
plt.show()<jupyter_output><empty_output><jupyter_text>For the four graphs we see above:The Y and Fitted vs. X graph plots the dependent variable against our predicted values with a confidence interval. The positive relationship shows that height and weight are correlated, i.e., when one variable increases the other increases.The Residuals versus height graph shows our model's errors versus the specified predictor variable. Each dot is an observed value; the line represents the mean of those observed values. Since there's no pattern in the distance between the dots and the mean value, the OLS assumption of homoskedasticity holds.The Partial regression plot shows the relationship between height and weight, taking in to account the impact of adding other independent variables on our existing height coefficient. You'll later learn how this same graph changes when you add more variables.The Component and Component Plus Residual (CCPR) plot is an extension of the partial regression plot. It shows where the trend line would lie after adding the impact of adding our other independent variables on the weight. Q-Q PlotsTo check for the normality assumption, you can obtain error terms (residuals) from the model and draw Q-Q Plot against a standard normal distribution as shown below. While the residuals do not seem to match up perfectly with the red line, there seem to be no super clear deviations from the red line. So you can assume that you're OK for the normality assumption.<jupyter_code>import scipy.stats as stats
residuals = model.resid
fig = sm.graphics.qqplot(residuals, dist=stats.norm, line='45', fit=True)
fig.show()<jupyter_output><ipython-input-15-8c4a1944e451>:4: UserWarning: Matplotlib is currently using module://ipykernel.pylab.backend_inline, which is a non-GUI backend, so cannot show the figure.
  fig.show()
{'fraction_non_alphanumeric': 0.04293990684993752, 'fraction_numerical': 0.0036351243894127004, 'mean_word_length': 5.2175141242937855, 'pattern_counts': {'":': 0, '<': 21, '<?xml version=': 0, '>': 23, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '30896', 'license': 'MIT', 'n_tokens_mistral': 2126, 'n_tokens_neox': 2006, 'n_words': 1268, 'path': 'Linear_Regression/OLS_Statsmodels.ipynb', 'repo_name': 'akladyous/Machine-Learning', 'size': '8803'}