<jupyter_start><jupyter_text>Package Expo: statsmodels  Installation<jupyter_code>#pip install statsmodels<jupyter_output>Requirement already satisfied: statsmodels in c:\users\jonathan\anaconda3\lib\site-packages (0.12.0)
Requirement already satisfied: numpy>=1.15 in c:\users\jonathan\anaconda3\lib\site-packages (from statsmodels) (1.19.2)
Requirement already satisfied: scipy>=1.1 in c:\users\jonathan\anaconda3\lib\site-packages (from statsmodels) (1.5.2)
Requirement already satisfied: pandas>=0.21 in c:\users\jonathan\anaconda3\lib\site-packages (from statsmodels) (1.1.3)
Requirement already satisfied: patsy>=0.5 in c:\users\jonathan\anaconda3\lib\site-packages (from statsmodels) (0.5.1)
Requirement already satisfied: python-dateutil>=2.7.3 in c:\users\jonathan\anaconda3\lib\site-packages (from pandas>=0.21->statsmodels) (2.8.1)
Requirement already satisfied: pytz>=2017.2 in c:\users\jonathan\anaconda3\lib\site-packages (from pandas>=0.21->statsmodels) (2020.1)
Requirement already satisfied: six in c:\users\jonathan\anaconda3\lib\site-packages (from patsy>=0.5->statsmodels) (1.15.0)<jupyter_text>Linear Regression with Scikit-Learn  Data Loading<jupyter_code>import pandas as pd
import numpy as np
from sklearn.linear_model import LinearRegression
from sklearn.metrics import r2_score
import statsmodels.api as sm
# insurance dataset
data = pd.read_csv("insurance.csv")
data.head()
# convert categorical variables to indicators; clean up data
data = pd.get_dummies(data).drop(["sex_female", "smoker_no", "region_northeast"], axis=1)
data.head()
# set up feature matrix and target array
X = data.drop("charges", axis=1)
y = data[["charges"]]<jupyter_output><empty_output><jupyter_text>Analysis<jupyter_code>sk_linear = LinearRegression(fit_intercept=True).fit(X, y)
sk_linear.coef_
sk_linear.intercept_
sk_yfit = sk_linear.predict(X)
r2_score(y, sk_yfit)<jupyter_output><empty_output><jupyter_text>Linear Regression with statsmodels  Data Loading<jupyter_code># insurance dataset again
data = pd.read_csv("insurance.csv")
data.head()
# no cleanup needed
# data = pd.get_dummies(data).drop(["sex_female", "smoker_no", "region_northeast"], axis=1)
# data.head()<jupyter_output><empty_output><jupyter_text>Analysis<jupyter_code># predict charges as a function of other variables
sm_linear = sm.OLS.from_formula("charges ~ age + sex + bmi + children + smoker + region", data=data).fit()
# alternative method, similar to scikit-learn
# X = sm.add_constant(X)
# sm_linear = sm.OLS(y, X).fit()
print(sm_linear.summary())
sm_linear.params
sm_linear.rsquared<jupyter_output><empty_output><jupyter_text>Aside: Vectorized Operations Example<jupyter_code># assume age is curvilinearly related to charges
sm_linear = sm.OLS.from_formula("charges ~ np.log(age) + sex + bmi + children + smoker + region", data=data).fit()
print(sm_linear.summary())<jupyter_output>OLS Regression Results                            
==============================================================================
Dep. Variable:                charges   R-squared:                       0.744
Model:                            OLS   Adj. R-squared:                  0.743
Method:                 Least Squares   F-statistic:                     483.0
Date:                Mon, 03 May 2021   Prob (F-statistic):               0.00
Time:                        21:20:28   Log-Likelihood:                -13566.
No. Observations:                1338   AIC:                         2.715e+04
Df Residuals:                    1329   BIC:                         2.720e+04
Df Model:                           8                                         
Covariance Type:            nonrobust                                         
=======================================================================================
                          coef    std err  [...]<jupyter_text>Logistic Regression with statsmodels   Data Loading<jupyter_code># diabetes dataset
data2 = pd.read_csv("diabetes.csv")
data2.head()<jupyter_output><empty_output><jupyter_text>Analysis<jupyter_code># predict probability of diabetes (outcome) as function of other variables
sm_logit = sm.Logit.from_formula(
    "Outcome ~ Pregnancies + Glucose + BloodPressure + SkinThickness + Insulin + BMI + DiabetesPedigreeFunction + Age",
    data = data2).fit()
print(sm_logit.summary())<jupyter_output>Logit Regression Results                           
==============================================================================
Dep. Variable:                Outcome   No. Observations:                  768
Model:                          Logit   Df Residuals:                      759
Method:                           MLE   Df Model:                            8
Date:                Mon, 03 May 2021   Pseudo R-squ.:                  0.2718
Time:                        21:20:28   Log-Likelihood:                -361.72
converged:                       True   LL-Null:                       -496.74
Covariance Type:            nonrobust   LLR p-value:                 9.652e-54
============================================================================================
                               coef    std err          z      P>|z|      [0.025      0.975]
--------------------------------------------------------------------------------------------
Intercept [...]<jupyter_text>Saving and Loading Models<jupyter_code>from statsmodels.discrete.discrete_model import LogitResults
sm_logit.save("logit_model.pkl")
logit_2 = LogitResults.load("logit_model.pkl")
print(logit_2.summary())<jupyter_output>Logit Regression Results                           
==============================================================================
Dep. Variable:                Outcome   No. Observations:                  768
Model:                          Logit   Df Residuals:                      759
Method:                           MLE   Df Model:                            8
Date:                Mon, 03 May 2021   Pseudo R-squ.:                  0.2718
Time:                        21:20:28   Log-Likelihood:                -361.72
converged:                       True   LL-Null:                       -496.74
Covariance Type:            nonrobust   LLR p-value:                 9.652e-54
============================================================================================
                               coef    std err          z      P>|z|      [0.025      0.975]
--------------------------------------------------------------------------------------------
Intercept [...]
{'fraction_non_alphanumeric': 0.19403669724770642, 'fraction_numerical': 0.03394495412844037, 'mean_word_length': 2.218996062992126, 'pattern_counts': {'":': 0, '<': 33, '<?xml version=': 0, '>': 47, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 5, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '463770', 'license': 'MIT', 'n_tokens_mistral': 1995, 'n_tokens_neox': 1800, 'n_words': 398, 'path': 'package_expos/statsmodels/demo.ipynb', 'repo_name': 'hanisaf/advanced-data-management-and-analytics-spring2021', 'size': '6540'}