<commit_before>\documentclass[11pt]{article}
\usepackage[english]{babel}
\usepackage{amsmath,amsthm,amsfonts,amssymb,epsfig}
\usepackage[left=1.1in,top=1in,right=1.1in]{geometry}
\usepackage{url}
\usepackage{array}
\usepackage{datetime}
\usepackage{lipsum}
\usepackage{url}
\usepackage{spverbatim}
\usepackage{hyperref}
\hypersetup{colorlinks, urlcolor={blue}}
\usepackage{graphicx}
\graphicspath{ {images/} }
\newcommand{\argmin}{\operatornamewithlimits{argmin}}
\newcommand{\argmax}{\operatornamewithlimits{argmax}}


\begin{document}

\thispagestyle{empty} %removes page number

\begin{center}
\textsc{\Large\bf{Generalized Linear Modeling  with H2O's R Package}}
\\
\bigskip
\textsc{November 2014}
\end{center}
\bigskip
\bigskip 
\bigskip
\bigskip
\tableofcontents

\newpage


\section{Introduction} \label{1}

Generalized linear models (GLM) are some of the most commonly-used models for many types of data analysis use cases. While some data analysis can be done using general linear models, if the variables are more complex, general linear models may not be as accurate. For example, if the dependent variable has a non-continuous distribution or if the effect of the predictors is not linear, generalized linear models will produce more accurate results than general linear models.  

H2O's GLM algorithm fits the generalized linear model with elastic net penalties. The model fitting computation is distributed, extremely fast,and scales extremely well for models with a limited number (~ low thousands) of predictors with non-zero coefficients . The algorithm can compute models for a single value of a penalty argument or the full regularization path, similar to glmnet. It can compute gaussian (linear), logistic, poisson, and gamma regression models. This document describes the usage of H2O's implementation of GLM in R.

H2O's GLM fits the model by solving following problem:

\[ \min\limits_{\beta}\ {{1\over{N}} log-likelihood(family,\beta)  + \lambda (\alpha \| \beta \|_1}  + {1- \alpha \over 2} \| \beta \|_2) \]

The elastic net parameter $\alpha$ controls the penalty distribution between L1 and L2 penalty. It can have any value between 0 and 1. When $\alpha$ = 0, we have no L1 penalty and the problem becomes ridge regression. If $\alpha$ = 1, there is no L2 penalty and we have lasso.

The main advantage of an L1 penalty is that with sufficiently high $\lambda$, it produces a sparse solution; the L2-only penalty does not reduce coefficients to exactly 0. The two penalties also differ in the case of correlated predictors. The L2 penalty shrinks coefficients for correlated columns towards each other, while the L1 penalty will pick one and drive the others to zero. Using the elastic net argument $\alpha$, you can combine these two behaviors. It is also useful to always add a small L2 penalty to increase numerical stability.

As mentioned before, model-fitting works and scales well if numcols $<<$ numrows. However, in many cases we want to fit a GLM model over many predictors (10,000 or more) with an L1 penalty, selecting only a few of them for the model. The rest will have a zero coefficient and will not be part of the model. To maintain excellent performance and scaling properties in such cases, we employ strong rules  (as described in "Strong Rules for Discarding Predictors in Lasso-type Problems") to limit the set of active coefficients. The performance of the algorithm is then limited by the number of coefficients in the model, rather than the number of coefficients present in the training data set. 

The algorithm can handle categorical variables, which are treated as a special kind of sparse vectors: each categorical column is expanded on the fly into multiple binary vectors. 



To prevent such a correlation, we usually skip one of the levels (the first one). This is not necessary with a non-zero penalty and H2O has an option to include or factor levels in the model. All levels are always included when computing the whole regularization path. This is useful when interpreting the variable importance of the model. 

\section{Installation} 

You can load the latest CRAN H2O package by running:

\begin{spverbatim}
> install.packages("h2o")
\end{spverbatim}
\bigskip
\noindent
Alternatively, you can (and should for this tutorial) download the latest H2O build by following the ``Install in R" instructions in the H2O download table:
\underbar{http://s3.amazonaws.com/h2o-release/h2o/master/latest.html}. Open your R Console and run the following to install the latest H2O build in R:

\begin{spverbatim}
# The following two commands remove any previously installed H2O packages for R.
> if ("package:h2o" %in% search()) { detach("package:h2o", unload=TRUE) }
> if ("h2o" %in% rownames(installed.packages())) { remove.packages("h2o") }

# Next, we download, install and initialize the H2O package for R
# replace the *s below with the release number found on our download page
> install.packages("h2o", repos=(c("http://s3.amazonaws.com/h2o-release/h2o/
master/****/R", getOption("repos"))))

# Load h2o library in R
> library(h2o)

\end{spverbatim}
\noindent
To launch on a single node, initialize H2O on all the cores of your machine with

\begin{spverbatim}
> localH2O = h2o.init(nthreads = -1)

\end{spverbatim}
\noindent
With this command, the H2O R module will start an instance of H2O automatically at localhost:54321. Alternatively,  to specify a connection with an existing H2O cluster node (other than localhost at port 54321) you must explicitly state the IP address and port number in the \texttt{h2o.init()} call. An example is given below, but you should specify the IP and port number appropriate to your specific environment.

\noindent
Run the following command to observe an example classification model built through H2O's GLM:

\begin{spverbatim}
# Run GLM of CAPSULE ~ AGE + RACE + PSA + DCAPS
prostatePath = system.file("extdata", "prostate.csv", package = "h2o")
prostate.hex = h2o.importFile(localH2O, path = prostatePath, key = "prostate.hex")
h2o.glm(y = "CAPSULE", x = c("AGE","RACE","PSA","DCAPS"), data = prostate.hex,
        family = "binomial", nfolds = 0, alpha = 0.5, lambda_search = FALSE,
        use_all_factor_levels = FALSE, variable_importances = FALSE, higher_accuracy = FALSE)
\end{spverbatim}

\vspace{10 mm}
\noindent
You should see the following output:

\begin{spverbatim}       
IP Address: 127.0.0.1 
Port      : 54321 
Parsed Data Key: prostate.hex 

GLM2 Model Key: <KEY>

Coefficients:
      AGE      RACE     DCAPS       PSA Intercept 
 -0.01104  -0.63136   1.31888   0.04713  -1.10896 

Normalized Coefficients:
      AGE      RACE     DCAPS       PSA Intercept 
 -0.07208  -0.19495   0.40972   0.94253  -0.33707 

Degrees of Freedom: 379 Total (i.e. Null);  375 Residual
Null Deviance:     512.3
Residual Deviance: 449.5  AIC: 459.5
Deviance Explained: 0.12254 
 Best Threshold: 0.28

Confusion Matrix:
        Predicted
Actual   false true Error
  false     75  152 0.670
  true      18  135 0.118
  Totals    93  287 0.447

AUC =  0.7157151 (on train) 
\end{spverbatim}

\subsection{Support} 

Users of the H2O package may submit general enquiries and bug reports to the 0xdata support address:\underbar {<EMAIL>}. Alternatively, specific bugs or issues may be filed to the 0xdata JIRA: \underbar{https://0xdata.atlassian.net/secure/Dashboard.jspa}.

\section{Generalized Linear Modeling} 
This section contains a brief overview of generalized linear models and follows up with a few details for each model family.

The generalized linear model is a generalization of linear regression. Linear regression models the dependency of response y on a vector of predictors x ($y \sim x^T \beta + \beta_0$), with the assumption that y has a gaussian distribution, with the mean being a linear function of x (+ the offset) and a deviance of $\sigma$ (independent of x), i.e. $ y = \mathcal{N}(x^T \beta + \beta_0 , \sigma^2) $. In many situations, these assumptions are overly restrictive but we would still want to take advantage of linear models. GLM generalizes linear regression in following ways: 
\begin{itemize} 
\item adds a non-linear link function, so that $link(y) \sim x^T \beta + \beta_0$.
\item allows variance to depend on the predicted value (family argument).

\end{itemize}
This generalization allows us to apply GLM to problems such as binary classification (Logistic regression).
GLM models are fitted by maximizing the likelihood.

\subsubsection{Model Fitting}
GLM models are fitted by maximizing the likelihood. For gaussian family, maximum likelihood is simply minimal mean squared error which has an analytical solution and it can be solved with ordinary least squares. For all other families, the maximum likelihood problem has no analytical solution and we have to use an iterative method (typical examples are IRLSM, Newton method, gradient descent, L-BFGS). 
\subsubsection{Model Validation}
Evaluating quality of the model is critical part of any data-modelling/prediction task. There are several standard ways how to evalute quality of fitted GLM model. The most common is to judge by deviance statistic. Deviance is based on comparing log likelihood of the fitted model with log likelihood of the saturated model (theoretical perfect model).

\[ deviance = 2({ln(L_{s})} - {ln(L_{m})}) \]

Another metric frequently used for model selection is The Akaike information criterion (AIC). AIC is a measure of the relative quality of a statistical model for a given set of data. It is usefull for model comparison. Unlike deviance (which would have perfect value for saturated model), it takes into account the complexity of the given model. It is based on meassuring information loss when replacing original data with the model itself. Unlike deviance, it does not meassure absolute quality of the fit (comparison against null-hypothesis). AIC is defined as follows:

\[ aic = 2k - 2ln(L_{m})\]
, where k is number of model parameters and L is likelihood of the model over the dataset.

\subsection{Regularization with Elastic Net Penalty} 
We introduce penalty to model-fitting to avoid over-fitting, reduce variance of the prediction error and to deal with correlated predictors. There are two common penalized linear models, Ridge Regression and Lasso. Ridge regression provides greater numerical stability and is easier (faster) to compute. On the other hand, Lasso leads to a sparse solution which is big advantage in many situations as it can be used for feature selection and produce models with fewer parameters. When ecnoutering highly correlated columns, l2 penalty tends to push all of the coefficients towards each other, while L1 penalty will pick one and remove the others (0 coefficients).

Elastic net combines the two and adds another parameter $\alpha$ which controls distribution of the pnelaty between L1 and l2. The combination of the two penalties is benefitial since L1 gives sparsity while l2 gives stability and encourages grouping effect (group of correlated variables tends to be dropped or aded into the model all at once).One possible use of the $\alpha$ argument is to do lasso with very little l2 penalty ($\alpha$ almost 1) to stabilize the computation (improve convergence speed)

Model fitting problem with elastic net penalty becomes:

\[ \min\limits_{\beta,\beta_0}\ {{1\over{N}} ln(L(family,\beta,\beta_0))  + \lambda (\alpha \| \beta \|_1}  + {1- \alpha \over 2} \| \beta \|_2^2) \]


\subsection{GLM Models} 
The following subsection describes GLM families supported in h2o. 

\subsubsection{Linear Regression (Gaussian family) }
Linear regression refers to the gaussian family model. It is the simplest example of GLM, but it has many uses and several advantages over the other families, such as faster and more stable computation. 

It models the dependency as a purely linear function (with link = identity):
\[ \hat{y} = x^T\beta + \beta_0\]

The model is fitted by solving the least squares problem (maximum likelihood for gaussian family):

\[ \min\limits_{\beta,\beta_0} { {1 \over 2N}\sum\limits_{i=1}\limits^{N}(x_i^{T}\beta  + \beta_0- y_i)^T (x_i^{T}\beta + \beta_0 - y_i))  + \lambda (\alpha \|\beta \|_1 + {1-\alpha \over 2}) \| \beta \|_2^2} \]<commit_msg>section 2/3 of glm vignette edit for grammar and structure<commit_after>\documentclass[11pt]{article}
\usepackage[english]{babel}
\usepackage{amsmath,amsthm,amsfonts,amssymb,epsfig}
\usepackage[left=1.1in,top=1in,right=1.1in]{geometry}
\usepackage{url}
\usepackage{array}
\usepackage{datetime}
\usepackage{lipsum}
\usepackage{url}
\usepackage{spverbatim}
\usepackage{hyperref}
\hypersetup{colorlinks, urlcolor={blue}}
\usepackage{graphicx}
\graphicspath{ {images/} }
\newcommand{\argmin}{\operatornamewithlimits{argmin}}
\newcommand{\argmax}{\operatornamewithlimits{argmax}}


\begin{document}

\thispagestyle{empty} %removes page number

\begin{center}
\textsc{\Large\bf{Generalized Linear Modeling  with H2O's R Package}}
\\
\bigskip
\textsc{November 2014}
\end{center}
\bigskip
\bigskip 
\bigskip
\bigskip
\tableofcontents

\newpage


\section{Introduction} \label{1}

Generalized linear models (GLM) are some of the most commonly-used models for many types of data analysis use cases. While some data analysis can be done using general linear models, if the variables are more complex, general linear models may not be as accurate. For example, if the dependent variable has a non-continuous distribution or if the effect of the predictors is not linear, generalized linear models will produce more accurate results than general linear models.  

H2O's GLM algorithm fits the generalized linear model with elastic net penalties. The model fitting computation is distributed, extremely fast,and scales extremely well for models with a limited number (~ low thousands) of predictors with non-zero coefficients . The algorithm can compute models for a single value of a penalty argument or the full regularization path, similar to glmnet. It can compute gaussian (linear), logistic, poisson, and gamma regression models. This document describes the usage of H2O's implementation of GLM in R.

H2O's GLM fits the model by solving following problem:

\[ \min\limits_{\beta}\ {{1\over{N}} log-likelihood(family,\beta)  + \lambda (\alpha \| \beta \|_1}  + {1- \alpha \over 2} \| \beta \|_2) \]

The elastic net parameter $\alpha$ controls the penalty distribution between L1 and L2 penalty. It can have any value between 0 and 1. When $\alpha$ = 0, we have no L1 penalty and the problem becomes ridge regression. If $\alpha$ = 1, there is no L2 penalty and we have lasso.

The main advantage of an L1 penalty is that with sufficiently high $\lambda$, it produces a sparse solution; the L2-only penalty does not reduce coefficients to exactly 0. The two penalties also differ in the case of correlated predictors. The L2 penalty shrinks coefficients for correlated columns towards each other, while the L1 penalty will pick one and drive the others to zero. Using the elastic net argument $\alpha$, you can combine these two behaviors. It is also useful to always add a small L2 penalty to increase numerical stability.

As mentioned before, model-fitting works and scales well if numcols $<<$ numrows. However, in many cases we want to fit a GLM model over many predictors (10,000 or more) with an L1 penalty, selecting only a few of them for the model. The rest will have a zero coefficient and will not be part of the model. To maintain excellent performance and scaling properties in such cases, we employ strong rules  (as described in "Strong Rules for Discarding Predictors in Lasso-type Problems") to limit the set of active coefficients. The performance of the algorithm is then limited by the number of coefficients in the model, rather than the number of coefficients present in the training data set. 

The algorithm can handle categorical variables, which are treated as a special kind of sparse vectors: each categorical column is expanded on the fly into multiple binary vectors. 



To prevent such a correlation, we usually skip one of the levels (the first one). This is not necessary with a non-zero penalty and H2O has an option to include or factor levels in the model. All levels are always included when computing the whole regularization path. This is useful when interpreting the variable importance of the model. 

\section{Installation} 

You can load the latest CRAN H2O package by running:

\begin{spverbatim}
> install.packages("h2o")
\end{spverbatim}
\bigskip
\noindent
Alternatively, you can (and should for this tutorial) download the latest H2O build by following the ``Install in R" instructions in the H2O download table:

\indent \indent \indent \underbar{http://s3.amazonaws.com/h2o-release/h2o/master/latest.html}.

\noindent
Open your R Console and run the following to install the latest H2O build in R:

\begin{spverbatim}
# The following two commands remove any previously installed H2O packages for R.
> if ("package:h2o" %in% search()) { detach("package:h2o", unload=TRUE) }
> if ("h2o" %in% rownames(installed.packages())) { remove.packages("h2o") }

# Next, we download, install and initialize the H2O package for R
# replace the *s below with the release number found on our download page
> install.packages("h2o", repos=(c("http://s3.amazonaws.com/h2o-release/h2o/
master/****/R", getOption("repos"))))

# Load h2o library in R
> library(h2o)

\end{spverbatim}
\noindent
To launch on a single node, initialize H2O on all the cores of your machine with

\begin{spverbatim}
> localH2O = h2o.init(nthreads = -1)

\end{spverbatim}
\noindent
The function \texttt{h2o.init()} will initialize a H2O instance and instantiates a H2O client module. By default the H2O instance will launch on localhost:54321. Alternatively, to establish a connection to an existing H2O cluster node explicitly state the IP address (\texttt{ip = "localhost"}) and port number (\texttt{port = 54321}) in the \texttt{h2o.init()} call.

\noindent
Run the following command to observe an example classification model built through H2O's GLM:

\begin{spverbatim}
# Build a GLM model on prostate data with formula: CAPSULE ~ AGE + RACE + PSA + DCAPS
> prostatePath = system.file("extdata", "prostate.csv", package = "h2o")
> prostate.hex = h2o.importFile(localH2O, path = prostatePath, key = "prostate.hex")
> h2o.glm(y = "CAPSULE", x = c("AGE","RACE","PSA","DCAPS"), data = prostate.hex, family = "binomial", nfolds = 0, alpha = 0.5, lambda_search = FALSE, use_all_factor_levels = FALSE, variable_importances = FALSE, higher_accuracy = FALSE)
\end{spverbatim}

\vspace{10 mm}
\noindent
The output of the model build will include coefficients as well as some validation statistics:

\begin{spverbatim}       
IP Address: 127.0.0.1 
Port      : 54321 
Parsed Data Key: prostate.hex 

GLM2 Model Key: <KEY>

Coefficients:
      AGE      RACE     DCAPS       PSA Intercept 
 -0.01104  -0.63136   1.31888   0.04713  -1.10896 

Normalized Coefficients:
      AGE      RACE     DCAPS       PSA Intercept 
 -0.07208  -0.19495   0.40972   0.94253  -0.33707 

Degrees of Freedom: 379 Total (i.e. Null);  375 Residual
Null Deviance:     512.3
Residual Deviance: 449.5  AIC: 459.5
Deviance Explained: 0.12254 
 Best Threshold: 0.28

Confusion Matrix:
        Predicted
Actual   false true Error
  false     75  152 0.670
  true      18  135 0.118
  Totals    93  287 0.447

AUC =  0.7157151 (on train)
\end{spverbatim}

\subsection{Support} 

Users of the H2O package may submit general enquiries and bug reports privately to H2O via email: \underbar {<EMAIL>}; or publicly post them to: \underbar {<EMAIL>}. Specific bugs or issues will be filed to H2O's JIRA: 
\underbar{https://0xdata.atlassian.net/secure/Dashboard.jspa}.

\section{Generalized Linear Modeling} 
This section contains a brief overview of generalized linear models and follows up with a few details for each model family.

Generalized linear models are generalization of linear regressions. Linear regression models the dependency of response y on a vector of predictors x ($y \sim x^T \beta + \beta_0$). The models are built with the assumptions that y has a gaussian distribution with a variance of $\sigma^2$ and the mean is a linear function of x with an offset of some constant $\beta_0$, i.e. $ y = \mathcal{N}(x^T \beta + \beta_0 ,  \sigma^2) $. These assumptions can be overly restrictive for real world data that do not necessarily follow have a gaussian distribution. GLM generalizes linear regression in following ways: 
\begin{itemize} 
\item adds a non-linear link function which transforms the expectation of response variable, so that $link(y) \sim x^T \beta + \beta_0$.
\item allows variance to depend on the predicted value by specifying the conditional distribution of the response variable or the family argument.

\end{itemize}
This generalization allows us to use GLM on problems such as binary classification (Logistic regression).

\subsubsection{Model Fitting}
GLM models are fitted by maximizing the likelihood. For gaussian family, maximum likelihood is simply minimal mean squared error which has an analytical solution and can be solved with ordinary least squares. For all other families, the maximum likelihood problem has no analytical solution so we must use an iterative method such as IRLSM, Newton method, gradient descent, and L-BFGS.
\subsubsection{Model Validation}
Evaluating the quality of the model is a critical part of any data-modeling and scoring process. There are several standard ways on how to evalute the quality of the fitted GLM model; the most common method is to use the resulting deviance. Deviance is calculated by comparing the log likelihood of the fitted model with log likelihood of the saturated model (or theoretically perfect model).

\[ deviance = 2({ln(L_{s})} - {ln(L_{m})}) \]

Another metric frequently used for model selection is the Akaike information criterion (AIC). AIC is a measure of the relative quality of a statistical model for a given set of data by calculating the information loss when replacing the original data with the model itself. Unlike deviance which would assign a perfect value for the saturated model and measures the absolute quality of the fit with a comparison against the null-hypothesis, it takes into account the complexity of the given model. AIC is defined as follows:

\[ aic = 2(k - ln(L_{m}))\]

\noindent
where k is the number of model parameters and $ln(L_{m})$ is log likelihood of the fitted model over the data.

\subsection{Regularization with Elastic Net Penalty} 
We introduce penalty to model-fitting to avoid over-fitting, reduce variance of the prediction error and to deal with correlated predictors. There are two common penalized linear models, Ridge Regression and Lasso. Ridge regression provides greater numerical stability and is easier (faster) to compute. On the other hand, Lasso leads to a sparse solution which is big advantage in many situations as it can be used for feature selection and produce models with fewer parameters. When ecnoutering highly correlated columns, l2 penalty tends to push all of the coefficients towards each other, while L1 penalty will pick one and remove the others (0 coefficients).

Elastic net combines the two and adds another parameter $\alpha$ which controls distribution of the pnelaty between L1 and l2. The combination of the two penalties is benefitial since L1 gives sparsity while l2 gives stability and encourages grouping effect (group of correlated variables tends to be dropped or aded into the model all at once).One possible use of the $\alpha$ argument is to do lasso with very little l2 penalty ($\alpha$ almost 1) to stabilize the computation (improve convergence speed)

Model fitting problem with elastic net penalty becomes:

\[ \min\limits_{\beta,\beta_0}\ {{1\over{N}} ln(L(family,\beta,\beta_0))  + \lambda (\alpha \| \beta \|_1}  + {1- \alpha \over 2} \| \beta \|_2^2) \]


\subsection{GLM Models} 
The following subsection describes GLM families supported in h2o. 

\subsubsection{Linear Regression (Gaussian family) }
Linear regression refers to the gaussian family model. It is the simplest example of GLM, but it has many uses and several advantages over the other families, such as faster and more stable computation. 

It models the dependency as a purely linear function (with link = identity):
\[ \hat{y} = x^T\beta + \beta_0\]

The model is fitted by solving the least squares problem (maximum likelihood for gaussian family):

\[ \min\limits_{\beta,\beta_0} { {1 \over 2N}\sum\limits_{i=1}\limits^{N}(x_i^{T}\beta  + \beta_0- y_i)^T (x_i^{T}\beta + \beta_0 - y_i))  + \lambda (\alpha \|\beta \|_1 + {1-\alpha \over 2}) \| \beta \|_2^2} \]
{'fraction_non_alphanumeric': 0.07592532729917569, 'fraction_numerical': 0.02161790851786003, 'mean_word_length': 4.384899912967798, 'pattern_counts': {'":': 0, '<': 12, '<?xml version=': 0, '>': 23, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 2, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 8, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '5868059', 'n_tokens_mistral': 7175, 'n_tokens_neox': 6639, 'n_words': 3345}