<jupyter_start><jupyter_text>![Klee - Visual Analytics](https://github.com/nikbearbrown/Visual_Analytics/blob/main/IMG/Klee_Visual_Analytics.png?raw=true)YouTube - https://www.youtube.com/c/NikBearBrown    GitHub - https://github.com/nikbearbrown/Visual_Analytics   Kaggle - https://www.kaggle.com/nikbearbrown   Klee.ai (Visual AI) - http://klee.ai   AutoML with H2O.ai_Lessons from Kaggle – Ensemble ML and Feature Engineering_99.9% of high ranking Kaggle submissions shared two approaches. Stacking and feature engineering. In this notebook, we will use indivdual models and stacked models to predict lift. Stacking is a type of ensemble, creating a ”super-model” by combining many complementary models.We will use generate thousands on individual models, select the best models and combine the best models into a ”super-model” to predict lift._Models and hyperparamter optimization_A model is an algorithm with a given set of hyperparamters. For example, a random forest estimator that uses 10 trees and one that uses 20 trees are two different models. Using a few algorithms and important tuning paramters (hyperparamters) we will try many combination and select rank the models on some metric like AUC, mean residual deviance, RSME as approriate for the analysis.  _The machine learning algorithms_We will use the following algorithms as our base:* Deep Learning (Neural Networks)    * Generalized Linear Model (GLM)  * Extreme Random Forest (XRT) * Distributed Random Forest (DRF)     * Gradient Boosting Machine (GBM)     * XGBoost   _Deep Learning (Neural Networks)_  The are simple Multiclass perceptrons (MLPs) as discussed in the first notebook.  _Generalized Linear Model (GLM)_   The generalized linear model (GLM) is a flexible generalization of ordinary linear regression that allows for response variables that have error distribution models other than a normal distribution. The GLM generalizes linear regression by allowing the linear model to be related to the response variable via a link function and by allowing the magnitude of the variance of each measurement to be a function of its predicted value.In our case, we will assume that the the distribution of errors is normal and that the link function is the identity, which means the will will be performing simple linear regression.   Linear regression predicts the response variable $y$ assuming it has a linear relationship with predictor variable(s) $x$ or $x_1, x_2, ,,, x_n$.$$y = \beta_0 + \beta_1 x + \varepsilon .$$_Distributed Random Forest (DRF)_    A Distributed Random Forest (DRF) is a powerful low-bias classification and regression tool that can fit highly non-linear data. To prevent overfitting a DRF generates a forest of classification or regression trees, rather than a single classification or regression tree through a process called bagging. The variance of estimates can be adjusted by the number of trees used. _Extreme Random Forest (XRT)_Extreme random forests are nearly identical to standard random forests except that the splits, both attribute and cut-point, are chosen totally or partially at random. Bias/varianceanalysis has shown that XRTs work by decreasing variance while at the same time increasing bias. Once the randomization level is properly adjusted, the variance almost vanishes while bias only slightly increases with respect to standard trees. _Gradient Boosting Machine (GBM)_   Gradient Boosting Machine (for Regression and Classification) is a forward learning ensemble method. The guiding heuristic is that good predictive results can be obtained through increasingly refined approximations. Boosting can create more accurate models than bagging but doesn’t help to avoid overfitting as much as bagging does.Unlike a DRF which uses bagging to prevent overfitting a GBM uses boosting to sequentially refine a regression or classification tree. However as each tree is built in parallel it allows for multi-threading (asynchronous) training large data sets.As with all tree based methods it creates decision trees and is highly interpretable._XGBoost_XGBoost is a supervised learning algorithm that implements a process called boosting to yield accurate models. Boosting refers to the ensemble learning technique of building many models sequentially, with each new model attempting to correct for the deficiencies in the previous model. Both XGBoost and GBM follows the principle of gradient boosting. However, XGBoost has a more regularized model formalization to control overfitting. Boosting does not prevent overfitting the way bagging does, but typically gives better accuracy. XGBoost corrects for the deficiencies of boosting by ensembling regularized trees.Like a GBM, each tree is built in parallel it allows for multi-threading (asynchronous) training large data sets.As with all tree based methods it creates decision trees and is highly interpretable.  H2O.ai AutomlH2O’s AutoML can be used for automating the machine learning workflow, which includes automatic training and tuning of many models within a user-specified time-limit. Stacked Ensembles – one based on all previously trained models, another one on the best model of each family – will be automatically trained on collections of individual models to produce highly predictive ensemble models which, in most cases, will be the top performing models in the AutoML Leaderboard.You will need to install H2O.ai Automl for python to run this notebook. ```bashpip install requestspip install tabulatepip install "colorama>=0.3.8"pip install futurepip uninstall h2opip install -f http://h2o-release.s3.amazonaws.com/h2o/latest_stable_Py.html h2o```Note: When installing H2O from pip in OS X El Capitan, users must include the --user flag.```bashpip install -f http://h2o-release.s3.amazonaws.com/h2o/latest_stable_Py.html h2o --user```See Downloading & Installing H2O [http://docs.h2o.ai/h2o/latest-stable/h2o-docs/downloading.html](http://docs.h2o.ai/h2o/latest-stable/h2o-docs/downloading.html)<jupyter_code># Import libraries
# Use pip install or conda install if missing a library
import h2o
from h2o.automl import H2OAutoML
import random, os, sys
from datetime import datetime
import pandas as pd
import logging
import csv
import optparse
import time
import json
from distutils.util import strtobool
import psutil
import numpy as np
import matplotlib.pyplot as plt
# Set a minimum memory size and a run time in seconds
min_mem_size=6
run_time=300
# Use 50% of availible resources
pct_memory=0.5
virtual_memory=psutil.virtual_memory()
min_mem_size=int(round(int(pct_memory*virtual_memory.available)/1073741824,0))
print(min_mem_size)
# 65535 Highest port no
# Start the H2O server on a random port
port_no=random.randint(5555,55555)

#  h2o.init(strict_version_check=False,min_mem_size_GB=min_mem_size,port=port_no) # start h2o
try:
  h2o.init(strict_version_check=False,min_mem_size_GB=min_mem_size,port=port_no) # start h2o
except:
  logging.critical('h2o.init')
  h2o.download_all_logs(dirname=logs_path, filename=logfile)      
  h2o.cluster().shutdown()
  sys.exit(2)<jupyter_output>Checking whether there is an H2O instance running at http://localhost:42708 ..... not found.
Attempting to start a local H2O server...
  Java Version: java version "1.8.0_311"; Java(TM) SE Runtime Environment (build 1.8.0_311-b11); Java HotSpot(TM) 64-Bit Server VM (build 25.311-b11, mixed mode)
  Starting server from /opt/anaconda3/lib/python3.9/site-packages/h2o/backend/bin/h2o.jar
  Ice root: /<KEY>
  JVM stdout: /var/folders/5q/w_y2sfjj2bv130sh86slry7r0000gq/T/tmp__1_yumb/h2o_work_started_from_python.out
  JVM stderr: /var/folders/5q/w_y2sfjj2bv130sh86slry7r0000gq/T/tmp__1_yumb/h2o_work_started_from_python.err
  Server is running at http://127.0.0.1:42708
Connecting to H2O server at http://127.0.0.1:42708 ... successful.<jupyter_text>Import data and Manage Data TypesThis exploration of H2O will use a version of<jupyter_code># Import the processed data from notebook One
url = "https://raw.githubusercontent.com/sanapsanket/Bank-Loan-Status-Predictive-Analysis/main/KaggleDataset/BankLoanStatusDataset/credit_train.csv"
df = h2o.import_file(path = url)
df.head(1)
df = df.na_omit()
#best.varimp(use_pandas=True)
#df.describe()
#df=df.drop('Loan ID')
#df=df.drop('Customer ID')
#df=df.drop('Tax Liens')
#df=df.drop('Bankruptcies')
#df=df.drop('Number of Credit Problems')
#df=df.drop('Term')
#df=df.drop('Home Ownership')
#df=df.drop('Maximum Open Credit')
#df=df.drop('Purpose')
#df=df.drop('Years in current job')
# Create a 80/20 train/test splie
pct_rows=0.80
df_train, df_test = df.split_frame([pct_rows])
print(df_train.shape)
print(df_test.shape)
df_train.head(1)<jupyter_output><empty_output><jupyter_text>Train Models Using H2O's AutoML<jupyter_code># Set the features and target
X=df.columns
# Set target and predictor variables
target ='Loan Status'
X.remove(target)
df_train[target]=df_train[target].asfactor()
df_test[target]=df_test[target].asfactor()
print(X)<jupyter_output>['Loan ID', 'Customer ID', 'Current Loan Amount', 'Term', 'Credit Score', 'Annual Income', 'Years in current job', 'Home Ownership', 'Purpose', 'Monthly Debt', 'Years of Credit History', 'Months since last delinquent', 'Number of Open Accounts', 'Number of Credit Problems', 'Current Credit Balance', 'Maximum Open Credit', 'Bankruptcies', 'Tax Liens']<jupyter_text>RegressionH20 AutoML will automatically perform regression or classification depedending on the target data type.<jupyter_code># Set up AutoML
aml = H2OAutoML(seed=1,max_runtime_secs=run_time,verbosity="info")
aml.train(x=X,y=target,training_frame=df_train)<jupyter_output>AutoML progress: |
15:42:20.811: Project: AutoML_6_20220211_154220
15:42:20.812: 5-fold cross-validation will be used.
15:42:20.816: Setting stopping tolerance adaptively based on the training frame: 0.0035241429027194466
15:42:20.816: Build control seed: 1
15:42:20.818: training frame: Frame key: AutoML_6_20220211_154220_training_py_27_sid_9826    cols: 19    rows: 80518  chunks: 32    size: 5237594  checksum: -4210142017087432970
15:42:20.818: validation frame: NULL
15:42:20.818: leaderboard frame: NULL
15:42:20.818: blending frame: NULL
15:42:20.818: response column: Loan Status
15:42:20.818: fold column: null
15:42:20.818: weights column: null
15:42:20.821: Loading execution steps: [{XGBoost : [def_2 (1g, 10w), def_1 (2g, 10w), def_3 (3g, 10w), grid_1 (4g, 90w), lr_search (6g, 30w)]}, {GLM : [def_1 (1g, 10w)]}, {DRF : [def_1 (2g, 10w), XRT (3g, 10w)]}, {GBM : [def_5 (1g, 10w), def_2 (2g, 10w), def_3 (2g, 10w), def_4 (2g, 10w), def_1 (3g, 10w), grid_1 (4g, 60w), lr_annealing (6g, 10[...]<jupyter_text>Classifying Loan Status<jupyter_code>print(aml.leaderboard)
df=df.drop('Loan ID')
df=df.drop('Customer ID')
df = df.na_omit()
# Create a 80/20 train/test splie
pct_rows=0.80
df_train, df_test = df.split_frame([pct_rows])
# Set the features and target
X=df.columns
# Set target and predictor variables
target ='Loan Status'
X.remove(target)
df_train[target]=df_train[target].asfactor()
df_test[target]=df_test[target].asfactor()
print(X)
# Set up AutoML
aml2 = H2OAutoML(seed=1,max_runtime_secs=run_time, exclude_algos=["StackedEnsemble"],verbosity="info")
aml2.train(x=X,y=target,training_frame=df_train)
print(aml2.leaderboard)
aml.corr
best=aml.get_best_model()
best.accuracy?
best.varimp(use_pandas=True)
best.accuracy()
best.algo
import pandas as pd

k =pd.read_csv("https://raw.githubusercontent.com/sanapsanket/Bank-Loan-Status-Predictive-Analysis/main/KaggleDataset/BankLoanStatusDataset/credit_train.csv")
k.corr()<jupyter_output><empty_output><jupyter_text>RSME comparison and understanding the leader boardThe best models after running for a little under four minutes is around 0.005 about half of that of the 0.010 RSME that we got our simple MLP in notebook one and a quarter of the  0.017 RSME that we got with a simple MLP with the same independent variables. When we run for a short time, under 10 minutes, out leaderboard with be biased towards trre-based methods as the deep learners take much more time to converge.  It is rare to see deep learners in the top 500 models when we run for less than 5 moinutes.We should still plot the results but before we do that let's discuss a big advantage of these models, model interpretability.<jupyter_code>model_index=0
glm_index=0
glm_model=''
aml_leaderboard_df=aml.leaderboard.as_data_frame()

models_dict={}
for m in aml_leaderboard_df['model_id']:
  models_dict[m]=model_index
  if 'StackedEnsemble' not in m:
    break
  model_index=model_index+1

for m in aml_leaderboard_df['model_id']:
  if 'GLM' in m:
    models_dict[m]=glm_index
    break  
  glm_index=glm_index+1     
models_dict<jupyter_output><empty_output><jupyter_text>Examine the Best Model<jupyter_code>print(0)
StackedEnsemble = h2o.get_model(aml.leaderboard[0,'model_id'])
DRF = h2o.get_model(aml.leaderboard[6,'model_id'])
DRF.algo
perf = aml.leader.model_performance(df_test)
perf.auc()
StackedEnsemble.algo
import matplotlib.pyplot as plt
%matplotlib inline

import warnings
import matplotlib.cbook
warnings.filterwarnings("ignore", category = matplotlib.cbook.mplDeprecation)<jupyter_output><empty_output><jupyter_text>Variable importance plotVariable importance plots in tree-based methods provides a list of the most significant variables in descending order by a measure of the information in each variable. Remember that tree calculates the information content of each variable. A variable importance plot is just a bar chart of each variables information content in decreasing order.It can show actual information estimates or standardized plots like the one below. In a standardized plot the most important variable is always given a value of 1.0. The other variables scores represent their percentage of information relative to the most important variable.Notice that some varibales have almost no information content. Knowing this allows for feature selection by removing unimportant variables. This makes a model more effecient to run and helps prevent overfitting as the unimportant variables can fit noise and, as we saw in notebook one, make strange predictions.<jupyter_code>if DRF.algo in ['gbm','drf','xrt','xgboost']:
  DRF.varimp_plot()
if glm_index != 0:
  print(glm_index)
  glm_model=h2o.get_model(aml.leaderboard[glm_index,'model_id'])
  print(glm_model.algo) 
  glm_model.std_coef_plot()
print("RMSE:")
print("StackedEnsemble 0: ",StackedEnsemble.rmse(train = True))
print("DRF: ",DRF.rmse(train = True))
def model_performance_stats(perf):
    d={}
    try:    
      d['mse']=perf.mse()
    except:
      pass      
    try:    
      d['rmse']=perf.rmse() 
    except:
      pass      
    try:    
      d['null_degrees_of_freedom']=perf.null_degrees_of_freedom()
    except:
      pass      
    try:    
      d['residual_degrees_of_freedom']=perf.residual_degrees_of_freedom()
    except:
      pass      
    try:    
      d['residual_deviance']=perf.residual_deviance() 
    except:
      pass      
    try:    
      d['null_deviance']=perf.null_deviance() 
    except:
      pass      
    try:    
      d['aic']=perf.aic() 
    except:
      pass      
    try:
      d['logloss']=perf.logloss() 
    except:
      pass    
    try:
      d['auc']=perf.auc()
    except:
      pass  
    try:
      d['gini']=perf.gini()
    except:
      pass    
    return d
mod_perf=StackedEnsemble.model_performance(df_test)
stats_test={}
stats_test=model_performance_stats(mod_perf)
stats_test
predictions = DRF.predict(df_test)
y_pred=h2o.as_list(predictions)
y_pred[:5]<jupyter_output><empty_output><jupyter_text>Partial Dependence PlotsPartial dependence plots (PDP) show the dependence between the target response and a set of features, marginalizing over the values of all other features. Intuitively, we can interpret the partial dependence as the expected target response as a function of the feature.The partial dependence plot gives a graphical depiction of the marginal effect of a variable on the response. The effect of a variable is measured in change in the mean response. This helps one answer the question of how changing a variables values would change the outcome.The partial dependence plots show only impact of single variable if others are kept constant. But in many cases, there is interaction between variables. Never-the-less, they are very useful in estimating whether, for example, doubling some predictor varible will double a response or whether that predictor varible is already saturated.<jupyter_code>print(X)
best_model.partial_plot(df, cols=['Credit Score'])
best_model.partial_plot(df, cols=['total_night_minutes', 'total_night_calls', 'total_night_charge', 'total_intl_minutes', 'total_intl_calls', 'total_intl_charge', 'number_customer_service_calls'])<jupyter_output>PartialDependencePlot progress: |████████████████████████████████████████████████| (done) 100%

PartialDependence: Partial Dependence Plot of model XGBoost_grid_1_AutoML_2_20220105_04416_model_38 on column 'total_night_minutes'.<jupyter_text>Shutdown H2O Cluster<jupyter_code>h2o.cluster().shutdown()<jupyter_output>H2O session _sid_840b closed.<jupyter_text>Appendix - Generalized Linear Model (GLM)  The generalized linear model (GLM) is a flexible generalization of ordinary linear regression that allows for response variables that have error distribution models other than a normal distribution. The GLM generalizes linear regression by allowing the linear model to be related to the response variable via a link function and by allowing the magnitude of the variance of each measurement to be a function of its predicted value.Generalized Linear Models (GLM) estimate regression models for various probability distributions. In addition to the Gaussian (i.e. normal) distribution, these include Poisson, binomial, and gamma distributions. This can be used either for prediction or classification.Overfitting is prevented using L1, L2 and elastic-net regularization.While not easily parallelable, maximum likelihood estimates of coefficients is very efficient.GLMs are highly interpretable as the coefficients (i.e. the slopes) as they directly related the degree the dependent variable changes in response to a change in each independent variable.For these data and the regression case, we can think of this as linear regression. to In linear regression, the use of the least-squares estimator is justified by the Gauss--Markov theorem, which does not assume that the distribution is normal. From the perspective of generalized linear models, however, it is useful to suppose that the distribution function is the normal distribution with constant variance and the link function is the identity, which isthe canonical link if the variance is known.In our case, we will assume that the the distribution of errors is normal and that the link function is the identity, which means the will will be performing simple linear regression.   Linear regression predicts the response variable $y$ assuming it has a linear relationship with predictor variable(s) $x$ or $x_1, x_2, ,,, x_n$.$$y = \beta_0 + \beta_1 x + \varepsilon .$$*Simple* regression use only one predictor variable $x$. *Mulitple* regression uses a set of predictor variables $x_1, x_2, ,,, x_n$.The *response variable* $y$ is also called the regressand, forecast, dependent or explained variable. The *predictor variable* $x$ is also called the regressor, independent or explanatory variable.The parameters $\beta_0$ and $\beta_1$ determine the intercept and the slope of the line respectively. The intercept $\beta_0$ represents the predicted value of $y$ when $x=0$. The slope $\beta_1$ represents the predicted increase in $Y$ resulting from a one unit increase in $x$.Note that the regression equation is just our famliar equation for a line with an error term.The equation for a line:  $$ Y = bX + a $$$$y = \beta_0 + \beta_1 x $$The equation for a line with an error term:  $$ Y = bX + a + \varepsilon $$$$y = \beta_0 + \beta_1 x + \varepsilon .$$- $b$ = $\beta_1$ = slope- $a$ = $\beta_0$ = $Y$ intercept- $\varepsilon$ = error termWe can think of each observation $y_i$ consisting of the systematic or explained part of the model, $\beta_0+\beta_1x_i$, and the random *error*, $\varepsilon_i$._Zero Slope_Note that when  $\beta_1 = 0$ then response does not change as the predictor changes.For multiple regression $x$ is a $X$ to produce a system of equations:  $$ Y = \beta_0 + \beta_1 X  + \varepsilon $$_The error $\varepsilon_i$_The error term is a catch-all for anything that may affect $y_i$ other than $x_i$. We assume that these errors:* have mean zero; otherwise the forecasts will be systematically biased.* statistical independence of the errors (in particular, no correlation between consecutive errors in the case of time series data).* homoscedasticity (constant variance) of the errors.* normality of the error distribution.If any of these assumptions is violated then the robustness of the model to be taken with a grain of salt._Least squares estimation_In a linear model, the values of $\beta_0$ and $\beta_1$. These need to be estimated from the data. We call this *fitting a model*.The least squares method iis the most common way of estimating $\beta_0$ and $\beta_1$ by minimizing the sum of the squared errors. The values of $\beta_0$ and $\beta_1$ are chosen so that that minimize$$\sum_{i=1}^N \varepsilon_i^2 = \sum_{i=1}^N (y_i - \beta_0 - \beta_1x_i)^2. $$Using mathematical calculus, it can be shown that the resulting **least squares estimators** are$$\hat{\beta}_1=\frac{ \sum_{i=1}^{N}(y_i-\bar{y})(x_i-\bar{x})}{\sum_{i=1}^{N}(x_i-\bar{x})^2} $$ and$$\hat{\beta}_0=\bar{y}-\hat{\beta}_1\bar{x}, $$where $\bar{x}$ is the average of the $x$ observations and $\bar{y}$ is the average of the $y$ observations. The estimated line is known as the *regression line*.To solve least squares with gradient descent or stochastic gradient descent (SGD) or losed Form (set derivatives equal to zero and solve for parameters)._Fitted values and residuals_The response values of $y$ obtained from the observed $x$ values arecalled *fitted values*: $\hat{y}_i=\hat{\beta}_0+\hat{\beta}_1x_i$, for$i=1,\dots,N$. Each $\hat{y}_i$ is the point on the regressionline corresponding to $x_i$.The difference between the observed $y$ values and the corresponding fitted values are the *residuals*:$$e_i = y_i - \hat{y}_i = y_i -\hat{\beta}_0-\hat{\beta}_1x_i. $$The residuals have some useful properties including the following two:$$\sum_{i=1}^{N}{e_i}=0 \quad\text{and}\quad \sum_{i=1}^{N}{x_ie_i}=0. $$Residuals are the errors that we cannot predict.Residuals are highly useful for studying whether a given regression model is an appropriate statistical technique for analyzing the relationship.   Appendix - _Decision-tree based methods (DRF, XRT, GBM, and XGBoost)_**What is a decision-tree?**What is a tree?  In mathematics, and more specifically in graph theory, a [tree](https://en.wikipedia.org/wiki/Tree_(graph_theory)) is a directed or an undirected graph in which any two vertices are connected by exactly one path. In other words, any acyclic connected graph is a tree.A tree is an undirected graph G that satisfies any of the following equivalent conditions:  * G is connected and has no cycles.   * G is acyclic, and a simple cycle is formed if any edge is added to G.  * G is connected, but is not connected if any single edge is removed from G.  A rooted tree is a tree in which one vertex/node has been designated the root. The edges of a rooted tree can be assigned a natural orientation, either away from or towards the root, in which case the structure becomes a directed rooted tree. A vertex/node that does not split is called Leaf or Terminal node.     A sub section of entire tree is called branch or sub-tree.  A vertex/node, which is divided into sub-nodes is called parent node of sub-nodes where as sub-nodes are the child of parent node.  A [decision tree](https://en.wikipedia.org/wiki/Decision_tree) is a [supervised learning](https://en.wikipedia.org/wiki/Supervised_learning) algorithm that uses a tree-like graph or model of decisions and their outcomes.  The decision tree can be linearized into decision rules, where the outcome is the contents of the leaf node, and the conditions along the path form a conjunction in the if clause. In general, the rules have the form:$if \quad condition1 \quad and \quad condition2 \quad and \quad condition3 \quad then \quad outcome$Each node in the tree is a decisions/tests. Each path from the tree root to a leaf corresponds to a conjunction of attribute decisions/tests. The tree itself corresponds to a disjunction of these conjunctions.**The 20 Questions of machine learning**  In the traditional [20 Questions](https://en.wikipedia.org/wiki/Twenty_Questions) game, one player is chosen to be the answerer. That person chooses a subject (object) but does not reveal this to the others. All other players are questioners. They each take turns asking a question which can be answered with a simple "Yes" or "No." The questioners try to guess the answerers subject (object).The Two Rules   Rule 1: Questioners ask Yes-or-No questions       Rule 2: Answerer responds with a Yes or a No    Traditionally,first question is something like the following:   * "Is it animal?"    * "Is it vegetable?"    * "Is it mineral?"    Suppose the answer is "<NAME>?"Which would be a better first question?"Is it Taylor Swift?" or "Is it animal?"   **Estimating the information in a data split?**Like 20 questions we want to split data in such a way as to maximize the information generated from the split.To calculate entropy, we can calculate the information difference, $-p_1 \log p_1 - p_2 \log p_2$. Generalizing this to n events, we get:$$entropy(p_1, p_2, ... p_n) = -p_1 \log p_1 - p_2 \log p_2 ...  - p_n \log p_n $$which is just the Shannon entropy$$H_1 (X) = - \sum_{i=1}^n p_i \log p_i. $$For example, if entropy = $-1.0 \log (1.0) - 0.0 \log (0.0) = 0$ then this provides no information. If entropy = $-0.5 \log (0.5) - 0.5 \log (0.5) = 1.0$ then this provides one “bit” of information.  Note that when $P(X)$ is 0.5 one is most uncertain and the Shannon entropy is highest (i.e. 1). When $P(X)$ is either 0.0 or 1.0 one is most certain and the Shannon entropy is lowest (i.e. 0)_Shannon entropy_  The notion of using entropy as a measure of change in system state and dynamics comes both from [statistical physics](https://en.wikipedia.org/wiki/Entropy) and from [information theory](https://en.wikipedia.org/wiki/Entropy_(information_theory)). In statistical physics, entropy is a measure of disorder and uncertainty in a random variable; the higher the entropy, the greater the disorder. In the statistical physics context, the term usually refers to [Gibbs entropy](https://en.wikipedia.org/wiki/Entropy_(statistical_thermodynamics)), which measures the macroscopic state of the system as defined by a distribution of atoms and molecules in a thermodynamic system. Gibbs entropy is a measure of the disorder in the arrangements of its particles. As the position of a particle becomes less predictable, the entropy increases. For a classical system (i.e., a collection of classical particles) with a discrete set of microstates, if $E_i$ is the energy of microstate $i$, and $p_i$ is the probability that it occurs during the system's fluctuations, then the entropy of the system is$$S = -k_\text{B}\,\sum_i p_i \ln \,p_i$$The quantity $k_\text{B}$ is a physical constant known as [Boltzmann's constant](https://en.wikipedia.org/wiki/Boltzmann_constant), which, like the entropy, has units of heat capacity. The logarithm is dimensionless.In information theory, entropy is also a measure of the uncertainty in a random variable. In this context, however, the term usually refers to the [Shannon entropy](https://en.wikipedia.org/wiki/Entropy_(information_theory)), which quantifies the expected value of the information contained in a message (or the expected value of the information of the probability distribution). The concept was introduced by [<NAME>](https://en.wikipedia.org/wiki/Claude_Shannon) in his 1948 paper "A Mathematical Theory of Communication." Shannon entropy establishes the limits to possible data compression and channel capacity.  That is, the entropy gives a lower bound for the efficiency of an encoding scheme (in other words, a lower bound on the possible compression of a data stream). Typically this is expressed in the number of ‘bits’ or ‘nats’ that are required to encode a given message. Given the probability of each of n events, the information required to predict an event is the distribution’s entropy. Low entropy means the system is very ordered, that is, very predictable. High entropy means the system is mixed, that is, very unpredictable; a lot of information is needed for prediction. The Shannon entropy can explicitly be written as$$E(X) = \sum_{i} {\mathrm{P}(x_i)\,\mathrm{I}(x_i)} = -\sum_{i} {\mathrm{P}(x_i) \log_b \mathrm{P}(x_i)},$$where b is the base of the logarithm used. Common values of b are 2, Euler's number $e$, and 10, and the unit of entropy is shannon for b = 2, nat for b = e, and hartley for b = 10.When b = 2, the units of entropy are also commonly referred to as bits.The Shannon entropy is by far the most common information-theoretic measure there are others. Other information-theoretic measures include: plog,Rényi entropy, Hartley entropy, collision entropy, min-entropy, Kullback-Leibler divergence and the information dimension.The Shannon entropy is the Rényi entropy with an alpha of one (see appendix). The Shannon entropy is a simple estimate of the expected value of the information contained in a message. It assumes independence and identically distributed random variables, which is a simplification when applied to word counts. In this sense it is analogous to naïve Bayes, in that it is very commonly used and thought to work well in spite of violating some assumptions upon which it is based.The limiting value of $H_\alpha as \alpha \rightarrow 1$ is the Shannon entropy:$$H_1(X) = - \sum_{i=1}^n p_i \log p_i. $$**Classification vs Regression Trees**  Types of decision tree is based on the type of target variable we have. It can be of two types:Classification Tree (Categorical Response Variable Decision Tree): Decision Tree which separates the dataset into classes belonging to the categorical target variable. Usually the response variable has two classes: Yes or No (1 or 0).  Regression trees (Continuous Response Variable Decision Tree): If a decision Tree has continuous target variable it is applicable for prediction type of problems as opposed to classification.<jupyter_code>def shannon_entropy(p):   
    return (-p *np.log2(p) - (1-p)*np.log2(1-p))

base=0.0000000001
x = np.arange(base, 1.0-base, 0.01)


plt.figure(1)
plt.plot(x, shannon_entropy(x), 'go', x, shannon_entropy(x), 'k')
plt.ylabel('Shannon entropy(X)')
plt.xlabel('X')
plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08203426951816542, 'fraction_numerical': 0.02103195052821438, 'mean_word_length': 4.992858521520942, 'pattern_counts': {'":': 0, '<': 43, '<?xml version=': 0, '>': 44, 'https://': 16, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 2, 'substrings_counts': 1, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '396734', 'license': 'MIT', 'n_tokens_mistral': 9392, 'n_tokens_neox': 8727, 'n_words': 3832, 'path': 'src/BankStatusClassificationAutoML.ipynb', 'repo_name': 'sanketsanap5/Bank-Loan-Status-Predictive-Analysis', 'size': '31120'}