<jupyter_start><jupyter_text>Analysis IIIn this thesis that I am preparing is another method for analyzing the importance of hyperparameter. In this method, my new aproach is considering the below assumption: Assumtion* Why not the Data Base of `Hyperparameter` that will be turned in by the BD team be itself considered as a Individual Dataset - This approach is called as building a `Surrogate Model`* The dataset will consist of MetaData, Algorithm, Leaderboard(collected from H2O), Hyperparameter for the respective Algorithms All arranged as a columns with it's corresponding respective values in it. Now, this I am going o use it to build a model around it.Below are the columns, which are dummy as far now, as I have built it individually according to my assumptions. The DB team is yet to turn in it's DataBase, I am waiting for the good model amongst what they turn in to be considered for my thesis - to build a model.The columns that I have taken till now are as below with it's description:* runid	- Run id for a dataset and its iteration* dataset	- The dataset and its link from where it's being fetched* problem	- if it is a classification/regression* runtime	- runtime for which H2O was run on this dataset* columns	- number of columns in the dataset* rows	- number rows in the dataset* tags	- What genre it belongs to* algo	- algorithm applied on this dataset by H2O* model_id	- the model id generated* ntree	- value for n_estimator* max_depth	- value for the hyperaparameter* learn_rate	- value for the hyperaparameter* mean_residual_deviance	- value for the hyperaparameter* rmse	- value for the metric* mse	- value for the metric* mae	- value for the metric* rmsle - value for the metric![image.png](attachment:image.png)<jupyter_code>import h2o
from h2o.automl import H2OAutoML
import pandas as pd
from sklearn.model_selection import train_test_split
from xgboost import XGBClassifier
from sklearn.model_selection import GridSearchCV
from sklearn.model_selection import StratifiedKFold
import matplotlib.pyplot as plt
h2o.init(min_mem_size_GB=2)
df = h2o.import_file('./hyperparamter_db_test.csv')<jupyter_output>Parse progress: |█████████████████████████████████████████████████████████| 100%<jupyter_text>This dataset is created on own, and the data are not true. This is just t test the thesis and the hypothesis to check if this will give us a good analysis results<jupyter_code>df.head()<jupyter_output><empty_output><jupyter_text>I am running for a simple AutoML runtime of 100 seconds. The results will be better if I run for more time.<jupyter_code>aml = H2OAutoML(max_runtime_secs=100)
X=df.columns<jupyter_output><empty_output><jupyter_text>Below are the predictors we have considered<jupyter_code>X = ['runid',
 'dataset',
 'problem',
 'runtime',
 'columns',
 'rows',
 'tags',
 'model_id',
 'ntree',
 'max_depth',
 'learn_rate',
 'mean_residual_deviance',
 'rmse',
 'mse',
 'mae',
 'rmsle']
y='algo'
aml.train(x=X,y=y,training_frame=df)<jupyter_output>AutoML progress: |████████████████████████████████████████████████████████| 100%<jupyter_text>On running H2O AutoML for 100 seconds, for this simple Surrogate Dataset, we are getting 68 models generated by H2O<jupyter_code>aml_leaderboard_df=aml.leaderboard.as_data_frame()
aml_leaderboard_df
model_set=aml_leaderboard_df['model_id']
mod_best=h2o.get_model(model_set[0])
mod_best<jupyter_output>Model Details
=============
H2OGradientBoostingEstimator :  Gradient Boosting Machine
Model Key:  GBM_grid_0_AutoML_20190423_134333_model_32


ModelMetricsMultinomial: gbm
** Reported on train data. **

MSE: 0.005379815592386542
RMSE: 0.07334722620785698
LogLoss: 0.07493269418958665
Mean Per-Class Error: 0.0
Confusion Matrix: Row labels: Actual class; Column labels: Predicted class<jupyter_text>Now, what we perform here is important and is going to give us a definite result for our analysis. Something called Variable importance is brought into the picture to analysis our thesis. `Variable importance is an indication of which predictors are most useful for predicting the response variable. Various measures of variable importance have been proposed in the data mining literature. The most important variables might not be the ones near the top of the tree.`Below is something that H2O turns in when one of the models except `StackedEnsemble` is called<jupyter_code>mod_best.varimp_plot()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.0894844424256189, 'fraction_numerical': 0.024301612536906655, 'mean_word_length': 5.870514820592824, 'pattern_counts': {'":': 0, '<': 22, '<?xml version=': 0, '>': 22, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '509721', 'license': 'MIT', 'n_tokens_mistral': 1474, 'n_tokens_neox': 1318, 'n_words': 537, 'path': 'Phase II/Surrogate Model using VarrImp Plot.ipynb', 'repo_name': 'prabhuSub/BDIA_Project', 'size': '4403'}