<jupyter_start><jupyter_text>ModelDB Features Demo -- couture.aiThis notebook runs a sample sci-kit model and connects it to a running ModelDB instance -- using ModelDB for model repository and expriment logging.The scikit model used is available in [scikit documentatation](https://scikit-learn.org/stable/tutorial/basic/tutorial.html)  ModelDB references:- [this blog by the founder](https://blog.verta.ai/blog/model-versioning-done-right-a-modeldb-2.0-walkthrough)- [webinar 1 code](https://github.com/VertaAI/modeldb/blob/master/demos/webinar-2020-5-6/02-mdb_versioned/01-train/02%20Positive%20Data%20NLP.ipynb)- [webinar 2 code]()  Setting up Git for Notebook  and other settings ModelDB uses **git** for tracking model versions, this notebook should be in a proper git repository. Ensure that you've setup the repository with a git origin URL.Before running this notebook:- make sure ModelDB is running at localhost:3000- create a new test folder for this repo and run ```git init``` in it- in this new folder, set an origin ```git remote add origin .git```   Imports- Scikit Learn (Dataset and models)- Verta (ModelDB API calls)- Itertools<jupyter_code># imports and package installations
import itertools
try:
    import sklearn
except:
    !pip3 install scikit-learn
try:
    import verta
except:
    !pip3 install<jupyter_output><empty_output><jupyter_text>Importing a sample datasetThe sklearn [digits](https://scikit-learn.org/stable/modules/generated/sklearn.datasets.load_digits.html) dataset is used and split into test and training components in a 1:2 ratio.<jupyter_code># getting sample dataset
from sklearn import datasets
from sklearn.model_selection import train_test_split

digits = datasets.load_digits()
X_train, X_test, y_train, y_test = train_test_split(digits.data, digits.target, test_size=0.33)<jupyter_output><empty_output><jupyter_text>Constants for ModelDB Naming and BehaviourSpecify the the following:- Project Name- Experiment Name- Repository Name- Branch Name- *whether to marge branch to master*<jupyter_code># Projects Constants
HOST = "http://localhost:3000"
PROJECT_NAME = "Couture Sklearn Test"
EXPERIMENT_NAME = "Experiment 1"
REPOSITORY_NAME = "couture-sklearn-repo"
BRANCH_NAME = "svm-test"
MERGE_TO_MASTER = True<jupyter_output><empty_output><jupyter_text>Setting up ModelDB Project, Experiment and Repository and Dataset<jupyter_code># setting up ModelDB project, experiment and repository
from verta import Client
from verta.utils import ModelAPI

client = Client(HOST)
proj = client.set_project(PROJECT_NAME)
expt = client.set_experiment(EXPERIMENT_NAME)

repo = client.set_repository(REPOSITORY_NAME)
commit = repo.get_commit(branch='master').new_branch(BRANCH_NAME)

dataset = client.set_dataset(name="Test Dataset 1",
                             type="local")

# arguments for create version depends on type of dataset
dataset_version = dataset.create_version(path="digits.csv")

# run.log_dataset_version("training_data", dataset_version)<jupyter_output>connection successfully established
got existing Project: Couture Sklearn Test v9
got existing Experiment: Experiment 1
set existing Repository: couture-sklearn-v9 from personal workspace
set existing Dataset: Test Dataset 1 from personal workspace
created new DatasetVersion: 1<jupyter_text>HyperparametersMultiple values can be specified for each hyperparameter and an experiment run would be generated for each possible combination of hyperparameter values. For example, if ```n``` values for *parameter A*, ```m``` values for *parameter B* and ```p``` values for *parameter C* are declared, then there would be a total fo ```m*n*p``` runs in the experiment.This behaviour can be changed by specifying what's wanted for ```itertools```. Check the documentation [here](https://docs.python.org/3/library/itertools.html)<jupyter_code># set model hyperparameters
hyperparam_candidates = {
    'C': [90., 95.],
    'gamma': [0.002, 0.001],
}
hyperparam_sets = [dict(zip(hyperparam_candidates.keys(), values))
                   for values
                   in itertools.product(*hyperparam_candidates.values())]<jupyter_output><empty_output><jupyter_text>Experiment RunsThis section is the heart of the code file where the model is defined, trained and experiment runs are conducted for provided combination of hyperparameters. The code is largely self-explanatory. In-line comments have been provided for readability.<jupyter_code>from verta.code import Notebook
from verta.configuration import Hyperparameters
from verta.environment import Python

from sklearn import svm

def run_experiment(hyperparams, index):
    ...
    code_ver = Notebook()
    config_ver = Hyperparameters(hyperparams)
    train_ver = digits.data
    test_ver = digits.target
    env_ver = Python(Python.read_pip_environment())

    commit.update("notebooks/couture-sklearn-test", code_ver)
    commit.update("config/hyperparams", config_ver)
#     commit.update("data/train", train_ver)
#     commit.update("data/test", test_ver)
    commit.update("env/python", env_ver)
    commit.save("Hyperparameter tuning Run: " + str(index))

    # create object to track experiment run
    run = client.set_experiment_run("Run : " + str(index+1))
    
    # log hyperparameters
    run.log_hyperparameters(hyperparams)
    
    # model definition
    model = svm.SVC(**hyperparams)
    
    # model training
    model.fit(X_train, y_train)
    
    # calculate and log validation accuracy
    val_acc = model.score(X_test, y_test)
    run.log_metric("val_acc", val_acc)
    print("Validation accuracy: {:.4f}".format(val_acc))
    
    # save and log model
    run.log_model(model)
    
    # log dataset snapshot as version
    run.log_dataset_version("0.67 training set", dataset_version)
    
#     log Git information as code version
    run.log_code()
    
    run.log_commit(
        commit,
        {
            'notebook': "notebooks/couture-sklearn-test",
            'hyperparameters': "config/hyperparams",
#             'training_data': "data/train",
#             'test_data': "data/test",
            'python_env': "env/python",
        },
    )
     

# to run all experiments
for i, hyperparams in enumerate(hyperparam_sets):
    run_experiment(hyperparams, i)
# Print the commit log
for c in commit.log():
    print(c)
# merge to master if specified in config
master = repo.get_commit(branch='master')
if MERGE_TO_MASTER == True:
    master.merge(commit)<jupyter_output><empty_output><jupyter_text>Get Best RunLook through the accuracy of all model experiment runs to retrieve the most successful experiment.<jupyter_code>best_run = expt.expt_runs.sort("metrics.val_acc", descending=True)[0]
print("Validation Accuracy: {:.4f}".format(best_run.get_metric("val_acc")))

best_hyperparams = best_run.get_hyperparameters()
print("Hyperparameters: {}".format(best_hyperparams))<jupyter_output>Validation Accuracy: 0.9865
Hyperparameters: {'C': 90, 'gamma': 0.0010000000474974513}
{'fraction_non_alphanumeric': 0.09747344243468274, 'fraction_numerical': 0.013063451047947173, 'mean_word_length': 4.701309328968904, 'pattern_counts': {'":': 0, '<': 27, '<?xml version=': 0, '>': 27, 'https://': 5, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 1, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '31007', 'license': 'Apache-2.0', 'n_tokens_mistral': 2159, 'n_tokens_neox': 2063, 'n_words': 606, 'path': 'demos/couture-demo/couture-sklearn-modeldb-test.ipynb', 'repo_name': 'coutureai/CoutureModelDB', 'size': '6966'}