<jupyter_start><jupyter_text>Table of Contents1&nbsp;&nbsp;DVC + MLflow = experiments managementm1.1&nbsp;&nbsp;Create a new log_metrics pipeline stage2&nbsp;&nbsp;Experimenting2.1&nbsp;&nbsp;Experiment 12.2&nbsp;&nbsp;Experiment 2: params tuning3&nbsp;&nbsp;Show metrics and params diffs with DVC  DVC + MLflow = experiments managementm  Create a new `log_metrics` pipeline stageThen we can add it to DVC as a stage and run together with other stades through `dvc repro` Install `catboost` flavour which is not supported out of the box:<jupyter_code>! pip install src/utils/cb_flavor/.<jupyter_output>Processing ./src/utils/cb_flavor
Using legacy 'setup.py install' for cb-flavor, since package 'wheel' is not installed.
Installing collected packages: cb-flavor
    Running setup.py install for cb-flavor ... [?25ldone
[?25hSuccessfully installed cb-flavor-0.0.1
[33mWARNING: You are using pip version 21.0; however, version 21.0.1 is available.
You should consider upgrading via the '/Users/antongusarov/ML_REPA/tutorial-predict-device-change/venv/bin/python -m pip install --upgrade pip' command.[0m<jupyter_text>Check that the newly created `log_metrics` pipeline stage is functioning properly:<jupyter_code>! python -m src.pipelines.log_metrics --config=config/params.yaml<jupyter_output>2021-02-25 23:18:53,090 — LOG_METRICS — INFO — Start logging
2021-02-25 23:18:53,100 — LOG_METRICS — DEBUG — Experiment ID: 3, Run ID: f5a3393ee58544a0aef78787a8e41fa9
2021-02-25 23:18:53,104 — LOG_METRICS — INFO — Log train metrics
2021-02-25 23:18:53,109 — LOG_METRICS — INFO — Log cross-validation metrics (raw metrics, by folds
2021-02-25 23:18:53,112 — LOG_METRICS — DEBUG — lift: {'2020-05-31': 2.1466859497062547, '2020-06-30': 2.1505471897179307, '2020-07-31': 2.1445188225607903, '2020-08-31': 2.1383669439425}
2021-02-25 23:18:53,117 — LOG_METRICS — DEBUG — precision_at_k: {'2020-05-31': 0.8380311652395894, '2020-06-30': 0.8377120672233155, '2020-07-31': 0.8346540445673563, '2020-08-31': 0.8354251981066851}
2021-02-25 23:18:53,124 — LOG_METRICS — DEBUG — recall_at_k: {'2020-05-31': 1.0, '2020-06-30': 1.0, '2020-07-31': 1.0, '2020-08-31': 1.0}
2021-02-25 23:18:53,132 — LOG_METRICS — INFO — Log params
2021-02-25 23:18:53,136 — LOG_METRICS — INFO — Log artifacts
2021-02-25 23:18:53,61[...]<jupyter_text>! mlflow ui Add this new pipeline stage into DVC (`dvc.yaml`):<jupyter_code>%%bash
dvc run -f \
    -n log_metrics \
    -d data/processed/features.feather \
    -d models/model.joblib \
    -d reports/train_metrics.json \
    -d reports/raw_metrics.csv \
    -O reports/mlflow_report.md\
    -p config/params.yaml:train,log_metrics \
    python -m src.pipelines.log_metrics --config=config/params.yaml
! dvc dag<jupyter_output>7[?47h[?1h=






















[H[2J[H[H[2J[H+-----------------+  [m
| load_data_stage |  [m
+-----------------+  [m
+-----------------+  [m
| featurize_stage |  [m
+-----------------+  [m
+--------------------+               +-------------+  [m
| data/processed.dvc |               | train_stage |  [m
+--------------------+               +-------------+  [m
                   **               **                [m
                     ***         ***                  [m
                        **     **                     [m
                    +-------------+                   [m
                    | log_metrics |                   [m
                    +-------------+                   [m
[1m~[m
[1m~[m
[1m~[m
[1m~[m
[1m~[m
[1m~[m
[1m~[m
[1m~[m
[K[7m(END)[m[Kp/zg2nfhts6q74ptj2dn93gxfr0000gn/T/tmpnelsbodb (END)[m[K<jupyter_text>Experimenting  Experiment 1 - Create a new branch `experiment_1`<jupyter_code>! git checkout -b "experiment_1"<jupyter_output>fatal: A branch named 'experiment_1' already exists.<jupyter_text>- Modify any train parameters in `params.yaml`For example:```yamltrain:  estimator: catboost  catboost_params:    iterations: 10    thread_count: 50    has_time: true    depth: 6    learning_rate: 0.15    allow_writing_files: false``` - Run experiment:<jupyter_code>! dvc repro<jupyter_output>Stage 'load_data_stage' didn't change, skipping                       core[39m>
'data/processed.dvc' didn't change, skipping
Stage 'train_stage' didn't change, skipping
Stage 'log_metrics' didn't change, skipping
Stage 'featurize_stage' didn't change, skipping
Data and pipelines are up to date.
[0m<jupyter_text>Note that `dvc.lock` has changed accordingly reflecting the updated model paramaters.- Commit the experiment: ```bashgit add .git commit -m "Make experiment 1"``` - **Add tag**Use MLflow `run_id` as a part of the tag ```bashgit tag exp-1-[MLFLOW_RUN_ID] -a``` ```bash ! git tag exp-1-89997d6ead9845babd4d6263b85bc217 -a```<jupyter_code>! git tag<jupyter_output>exp-1-89997d6ead9845babd4d6263b85bc217
exp_1.0
v0.1
v0.2<jupyter_text>- **DVC Push** using `dvc push`:<jupyter_code>! dvc push -r myremote<jupyter_output>Everything is up to date.                                                       
[0m<jupyter_text>- **Push to `origin experiments`**:<jupyter_code>! git push origin experiment_1<jupyter_output>Enumerating objects: 19, done.
Counting objects: 100% (19/19), done.
Delta compression using up to 8 threads
Compressing objects: 100% (10/10), done.
Writing objects: 100% (10/10), 29.94 KiB | 5.99 MiB/s, done.
Total 10 (delta 4), reused 0 (delta 0)
remote: Resolving deltas: 100% (4/4), completed with 4 local objects.[K
remote: This repository moved. Please use the new location:[K
remote:   <EMAIL>:AntonGusarov/MLOps-demo-project.git[K
remote: 
remote: Create a pull request for 'experiment_1' on GitHub by visiting:[K
remote:      https://github.com/AntonGusarov/MLOps-demo-project/pull/new/experiment_1[K
remote: 
To github.com:AntonGusarov/predict-device-change.git
 * [new branch]      experiment_1 -> experiment_1<jupyter_text>**Create a Merge Request into & merge `dev`**  Experiment 2: params tuning **GitHub**:- Create an issue: `Add experiment 2`- Create a new branch from `exeriments`**Git fetch** the experiment branch to local workspace```bashgit fetchgit checkout ```<jupyter_code>! git branch
! git fetch
! git branch -a
! git checkout remotes/origin/experiment_2<jupyter_output>D	1.1-antong_step_3_unittests.ipynb
D	2.1-antong_step4_dvc.ipynb
A	Untitled1.ipynb
A	step-4.1-metrics-tracking.ipynb
A	step-4.2-model-life-cycle-management.ipynb
A	step-4.3-experiment-management.ipynb
Note: checking out 'remotes/origin/experiment_2'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:

  git checkout -b <new-branch-name>

HEAD is now at 92d2082 Merge pull request #3 from AntonGusarov/experiment_1<jupyter_text>Used PyCharm to checkout from a remote branch (automatically creates a local copy of a remote branch).<jupyter_code>! git branch -a<jupyter_output>dvc_pipelines[m
  experiment_1[m
* [32mexperiment_2[m
  master[m
  [31mremotes/AntonGusarov/dvc_pipelines[m
  [31mremotes/AntonGusarov/experiment_1[m
  [31mremotes/AntonGusarov/master[m
  [31mremotes/origin/HEAD[m -> origin/master
  [31mremotes/origin/dvc_pipelines[m
  [31mremotes/origin/experiment_1[m
  [31mremotes/origin/experiment_2[m
  [31mremotes/origin/master[m<jupyter_text>**Modify any train parameters in `params.yaml`**For example:```yamlbase:  ...  exp_name: 'Experiment 2'  catboost_params:    iterations: 30``` - **Run experiment** via DVC pipeline<jupyter_code>! dvc repro<jupyter_output>'data/processed.dvc' didn't change, skipping                          core[39m>
Running stage 'train_stage' with command:
	python -m src.pipelines.train --config=config/params.yaml
{'project_dir': '.', 'random_state': 42, 'log_level': 'DEBUG', 'exp_name': 'Experiment 1'}
2021-02-26 13:04:57,784 — TRAIN — INFO — Load data
2021-02-26 13:04:58,358 — TRAIN — INFO — Instantiate model
2021-02-26 13:04:58,445 — TRAIN — INFO — Top_K 5.0% of the dataset size: 37606
2021-02-26 13:04:58,446 — TRAIN — INFO — Fold 1:
2021-02-26 13:04:58,446 — TRAIN — INFO — Train: 2020-04-30 00:00:00 - 2020-04-30 00:00:00
2021-02-26 13:04:58,446 — TRAIN — INFO — Test: 2020-05-31 00:00:00 

2021-02-26 13:04:58,566 — TRAIN — INFO — Train shapes: X - (150484, 30), y - (150484,)
2021-02-26 13:04:58,567 — TRAIN — INFO — Test shapes: X - (150411, 30), y - (150411,)
0:	learn: 0.6627438	total: 192ms	remaining: 5.58s
1:	learn: 0.6299553	total: 297ms	remaining: 4.15s
2:	learn: 0.6057897	total: 409ms	remaining: 3.68s
3:	lear[...]<jupyter_text>We can now check the results in MLflow server and `mlflow_report.md` to check the current experiment and run details.  - **Commit experiment** - **Add tag** using PyCharm<jupyter_code>! git tag<jupyter_output>exp-1-89997d6ead9845babd4d6263b85bc217
exp-2-dca173dc365e4ab5b4bb7decc0e83bab
exp_1.0
v0.1
v0.2<jupyter_text>- **git push**<jupyter_code>! git branch
! git push<jupyter_output>Enumerating objects: 13, done.
Counting objects: 100% (13/13), done.
Delta compression using up to 8 threads
Compressing objects: 100% (7/7), done.
Writing objects: 100% (7/7), 848 bytes | 424.00 KiB/s, done.
Total 7 (delta 4), reused 0 (delta 0)
remote: Resolving deltas: 100% (4/4), completed with 4 local objects.[K
remote: This repository moved. Please use the new location:[K
remote:   <EMAIL>:AntonGusarov/MLOps-demo-project.git[K
To github.com:AntonGusarov/predict-device-change.git
   92d2082..dfd70aa  experiment_2 -> experiment_2<jupyter_text>Show metrics and params diffs with DVC<jupyter_code>! dvc metrics show
!dvc metrics diff exp-1-89997d6ead9845babd4d6263b85bc217 exp-2-dca173dc365e4ab5b4bb7decc0e83bab<jupyter_output>Path                        Metric               Old      New      Changee[39m>
reports/train_metrics.json  lift_max             2.0829   2.14905  0.06615
reports/train_metrics.json  lift_mean            2.07286  2.14195  0.06908
reports/train_metrics.json  lift_min             2.06383  2.13258  0.06875
reports/train_metrics.json  lift_std             0.00785  0.00706  -0.00079
reports/train_metrics.json  precision_at_k_max   0.81136  0.83745  0.02609
reports/train_metrics.json  precision_at_k_mean  0.80832  0.83525  0.02694
reports/train_metrics.json  precision_at_k_min   0.80325  0.83316  0.02992
reports/train_metrics.json  precision_at_k_std   0.00356  0.00235  -0.0012
[0m
{'fraction_non_alphanumeric': 0.13851510303942788, 'fraction_numerical': 0.11094382233932436, 'mean_word_length': 3.5264054514480407, 'pattern_counts': {'":': 0, '<': 46, '<?xml version=': 0, '>': 52, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 7, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '14', 'dir': 'jupyter-structured-clean-dedup', 'id': '511833', 'license': 'MIT', 'n_tokens_mistral': 4707, 'n_tokens_neox': 3844, 'n_words': 981, 'path': '3.2-antong-step-5-MLflow-experiment-management.ipynb', 'repo_name': 'mnrozhkov/MLOps-demo-project', 'size': '10641'}