<jupyter_start><jupyter_text>View `params.yaml` config<jupyter_code># Look on pipelines config 

with open('params.yaml') as conf_file:
    config = conf_file.read()
    
print(config)<jupyter_output>base:
  project: dvc-5-demo-project-iris

  model:
    model_name: model.joblib
    models_folder: models

  reports:
    reports_folder: reports

  random_state: 42 # random state for train/test split


data_load:
  dataset_csv: data/raw/iris.csv


featurize:
  features_path: data/processed/featured_iris.csv
  target_column: target


data_split:
  folder: experiments
  train_path: data/processed/train_iris.csv
  test_path: data/processed/test_iris.csv
  test_size: 0.3


train:
  cv: 3
  estimator_name: svm
  estimators:

    logreg: # sklearn.linear_model.LogisticRegression
      param_grid: # params of GridSearchCV constructor
        C: [0.001, 0.01]
        max_iter: [100]
        solver: ['lbfgs']
        multi_class: ['multinomial']

    svm: # sklearn.svm.SVC
      param_grid:
        C: [0.1, 1.0]
        kernel: ["rbf", "linear"]
        gamma: ["scale"]
        degree: [3, 5]

evaluate:
  metrics_file: metrics.json
  confusion_matrix_png: confusion_matrix.png
  classes_path: [...]<jupyter_text>Load data- src/pipelines/data_load.py  Run as Python function<jupyter_code>from src.pipelines.data_load import data_load

data_load('params.yaml')<jupyter_output><empty_output><jupyter_text>Run as Python module<jupyter_code>!python src/pipelines/data_load.py --config=params.yaml
%%bash 

# Raw Iris dataset saved

ls data/raw<jupyter_output>iris.csv<jupyter_text>Extract features- `src/pipelines/featurize.py`<jupyter_code>!python src/pipelines/featurize.py --config=params.yaml
# Features.csv file has created

!ls data/processed<jupyter_output>featured_iris.csv<jupyter_text>Split train/test dataset- `src/pipelines/data_split.py`<jupyter_code>!python src/pipelines/data_split.py --config=params.yaml
# Train and Test datsets have created

!ls data/processed/<jupyter_output>featured_iris.csv  test_iris.csv  train_iris.csv<jupyter_text>Train model- `src/pipelines/train.py`<jupyter_code>!python src/pipelines/train.py --config=params.yaml
# model is created
!ls models/<jupyter_output>model.joblib<jupyter_text>Evaluate model - add `reports/` folder - add `src/pipelines/evaluate.py`<jupyter_code>import sys
sys.path.insert(0, '/home/jay/PycharmProjects/')
sys.path.append('/home/jay/PycharmProjects/')
import os
os.getcwd()
os.chdir('/home/jay/PycharmProjects/DVC_pract01/')
!export PYTHONPATH='/home/jay/PycharmProjects/DVC_pract01'
!python src/pipelines/evaluate.py --config=params.yaml
# metrics file eval.txt is created
!ls reports
!cat reports/metrics.json
# Display confusion matrix

from IPython.display import Image
Image('reports/confusion_matrix.png')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11616341030195382, 'fraction_numerical': 0.009591474245115453, 'mean_word_length': 4.711967545638945, 'pattern_counts': {'":': 0, '<': 24, '<?xml version=': 0, '>': 24, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '174586', 'license': 'MIT', 'n_tokens_mistral': 1081, 'n_tokens_neox': 1027, 'n_words': 183, 'path': 'notebooks/Step-3-Create-pipelines.ipynb', 'repo_name': 'kumarjay/DVC_pract02', 'size': '2815'}