{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "view-in-github"
   },
   "source": [
    "<a href=\"https://colab.research.google.com/github/Tessellate-Imaging/monk_v1/blob/master/study_roadmaps/1_getting_started_roadmap/6_hyperparameter_tuning/1)%20Analyse%20Learning%20Rates.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Goals\n",
    "\n",
    "\n",
    "### Learn how to use hyper parameter analyser for learning rates"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Table of Contents\n",
    "\n",
    "\n",
    "## [Install](#0)\n",
    "\n",
    "\n",
    "## [Load experiment in default mode](#1)\n",
    "\n",
    "\n",
    "## [Run Analyser](#2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='0'></a>\n",
    "# Install Monk"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Using pip (Recommended)\n",
    "\n",
    "  - colab (gpu) \n",
    "      - All bakcends: `pip install -U monk-colab`\n",
    "      \n",
    "\n",
    "  - kaggle (gpu) \n",
    "      - All backends: `pip install -U monk-kaggle`\n",
    "      \n",
    "\n",
    "  - cuda 10.2\t\n",
    "      - All backends: `pip install -U monk-cuda102`\n",
    "      - Gluon bakcned: `pip install -U monk-gluon-cuda102`\n",
    "\t  - Pytorch backend: `pip install -U monk-pytorch-cuda102`\n",
    "      - Keras backend: `pip install -U monk-keras-cuda102`\n",
    "      \n",
    "\n",
    "  - cuda 10.1\t\n",
    "      - All backend: `pip install -U monk-cuda101`\n",
    "\t  - Gluon bakcned: `pip install -U monk-gluon-cuda101`\n",
    "\t  - Pytorch backend: `pip install -U monk-pytorch-cuda101`\n",
    "\t  - Keras backend: `pip install -U monk-keras-cuda101`\n",
    "      \n",
    "\n",
    "  - cuda 10.0\t\n",
    "      - All backend: `pip install -U monk-cuda100`\n",
    "\t  - Gluon bakcned: `pip install -U monk-gluon-cuda100`\n",
    "\t  - Pytorch backend: `pip install -U monk-pytorch-cuda100`\n",
    "\t  - Keras backend: `pip install -U monk-keras-cuda100`\n",
    "      \n",
    "\n",
    "  - cuda 9.2\t\n",
    "      - All backend: `pip install -U monk-cuda92`\n",
    "\t  - Gluon bakcned: `pip install -U monk-gluon-cuda92`\n",
    "\t  - Pytorch backend: `pip install -U monk-pytorch-cuda92`\n",
    "\t  - Keras backend: `pip install -U monk-keras-cuda92`\n",
    "      \n",
    "\n",
    "  - cuda 9.0\t\n",
    "      - All backend: `pip install -U monk-cuda90`\n",
    "\t  - Gluon bakcned: `pip install -U monk-gluon-cuda90`\n",
    "\t  - Pytorch backend: `pip install -U monk-pytorch-cuda90`\n",
    "\t  - Keras backend: `pip install -U monk-keras-cuda90`\n",
    "      \n",
    "\n",
    "  - cpu \t\t\n",
    "      - All backend: `pip install -U monk-cpu`\n",
    "\t  - Gluon bakcned: `pip install -U monk-gluon-cpu`\n",
    "\t  - Pytorch backend: `pip install -U monk-pytorch-cpu`\n",
    "\t  - Keras backend: `pip install -U monk-keras-cpu`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Install Monk Manually (Not recommended)\n",
    " \n",
    "### Step 1: Clone the library\n",
    " - git clone https://github.com/Tessellate-Imaging/monk_v1.git\n",
    " \n",
    " \n",
    " \n",
    " \n",
    "### Step 2: Install requirements \n",
    " - Linux\n",
    "     - Cuda 9.0\n",
    "         - `cd monk_v1/installation/Linux && pip install -r requirements_cu90.txt`\n",
    "     - Cuda 9.2\n",
    "         - `cd monk_v1/installation/Linux && pip install -r requirements_cu92.txt`\n",
    "     - Cuda 10.0\n",
    "         - `cd monk_v1/installation/Linux && pip install -r requirements_cu100.txt`\n",
    "     - Cuda 10.1\n",
    "         - `cd monk_v1/installation/Linux && pip install -r requirements_cu101.txt`\n",
    "     - Cuda 10.2\n",
    "         - `cd monk_v1/installation/Linux && pip install -r requirements_cu102.txt`\n",
    "     - CPU (Non gpu system)\n",
    "         - `cd monk_v1/installation/Linux && pip install -r requirements_cpu.txt`\n",
    " \n",
    " \n",
    "  - Windows\n",
    "     - Cuda 9.0 (Experimental support)\n",
    "         - `cd monk_v1/installation/Windows && pip install -r requirements_cu90.txt`\n",
    "     - Cuda 9.2 (Experimental support)\n",
    "         - `cd monk_v1/installation/Windows && pip install -r requirements_cu92.txt`\n",
    "     - Cuda 10.0 (Experimental support)\n",
    "         - `cd monk_v1/installation/Windows && pip install -r requirements_cu100.txt`\n",
    "     - Cuda 10.1 (Experimental support)\n",
    "         - `cd monk_v1/installation/Windows && pip install -r requirements_cu101.txt`\n",
    "     - Cuda 10.2 (Experimental support)\n",
    "         - `cd monk_v1/installation/Windows && pip install -r requirements_cu102.txt`\n",
    "     - CPU (Non gpu system)\n",
    "         - `cd monk_v1/installation/Windows && pip install -r requirements_cpu.txt`\n",
    "         \n",
    "         \n",
    "  - Mac\n",
    "     - CPU (Non gpu system)\n",
    "         - `cd monk_v1/installation/Mac && pip install -r requirements_cpu.txt`\n",
    "         \n",
    "         \n",
    "  - Misc\n",
    "      - Colab (GPU)\n",
    "          - `cd monk_v1/installation/Misc && pip install -r requirements_colab.txt`\n",
    "      - Kaggle (GPU)\n",
    "          - `cd monk_v1/installation/Misc && pip install -r requirements_kaggle.txt`\n",
    " \n",
    " \n",
    " \n",
    "### Step 3: Add to system path (Required for every terminal or kernel run)\n",
    " - `import sys`\n",
    " - `sys.path.append(\"monk_v1/\");`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dataset - Caltech-256\n",
    "    - https://www.kaggle.com/jessicali9530/caltech256"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "! wget --load-cookies /tmp/cookies.txt \"https://docs.google.com/uc?export=download&confirm=$(wget --save-cookies /tmp/cookies.txt --keep-session-cookies --no-check-certificate 'https://docs.google.com/uc?export=download&id=1Lltrl2U4L8WJkyBjMBFHSaoK8dLhoItl' -O- | sed -rn 's/.*confirm=([0-9A-Za-z_]+).*/\\1\\n/p')&id=1Lltrl2U4L8WJkyBjMBFHSaoK8dLhoItl\" -O caltech256.zip && rm -rf /tmp/cookies.txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "! unzip -qq caltech256.zip"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Using gluon backend \n",
    "\n",
    "# When installed using pip\n",
    "from monk.gluon_prototype import prototype\n",
    "\n",
    "\n",
    "# When installed manually (Uncomment the following)\n",
    "#import os\n",
    "#import sys\n",
    "#sys.path.append(\"monk_v1/\");\n",
    "#sys.path.append(\"monk_v1/monk/\");\n",
    "#from monk.gluon_prototype import prototype"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='1'></a>\n",
    "# Load experiment in default mode"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Mxnet Version: 1.5.0\n",
      "\n",
      "Experiment Details\n",
      "    Project: Project\n",
      "    Experiment: analyser_lr\n",
      "    Dir: /home/abhi/Desktop/Work/tess_tool/gui/v0.3/finetune_models/Organization/development/v5.3_roadmaps/1_getting_started_roadmap/6_hyperparameter_tuning/workspace/Project/analyser_lr/\n",
      "\n"
     ]
    }
   ],
   "source": [
    "gtf = prototype(verbose=1);\n",
    "gtf.Prototype(\"Project\", \"analyser_lr\");"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Dataset Details\n",
      "    Train path:     caltech256/train\n",
      "    Val path:       None\n",
      "    CSV train path: None\n",
      "    CSV val path:   None\n",
      "\n",
      "Dataset Params\n",
      "    Input Size:   224\n",
      "    Batch Size:   4\n",
      "    Data Shuffle: True\n",
      "    Processors:   4\n",
      "    Train-val split:   0.7\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/abhi/.virtualenvs/finetune_py36/lib/python3.6/site-packages/mxnet/gluon/data/vision/datasets.py:312: UserWarning: Ignoring caltech256/train/198.spider/RENAME2 of type . Only support .jpg, .jpeg, .png\n",
      "  filename, ext, ', '.join(self._exts)))\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Pre-Composed Train Transforms\n",
      "[{'RandomHorizontalFlip': {'p': 0.8}}, {'Normalize': {'mean': [0.485, 0.456, 0.406], 'std': [0.229, 0.224, 0.225]}}]\n",
      "\n",
      "Pre-Composed Val Transforms\n",
      "[{'RandomHorizontalFlip': {'p': 0.8}}, {'Normalize': {'mean': [0.485, 0.456, 0.406], 'std': [0.229, 0.224, 0.225]}}]\n",
      "\n",
      "Dataset Numbers\n",
      "    Num train images: 21424\n",
      "    Num val images:   9183\n",
      "    Num classes:      257\n",
      "\n",
      "Model Params\n",
      "    Model name:           resnet18_v1\n",
      "    Use Gpu:              True\n",
      "    Use pretrained:       True\n",
      "    Freeze base network:  False\n",
      "\n",
      "Model Details\n",
      "    Loading pretrained model\n",
      "    Model Loaded on device\n",
      "        Model name:                           resnet18_v1\n",
      "        Num of potentially trainable layers:  41\n",
      "        Num of actual trainable layers:       41\n",
      "\n",
      "Optimizer\n",
      "    Name:          sgd\n",
      "    Learning rate: 0.01\n",
      "    Params:        {'lr': 0.01, 'momentum': 0, 'weight_decay': 0, 'momentum_dampening_rate': 0, 'clipnorm': 0.0, 'clipvalue': 0.0}\n",
      "\n",
      "\n",
      "\n",
      "Learning rate scheduler\n",
      "    Name:   steplr\n",
      "    Params: {'step_size': 1, 'gamma': 0.98, 'last_epoch': -1}\n",
      "\n",
      "Loss\n",
      "    Name:          softmaxcrossentropy\n",
      "    Params:        {'weight': None, 'batch_axis': 0, 'axis_to_sum_over': -1, 'label_as_categories': True, 'label_smoothing': False}\n",
      "\n",
      "Training params\n",
      "    Num Epochs: 5\n",
      "\n",
      "Display params\n",
      "    Display progress:          True\n",
      "    Display progress realtime: True\n",
      "    Save Training logs:        True\n",
      "    Save Intermediate models:  True\n",
      "    Intermediate model prefix: intermediate_model_\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "monk_v1/monk/system/imports.py:160: UserWarning: ArgumentWarning: clipnorm and clipvalue are active only for keras in current version of Monk\n",
      "  warnings.warn(msg)\n",
      "monk_v1/monk/system/imports.py:160: UserWarning: ArgumentWarning: momentum_dampening_rate is active only for pytorch in current version of Monk\n",
      "  warnings.warn(msg)\n"
     ]
    }
   ],
   "source": [
    "gtf.Default(dataset_path=\"caltech256/train\", \n",
    "            model_name=\"resnet18_v1\", \n",
    "            freeze_base_network=False,\n",
    "            num_epochs=5);\n",
    "\n",
    "#Read the summary generated once you run this cell. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='2'></a>\n",
    "# Analyse Learning Rates"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Analysis Project Name\n",
    "analysis_name = \"analyse_learning_rates\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Learning rates to explore\n",
    "lrs = [0.1, 0.05, 0.01, 0.005, 0.0001];"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Num epochs for each sub-experiment to run\n",
    "epochs=10"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Percentage of original dataset to take in for experimentation\n",
    "percent_data=10"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Running Learning rate analysis\n",
      "Analysis Name      : analyse_learning_rates\n",
      "\n",
      "Running experiment : 1/5\n",
      "Experiment name    : Learning_Rate_0.1\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/abhi/.virtualenvs/finetune_py36/lib/python3.6/site-packages/mxnet/gluon/data/vision/datasets.py:312: UserWarning: Ignoring caltech256/train/198.spider/RENAME2 of type . Only support .jpg, .jpeg, .png\n",
      "  filename, ext, ', '.join(self._exts)))\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Estimated time     : 3 min\n",
      "Experiment Complete\n",
      "\n",
      "\n",
      "Running experiment : 2/5\n",
      "Experiment name    : Learning_Rate_0.05\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/abhi/.virtualenvs/finetune_py36/lib/python3.6/site-packages/mxnet/gluon/data/vision/datasets.py:312: UserWarning: Ignoring caltech256/train/198.spider/RENAME2 of type . Only support .jpg, .jpeg, .png\n",
      "  filename, ext, ', '.join(self._exts)))\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Estimated time     : 3 min\n",
      "Experiment Complete\n",
      "\n",
      "\n",
      "Running experiment : 3/5\n",
      "Experiment name    : Learning_Rate_0.01\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/abhi/.virtualenvs/finetune_py36/lib/python3.6/site-packages/mxnet/gluon/data/vision/datasets.py:312: UserWarning: Ignoring caltech256/train/198.spider/RENAME2 of type . Only support .jpg, .jpeg, .png\n",
      "  filename, ext, ', '.join(self._exts)))\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Estimated time     : 3 min\n",
      "Experiment Complete\n",
      "\n",
      "\n",
      "Running experiment : 4/5\n",
      "Experiment name    : Learning_Rate_0.005\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/abhi/.virtualenvs/finetune_py36/lib/python3.6/site-packages/mxnet/gluon/data/vision/datasets.py:312: UserWarning: Ignoring caltech256/train/198.spider/RENAME2 of type . Only support .jpg, .jpeg, .png\n",
      "  filename, ext, ', '.join(self._exts)))\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Estimated time     : 3 min\n",
      "Experiment Complete\n",
      "\n",
      "\n",
      "Running experiment : 5/5\n",
      "Experiment name    : Learning_Rate_0.0001\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/abhi/.virtualenvs/finetune_py36/lib/python3.6/site-packages/mxnet/gluon/data/vision/datasets.py:312: UserWarning: Ignoring caltech256/train/198.spider/RENAME2 of type . Only support .jpg, .jpeg, .png\n",
      "  filename, ext, ', '.join(self._exts)))\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Estimated time     : 3 min\n",
      "Experiment Complete\n",
      "\n",
      "\n",
      "Comparing Experiments\n",
      "Comparison ID:      Comparison_analyse_learning_rates\n",
      "Generated statistics post all epochs\n",
      "| Experiment Name      |   Train Acc |   Val Acc |   Train Loss |   Val Loss |\n",
      "|----------------------+-------------+-----------+--------------+------------|\n",
      "| Learning_Rate_0.1    |   0.0746437 | 0.0639731 |     5.10316  |    5.27679 |\n",
      "| Learning_Rate_0.05   |   0.358965  | 0.23569   |     2.78242  |    3.77081 |\n",
      "| Learning_Rate_0.01   |   0.978995  | 0.47138   |     0.2577   |    2.3881  |\n",
      "| Learning_Rate_0.005  |   0.945611  | 0.491582  |     0.599715 |    2.53947 |\n",
      "| Learning_Rate_0.0001 |   0.0982746 | 0.0808081 |     5.119    |    5.32924 |\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<Figure size 432x288 with 0 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "<Figure size 1440x720 with 0 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "<Figure size 1440x720 with 0 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "<Figure size 1440x720 with 0 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "<Figure size 1440x720 with 0 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# \"keep_all\" - Keep all the sub experiments created\n",
    "# \"keep_non\" - Delete all sub experiments created\n",
    "analysis = gtf.Analyse_Learning_Rates(analysis_name, lrs, percent_data, \n",
    "                                      num_epochs=epochs, state=\"keep_none\"); "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Analysis\n",
    "\n",
    "    - LR as 0.1 doesnt work\n",
    "    - Same is the case with 0.0001\n",
    "    \n",
    "    - Of the other's lr as 0.01 produces least validation loss"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Update learning rate"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Update: Learning Rate - 0.01\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/abhi/.virtualenvs/finetune_py36/lib/python3.6/site-packages/mxnet/gluon/data/vision/datasets.py:312: UserWarning: Ignoring caltech256/train/198.spider/RENAME2 of type . Only support .jpg, .jpeg, .png\n",
      "  filename, ext, ', '.join(self._exts)))\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Pre-Composed Train Transforms\n",
      "[{'RandomHorizontalFlip': {'p': 0.8}}, {'Normalize': {'mean': [0.485, 0.456, 0.406], 'std': [0.229, 0.224, 0.225]}}]\n",
      "\n",
      "Pre-Composed Val Transforms\n",
      "[{'RandomHorizontalFlip': {'p': 0.8}}, {'Normalize': {'mean': [0.485, 0.456, 0.406], 'std': [0.229, 0.224, 0.225]}}]\n",
      "\n",
      "Dataset Numbers\n",
      "    Num train images: 21424\n",
      "    Num val images:   9183\n",
      "    Num classes:      257\n",
      "\n",
      "Model Details\n",
      "    Loading pretrained model\n",
      "    Model Loaded on device\n",
      "        Model name:                           resnet18_v1\n",
      "        Num of potentially trainable layers:  41\n",
      "        Num of actual trainable layers:       41\n",
      "\n"
     ]
    }
   ],
   "source": [
    "gtf.update_learning_rate(0.01);\n",
    "\n",
    "# Very important to reload post updates\n",
    "gtf.Reload();"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Start Training\n",
    "gtf.Train();\n",
    "\n",
    "#Read the training summary generated once you run the cell and training is completed"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Goals Completed\n",
    "\n",
    "### Learn how to use hyper parameter analyser for learning rates"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
