<commit_before>{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "64203dd8",
   "metadata": {},
   "outputs": [],
   "source": [
    "#|hide\n",
    "#|skip\n",
    "! [ -e /content ] && pip install -Uqq fastai  # upgrade fastai on colab"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e32a6fbd",
   "metadata": {},
   "source": [
    "# Tutorial - Distributed training in a notebook!\n",
    "> Using `Accelerate` to launch a training script from your notebook"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d3909dd4-673e-4eaf-a010-26ecdbd76d5f",
   "metadata": {},
   "outputs": [],
   "source": [
    "#|all_multicuda"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2322476f",
   "metadata": {},
   "source": [
    "## Overview\n",
    "\n",
    "In this tutorial we will see how to use [Accelerate](https://github.com/huggingface/accelerate) to launch a training function on a distributed system, from inside your **notebook**! \n",
    "\n",
    "To keep it easy, this example will follow training PETs, showcasing how all it takes is 3 new lines of code to be on your way!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ef9bb971",
   "metadata": {},
   "source": [
    "## Setting up imports and building the DataLoaders\n",
    "\n",
    "First, make sure that Accelerate is installed on your system by running:\n",
    "```bash\n",
    "pip install accelerate -U\n",
    "```\n",
    "\n",
    "In your code, along with the normal `from fastai.module.all import *` imports two new ones need to be added:\n",
    "```diff\n",
    "+ from fastai.accelerate import *\n",
    "from fastai.vision.all import *\n",
    "from fastai.vision.models.xresnet import *\n",
    "\n",
    "+ from accelerate import notebook_launcher\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b6dde5b3",
   "metadata": {},
   "source": [
    "The first brings in a new `Learner.accelerate_ctx` context manager, similar to the `Learner.distrib_ctx`, however only it is capible to be ran from a Jupyter Notebook. The second brings in Accelerate's [notebook_launcher](https://huggingface.co/docs/accelerate/launcher), the key function we will call to run what we want."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2cfcb5de",
   "metadata": {},
   "outputs": [],
   "source": [
    "#hide\n",
    "from fastai.vision.all import *\n",
    "from fastai.accelerate import *\n",
    "from fastai.vision.models.xresnet import *\n",
    "\n",
    "from accelerate import notebook_launcher\n",
    "from accelerate.utils import write_basic_config"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c11155c3-8054-4168-a4e9-a28cc70e8893",
   "metadata": {},
   "source": [
    "We need to setup `Accelerate` to use all of our GPUs. We can do so quickly with `write_basic_config ()`:\n",
    "\n",
    "> Note: Since this checks `torch.cuda.device_count`, you will need to restart your notebook and skip calling this again to continue. It only needs to be ran once!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "72a1427a-106e-4cc4-9d5e-123b1bb3a082",
   "metadata": {},
   "outputs": [],
   "source": [
    "#write_basic_config()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8003b603",
   "metadata": {},
   "source": [
    "Next let's download some data to train on. You don't need to worry about using `rank0_first`, as since we're in our Jupyter Notebook it will only run on one process like normal:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "481ef1c9",
   "metadata": {},
   "outputs": [],
   "source": [
    "path = untar_data(URLs.PETS)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c6250dcc",
   "metadata": {},
   "source": [
    "We wrap the creation of the `DataLoaders`, our `vision_learner`, and call to `fine_tune` inside of a `train` function. \n",
    "\n",
    "> Note: It is important to **not** build the `DataLoaders` outside of the function, as absolutely *nothing* can be loaded onto CUDA beforehand."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "055fc7cd",
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_y(o): return o[0].isupper()\n",
    "def train(path):\n",
    "    dls = ImageDataLoaders.from_name_func(\n",
    "        path, get_image_files(path), valid_pct=0.2,\n",
    "        label_func=get_y, item_tfms=Resize(224))\n",
    "    learn = vision_learner(dls, resnet34, metrics=error_rate).to_fp16()\n",
    "    learn.fine_tune(1)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ae6eece1",
   "metadata": {},
   "source": [
    "The last addition to the `train` function needed is to use our context manager before calling `fine_tune` and setting `in_notebook` to `True`:\n",
    "\n",
    "> Note: for this example `sync_bn` is disabled for compatibility purposes with `torchvision`'s resnet34"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cf0529db",
   "metadata": {},
   "outputs": [],
   "source": [
    "def train(path):\n",
    "    dls = ImageDataLoaders.from_name_func(\n",
    "        path, get_image_files(path), valid_pct=0.2,\n",
    "        label_func=get_y, item_tfms=Resize(224))\n",
    "    learn = vision_learner(dls, resnet34, metrics=error_rate).to_fp16()\n",
    "    with learn.accelerate_ctx(sync_bn=False, in_notebook=True):\n",
    "        learn.fine_tune(1)\n",
    "    if not rank_distrib():\n",
    "        learn.export(\"pets\")"
   ]
  },
  {
   "cell_type": "markdown",<commit_msg>Update tutorial w/ new config<commit_after>{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "64203dd8",
   "metadata": {},
   "outputs": [],
   "source": [
    "#|hide\n",
    "#|skip\n",
    "! [ -e /content ] && pip install -Uqq fastai  # upgrade fastai on colab"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e32a6fbd",
   "metadata": {},
   "source": [
    "# Tutorial - Distributed training in a notebook!\n",
    "> Using `Accelerate` to launch a training script from your notebook"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d3909dd4-673e-4eaf-a010-26ecdbd76d5f",
   "metadata": {},
   "outputs": [],
   "source": [
    "#|all_multicuda"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2322476f",
   "metadata": {},
   "source": [
    "## Overview\n",
    "\n",
    "In this tutorial we will see how to use [Accelerate](https://github.com/huggingface/accelerate) to launch a training function on a distributed system, from inside your **notebook**! \n",
    "\n",
    "To keep it easy, this example will follow training PETs, showcasing how all it takes is 3 new lines of code to be on your way!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ef9bb971",
   "metadata": {},
   "source": [
    "## Setting up imports and building the DataLoaders\n",
    "\n",
    "First, make sure that Accelerate is installed on your system by running:\n",
    "```bash\n",
    "pip install accelerate -U\n",
    "```\n",
    "\n",
    "In your code, along with the normal `from fastai.module.all import *` imports two new ones need to be added:\n",
    "```diff\n",
    "+ from fastai.distributed import *\n",
    "from fastai.vision.all import *\n",
    "from fastai.vision.models.xresnet import *\n",
    "\n",
    "+ from accelerate import notebook_launcher\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b6dde5b3",
   "metadata": {},
   "source": [
    "The first brings in the  `Learner.distrib_ctx` context manager. The second brings in Accelerate's [notebook_launcher](https://huggingface.co/docs/accelerate/launcher), the key function we will call to run what we want."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2cfcb5de",
   "metadata": {},
   "outputs": [],
   "source": [
    "#hide\n",
    "from fastai.vision.all import *\n",
    "from fastai.distributed import *\n",
    "from fastai.vision.models.xresnet import *\n",
    "\n",
    "from accelerate import notebook_launcher\n",
    "from accelerate.utils import write_basic_config"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c11155c3-8054-4168-a4e9-a28cc70e8893",
   "metadata": {},
   "source": [
    "We need to setup `Accelerate` to use all of our GPUs. We can do so quickly with `write_basic_config ()`:\n",
    "\n",
    "> Note: Since this checks `torch.cuda.device_count`, you will need to restart your notebook and skip calling this again to continue. It only needs to be ran once!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "72a1427a-106e-4cc4-9d5e-123b1bb3a082",
   "metadata": {},
   "outputs": [],
   "source": [
    "#from accelerate.utils import write_basic_config\n",
    "#write_basic_config()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8003b603",
   "metadata": {},
   "source": [
    "Next let's download some data to train on. You don't need to worry about using `rank0_first`, as since we're in our Jupyter Notebook it will only run on one process like normal:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "481ef1c9",
   "metadata": {},
   "outputs": [],
   "source": [
    "path = untar_data(URLs.PETS)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c6250dcc",
   "metadata": {},
   "source": [
    "We wrap the creation of the `DataLoaders`, our `vision_learner`, and call to `fine_tune` inside of a `train` function. \n",
    "\n",
    "> Note: It is important to **not** build the `DataLoaders` outside of the function, as absolutely *nothing* can be loaded onto CUDA beforehand."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "055fc7cd",
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_y(o): return o[0].isupper()\n",
    "def train(path):\n",
    "    dls = ImageDataLoaders.from_name_func(\n",
    "        path, get_image_files(path), valid_pct=0.2,\n",
    "        label_func=get_y, item_tfms=Resize(224))\n",
    "    learn = vision_learner(dls, resnet34, metrics=error_rate).to_fp16()\n",
    "    learn.fine_tune(1)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ae6eece1",
   "metadata": {},
   "source": [
    "The last addition to the `train` function needed is to use our context manager before calling `fine_tune` and setting `in_notebook` to `True`:\n",
    "\n",
    "> Note: for this example `sync_bn` is disabled for compatibility purposes with `torchvision`'s resnet34"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cf0529db",
   "metadata": {},
   "outputs": [],
   "source": [
    "def train(path):\n",
    "    dls = ImageDataLoaders.from_name_func(\n",
    "        path, get_image_files(path), valid_pct=0.2,\n",
    "        label_func=get_y, item_tfms=Resize(224))\n",
    "    learn = vision_learner(dls, resnet34, metrics=error_rate).to_fp16()\n",
    "    with learn.distrib_ctx(sync_bn=False, in_notebook=True):\n",
    "        learn.fine_tune(1)\n",
    "    if not rank_distrib():\n",
    "        learn.export(\"pets\")"
   ]
  },
  {
   "cell_type": "markdown",
{'fraction_non_alphanumeric': 0.17427830905353472, 'fraction_numerical': 0.024667083742961837, 'mean_word_length': 2.997856377277599, 'pattern_counts': {'":': 152, '<': 3, '<?xml version=': 0, '>': 11, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1667581', 'n_tokens_mistral': 4298, 'n_tokens_neox': 4080, 'n_words': 1034}