Updating with the correct notebooks

01_1_TGI-launcher.ipynb

@@ -7,7 +7,7 @@
    "source": [
     "Here we can see the different settings for TGI. Be sure to read through them and decide which settings are most important for your use-case.\n",
     "\n",
-    "Here are some of the most important ones for our purposes:\n",
+    "Here are some of the most important ones:\n",
     "- `--model-id`\n",
     "- `--quantize` Most of the time you want to quantize to save memory\n",
     "- `--max-input-tokens` We will use a high number for RAG and a low number for Chat\n",
@@ -15,7 +15,7 @@
     "- `--max-batch-size` \n",
     "- `--max-batch-total-tokens`\n",
     "\n",
-    "These are just used because we are on spaces, and dont want to conflict with the spaces server:\n",
+    "These are changed because we are on spaces, and dont want to conflict with the spaces server:\n",
     "- `--hostname`\n",
     "- `--port`"
    ]
@@ -143,43 +143,30 @@
    "id": "598435df-0cdb-411b-96c2-3930fc222f8e",
    "metadata": {},
    "source": [
-    "We can launch directly from the notebook since we dont need the command to be interactive."
+    "We can launch directly from the notebook since we dont need the command to be interactive.\n",
+    "\n",
+    "We will just be using defaults in this notebook as the intent is to understand the benchmark tool."
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": 2,
-   "id": "f8856772-5173-4549-aebd-0f4564cce143",
-   "metadata": {
-    "tags": []
-   },
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "zsh:1: command not found: text-generation-launcher\n"
-     ]
-    }
-   ],
+   "execution_count": null,
+   "id": "de3b9694-d417-4d17-a28e-e481703344ec",
+   "metadata": {},
+   "outputs": [],
    "source": [
     "!RUST_BACKTRACE=1 \\\n",
     "text-generation-launcher \\\n",
     "--model-id astronomer/Llama-3-8B-Instruct-GPTQ-8-Bit \\\n",
     "--quantize gptq \\\n",
-    "--max-input-tokens 3000 \\\n",
-    "--max-total-tokens 3300 \\\n",
-    "--max-batch-size 5 \\\n",
-    "--max-total-tokens 100000 \\\n",
     "--hostname 0.0.0.0 \\\n",
-    "--port 1337 # We need to change it from the default to play well with spaces \n",
-    "\n"
+    "--port 1337 # We need to change it from the default to play well with spaces \n"
    ]
   },
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "de3b9694-d417-4d17-a28e-e481703344ec",
+   "id": "9de093ba-b39a-4881-ab09-106de58dcd16",
    "metadata": {},
    "outputs": [],
    "source": []
@@ -201,7 +188,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.10.8"
+   "version": "3.9.5"
   }
  },
  "nbformat": 4,

01_2_TGI-benchmark.ipynb

@@ -6,10 +6,7 @@
    "metadata": {},
    "source": [
     "# Introduction\n",
-    "This notebook is to show how to launch the TGI Benchmark tool. \n",
-    "\n",
-    "## Warning\n",
-    "Please note that the TGI Benchmark tool is designed to work in a terminal, not a jupyter notebook. This means you will need to copy/paste the command in a jupyter terminal tab. I am putting them here for convenience."
+    "This notebook is to show how to launch the TGI Benchmark tool. \n"
    ]
   },
   {
@@ -19,13 +16,17 @@
    "source": [
     "Here we can see the different settings for TGI Benchmark. \n",
     "\n",
-    "Here are some of the most important ones:\n",
+    "Here are some of the more important ones:\n",
     "\n",
     "- `--tokenizer-name` This is required so the tool knows what tokenizer to use\n",
     "- `--batch-size` This is important for load testing. We should use more and more values to see what happens to throughput and latency\n",
     "- `--sequence-length` AKA input tokens, it is important to match your use-case needs\n",
     "- `--decode-length` AKA output tokens, it is important to match your use-case needs\n",
-    "- `--runs` Use fewer when you are exploring, but the default 10 is good for your final estimate"
+    "- `--runs` 10 is the default\n",
+    "\n",
+    "<blockquote style=\"border-left: 5px solid #80CBC4; background: #263238; color: #CFD8DC; padding: 0.5em 1em; margin: 1em 0;\">\n",
+    "  <strong>💡 Tip:</strong> Use a low number for <code style=\"background: #37474F; color: #FFFFFF; padding: 2px 4px; border-radius: 4px;\">--runs</code> when you are exploring but a higher number as you finalize to get more precise statistics\n",
+    "</blockquote>\n"
    ]
   },
   {
@@ -96,103 +97,34 @@
    "metadata": {},
    "source": [
     "Here is an example command. Notice that I add the batch sizes of interest repeatedly to make sure all of them are used by the benchmark tool.\n",
-    "```bash\n",
     "\n",
+    "<blockquote style=\"border-left: 5px solid #FFAB91; background: #37474F; color: #FFCCBC; padding: 0.5em 1em; margin: 1em 0;\">\n",
+    "  <strong>⚠️ Warning:</strong> Please note that the TGI Benchmark tool is designed to work in a terminal, not a jupyter notebook. This means you will need to copy/paste the command in a jupyter terminal tab. I am putting them here for convenience.\n",
+    "</blockquote>\n",
+    "\n",
+    "```bash\n",
     "text-generation-benchmark \\\n",
     "--tokenizer-name astronomer/Llama-3-8B-Instruct-GPTQ-8-Bit \\\n",
-    "--sequence-length 3000 \\\n",
-    "--decode-length 300 \\\n",
+    "--sequence-length 70 \\\n",
+    "--decode-length 50 \\\n",
     "--batch-size 1 \\\n",
     "--batch-size 2 \\\n",
-    "--batch-size 3 \\\n",
     "--batch-size 4 \\\n",
-    "--batch-size 5\n",
+    "--batch-size 8 \\\n",
+    "--batch-size 16 \\\n",
+    "--batch-size 32 \\\n",
+    "--batch-size 64 \\\n",
+    "--batch-size 128 \n",
     "```"
    ]
   },
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "c821081f-ff78-4046-88a6-3652f7322b85",
+   "id": "13ac475b-44e1-47e4-85ce-def2db6879c9",
    "metadata": {},
    "outputs": [],
-   "source": [
-    "import subprocess\n",
-    "import time\n",
-    "\n",
-    "def launch_server(tokens):\n",
-    "    try:\n",
-    "        command = [\n",
-    "            'text-generation-launcher',\n",
-    "            '--model-id', 'astronomer/Llama-3-8B-Instruct-GPTQ-8-Bit',\n",
-    "            '--quantize', 'gptq',\n",
-    "            '--max-input-tokens', '3000',\n",
-    "            '--max-batch-size', '5',\n",
-    "            '--max-batch-total-tokens', f'{tokens}',\n",
-    "            '--hostname', '0.0.0.0',\n",
-    "            '--port', '1337',\n",
-    "        ]\n",
-    "        \n",
-    "        # Launch the subprocess with a text command (without shell=True for safety)\n",
-    "        process = subprocess.Popen(command, stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True)\n",
-    "        \n",
-    "        # Set a time limit to wait for the process to stabilize\n",
-    "        time_limit = 120\n",
-    "        start_time = time.time()\n",
-    "        \n",
-    "        while time.time() - start_time < time_limit:\n",
-    "            output = process.stdout.readline()\n",
-    "            if \"Connected\" in output:\n",
-    "                print(f\"Success message found for {tokens} tokens.\")\n",
-    "                process.terminate()  # Gracefully terminate if successful\n",
-    "                return True\n",
-    "            if \"RuntimeError\" in output or \"OutOfMemory\" in output:\n",
-    "                print(f\"Failure message found for {tokens} tokens.\")\n",
-    "                process.terminate()\n",
-    "                return False\n",
-    "        \n",
-    "        # If no specific message was found but the process is still running\n",
-    "        if process.poll() is None:\n",
-    "            print(f\"No specific message but process is still running for {tokens} tokens.\")\n",
-    "            process.terminate()\n",
-    "            return True\n",
-    "        else:\n",
-    "            return False\n",
-    "    except Exception as e:\n",
-    "        print(f\"Error launching server with {tokens} tokens: {e}\")\n",
-    "        return False"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "6622e91a-dc01-4401-8036-f1bf3c85d655",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "%%time\n",
-    "def find_max_prefill_tokens():\n",
-    "    low, high = 0, 1_000_000  # Adjust the upper bound as needed\n",
-    "    best_valid = 0\n",
-    "    \n",
-    "    while low <= high:\n",
-    "        mid = (low + high) // 2\n",
-    "        print(f\"Testing with {mid} max-total-tokens...\")\n",
-    "        \n",
-    "        if launch_server(mid):\n",
-    "            print(f\"Success with {mid} max-total-tokens.\")\n",
-    "            best_valid = mid  # Update best known valid count\n",
-    "            low = mid + 1\n",
-    "        else:\n",
-    "            print(f\"Failed with {mid} max-total-tokens.\")\n",
-    "            high = mid - 1\n",
-    "    \n",
-    "    print(f\"Maximum manageable prefill tokens: {best_valid}\")\n",
-    "    return best_valid\n",
-    "\n",
-    "# Call the function\n",
-    "max_tokens = find_max_prefill_tokens()"
-   ]
+   "source": []
   }
  ],
  "metadata": {
@@ -211,7 +143,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.10.8"
+   "version": "3.9.5"
   }
  },
  "nbformat": 4,