<commit_before>{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# hide\n",
    "%load_ext autoreload\n",
    "%autoreload 2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Vespa - Application Package\n",
    "\n",
    "> Python API to create, modify and deploy application packages"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our goal is to create, modify and deploy simple application packages using our python API. This enables us to run data analysis experiments that are fully integrated with Vespa. As an example, we want to create the application package we used in our [text search tutorial](https://docs.vespa.ai/documentation/tutorials/text-search.html). "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Application spec"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our goal in this section is to create the following `msmarco` schema using our python API."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```\n",
    "schema msmarco {\n",
    "    document msmarco {\n",
    "        field id type string {\n",
    "            indexing: attribute | summary\n",
    "        }\n",
    "        field title type string {\n",
    "            indexing: index | summary\n",
    "            index: enable-bm25\n",
    "        }\n",
    "        field body type string {\n",
    "            indexing: index | summary\n",
    "            index: enable-bm25\n",
    "        }\n",
    "    }\n",
    "\n",
    "    fieldset default {\n",
    "        fields: title, body\n",
    "    }\n",
    "\n",
    "    rank-profile default {\n",
    "        first-phase {\n",
    "            expression: nativeRank(title, body)\n",
    "        }\n",
    "    }\n",
    "\n",
    "    rank-profile bm25 inherits default {\n",
    "        first-phase {\n",
    "            expression: bm25(title) + bm25(body)\n",
    "        }\n",
    "    }\n",
    "\n",
    "}\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Schema API"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from vespa.package import Document, Field, Schema, FieldSet, RankProfile, ApplicationPackage\n",
    "\n",
    "document = Document(\n",
    "    fields=[\n",
    "        Field(name = \"id\", type = \"string\", indexing = [\"attribute\", \"summary\"]),\n",
    "        Field(name = \"title\", type = \"string\", indexing = [\"index\", \"summary\"], index = \"enable-bm25\"),\n",
    "        Field(name = \"body\", type = \"string\", indexing = [\"index\", \"summary\"], index = \"enable-bm25\")        \n",
    "    ]\n",
    ")\n",
    "\n",
    "msmarco_schema = Schema(\n",
    "    name = \"msmarco\", \n",
    "    document = document, \n",
    "    fieldsets = [FieldSet(name = \"default\", fields = [\"title\", \"body\"])],\n",
    "    rank_profiles = [RankProfile(name = \"default\", first_phase = \"nativeRank(title, body)\")]\n",
    ")\n",
    "\n",
    "app_package = ApplicationPackage(name = \"msmarco\", schema=msmarco_schema)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Deploy it locally"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "app_package.deploy_locally(disk_folder=\"~/sample_application\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Change the application package and redeploy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can add a new rank profile and redeploy our application"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "app_package.schema.add_rank_profile(\n",
    "    RankProfile(name = \"bm25\", inherits = \"default\", first_phase = \"bm25(title) + bm25(body)\")\n",
    ")"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "app_package.deploy_locally(disk_folder=\"~/sample_application\")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "vespa",
   "language": "python",
   "name": "vespa"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}<commit_msg>update application package notebook<commit_after>{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# hide\n",
    "%load_ext autoreload\n",
    "%autoreload 2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Vespa - Application Package\n",
    "\n",
    "> Python API to create, modify and deploy application packages"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our goal is to create, modify and deploy simple application packages using our python API. This enables us to run data analysis experiments that are fully integrated with Vespa. As an example, we want to create the application package we used in our [text search tutorial](https://docs.vespa.ai/documentation/tutorials/text-search.html). "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Application spec"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our goal in this section is to create the following `msmarco` schema using our python API."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```\n",
    "schema msmarco {\n",
    "    document msmarco {\n",
    "        field id type string {\n",
    "            indexing: attribute | summary\n",
    "        }\n",
    "        field title type string {\n",
    "            indexing: index | summary\n",
    "            index: enable-bm25\n",
    "        }\n",
    "        field body type string {\n",
    "            indexing: index | summary\n",
    "            index: enable-bm25\n",
    "        }\n",
    "    }\n",
    "\n",
    "    fieldset default {\n",
    "        fields: title, body\n",
    "    }\n",
    "\n",
    "    rank-profile default {\n",
    "        first-phase {\n",
    "            expression: nativeRank(title, body)\n",
    "        }\n",
    "    }\n",
    "\n",
    "    rank-profile bm25 inherits default {\n",
    "        first-phase {\n",
    "            expression: bm25(title) + bm25(body)\n",
    "        }\n",
    "    }\n",
    "\n",
    "}\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Schema API"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from vespa.package import Document, Field, Schema, FieldSet, RankProfile, ApplicationPackage\n",
    "\n",
    "document = Document(\n",
    "    fields=[\n",
    "        Field(name = \"id\", type = \"string\", indexing = [\"attribute\", \"summary\"]),\n",
    "        Field(name = \"title\", type = \"string\", indexing = [\"index\", \"summary\"], index = \"enable-bm25\"),\n",
    "        Field(name = \"body\", type = \"string\", indexing = [\"index\", \"summary\"], index = \"enable-bm25\")        \n",
    "    ]\n",
    ")\n",
    "\n",
    "msmarco_schema = Schema(\n",
    "    name = \"msmarco\", \n",
    "    document = document, \n",
    "    fieldsets = [FieldSet(name = \"default\", fields = [\"title\", \"body\"])],\n",
    "    rank_profiles = [RankProfile(name = \"default\", first_phase = \"nativeRank(title, body)\")]\n",
    ")\n",
    "\n",
    "app_package = ApplicationPackage(name = \"msmarco\", schema=msmarco_schema)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Deploy it locally"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "from vespa.package import VespaDocker\n",
    "\n",
    "vespa_docker = VespaDocker(application_package=app_package)\n",
    "vespa_docker.deploy(disk_folder=\"/Users/username/sample_application\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Change the application package and redeploy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can add a new rank profile and redeploy our application"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "app_package.schema.add_rank_profile(\n",
    "    RankProfile(name = \"bm25\", inherits = \"default\", first_phase = \"bm25(title) + bm25(body)\")\n",
    ")"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "vespa_docker.deploy(disk_folder=\"/Users/username/sample_application\")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "vespa",
   "language": "python",
   "name": "vespa"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
{'fraction_non_alphanumeric': 0.20522222222222222, 'fraction_numerical': 0.005111111111111111, 'mean_word_length': 2.1873229461756374, 'pattern_counts': {'":': 112, '<': 3, '<?xml version=': 0, '>': 5, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1782935', 'n_tokens_mistral': 3162, 'n_tokens_neox': 3078, 'n_words': 678}