hw description update

rag-homework.ipynb

@@ -62,20 +62,17 @@
     "Your task is to implement and compare two chunking strategies: fixed-sized chunking and content-aware chunking. For content-aware you could split by sentences, paragraphs or in some other way that makes sence.\n",
     "\n",
     "The deliverables are:\n",
-    "\n",
-    "1. The code for chunk splitting\n",
-    "2. The illustrative examples of how retrived chunks differ depending on the chunking strategy.\n",
-    "3. The analysis of pros and cons of each strategy\n"
+    "- The code for chunk splitting"
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": null,
+   "execution_count": 1,
    "id": "f7bad8c8",
    "metadata": {},
    "outputs": [],
    "source": [
-    "# chunk splitting"
+    "# Chunk splitting deliverables"
    ]
   },
   {
@@ -87,24 +84,28 @@
     "\n",
     "Chunks need to be vectorized and made accessible to an LLM to enable semantic search with embedding models. A current industry standard is to use a vector database to store and retrieve texts both conveniently and efficiently. There are many products out there, we'll be using [LanceDB](https://lancedb.github.io/lancedb/). LanceDB is a young product, one way it stands out is that it's embedded - it's designed not to be a standalone service but rather a part of an application, more on this [here](https://lancedb.github.io/lancedb/basic/).\n",
     "\n",
-    "Your task is to vectorize and ingest chunked documents into the database. Use prep_scrips/lancedb_setup.py to vectorize chunks and store vector representations along with raw text in a Lancedb instance. The script also creates an index for fast ANN retrieval (not really needed for this exercise but necessary at scale). Try different embedding models and see how results differ. The options are:\n",
+    "Find more details on how different databases compare in [this](https://thedataquarry.com/tags/vector-db/) series of posts. \n",
+    "\n",
+    "Your task is to vectorize and ingest chunked documents into the database. \n",
+    "**For each chunking strategy from the previous step create a separate table with one of the embedding models. Compare the chunking strategies and choose one. Perform vectorization+ingestion with the second model only with one chunking strategy of your choice**.\n",
+    "Use prep_scrips/lancedb_setup.py to vectorize chunks and store vector representations along with raw text in a Lancedb instance. The script also creates an index for fast ANN retrieval (not really needed for this exercise but necessary at scale). Try different embedding models and see how results differ. The options are:\n",
     "\n",
     "- `sentence-transformers/all-MiniLM-L6-v2`: a light model, produces vectors of length 384\n",
     "- `BAAI/bge-large-en-v1.5`: a much heavier model, embedding vector length is 1024\n",
     "\n",
     "Feel free to explore other embedding models and justify your choice.\n",
-    "For different embedding model create different tables in the database so you can easily switch between them and compare.\n",
+    "For different embedding models and different chunking strategies create different tables in the database so you can easily switch between them and compare.\n",
     "\n",
-    "Run the embedding+ingestion script as follows, make sure to look into the script and see all the arguments. Note that the number of sub-vectors for indexing must be a divisor of the model embedding size.\n",
+    "Run the embedding+ingestion script as follows, make sure to look into the script and go over the arguments. Note that the number of sub-vectors for indexing must be a divisor of the model embedding size.\n",
     "\n",
     "```\n",
-    "python prep_scrips/lancedb_setup.py --emb-model <model name> --table <db table name> --input-dir <folder with docs> --num-sub-vectors <a number which is a divisor of the embedding dim>\n",
+    "python prep_scrips/lancedb_setup.py --emb-model <model name> --table <db table name> --input-dir <folder with chunked docs> --num-sub-vectors <a number which is a divisor of the embedding dim>\n",
     "```\n",
     "\n",
     "Before committing to your space set up environment variables on the settings tab of your space, use `.env` as a ference list of all the things you can customize. Make sure to add HF_TOKEN and OPENAI_API_KEY as secrets.\n",
     "Not all the parameters are required to set via environment variables, most have default values.\n",
     "\n",
-    "!!The database is expected to be in the `gradio_app` folder under `.lancedb`, make sure to move it there if was initialized elsewhere. It can be parametrized but it's unnecessary here.\n",
+    "*The database is expected to be in the `gradio_app` folder under `.lancedb`, make sure to move it there if was initialized elsewhere.* It can be parametrized but it's unnecessary here.\n",
     "\n",
     "To commit large files to Github use `git lfs`:\n",
     "```\n",
@@ -115,11 +116,13 @@
     "```\n",
     "Then proceed as usual.\n",
     "\n",
-    "For experimenting you can easily switch between embedding models/tables by changing the values of the corresponding env variables in your space (EMB_MODEL, TABLE_NAME). Overall, every time you change the value of an environment variable a space gets automatically rebuilt.\n",
+    "For experimenting you can easily switch between embedding models/tables by changing the values of the corresponding env variables in your space (`EMB_MODEL`, `TABLE_NAME`). Overall, every time you change the value of an environment variable a space gets automatically rebuilt.\n",
     "\n",
     "The deliverables are:\n",
-    "1. The illustration of how retrieved documents differ depending on the embedding model\n",
-    "2. The analysis of how the embedding time differs between models\n"
+    "1. The illustration of how retrieved documents differ depending on the embedding model and the chunking strategy. You should create at least 3 tables: model_1 + chunking_strategy_1, model_1 + chunking_strategy_2, model_2 + chunking_strategy_<1 or 2>\n",
+    "2. The analysis of pros and cons of chunking strategies\n",
+    "3. The analysis of how retrieved document differ between embedding models (is one better than the other?)\n",
+    "4. The analysis of how the embedding time differs between models"
    ]
   },
   {
@@ -129,33 +132,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "# Embed documents and ingest into the database "
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "c929fe99-7f3f-49cf-af9f-07bac9668c2f",
-   "metadata": {},
-   "source": [
-    "### Step 3: DB retrieval\n",
-    "\n",
-    "To augment an LLM with a knowledge base the context provided in a prompt is enriched with relevant pieces from a DB. An incoming query is vectorized with an embedding model and top-k relevant pieces are retrieved based on similarity to a query vector. Your task is to implement database retrieval and adjust the prompt to include the documents from the DB.\n",
-    "\n",
-    "The deliverables are:\n",
-    "\n",
-    "1. The new code that enables RAG. Paste the code added and some surrounding context; indicate which file it comes from.\n",
-    "2. A new prompt template\n",
-    "3. Examples of how the output changes\n"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "5bc1399d-1c3b-48b7-a9fb-ed48a0dfaa6a",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "# DB retrieval implementation"
+    "# Embed documents with different chunking strategies and ingest into the database "
    ]
   },
   {
@@ -163,9 +140,9 @@
    "id": "7d818b4f-ba5a-4c81-b6d7-f3474c398d9c",
    "metadata": {},
    "source": [
-    "### Step 4: Add a reranker\n",
+    "### Step 3: Add a reranker\n",
     "\n",
-    "A reranker is a second-level model which produces similarity scores for pairs of (input query + retrieved document). Cross-encoders are conventionally used for reranking, their architecture is slightly different from retrieval models (more on it [here]). Cross-encoders are much more costly to run, therefore a retrieval model is used to get a few (dozens) highest-scoring items, and a reranker picks the best among these. The overall pipeline is similar to the recommender system indudustry standard: a light model retrieves top-n, a precise and heavy model reranks n to get top k, n is orders of magnitude larger than k.\n",
+    "A reranker is a second-level model which produces similarity scores for pairs of (input query + retrieved document). Cross-encoders are conventionally used for reranking, their architecture is slightly different from retrieval models (more on it [here] and [here](https://www.sbert.net/examples/applications/retrieve_rerank/README.html)). Cross-encoders are much more costly to run, therefore a retrieval model is used to get a few (dozens) highest-scoring items, and a reranker picks the best among these. The overall pipeline is similar to the recommender system indudustry standard: a light model retrieves top-n, a precise and heavy model reranks n to get top k, n is orders of magnitude larger than k.\n",
     "\n",
     "Cross-encoders are optional because of the overhead their usage implies. Your task is to implement a reranker using a cross-encoder and assess pros and cons of having it. Do not forget that the process of pulling the most relevant documents becomes two-staged: retrieve a larger number of items first, than rerank and keep the best top-k for context.\n",
     "\n",
@@ -178,8 +155,8 @@
     "The deliverables are:\n",
     "\n",
     "1. The code that enables a reranker.\n",
-    "2. A comparison of how the prompt and the model output change after adding a reranker\n",
-    "3. The analysis of pros and cons. The evaluation aspects should include the relevance of the top-k documents, the response time.\n"
+    "3. A comparison of how the prompt and the model output change after adding a reranker\n",
+    "4. The analysis of pros and cons. The evaluation aspects should include the relevance of the top-k documents, the response time.\n"
    ]
   },
   {
@@ -189,7 +166,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "# Implement code for selecting the final documents using a cross-encoder"
+    "# Implement code for selecting the final documents using a cross-encoder and compare with and without"
    ]
   },
   {
@@ -197,18 +174,23 @@
    "id": "f5816c54-a290-4cb0-b7db-3b8901998cb0",
    "metadata": {},
    "source": [
-    "### Step 5: Try a different LLM\n",
+    "### Step 4: Try a different LLM\n",
     "\n",
-    "The suggested Mistral-7b is a great but small model for an LLM. A larger model can be applied to a wider range of problems and do more complex reasoning. Within the scope of this project a larger model may not be beneficial but for more complex cases the difference would become apparent. \n",
+    "The suggested `Mistral-7b-instruct` is a great but small model for an LLM. A larger model can be applied to a wider range of problems and do more complex reasoning. Within the scope of this project a larger model may not be beneficial but for more complex cases the difference would become apparent. Another dimension to explore is a base model which was not instruction fine-tuned - it won't respond to your queries the way you'd expect. It may be a great exercise to see the value of fine-tuning.\n",
     "\n",
-    "A good option for a large model is gpt-4 but the inference time comparison will not be representative due to the hosting infrastructure differences. Try a larger model hosted by HF if you have access.\n",
+    "The task here is to try out an alternative LLM to explore the differencies.\n",
     "\n",
-    "The task here is to try out a large LLM and describe the differencies.\n",
+    "The options are:\n",
+    "1. mistralai/Mistral-7B-v0.1\n",
+    "2. mistralai/Mixtral-8x7B-Instruct-v0.1\n",
+    "\n",
+    "Of couse, feel free to choose another one and give some details on how different it is from the initial model.\n",
     "\n",
     "The deliverables are:\n",
     "\n",
-    "1. The comparison between outputs with Mistral-7b and a larger model\n",
-    "2. The difference in response times\n"
+    "1. The comparison between outputs of the Mistral-7b-instuct and a different model of your choice.\n",
+    "2. The difference in response times if a larger model was chosen. Make sure to make multiple queries to make the comparison meaningful.\n",
+    "3. Analyse the differencies between outputs and share the conclusions.\n"
    ]
   },
   {
@@ -226,7 +208,7 @@
    "id": "70c16440",
    "metadata": {},
    "source": [
-    "### Step 6 (Bonus): Use an LLM to quantitatively compare outputs of different variants of the system (LLM as a Judge)\n",
+    "### Step 5 (Bonus): Use an LLM to quantitatively compare outputs of different variants of the system (LLM as a Judge)\n",
     "\n",
     "Use a powerful LLM (e.g. GPT-4) to quantitatively evaluate outputs of two alternative setups (different embedding models, different LLMs, both etc.). For inspiration and for prompts refer to [1](https://arxiv.org/pdf/2306.05685.pdf), [2](https://arxiv.org/pdf/2401.10020.pdf), [3](https://www.airtrain.ai/blog/the-comprehensive-guide-to-llm-evaluation#high-level-approach)\n",
     "\n",
@@ -243,7 +225,9 @@
    "id": "39c18ba0-e54a-478f-9e60-0ea65c29238a",
    "metadata": {},
    "outputs": [],
-   "source": []
+   "source": [
+    "# The code implementing LLM-as-a-Judge and "
+   ]
   }
  ],
  "metadata": {