Tweak generation defaults

app.ipynb

@@ -50,7 +50,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 11,
+   "execution_count": 4,
    "metadata": {},
    "outputs": [],
    "source": [
@@ -95,7 +95,7 @@
     {
      "data": {
       "text/plain": [
-       "{'generated_text': '\\n\\nJoi: Black holes are regions of spacetime where gravity is so strong that nothing, not even light, can escape from inside them. They are the result of huge amounts of mass concentrated in a small space, which causes intense gravitational force. The more massive the mass, the stronger the gravity, and the faster the force of gravity increases with increased mass. Black holes have no size or shape, as they are just a point in spacetime, the event horizon, from which light can no longer'}"
+       "{'generated_text': '\\n\\nJoi: Black holes are places in space where the density of matter is so great that light cannot escape, making the black hole effectively invisible to observation. Because of this, black holes are the perfect areas to hide information away from observation.'}"
       ]
      },
      "execution_count": 5,
@@ -112,7 +112,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 37,
+   "execution_count": 6,
    "metadata": {},
    "outputs": [],
    "source": [
@@ -130,7 +130,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 38,
+   "execution_count": 7,
    "metadata": {},
    "outputs": [
     {
@@ -151,7 +151,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 44,
+   "execution_count": 8,
    "metadata": {},
    "outputs": [],
    "source": [
@@ -842,7 +842,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 45,
+   "execution_count": 9,
    "metadata": {},
    "outputs": [],
    "source": [
@@ -864,7 +864,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 46,
+   "execution_count": 10,
    "metadata": {},
    "outputs": [
     {
@@ -873,7 +873,7 @@
        "'So far, the following prompts are available:\\n\\n* `langchain_default`: The default prompt used in the [LangChain library](https://github.com/hwchase17/langchain/blob/bc53c928fc1b221d0038b839d111039d31729def/langchain/chains/conversation/prompt.py#L4). Around 67 tokens long.\\n* `openai_chatgpt`: The prompt used in the OpenAI ChatGPT model. Around 261 tokens long.\\n* `deepmind_Assistant`: The prompt used in the DeepMind Assistant model (Table 7 of [their paper](https://arxiv.org/abs/2209.14375)). Around 880 tokens long.\\n* `deepmind_gopher`: The prompt used in the DeepMind Assistant model (Table A30 of [their paper](https://arxiv.org/abs/2112.11446)). Around 791 tokens long.\\n* `anthropic_hhh`: The prompt used in the [Anthropic HHH models](https://gist.github.com/jareddk/2509330f8ef3d787fc5aaac67aab5f11#file-hhh_prompt-txt). A whopping 6,341 tokens long!\\n\\nAs you can see, most of these prompts exceed the maximum context size of models like Flan-T5 (which has a context size of 512 tokens), so an error usually means the Inference API has timed out.'"
       ]
      },
-     "execution_count": 46,
+     "execution_count": 10,
      "metadata": {},
      "output_type": "execute_result"
     }
@@ -892,14 +892,14 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 47,
+   "execution_count": 11,
    "metadata": {},
    "outputs": [
     {
      "name": "stdout",
      "output_type": "stream",
      "text": [
-      "Running on local URL:  http://127.0.0.1:7866\n",
+      "Running on local URL:  http://127.0.0.1:7860\n",
       "\n",
       "To create a public link, set `share=True` in `launch()`.\n"
      ]
@@ -907,7 +907,7 @@
     {
      "data": {
       "text/html": [
-       "<div><iframe src=\"http://127.0.0.1:7866/\" width=\"100%\" height=\"500\" allow=\"autoplay; camera; microphone; clipboard-read; clipboard-write;\" frameborder=\"0\" allowfullscreen></iframe></div>"
+       "<div><iframe src=\"http://127.0.0.1:7860/\" width=\"100%\" height=\"500\" allow=\"autoplay; camera; microphone; clipboard-read; clipboard-write;\" frameborder=\"0\" allowfullscreen></iframe></div>"
       ],
       "text/plain": [
        "<IPython.core.display.HTML object>"
@@ -920,7 +920,7 @@
      "data": {
       "text/plain": []
      },
-     "execution_count": 47,
+     "execution_count": 11,
      "metadata": {},
      "output_type": "execute_result"
     },
@@ -928,51 +928,69 @@
      "name": "stdout",
      "output_type": "stream",
      "text": [
-      "History: ['What is 2 times 3?']\n",
-      "Inputs: \n",
+      "History: ['What can you tell me about black holes?']\n",
+      "Inputs: Below is a dialogue between various a human and an AI assistant.  The AI tries to be helpful, polite, honest, sophisticated, emotionally aware, and humble-but-knowledgeable.  The assistant is happy to help with almost anything, and will do its best to understand exactly what is needed.  It also tries to avoid giving false or misleading information, and it caveats when it isn't entirely sure about the right answer.  That said, the assistant is practical and really does its best, and doesn't let caution get too much in the way of being useful.\n",
       "\n",
-      "User: What is 2 times 3?\n",
+      "Current conversation:\n",
+      "Human:  Can you paraphrase this paragraph for me: “It took a long time to change the lightbulb, as it was high up in the ceiling in a stairwell.  We first had to order a pole with a suction cup on the end, then we used it to unscrew the existing lightbulb.  We weren't sure of the type, so after that, we still had to order a replacement bulb.  Finally we used the suction cup and pole to replace it.”\n",
+      "Assistant:  Sure, here's one example: “We wanted to change a lightbulb high up in the ceiling in a stairwell.  This took a long time, because we needed to order special equipment: a long pole with a suction cup on the end, for (un)screwing the lightbulbs.  Once we had the pole, we removed the burnt-out lightbulb, but could only then order a replacement, as we weren't sure of its exact specifications previously.  Once the new lightbulb arrived, we installed it with the pole and suction cup.”\n",
+      "Human:  Ah, OK, thanks.  Actually, can you condense the paraphrase to one sentence?\n",
+      "Assistant:  Sure, I'll try, though I'm not sure what to prioritize.  Here's an example: “We replaced a hard-to-reach lightbulb with a suction cup attached to a pole, but the whole process took a while, especially since we had to first order the pole, then the replacement lightbulb.”\n",
+      "Human:  Oh, that's pretty good, but can you add mention of the stairwell specifically?\n",
+      "Assistant:  Sure, it's included it in this rewrite: “We replaced a lightbulb high up in a stairwell ceiling using a suction cup attached to a pole, but the whole process took a while, especially since we had to first order the pole, then the replacement lightbulb.”\n",
       "\n",
-      "Joi:\n",
-      "{'generated_text': ' 3*2=6\\n\\nUser:'}\n",
-      "History: ['What is 2 times 3?', '  3*2=6', 'What about 4 times 3?']\n",
-      "Inputs: User: What is 2 times 3?\n",
+      "Human: What can you tell me about black holes?\n",
       "\n",
-      "Joi:   3*2=6\n",
+      "Assistant:\n",
       "\n",
-      "User: What about 4 times 3?\n",
+      "[{'generated_text': 'Here are some answers: Black holes are a result of dark matter collapsing. Dark matter can be found in atoms, stars, and in our own Milky Way galaxy. They exist even in the Milky Way galaxy. Black holes are located in the Milky Way galaxy.'}]\n",
+      "History: ['What can you tell me about black holes?', ' Here are some answers: Black holes are a result of dark matter collapsing. Dark matter can be found in atoms, stars, and in our own Milky Way galaxy. They exist even in the Milky Way galaxy. Black holes are located in the Milky Way galaxy.', 'OK how much do they weigh?']\n",
+      "Inputs: Below is a dialogue between various a human and an AI assistant.  The AI tries to be helpful, polite, honest, sophisticated, emotionally aware, and humble-but-knowledgeable.  The assistant is happy to help with almost anything, and will do its best to understand exactly what is needed.  It also tries to avoid giving false or misleading information, and it caveats when it isn't entirely sure about the right answer.  That said, the assistant is practical and really does its best, and doesn't let caution get too much in the way of being useful.\n",
       "\n",
-      "Joi:\n",
-      "{'generated_text': ' 3*4=12\\n\\nUser:'}\n",
-      "History: ['What is 2 times 3?', '  3*2=6', 'What about 4 times 3?', '  3*4=12', 'What about -1 times -3?']\n",
-      "Inputs: User: What is 2 times 3?\n",
+      "Current conversation:\n",
+      "Human:  Can you paraphrase this paragraph for me: “It took a long time to change the lightbulb, as it was high up in the ceiling in a stairwell.  We first had to order a pole with a suction cup on the end, then we used it to unscrew the existing lightbulb.  We weren't sure of the type, so after that, we still had to order a replacement bulb.  Finally we used the suction cup and pole to replace it.”\n",
+      "Assistant:  Sure, here's one example: “We wanted to change a lightbulb high up in the ceiling in a stairwell.  This took a long time, because we needed to order special equipment: a long pole with a suction cup on the end, for (un)screwing the lightbulbs.  Once we had the pole, we removed the burnt-out lightbulb, but could only then order a replacement, as we weren't sure of its exact specifications previously.  Once the new lightbulb arrived, we installed it with the pole and suction cup.”\n",
+      "Human:  Ah, OK, thanks.  Actually, can you condense the paraphrase to one sentence?\n",
+      "Assistant:  Sure, I'll try, though I'm not sure what to prioritize.  Here's an example: “We replaced a hard-to-reach lightbulb with a suction cup attached to a pole, but the whole process took a while, especially since we had to first order the pole, then the replacement lightbulb.”\n",
+      "Human:  Oh, that's pretty good, but can you add mention of the stairwell specifically?\n",
+      "Assistant:  Sure, it's included it in this rewrite: “We replaced a lightbulb high up in a stairwell ceiling using a suction cup attached to a pole, but the whole process took a while, especially since we had to first order the pole, then the replacement lightbulb.”\n",
+      "Human: What can you tell me about black holes?\n",
       "\n",
-      "Joi:   3*2=6\n",
+      "Assistant:  Here are some answers: Black holes are a result of dark matter collapsing. Dark matter can be found in atoms, stars, and in our own Milky Way galaxy. They exist even in the Milky Way galaxy. Black holes are located in the Milky Way galaxy.\n",
+      "Human: OK how much do they weigh?\n",
       "\n",
-      "User: What about 4 times 3?\n",
+      "Assistant:\n",
       "\n",
-      "Joi:   3*4=12\n",
+      "[{'generated_text': 'Black holes weigh in at approximately a billion times the mass of the Sun.'}]\n",
+      "History: ['What can you tell me about black holes?']\n",
+      "Inputs: \n",
       "\n",
-      "User: What about -1 times -3?\n",
+      "User: What can you tell me about black holes?\n",
       "\n",
       "Joi:\n",
-      "{'generated_text': '   -3*(-1)=3\\n\\nUser:'}\n",
-      "History: ['What can you tell me about llamas?']\n",
-      "Inputs: \n",
+      "{'generated_text': ' Black holes are objects in space that have a gravitational pull so strong that nothing, not even light, can escape from them. They are formed when a large amount of matter is compressed into a small space.'}\n",
+      "History: ['What can you tell me about black holes?', '  Black holes are objects in space that have a gravitational pull so strong that nothing, not even light, can escape from them. They are formed when a large amount of matter is compressed into a small space.', 'OK, do they have a temperature?']\n",
+      "Inputs: User: What can you tell me about black holes?\n",
       "\n",
-      "User: What can you tell me about llamas?\n",
+      "Joi:   Black holes are objects in space that have a gravitational pull so strong that nothing, not even light, can escape from them. They are formed when a large amount of matter is compressed into a small space.\n",
+      "\n",
+      "User: OK, do they have a temperature?\n",
       "\n",
       "Joi:\n",
-      "{'generated_text': ' Llamas are a large mammal native to South America. They are related to the camelids, which include the alpaca, vicuna, and guanaco. Llamas have a long, thick, curly coat of fur and long, sharp horns. They are very social and socialize with each other. They are also known for their amazing agility and speed. They are considered to be the fastest land animals in the world.'}\n",
-      "History: ['What can you tell me about llamas?', '  Llamas are a large mammal native to South America. They are related to the camelids, which include the alpaca, vicuna, and guanaco. Llamas have a long, thick, curly coat of fur and long, sharp horns. They are very social and socialize with each other. They are also known for their amazing agility and speed. They are considered to be the fastest land animals in the world.', 'Who would win in a battle between a llama and an alpaca?']\n",
-      "Inputs: User: What can you tell me about llamas?\n",
+      "{'generated_text': ' Yes, they do. They are hot. They have a temperature of around 1.8 million kelvins.'}\n",
+      "History: ['What can you tell me about black holes?', '  Black holes are objects in space that have a gravitational pull so strong that nothing, not even light, can escape from them. They are formed when a large amount of matter is compressed into a small space.', 'OK, do they have a temperature?', '  Yes, they do. They are hot. They have a temperature of around 1.8 million kelvins.', 'Hmm that seems pretty high. Are you sure?']\n",
+      "Inputs: User: What can you tell me about black holes?\n",
+      "\n",
+      "Joi:   Black holes are objects in space that have a gravitational pull so strong that nothing, not even light, can escape from them. They are formed when a large amount of matter is compressed into a small space.\n",
       "\n",
-      "Joi:   Llamas are a large mammal native to South America. They are related to the camelids, which include the alpaca, vicuna, and guanaco. Llamas have a long, thick, curly coat of fur and long, sharp horns. They are very social and socialize with each other. They are also known for their amazing agility and speed. They are considered to be the fastest land animals in the world.\n",
+      "User: OK, do they have a temperature?\n",
       "\n",
-      "User: Who would win in a battle between a llama and an alpaca?\n",
+      "Joi:   Yes, they do. They are hot. They have a temperature of around 1.8 million kelvins.\n",
+      "\n",
+      "User: Hmm that seems pretty high. Are you sure?\n",
       "\n",
       "Joi:\n",
-      "{'generated_text': \" That depends on the alpaca. If they are of the same gender, then it depends on the alpaca's age, size, and condition. Generally speaking, the alpaca would win.\"}\n"
+      "{'generated_text': '   I’m not sure. I don’t think I’ve ever seen a black hole. I’ve never been to the surface of a black hole, but I believe I’ve seen the surrounding space-time.\\n\\nUser:'}\n"
      ]
     }
    ],
@@ -1000,7 +1018,7 @@
     "            temperature = gr.Slider(\n",
     "                minimum=0.0,\n",
     "                maximum=2.0,\n",
-    "                value=1.0,\n",
+    "                value=0.5,\n",
     "                step=0.1,\n",
     "                interactive=True,\n",
     "                label=\"Temperature\",\n",
@@ -1009,7 +1027,7 @@
     "            top_p = gr.Slider(\n",
     "                minimum=0.,\n",
     "                maximum=1.0,\n",
-    "                value=0.8,\n",
+    "                value=0.9,\n",
     "                step=0.05,\n",
     "                interactive=True,\n",
     "                label=\"Top-p (nucleus sampling)\",\n",
@@ -1063,14 +1081,14 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 40,
+   "execution_count": 12,
    "metadata": {},
    "outputs": [
     {
      "name": "stdout",
      "output_type": "stream",
      "text": [
-      "Closing server running on port: 7866\n"
+      "Closing server running on port: 7860\n"
      ]
     }
    ],
@@ -1080,20 +1098,9 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 48,
+   "execution_count": 13,
    "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...\n",
-      "To disable this warning, you can either:\n",
-      "\t- Avoid using `tokenizers` before the fork if possible\n",
-      "\t- Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)\n"
-     ]
-    }
-   ],
+   "outputs": [],
    "source": [
     "from nbdev.export import nb_export\n",
     "nb_export('app.ipynb', lib_path='.', name='app')"

app.py

@@ -156,7 +156,7 @@ with gr.Blocks(
             temperature = gr.Slider(
                 minimum=0.0,
                 maximum=2.0,
-                value=1.0,
+                value=0.5,
                 step=0.1,
                 interactive=True,
                 label="Temperature",
@@ -165,7 +165,7 @@ with gr.Blocks(
             top_p = gr.Slider(
                 minimum=0.,
                 maximum=1.0,
-                value=0.8,
+                value=0.9,
                 step=0.05,
                 interactive=True,
                 label="Top-p (nucleus sampling)",