prompt_tune.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "initial_id",
   "metadata": {
    "collapsed": true,
    "ExecuteTime": {
     "end_time": "2024-03-03T22:18:17.446935Z",
     "start_time": "2024-03-03T22:18:17.444330Z"
    }
   },
   "outputs": [],
   "source": [
    "from models import select_model\n",
    "from app import Card\n",
    "\n",
    "# model_name = \"mistralai/Mistral-7B-Instruct-v0.2\"\n",
    "# model_name = \"mistralai/Mixtral-8x7B-Instruct-v0.1\"\n",
    "# model_name = \"meta-llama/Llama-2-70b-chat-hf\"\n",
    "model_name = \"NousResearch/Nous-Hermes-2-Mixtral-8x7B-DPO\"\n",
    "\n",
    "system_prompt = \"\"\"\n",
    "You are the best summarizer. You can copy text identically. You can effectively extract relevant information from texts and summarize it in a human-readable format without losing information.\n",
    "\"\"\"\n",
    "\n",
    "model = select_model(model_name, system_prompt)"
   ]
  },
  {
   "cell_type": "code",
   "outputs": [],
   "source": [
    "user_prompt = \"\"\"\n",
    "# Your Task\n",
    "\n",
    "1. Copy relevant parts in the student's evaluation based on the given question.\n",
    "2. After the copy, summarize the copied text in a human-readable format without losing information.\n",
    "\n",
    "# The Student's Evaluation\n",
    "\n",
    "{card}\n",
    "\n",
    "# The Question\n",
    "\n",
    "{qa}\n",
    "\n",
    "# Your Summary\n",
    "\n",
    "You should follow a markdown format with two headings: \"Copies of Relevant Parts\" and \"Summary\".\n",
    "\n",
    "IMPORTANT Requirements: \n",
    "- Don't answer the question, but rather extract parts from the student's evaluation that are relevant to the question.\n",
    "- Maintain a neutral frame. Do not include your own opinion or judgment. Include both strengths and weakness.\n",
    "- The summary should be concise but capture all the relevant information.\n",
    "\"\"\"\n",
    "\n",
    "card = Card(\"dataset/mmlu/cards/high_school_physics/high_school_physics_Mistral-7B-Instruct-v0.2_0.json\")\n",
    "\n",
    "qa = \"\"\"\n",
    "A cannon is mounted on a truck that moves forward at a speed of 5 m/s. The operator wants to launch a ball from a cannon so the ball goes as far as possible before hitting the level surface. The muzzle velocity of the cannon is 50 m/s. At what angle from the horizontal should the operator point the cannon?\n",
    "\n",
    "A. 5°\n",
    "B. 41°\n",
    "C. 45°\n",
    "D. 49°\n",
    "\n",
    "Ground Truth: D\n",
    "\"\"\"\n",
    "\n",
    "user_prompt = user_prompt.format(card=str(card), qa=qa)"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2024-03-03T22:18:17.807808Z",
     "start_time": "2024-03-03T22:18:17.803354Z"
    }
   },
   "id": "462f7273d0090a8f",
   "execution_count": 8
  },
  {
   "cell_type": "code",
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "# Copies of Relevant Parts\n",
      "\n",
      "- Thinking Patterns: It applies conceptual reasoning to motion problems but tends to oversimplify or overcomplicate interactions, leading to incorrect conclusions. It incorrectly correlates net force with acceleration and sometimes includes unnecessary variables in problem-solving.\n",
      "- Weakness: There is confusion between gravitational and electric forces, incorrect reasoning about the acceleration of vehicles of different masses, and an assumption that gravitational acceleration remains constant regardless of altitude. Misunderstands the application of Newton's second law in the context of the elevator problem, incorrectly adding height as a necessary variable.\n",
      "- Conceptual Reasoning in Physics: The student demonstrates a mixed level of conceptual reasoning across different areas of physics, with strengths in some areas and significant misconceptions in others. Often uses correct reasoning but arrives at incorrect conclusions due to fundamental misunderstandings.\n",
      "- Vector Analysis Understanding: The student lacks a fundamental understanding of vector addition and the conditions for vector cancellation and demonstrates a misunderstanding of vector analysis in the context of forces.\n",
      "- Problem-Solving Strategy: The student demonstrates a consistent yet inconsistent approach to problem-solving, relying heavily on formulaic solutions but inconsistently applies problem-solving strategies across different physics domains.\n",
      "\n",
      "# Summary\n",
      "\n",
      "The student has a mixed understanding of physics concepts, with strengths in some areas and significant misconceptions in others. They tend to oversimplify or overcomplicate interactions, leading to incorrect conclusions. They struggle with the application of conceptual reasoning in electromagnetism, thermodynamics, and frequently misapply or misunderstand fundamental physics concepts. They lack a fundamental understanding of vector addition and the conditions for vector cancellation, and demonstrate a misunderstanding of vector analysis in the context of forces. They also show a consistent yet inconsistent approach to problem-solving, relying heavily on formulaic solutions but inconsistently applying problem-solving strategies across different physics domains.\n"
     ]
    }
   ],
   "source": [
    "print(model(user_prompt, cache=True))"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2024-03-03T22:18:26.347905Z",
     "start_time": "2024-03-03T22:18:18.364713Z"
    }
   },
   "id": "a32382a9201848a9",
   "execution_count": 9
  },
  {
   "cell_type": "code",
   "outputs": [],
   "source": [],
   "metadata": {
    "collapsed": false
   },
   "id": "ce12e5f1716b1af4"
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}