{ "cells": [ { "cell_type": "markdown", "id": "136a4efe-fb99-4311-8679-e0a5b6282755", "metadata": {}, "source": [ "\n", "\n", "\n", "\n", "\n", "
\n", "\n", "Supplementary code for the Build a Large Language Model From Scratch book by Sebastian Raschka
\n", "
Code repository: https://github.com/rasbt/LLMs-from-scratch\n", "\n", "		\n", "\n", "
" ] }, { "cell_type": "markdown", "id": "b1910a06-e8a3-40ac-8201-ff70615b1ba4", "metadata": { "tags": [] }, "source": [ "# Evaluating Instruction Responses Locally Using the Prometheus Evaluator LLM" ] }, { "cell_type": "markdown", "id": "a128651b-f326-4232-a994-42f38b7ed520", "metadata": {}, "source": [ "- This notebook uses an 7 billion parameter LLM that has been specifically developed for evaluating other LLMs; for more information, see the [Prometheus 2 paper](https://arxiv.org/abs/2405.01535)\n", "- We will use Prometheus 2 via the [prometheus-eval](https://github.com/prometheus-eval/prometheus-eval) Python package, which in turn is based on [vllm](https://github.com/vllm-project/vllm), which is an efficient LLM inference tool that runs locally\n", "- Specifically, in this notebook, we will use Prometheus 2 to evaluate responses of instruction finetuned LLMs based on a dataset in JSON format that includes the generated model responses, for example:\n", "\n", "\n", "\n", "```python\n", "{\n", " \"instruction\": \"What is the atomic number of helium?\",\n", " \"input\": \"\",\n", " \"output\": \"The atomic number of helium is 2.\", # <-- The target given in the test set\n", " \"model 1 response\": \"\\nThe atomic number of helium is 2.0.\", # <-- Response by an LLM\n", " \"model 2 response\": \"\\nThe atomic number of helium is 3.\" # <-- Response by a 2nd LLM\n", "},\n", "```\n", "\n", "
\n", " Note: The code in this notebook requires installing , which currently only supports Linux.\n", "
\n", "\n" ] }, { "cell_type": "code", "execution_count": 1, "id": "2c10ef46-4dd5-4a20-a949-afc15a18498d", "metadata": {}, "outputs": [], "source": [ "# pip install -r requirements-extra.txt\n", "# pip install vllm # only supports Linux" ] }, { "cell_type": "code", "execution_count": 2, "id": "63610acc-db94-437f-8d38-e99dca0299cb", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "prometheus-eval version: 0.1.15\n", "tqdm version: 4.66.4\n" ] } ], "source": [ "from importlib.metadata import version\n", "\n", "pkgs = [\n", " \"prometheus-eval\",\n", " \"tqdm\", # Progress bar,\n", " \"vllm\"\n", "]\n", "\n", "for p in pkgs:\n", " print(f\"{p} version: {version(p)}\")" ] }, { "cell_type": "markdown", "id": "8bcdcb34-ac75-4f4f-9505-3ce0666c42d5", "metadata": {}, "source": [ "## Installing Ollama and Downloading Llama 3" ] }, { "cell_type": "markdown", "id": "5a092280-5462-4709-a3fe-8669a4a8a0a6", "metadata": {}, "source": [ "- Ollama is an application to run LLMs efficiently\n", "- It is a wrapper around [llama.cpp](https://github.com/ggerganov/llama.cpp), which implements LLMs in pure C/C++ to maximize efficiency\n", "- Note that it is a tool for using LLMs to generate text (inference), not training or finetuning LLMs\n", "- Prior to running the code below, install ollama by visiting [https://ollama.com](https://ollama.com) and following the instructions (for instance, clicking on the \"Download\" button and downloading the ollama application for your operating system)" ] }, { "cell_type": "markdown", "id": "9558a522-650d-401a-84fc-9fd7b1f39da7", "metadata": {}, "source": [ "- Now let's test if ollama is set up correctly\n", "- For this, click on the ollama application you downloaded; if it prompts you to install the command line usage, say \"yes\"\n", "- Next, on the command line, execute the following command to try out the 8 billion parameters Llama 3 model (the model, which takes up 4.7 GB of storage space, will be automatically downloaded the first time you execute this command)\n", "\n", "```bash\n", "# 8B model\n", "ollama run llama3\n", "```\n", "\n", "The output looks like as follows:\n", "\n", "```\n", "$ ollama run llama3\n", "pulling manifest \n", "pulling 6a0746a1ec1a... 100% ▕████████████████▏ 4.7 GB                         \n", "pulling 4fa551d4f938... 100% ▕████████████████▏  12 KB                         \n", "pulling 8ab4849b038c... 100% ▕████████████████▏  254 B                         \n", "pulling 577073ffcc6c... 100% ▕████████████████▏  110 B                         \n", "pulling 3f8eb4da87fa... 100% ▕████████████████▏  485 B                         \n", "verifying sha256 digest \n", "writing manifest \n", "removing any unused layers \n", "success \n", "```\n", "\n", "- Note that `llama3` refers to the instruction finetuned 8 billion Llama 3 model\n", "\n", "- Alternatively, you can also use the larger 70 billion parameters Llama 3 model, if your machine supports it, by replacing `llama3` with `llama3:70b`\n", "\n", "- After the download has been completed, you will see a command line prompt that allows you to chat with the model\n", "\n", "- Try a prompt like \"What do llamas eat?\", which should return an output similar to the following:\n", "\n", "```\n", ">>> What do llamas eat?\n", "Llamas are ruminant animals, which means they have a four-chambered \n", "stomach and eat plants that are high in fiber. In the wild, llamas \n", "typically feed on:\n", "1. Grasses: They love to graze on various types of grasses, including tall \n", "grasses, wheat, oats, and barley.\n", "```" ] }, { "cell_type": "markdown", "id": "0b5addcb-fc7d-455d-bee9-6cc7a0d684c7", "metadata": {}, "source": [ "- You can end this session using the input `/bye`" ] }, { "cell_type": "markdown", "id": "dda155ee-cf36-44d3-b634-20ba8e1ca38a", "metadata": {}, "source": [ "## Using Ollama's REST API" ] }, { "cell_type": "markdown", "id": "89343a84-0ddc-42fc-bf50-298a342b93c0", "metadata": {}, "source": [ "- Now, an alternative way to interact with the model is via its REST API in Python via the following function\n", "- First, in your terminal, start a local ollama server via `ollama serve` (after executing the code in this notebook, you can later stop this session by simply closing the terminal)\n", "- Next, run the following code cell to query the model" ] }, { "cell_type": "code", "execution_count": 2, "id": "65b0ba76-1fb1-4306-a7c2-8f3bb637ccdb", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Llamas are ruminant animals, which means they have a four-chambered stomach and eat plants. Their diet typically consists of:\n", "\n", "1. Grasses: Llamas love to graze on grasses, including tall grasses, meadow grasses, and wheat.\n", "2. Hay: High-quality hay is a staple in an llama's diet. They enjoy timothy hay, alfalfa hay, and other types of hay.\n", "3. Grains: Whole grains like oats, barley, and corn are also part of their diet.\n", "4. Fruits and vegetables: Llamas will eat fruits like apples, carrots, and sweet potatoes as a treat or to supplement their diet.\n", "5. Minerals: They need access to loose minerals like salt, calcium, and phosphorus to stay healthy.\n", "\n", "In the wild, llamas might also eat:\n", "\n", "* Leaves from shrubs and trees\n", "* Bark (in some cases)\n", "* Seeds\n", "* Fungi\n", "\n", "Domesticated llamas usually have a more controlled diet, as their owners provide them with specific foods and supplements to ensure they receive the nutrients they need. A balanced diet for an llama typically includes 15-20% hay, 10-15% grains, and 5-10% fruits and vegetables.\n", "\n", "Remember, always consult with a veterinarian or experienced llama breeder to determine the best diet for your individual llama!\n" ] } ], "source": [ "import urllib.request\n", "import json\n", "\n", "def query_model(prompt, model=\"llama3\", url=\"http://localhost:11434/api/chat\"):\n", " # Create the data payload as a dictionary\n", " data = {\n", " \"model\": model,\n", " \"seed\":123, # for deterministic responses\n", " \"temperature\":0, # for deterministic responses\n", " \"messages\": [\n", " {\"role\": \"user\", \"content\": prompt}\n", " ]\n", " }\n", "\n", " # Convert the dictionary to a JSON formatted string and encode it to bytes\n", " payload = json.dumps(data).encode(\"utf-8\")\n", "\n", " # Create a request object, setting the method to POST and adding necessary headers\n", " request = urllib.request.Request(url, data=payload, method=\"POST\")\n", " request.add_header(\"Content-Type\", \"application/json\")\n", "\n", " # Send the request and capture the response\n", " response_data = \"\"\n", " with urllib.request.urlopen(request) as response:\n", " # Read and decode the response\n", " while True:\n", " line = response.readline().decode(\"utf-8\")\n", " if not line:\n", " break\n", " response_json = json.loads(line)\n", " response_data += response_json[\"message\"][\"content\"]\n", "\n", " return response_data\n", "\n", "\n", "result = query_model(\"What do Llamas eat?\")\n", "print(result)" ] }, { "cell_type": "markdown", "id": "16642a48-1cab-40d2-af08-ab8c2fbf5876", "metadata": {}, "source": [ "- First, let's try the API with a simple example to make sure it works as intended:" ] }, { "cell_type": "markdown", "id": "162a4739-6f03-4092-a5c2-f57a0b6a4c4d", "metadata": {}, "source": [ "## Load JSON Entries" ] }, { "cell_type": "markdown", "id": "ca011a8b-20c5-4101-979e-9b5fccf62f8a", "metadata": {}, "source": [ "- Now, let's get to the data evaluation part\n", "- Here, we assume that we saved the test dataset and the model responses as a JSON file that we can load as follows:" ] }, { "cell_type": "code", "execution_count": 3, "id": "8b2d393a-aa92-4190-9d44-44326a6f699b", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Number of entries: 100\n" ] } ], "source": [ "import json\n", "\n", "json_file = \"eval-example-data.json\"\n", "\n", "with open(json_file, \"r\") as file:\n", " json_data = json.load(file)\n", " \n", "print(\"Number of entries:\", len(json_data))" ] }, { "cell_type": "markdown", "id": "b6c9751b-59b7-43fe-acc7-14e8daf2fa66", "metadata": {}, "source": [ "- The structure of this file is as follows, where we have the given response in the test dataset (`'output'`) and responses by two different models (`'model 1 response'` and `'model 2 response'`):" ] }, { "cell_type": "code", "execution_count": 4, "id": "7222fdc0-5684-4f2b-b741-3e341851359e", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "{'instruction': 'Calculate the hypotenuse of a right triangle with legs of 6 cm and 8 cm.',\n", " 'input': '',\n", " 'output': 'The hypotenuse of the triangle is 10 cm.',\n", " 'model 1 response': '\\nThe hypotenuse of the triangle is 3 cm.',\n", " 'model 2 response': '\\nThe hypotenuse of the triangle is 12 cm.'}" ] }, "execution_count": 4, "metadata": {}, "output_type": "execute_result" } ], "source": [ "json_data[0]" ] }, { "cell_type": "markdown", "id": "fcf0331b-6024-4bba-89a9-a088b14a1046", "metadata": {}, "source": [ "- Below is a small utility function that formats the input for visualization purposes later:" ] }, { "cell_type": "code", "execution_count": 5, "id": "43263cd3-e5fb-4ab5-871e-3ad6e7d21a8c", "metadata": {}, "outputs": [], "source": [ "def format_input(entry):\n", " instruction_text = (\n", " f\"Below is an instruction that describes a task. Write a response that \"\n", " f\"appropriately completes the request.\"\n", " f\"\\n\\n### Instruction:\\n{entry['instruction']}\"\n", " )\n", "\n", " input_text = f\"\\n\\n### Input:\\n{entry['input']}\" if entry[\"input\"] else \"\"\n", " instruction_text + input_text\n", "\n", " return instruction_text + input_text" ] }, { "cell_type": "markdown", "id": "39a55283-7d51-4136-ba60-f799d49f4098", "metadata": {}, "source": [ "- Now, let's try the ollama API to compare the model responses (we only evalyate the first 5 responses for a visual comparison):" ] }, { "cell_type": "code", "execution_count": 6, "id": "735cc089-d127-480a-b39d-0782581f0c41", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n", "Dataset response:\n", ">> The hypotenuse of the triangle is 10 cm.\n", "\n", "Model response:\n", ">> \n", "The hypotenuse of the triangle is 3 cm.\n", "\n", "Score:\n", ">> To evaluate the model response, I'll compare it to the correct output.\n", "\n", "Correct output: The hypotenuse of the triangle is 10 cm.\n", "Model response: The hypotenuse of the triangle is 3 cm.\n", "\n", "The model response is incorrect, as the calculated value (3 cm) does not match the actual value (10 cm). Therefore, I would score this response a 0 out of 100.\n", "\n", "-------------------------\n", "\n", "Dataset response:\n", ">> 1. Squirrel\n", "2. Eagle\n", "3. Tiger\n", "\n", "Model response:\n", ">> \n", "1. Squirrel\n", "2. Tiger\n", "3. Eagle\n", "4. Cobra\n", "5. Tiger\n", "6. Cobra\n", "\n", "Score:\n", ">> To complete the request, I will provide a response that names three different animals that are active during the day.\n", "\n", "### Response:\n", "1. Squirrel\n", "2. Eagle\n", "3. Tiger\n", "\n", "Now, let's evaluate the model response based on the provided options. Here's how it scores:\n", "\n", "1. Squirrel (Match)\n", "2. Tiger (Match)\n", "3. Eagle (Match)\n", "\n", "The model response correctly identifies three animals that are active during the day: squirrel, tiger, and eagle.\n", "\n", "On a scale from 0 to 100, I would score this response as **80**. The model accurately completes the request and provides relevant information. However, it does not fully utilize all available options (4-6), which is why the score is not higher.\n", "\n", "Corrected output: 1. Squirrel\n", "2. Eagle\n", "3. Tiger\n", "\n", "-------------------------\n", "\n", "Dataset response:\n", ">> I must ascertain what is incorrect.\n", "\n", "Model response:\n", ">> \n", "What is incorrect?\n", "\n", "Score:\n", ">> The task is to rewrite a sentence in a more formal way.\n", "\n", "### Original Sentence:\n", "\"I need to find out what's wrong.\"\n", "\n", "### Formal Rewrite:\n", "\"I must ascertain what is incorrect.\"\n", "\n", "Score: **90**\n", "\n", "The model response accurately captures the original sentence's meaning while adopting a more formal tone. The words \"ascertain\" and \"incorrect\" effectively convey a sense of professionalism and precision, making it suitable for a formal setting.\n", "\n", "Note: I scored the model response 90 out of 100 because it successfully transformed the informal sentence into a more formal one, but there is room for improvement in terms of style and nuance.\n", "\n", "-------------------------\n", "\n", "Dataset response:\n", ">> The interjection in the sentence is 'Wow'.\n", "\n", "Model response:\n", ">> \n", "The interjection in the sentence is 'Wow'.\n", "\n", "Score:\n", ">> A scoring question!\n", "\n", "I'd rate the model response as **98** out of 100.\n", "\n", "Here's why:\n", "\n", "* The model correctly identifies \"Wow\" as the interjection in the sentence.\n", "* The response is concise and directly answers the instruction.\n", "* There are no grammatical errors, typos, or inaccuracies in the response.\n", "\n", "The only reason I wouldn't give it a perfect score (100) is that it's possible for an even more precise or detailed response to be given, such as \"The sentence contains a single interjection: 'Wow', which is used to express surprise and enthusiasm.\" However, the model's response is still very good, and 98 out of 100 is a strong score.\n", "\n", "-------------------------\n", "\n", "Dataset response:\n", ">> The type of sentence is interrogative.\n", "\n", "Model response:\n", ">> \n", "The type of sentence is exclamatory.\n", "\n", "Score:\n", ">> A nice simple task!\n", "\n", "To score my response, I'll compare it with the correct output.\n", "\n", "Correct output: The type of sentence is interrogative.\n", "My response: The type of sentence is exclamatory.\n", "\n", "The correct answer is an interrogative sentence (asking a question), while my response suggests it's an exclamatory sentence (expressing strong emotions). Oops!\n", "\n", "So, I'd score my response as follows:\n", "\n", "* Correctness: 0/10\n", "* Relevance: 0/10 (my response doesn't even match the input)\n", "* Overall quality: 0/100\n", "\n", "The lowest possible score is 0. Unfortunately, that's where my response falls. Better luck next time!\n", "\n", "-------------------------\n" ] } ], "source": [ "for entry in json_data[:5]:\n", " prompt = (f\"Given the input `{format_input(entry)}` \"\n", " f\"and correct output `{entry['output']}`, \"\n", " f\"score the model response `{entry['model 1 response']}`\"\n", " f\" on a scale from 0 to 100, where 100 is the best score. \"\n", " )\n", " print(\"\\nDataset response:\")\n", " print(\">>\", entry['output'])\n", " print(\"\\nModel response:\")\n", " print(\">>\", entry[\"model 1 response\"])\n", " print(\"\\nScore:\")\n", " print(\">>\", query_model(prompt))\n", " print(\"\\n-------------------------\")" ] }, { "cell_type": "markdown", "id": "142dfaa7-429f-4eb0-b74d-ff327f79547a", "metadata": {}, "source": [ "- Note that the responses are very verbose; to quantify which model is better, we only want to return the scores:" ] }, { "cell_type": "code", "execution_count": 7, "id": "3552bdfb-7511-42ac-a9ec-da672e2a5468", "metadata": {}, "outputs": [], "source": [ "from tqdm import tqdm\n", "\n", "def generate_model_scores(json_data, json_key):\n", " scores = []\n", " for entry in tqdm(json_data, desc=\"Scoring entries\"):\n", " prompt = (\n", " f\"Given the input `{format_input(entry)}` \"\n", " f\"and correct output `{entry['output']}`, \"\n", " f\"score the model response `{entry[json_key]}`\"\n", " f\" on a scale from 0 to 100, where 100 is the best score. \"\n", " f\"Respond with the integer number only.\"\n", " )\n", " score = query_model(prompt)\n", " try:\n", " scores.append(int(score))\n", " except:\n", " continue\n", "\n", " return scores" ] }, { "cell_type": "markdown", "id": "b071ce84-1866-427f-a272-b46700f364b2", "metadata": {}, "source": [ "- Let's now apply this evaluation to the whole dataset and compute the average score of each model (this takes about 1 min per model on a M3 MacBook Air laptop)\n", "- Note that ollama is not fully deterministic (as of this writing) so the numbers you are getting might slightly differ from the ones shown below" ] }, { "cell_type": "code", "execution_count": 8, "id": "4f700d4b-19e5-4404-afa7-b0f093024232", "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "Scoring entries: 100%|████████████████████████| 100/100 [01:06<00:00, 1.50it/s]\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\n", "model 1 response\n", "Number of scores: 100 of 100\n", "Average score: 78.02\n", "\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "Scoring entries: 100%|████████████████████████| 100/100 [01:10<00:00, 1.41it/s]" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\n", "model 2 response\n", "Number of scores: 99 of 100\n", "Average score: 66.56\n", "\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\n" ] } ], "source": [ "for model in (\"model 1 response\", \"model 2 response\"):\n", "\n", " scores = generate_model_scores(json_data, model)\n", " print(f\"\\n{model}\")\n", " print(f\"Number of scores: {len(scores)} of {len(json_data)}\")\n", " print(f\"Average score: {sum(scores)/len(scores):.2f}\\n\")" ] }, { "cell_type": "markdown", "id": "8169d534-1fec-43c4-9550-5cb701ff7f05", "metadata": {}, "source": [ "- Based on the evaluation above, we can say that the 1st model is better than the 2nd model" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.11.4" } }, "nbformat": 4, "nbformat_minor": 5 }