"Supplementary code for the <a href=\"http://mng.bz/orYv\">Build a Large Language Model From Scratch</a> book by <a href=\"https://sebastianraschka.com\">Sebastian Raschka</a><br>\n",
"# Generating A Preference Dataset With Llama 3.1 70B And Ollama"
]
},
{
"cell_type": "markdown",
"id": "a128651b-f326-4232-a994-42f38b7ed520",
"metadata": {},
"source": [
"- Preference finetuning is a process to align an instruction-finetuned LLM with human preferences\n",
"- There are multiple ways to create a dataset for preference finetuning an LLM\n",
" 1. We use the instruction-finetuned LLM to generate multiple responses and have humans rank them based on their preference and/or given preference criteria\n",
" 2. We use the instruction-finetuned LLM to generate multiple responses and have LLMs rank them based on given preference criteria\n",
" 3. We use an LLM to generate preferred and dispreferred responses given certain preference criteria\n",
" \"instruction\": \"What is the state capital of California?\",\n",
" \"input\": \"\",\n",
" \"output\": \"The state capital of California is Sacramento.\",\n",
" },\n",
" {\n",
" \"instruction\": \"Provide a synonym for 'fast'.\",\n",
" \"input\": \"\",\n",
" \"output\": \"A synonym for 'fast' is 'quick'.\",\n",
" },\n",
" {\n",
" \"instruction\": \"What is the capital of Greece?\",\n",
" \"input\": \"\",\n",
" \"output\": \"The capital of Greece is Athens.\",\n",
"\n",
" },\n",
"...\n",
"]\n",
"```\n",
"\n",
"The output dataset will look as follows, where more polite responses are preferred (`'chosen'`), and more impolite responses are dispreferred (`'rejected'`):\n",
"- Ollama is an application to run LLMs efficiently\n",
"- It is a wrapper around [llama.cpp](https://github.com/ggerganov/llama.cpp), which implements LLMs in pure C/C++ to maximize efficiency\n",
"- Note that it is a tool for using LLMs to generate text (inference), not training or finetuning LLMs\n",
"- Prior to running the code below, install ollama by visiting [https://ollama.com](https://ollama.com) and following the instructions (for instance, clicking on the \"Download\" button and downloading the ollama application for your operating system)"
]
},
{
"cell_type": "markdown",
"id": "9558a522-650d-401a-84fc-9fd7b1f39da7",
"metadata": {},
"source": [
"- For macOS and Windows users, click on the ollama application you downloaded; if it prompts you to install the command line usage, say \"yes\"\n",
"- Linux users can use the installation command provided on the ollama website\n",
"\n",
"- In general, before we can use ollama from the command line, we have to either start the ollama application or run `ollama serve` in a separate terminal\n",
"- With the ollama application or `ollama serve` running, in a different terminal, on the command line, execute the following command to try out the 70-billion-parameter Llama 3.1 model \n",
"- Alternatively, you can also use the smaller, more resource-effiicent 8-billion-parameters Llama 3.1 model, by replacing `llama3.1:70b` with `llama3.1`\n",
"- After the download has been completed, you will see a command line prompt that allows you to chat with the model\n",
"\n",
"- Try a prompt like \"What do llamas eat?\", which should return an output similar to the following:\n",
"\n",
"```\n",
">>> What do llamas eat?\n",
"Llamas are ruminant animals, which means they have a four-chambered\n",
"stomach and eat plants that are high in fiber. In the wild, llamas\n",
"typically feed on:\n",
"1. Grasses: They love to graze on various types of grasses, including tall\n",
"grasses, wheat, oats, and barley.\n",
"```"
]
},
{
"cell_type": "markdown",
"id": "0b5addcb-fc7d-455d-bee9-6cc7a0d684c7",
"metadata": {},
"source": [
"- You can end this session using the input `/bye`"
]
},
{
"cell_type": "markdown",
"id": "dda155ee-cf36-44d3-b634-20ba8e1ca38a",
"metadata": {},
"source": [
"## Using Ollama's REST API"
]
},
{
"cell_type": "markdown",
"id": "89343a84-0ddc-42fc-bf50-298a342b93c0",
"metadata": {},
"source": [
"- Now, an alternative way to interact with the model is via its REST API in Python via the following function\n",
"- Before you run the next cells in this notebook, make sure that ollama is still running, as described above, via\n",
" - `ollama serve` in a terminal\n",
" - the ollama application\n",
"- Next, run the following code cell to query the model"
]
},
{
"cell_type": "markdown",
"id": "16642a48-1cab-40d2-af08-ab8c2fbf5876",
"metadata": {},
"source": [
"- First, let's try the API with a simple example to make sure it works as intended:"
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "65b0ba76-1fb1-4306-a7c2-8f3bb637ccdb",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Llamas are herbivores, which means they primarily eat plants and plant-based foods. Their diet consists of:\n",
"\n",
"1. **Grasses**: Various types of grasses, including timothy grass, orchard grass, and brome grass.\n",
"2. **Hay**: High-quality hay, such as alfalfa or clover hay, is a staple in a llama's diet.\n",
"3. **Leaves**: Leaves from trees and shrubs, like willow, cottonwood, and mesquite, are also eaten.\n",
"4. **Fruits and vegetables**: Llamas enjoy fruits like apples, carrots, and sweet potatoes, as well as leafy greens like kale and spinach.\n",
"5. **Grains**: In moderation, llamas can eat grains like oats, barley, and corn.\n",
"\n",
"It's essential to note that llamas have a unique digestive system, with a three-part stomach and a large cecum (a specialized part of the large intestine). This allows them to break down and extract nutrients from plant material more efficiently than many other animals.\n",
"\n",
"A typical llama diet might consist of:\n",
"\n",
"* 1-2% of their body weight in hay per day\n",
"* 0.5-1% of their body weight in grains per day (if fed)\n",
"* Free-choice access to fresh water\n",
"* Limited amounts of fruits and vegetables as treats\n",
"\n",
"It's also important to ensure that llamas have access to a mineral supplement, such as a salt lick or loose minerals, to help maintain optimal health.\n",
"\n",
"Remember, every llama is different, and their dietary needs may vary depending on factors like age, size, and activity level. Consult with a veterinarian or experienced llama breeder for specific guidance on feeding your llama.\n"
"- The structure of this file is as follows, where we have the given response in the test dataset (`'output'`) that we trained the model to generate via instruction finetuning based on the `'input'` and `'instruction'`"
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "7222fdc0-5684-4f2b-b741-3e341851359e",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"{'instruction': 'Evaluate the following phrase by transforming it into the spelling given.',\n",
" 'input': 'freind --> friend',\n",
" 'output': 'The spelling of the given phrase \"freind\" is incorrect, the correct spelling is \"friend\".'}"
]
},
"execution_count": 4,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"json_data[0]"
]
},
{
"cell_type": "markdown",
"id": "fcf0331b-6024-4bba-89a9-a088b14a1046",
"metadata": {},
"source": [
"- Below is a small utility function that formats the instruction and input:"
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "43263cd3-e5fb-4ab5-871e-3ad6e7d21a8c",
"metadata": {},
"outputs": [],
"source": [
"def format_input(entry):\n",
" instruction_text = (\n",
" f\"Below is an instruction that describes a task. Write a response that \"\n",
"- Let's now apply this evaluation to the whole dataset and compute the average score of each model (this takes about 1 minute per model on an M3 MacBook Air laptop)\n",
"- Note that ollama is not fully deterministic across operating systems (as of this writing) so the numbers you are getting might slightly differ from the ones shown below"
