{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Groq\n", "\n", "[Groq](https://groq.com/) is a cloud based platform serving a number of popular open weight models at high inference speeds. Models include Meta's Llama 3, Mistral AI's Mixtral, and Google's Gemma.\n", "\n", "Although Groq's API is aligned well with OpenAI's, which is the native API used by AutoGen, this library provides the ability to set specific parameters as well as track API costs.\n", "\n", "You will need a Groq account and create an API key. [See their website for further details](https://groq.com/)." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Groq provides a number of models to use, included below. See the list of [models here (requires login)](https://console.groq.com/docs/models).\n", "\n", "See the sample `OAI_CONFIG_LIST` below showing how the Groq client class is used by specifying the `api_type` as `groq`.\n", "\n", "```python\n", "[\n", " {\n", " \"model\": \"gpt-35-turbo\",\n", " \"api_key\": \"your OpenAI Key goes here\",\n", " },\n", " {\n", " \"model\": \"gpt-4-vision-preview\",\n", " \"api_key\": \"your OpenAI Key goes here\",\n", " },\n", " {\n", " \"model\": \"dalle\",\n", " \"api_key\": \"your OpenAI Key goes here\",\n", " },\n", " {\n", " \"model\": \"llama3-8b-8192\",\n", " \"api_key\": \"your Groq API Key goes here\",\n", " \"api_type\": \"groq\"\n", " },\n", " {\n", " \"model\": \"llama3-70b-8192\",\n", " \"api_key\": \"your Groq API Key goes here\",\n", " \"api_type\": \"groq\"\n", " },\n", " {\n", " \"model\": \"Mixtral 8x7b\",\n", " \"api_key\": \"your Groq API Key goes here\",\n", " \"api_type\": \"groq\"\n", " },\n", " {\n", " \"model\": \"gemma-7b-it\",\n", " \"api_key\": \"your Groq API Key goes here\",\n", " \"api_type\": \"groq\"\n", " }\n", "]\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "As an alternative to the `api_key` key and value in the config, you can set the environment variable `GROQ_API_KEY` to your Groq key.\n", "\n", "Linux/Mac:\n", "``` bash\n", "export GROQ_API_KEY=\"your_groq_api_key_here\"\n", "```\n", "\n", "Windows:\n", "``` bash\n", "set GROQ_API_KEY=your_groq_api_key_here\n", "```\n", "\n", "## API parameters\n", "\n", "The following parameters can be added to your config for the Groq API. See [this link](https://console.groq.com/docs/text-chat) for further information on them.\n", "\n", "- frequency_penalty (number 0..1)\n", "- max_tokens (integer >= 0)\n", "- presence_penalty (number -2..2)\n", "- seed (integer)\n", "- temperature (number 0..2)\n", "- top_p (number)\n", "\n", "Example:\n", "```python\n", "[\n", " {\n", " \"model\": \"llama3-8b-8192\",\n", " \"api_key\": \"your Groq API Key goes here\",\n", " \"api_type\": \"groq\",\n", " \"frequency_penalty\": 0.5,\n", " \"max_tokens\": 2048,\n", " \"presence_penalty\": 0.2,\n", " \"seed\": 42,\n", " \"temperature\": 0.5,\n", " \"top_p\": 0.2\n", " }\n", "]\n", "```\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Two-Agent Coding Example\n", "\n", "In this example, we run a two-agent chat with an AssistantAgent (primarily a coding agent) to generate code to count the number of prime numbers between 1 and 10,000 and then it will be executed.\n", "\n", "We'll use Meta's Llama 3 model which is suitable for coding." ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [], "source": [ "import os\n", "\n", "config_list = [\n", " {\n", " # Let's choose the Llama 3 model\n", " \"model\": \"llama3-8b-8192\",\n", " # Put your Groq API key here or put it into the GROQ_API_KEY environment variable.\n", " \"api_key\": os.environ.get(\"GROQ_API_KEY\"),\n", " # We specify the API Type as 'groq' so it uses the Groq client class\n", " \"api_type\": \"groq\",\n", " }\n", "]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Importantly, we have tweaked the system message so that the model doesn't return the termination keyword, which we've changed to FINISH, with the code block." ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "/usr/local/lib/python3.11/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n", " from .autonotebook import tqdm as notebook_tqdm\n" ] } ], "source": [ "from pathlib import Path\n", "\n", "from autogen import AssistantAgent, UserProxyAgent\n", "from autogen.coding import LocalCommandLineCodeExecutor\n", "\n", "# Setting up the code executor\n", "workdir = Path(\"coding\")\n", "workdir.mkdir(exist_ok=True)\n", "code_executor = LocalCommandLineCodeExecutor(work_dir=workdir)\n", "\n", "# Setting up the agents\n", "\n", "# The UserProxyAgent will execute the code that the AssistantAgent provides\n", "user_proxy_agent = UserProxyAgent(\n", " name=\"User\",\n", " code_execution_config={\"executor\": code_executor},\n", " is_termination_msg=lambda msg: \"FINISH\" in msg.get(\"content\"),\n", ")\n", "\n", "system_message = \"\"\"You are a helpful AI assistant who writes code and the user executes it.\n", "Solve tasks using your coding and language skills.\n", "In the following cases, suggest python code (in a python coding block) for the user to execute.\n", "Solve the task step by step if you need to. If a plan is not provided, explain your plan first. Be clear which step uses code, and which step uses your language skill.\n", "When using code, you must indicate the script type in the code block. The user cannot provide any other feedback or perform any other action beyond executing the code you suggest. The user can't modify your code. So do not suggest incomplete code which requires users to modify. Don't use a code block if it's not intended to be executed by the user.\n", "Don't include multiple code blocks in one response. Do not ask users to copy and paste the result. Instead, use 'print' function for the output when relevant. Check the execution result returned by the user.\n", "If the result indicates there is an error, fix the error and output the code again. Suggest the full code instead of partial code or code changes. If the error can't be fixed or if the task is not solved even after the code is executed successfully, analyze the problem, revisit your assumption, collect additional info you need, and think of a different approach to try.\n", "When you find an answer, verify the answer carefully. Include verifiable evidence in your response if possible.\n", "IMPORTANT: Wait for the user to execute your code and then you can reply with the word \"FINISH\". DO NOT OUTPUT \"FINISH\" after your code block.\"\"\"\n", "\n", "# The AssistantAgent, using Groq's model, will take the coding request and return code\n", "assistant_agent = AssistantAgent(\n", " name=\"Groq Assistant\",\n", " system_message=system_message,\n", " llm_config={\"config_list\": config_list},\n", ")" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\u001b[33mUser\u001b[0m (to Groq Assistant):\n", "\n", "Provide code to count the number of prime numbers from 1 to 10000.\n", "\n", "--------------------------------------------------------------------------------\n", "\u001b[33mGroq Assistant\u001b[0m (to User):\n", "\n", "Here's the plan to count the number of prime numbers from 1 to 10000:\n", "\n", "First, we need to write a helper function to check if a number is prime. A prime number is a number that is divisible only by 1 and itself.\n", "\n", "Then, we can use a loop to iterate through all numbers from 1 to 10000, check if each number is prime using our helper function, and count the number of prime numbers found.\n", "\n", "Here's the Python code to implement this plan:\n", "```python\n", "def is_prime(n):\n", " if n <= 1:\n", " return False\n", " for i in range(2, int(n ** 0.5) + 1):\n", " if n % i == 0:\n", " return False\n", " return True\n", "\n", "count = 0\n", "for i in range(2, 10001):\n", " if is_prime(i):\n", " count += 1\n", "\n", "print(count)\n", "```\n", "Please execute this code, and I'll wait for the result.\n", "\n", "--------------------------------------------------------------------------------\n", "\u001b[31m\n", ">>>>>>>> NO HUMAN INPUT RECEIVED.\u001b[0m\n", "\u001b[31m\n", ">>>>>>>> USING AUTO REPLY...\u001b[0m\n", "\u001b[31m\n", ">>>>>>>> EXECUTING CODE BLOCK (inferred language is python)...\u001b[0m\n", "\u001b[33mUser\u001b[0m (to Groq Assistant):\n", "\n", "exitcode: 0 (execution succeeded)\n", "Code output: 1229\n", "\n", "\n", "--------------------------------------------------------------------------------\n", "\u001b[33mGroq Assistant\u001b[0m (to User):\n", "\n", "FINISH\n", "\n", "--------------------------------------------------------------------------------\n", "\u001b[31m\n", ">>>>>>>> NO HUMAN INPUT RECEIVED.\u001b[0m\n" ] } ], "source": [ "# Start the chat, with the UserProxyAgent asking the AssistantAgent the message\n", "chat_result = user_proxy_agent.initiate_chat(\n", " assistant_agent,\n", " message=\"Provide code to count the number of prime numbers from 1 to 10000.\",\n", ")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Tool Call Example\n", "\n", "In this example, instead of writing code, we will show how we can use Meta's Llama 3 model to perform parallel tool calling, where it recommends calling more than one tool at a time, using Groq's cloud inference.\n", "\n", "We'll use a simple travel agent assistant program where we have a couple of tools for weather and currency conversion.\n", "\n", "We start by importing libraries and setting up our configuration to use Meta's Llama 3 model and the `groq` client class." ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [], "source": [ "import json\n", "import os\n", "from typing import Literal\n", "\n", "from typing_extensions import Annotated\n", "\n", "import autogen\n", "\n", "config_list = [\n", " {\"api_type\": \"groq\", \"model\": \"llama3-8b-8192\", \"api_key\": os.getenv(\"GROQ_API_KEY\"), \"cache_seed\": None}\n", "]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Create our two agents." ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [], "source": [ "# Create the agent for tool calling\n", "chatbot = autogen.AssistantAgent(\n", " name=\"chatbot\",\n", " system_message=\"\"\"For currency exchange and weather forecasting tasks,\n", " only use the functions you have been provided with.\n", " Output 'HAVE FUN!' when an answer has been provided.\"\"\",\n", " llm_config={\"config_list\": config_list},\n", ")\n", "\n", "# Note that we have changed the termination string to be \"HAVE FUN!\"\n", "user_proxy = autogen.UserProxyAgent(\n", " name=\"user_proxy\",\n", " is_termination_msg=lambda x: x.get(\"content\", \"\") and \"HAVE FUN!\" in x.get(\"content\", \"\"),\n", " human_input_mode=\"NEVER\",\n", " max_consecutive_auto_reply=1,\n", ")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Create the two functions, annotating them so that those descriptions can be passed through to the LLM.\n", "\n", "We associate them with the agents using `register_for_execution` for the user_proxy so it can execute the function and `register_for_llm` for the chatbot (powered by the LLM) so it can pass the function definitions to the LLM." ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [], "source": [ "# Currency Exchange function\n", "\n", "CurrencySymbol = Literal[\"USD\", \"EUR\"]\n", "\n", "# Define our function that we expect to call\n", "\n", "\n", "def exchange_rate(base_currency: CurrencySymbol, quote_currency: CurrencySymbol) -> float:\n", " if base_currency == quote_currency:\n", " return 1.0\n", " elif base_currency == \"USD\" and quote_currency == \"EUR\":\n", " return 1 / 1.1\n", " elif base_currency == \"EUR\" and quote_currency == \"USD\":\n", " return 1.1\n", " else:\n", " raise ValueError(f\"Unknown currencies {base_currency}, {quote_currency}\")\n", "\n", "\n", "# Register the function with the agent\n", "\n", "\n", "@user_proxy.register_for_execution()\n", "@chatbot.register_for_llm(description=\"Currency exchange calculator.\")\n", "def currency_calculator(\n", " base_amount: Annotated[float, \"Amount of currency in base_currency\"],\n", " base_currency: Annotated[CurrencySymbol, \"Base currency\"] = \"USD\",\n", " quote_currency: Annotated[CurrencySymbol, \"Quote currency\"] = \"EUR\",\n", ") -> str:\n", " quote_amount = exchange_rate(base_currency, quote_currency) * base_amount\n", " return f\"{format(quote_amount, '.2f')} {quote_currency}\"\n", "\n", "\n", "# Weather function\n", "\n", "\n", "# Example function to make available to model\n", "def get_current_weather(location, unit=\"fahrenheit\"):\n", " \"\"\"Get the weather for some location\"\"\"\n", " if \"chicago\" in location.lower():\n", " return json.dumps({\"location\": \"Chicago\", \"temperature\": \"13\", \"unit\": unit})\n", " elif \"san francisco\" in location.lower():\n", " return json.dumps({\"location\": \"San Francisco\", \"temperature\": \"55\", \"unit\": unit})\n", " elif \"new york\" in location.lower():\n", " return json.dumps({\"location\": \"New York\", \"temperature\": \"11\", \"unit\": unit})\n", " else:\n", " return json.dumps({\"location\": location, \"temperature\": \"unknown\"})\n", "\n", "\n", "# Register the function with the agent\n", "\n", "\n", "@user_proxy.register_for_execution()\n", "@chatbot.register_for_llm(description=\"Weather forecast for US cities.\")\n", "def weather_forecast(\n", " location: Annotated[str, \"City name\"],\n", ") -> str:\n", " weather_details = get_current_weather(location=location)\n", " weather = json.loads(weather_details)\n", " return f\"{weather['location']} will be {weather['temperature']} degrees {weather['unit']}\"" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We pass through our customers message and run the chat.\n", "\n", "Finally, we ask the LLM to summarise the chat and print that out." ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\u001b[33muser_proxy\u001b[0m (to chatbot):\n", "\n", "What's the weather in New York and can you tell me how much is 123.45 EUR in USD so I can spend it on my holiday? Throw a few holiday tips in as well.\n", "\n", "--------------------------------------------------------------------------------\n", "\u001b[33mchatbot\u001b[0m (to user_proxy):\n", "\n", "\u001b[32m***** Suggested tool call (call_hg7g): weather_forecast *****\u001b[0m\n", "Arguments: \n", "{\"location\":\"New York\"}\n", "\u001b[32m*************************************************************\u001b[0m\n", "\u001b[32m***** Suggested tool call (call_hrsf): currency_calculator *****\u001b[0m\n", "Arguments: \n", "{\"base_amount\":123.45,\"base_currency\":\"EUR\",\"quote_currency\":\"USD\"}\n", "\u001b[32m****************************************************************\u001b[0m\n", "\n", "--------------------------------------------------------------------------------\n", "\u001b[35m\n", ">>>>>>>> EXECUTING FUNCTION weather_forecast...\u001b[0m\n", "\u001b[35m\n", ">>>>>>>> EXECUTING FUNCTION currency_calculator...\u001b[0m\n", "\u001b[33muser_proxy\u001b[0m (to chatbot):\n", "\n", "\u001b[33muser_proxy\u001b[0m (to chatbot):\n", "\n", "\u001b[32m***** Response from calling tool (call_hg7g) *****\u001b[0m\n", "New York will be 11 degrees fahrenheit\n", "\u001b[32m**************************************************\u001b[0m\n", "\n", "--------------------------------------------------------------------------------\n", "\u001b[33muser_proxy\u001b[0m (to chatbot):\n", "\n", "\u001b[32m***** Response from calling tool (call_hrsf) *****\u001b[0m\n", "135.80 USD\n", "\u001b[32m**************************************************\u001b[0m\n", "\n", "--------------------------------------------------------------------------------\n", "\u001b[33mchatbot\u001b[0m (to user_proxy):\n", "\n", "\u001b[32m***** Suggested tool call (call_ahwk): weather_forecast *****\u001b[0m\n", "Arguments: \n", "{\"location\":\"New York\"}\n", "\u001b[32m*************************************************************\u001b[0m\n", "\n", "--------------------------------------------------------------------------------\n", "LLM SUMMARY: Based on the conversation, it's predicted that New York will be 11 degrees Fahrenheit. You also found out that 123.45 EUR is equal to 135.80 USD. Here are a few holiday tips:\n", "\n", "* Pack warm clothing for your visit to New York, as the temperature is expected to be quite chilly.\n", "* Consider exchanging your money at a local currency exchange or an ATM since the exchange rate might not be as favorable in tourist areas.\n", "* Make sure to check the estimated expenses for your holiday and adjust your budget accordingly.\n", "\n", "I hope you have a great trip!\n" ] } ], "source": [ "# start the conversation\n", "res = user_proxy.initiate_chat(\n", " chatbot,\n", " message=\"What's the weather in New York and can you tell me how much is 123.45 EUR in USD so I can spend it on my holiday? Throw a few holiday tips in as well.\",\n", " summary_method=\"reflection_with_llm\",\n", ")\n", "\n", "print(f\"LLM SUMMARY: {res.summary['content']}\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Using its fast inference, Groq required less than 2 seconds for the whole chat!\n", "\n", "Additionally, Llama 3 was able to call both tools and pass through the right parameters. The `user_proxy` then executed them and this was passed back for Llama 3 to summarise the whole conversation." ] } ], "metadata": { "kernelspec": { "display_name": "autogen", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.11.9" } }, "nbformat": 4, "nbformat_minor": 2 }