yujunjun/autogen
1
0
Fork 0
mirror of https://github.com/microsoft/autogen.git synced 2025-09-03 13:27:41 +00:00
Code Issues Packages Projects Releases Wiki Activity

LLM Observability documentation fixes: Broken links, grammar, and spelling (#2995)

Browse Source 
* update markdown hyperlinks to stable urls

* update notebook images and text

* re-write observability section

* Updated section

* update wording

* added newline

* update styling in image tags to be jsx compatible

* added text

* update link

* simplified text

---------

Co-authored-by: Braelyn Boynton <bboynton97@gmail.com>
This commit is contained in:
Alex Reibman 2024-06-21 20:05:11 -07:00 committed by GitHub
parent e4720bc6cf
commit 9b92093b00
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
3 changed files with 140 additions and 132 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

185
notebook/agentchat_agentops.ipynb
View File

@ -7,7 +7,7 @@
"collapsed": false "collapsed": false
}, },
"source": [ "source": [
"# AgentOps" "# Agent Tracking with AgentOps"
] ]
}, },
{ {
@ -15,7 +15,7 @@
"id": "a447802c88c8a240", "id": "a447802c88c8a240",
"metadata": {}, "metadata": {},
"source": [ "source": [
"![logo](https://raw.githubusercontent.com/AgentOps-AI/agentops/35d5682866921a9e28d8ef66ae3c3b3d92d8fa6b/img/logo.png)\n", "<img src=\"https://github.com/AgentOps-AI/agentops/blob/main/docs/images/external/logo/banner-badge.png?raw=true\"/>\n",
"\n", "\n",
"[AgentOps](https://agentops.ai/?=autogen) provides session replays, metrics, and monitoring for AI agents.\n", "[AgentOps](https://agentops.ai/?=autogen) provides session replays, metrics, and monitoring for AI agents.\n",
"\n", "\n",
@ -27,8 +27,11 @@
"id": "b354c068", "id": "b354c068",
"metadata": {}, "metadata": {},
"source": [ "source": [
"### Dashboard\n", "### Overview Dashboard\n",
"![Agent Dashboard](https://github.com/AgentOps-AI/agentops/assets/14807319/158e082a-9a7d-49b7-9b41-51a49a1f7d3d)" "<img src=\"https://raw.githubusercontent.com/AgentOps-AI/agentops/main/docs/images/external/app_screenshots/overview.gif\"/>\n",
"\n",
"### Session Replays\n",
"<img src=\"https://raw.githubusercontent.com/AgentOps-AI/agentops/main/docs/images/external/app_screenshots/drilldown.gif\"/>"
] ]
}, },
{ {
@ -39,7 +42,7 @@
"## Adding AgentOps to an existing Autogen service.\n", "## Adding AgentOps to an existing Autogen service.\n",
"To get started, you'll need to install the AgentOps package and set an API key.\n", "To get started, you'll need to install the AgentOps package and set an API key.\n",
"\n", "\n",
"AgentOps automatically configures itself when it's initialized. This means your agents will be tracked and logged to your AgentOps account right away." "AgentOps automatically configures itself when it's initialized meaning your agent run data will be tracked and logged to your AgentOps account right away."
] ]
}, },
{ {
@ -69,7 +72,7 @@
"\n", "\n",
"By default, the AgentOps `init()` function will look for an environment variable named `AGENTOPS_API_KEY`. Alternatively, you can pass one in as an optional parameter.\n", "By default, the AgentOps `init()` function will look for an environment variable named `AGENTOPS_API_KEY`. Alternatively, you can pass one in as an optional parameter.\n",
"\n", "\n",
"Create an account and API key at [AgentOps.ai](https://agentops.ai/)" "Create an account and obtain an API key at [AgentOps.ai](https://agentops.ai/settings/projects)"
] ]
}, },
{ {
@ -87,12 +90,14 @@
"name": "stderr", "name": "stderr",
"output_type": "stream", "output_type": "stream",
"text": [ "text": [
"🖇 AgentOps: \u001B[34m\u001B[34mSession Replay: https://app.agentops.ai/drilldown?session_id=8bfaeed1-fd51-4c68-b3ec-276b1a3ce8a4\u001B[0m\u001B[0m\n" "🖇 AgentOps: \u001b[34m\u001b[34mSession Replay: https://app.agentops.ai/drilldown?session_id=8bfaeed1-fd51-4c68-b3ec-276b1a3ce8a4\u001b[0m\u001b[0m\n"
] ]
}, },
{ {
"data": { "data": {
"text/plain": "UUID('8bfaeed1-fd51-4c68-b3ec-276b1a3ce8a4')" "text/plain": [
"UUID('8bfaeed1-fd51-4c68-b3ec-276b1a3ce8a4')"
]
}, },
"execution_count": 1, "execution_count": 1,
"metadata": {}, "metadata": {},
@ -104,7 +109,7 @@
"\n", "\n",
"from autogen import ConversableAgent, UserProxyAgent, config_list_from_json\n", "from autogen import ConversableAgent, UserProxyAgent, config_list_from_json\n",
"\n", "\n",
"agentops.init(api_key=\"7c94212b-b89d-47a6-a20c-23b2077d3226\") # or agentops.init(api_key=\"...\")" "agentops.init(api_key=\"...\")"
] ]
}, },
{ {
@ -144,19 +149,19 @@
"name": "stdout", "name": "stdout",
"output_type": "stream", "output_type": "stream",
"text": [ "text": [
"\u001B[33magent\u001B[0m (to user):\n", "\u001b[33magent\u001b[0m (to user):\n",
"\n", "\n",
"How can I help you today?\n", "How can I help you today?\n",
"\n", "\n",
"--------------------------------------------------------------------------------\n", "--------------------------------------------------------------------------------\n",
"\u001B[33muser\u001B[0m (to agent):\n", "\u001b[33muser\u001b[0m (to agent):\n",
"\n", "\n",
"2+2\n", "2+2\n",
"\n", "\n",
"--------------------------------------------------------------------------------\n", "--------------------------------------------------------------------------------\n",
"\u001B[31m\n", "\u001b[31m\n",
">>>>>>>> USING AUTO REPLY...\u001B[0m\n", ">>>>>>>> USING AUTO REPLY...\u001b[0m\n",
"\u001B[33magent\u001B[0m (to user):\n", "\u001b[33magent\u001b[0m (to user):\n",
"\n", "\n",
"2 + 2 equals 4.\n", "2 + 2 equals 4.\n",
"\n", "\n",
@ -168,7 +173,7 @@
"output_type": "stream", "output_type": "stream",
"text": [ "text": [
"🖇 AgentOps: This run's cost $0.000960\n", "🖇 AgentOps: This run's cost $0.000960\n",
"🖇 AgentOps: \u001B[34m\u001B[34mSession Replay: https://app.agentops.ai/drilldown?session_id=8bfaeed1-fd51-4c68-b3ec-276b1a3ce8a4\u001B[0m\u001B[0m\n" "🖇 AgentOps: \u001b[34m\u001b[34mSession Replay: https://app.agentops.ai/drilldown?session_id=8bfaeed1-fd51-4c68-b3ec-276b1a3ce8a4\u001b[0m\u001b[0m\n"
] ]
} }
], ],
@ -185,7 +190,7 @@
"# Create the agent that represents the user in the conversation.\n", "# Create the agent that represents the user in the conversation.\n",
"user_proxy = UserProxyAgent(\"user\", code_execution_config=False)\n", "user_proxy = UserProxyAgent(\"user\", code_execution_config=False)\n",
"\n", "\n",
"# Let the assistant start the conversation. It will end when the user types exit.\n", "# Let the assistant start the conversation. It will end when the user types \"exit\".\n",
"assistant.initiate_chat(user_proxy, message=\"How can I help you today?\")\n", "assistant.initiate_chat(user_proxy, message=\"How can I help you today?\")\n",
"\n", "\n",
"# Close your AgentOps session to indicate that it completed.\n", "# Close your AgentOps session to indicate that it completed.\n",
@ -204,13 +209,13 @@
}, },
{ {
"cell_type": "markdown", "cell_type": "markdown",
"source": [ "id": "cbd689b0f5617013",
"![Session_Overview](https://github.com/AgentOps-AI/agentops/assets/14807319/d7228019-1488-40d3-852f-a61e998658ad)"
],
"metadata": { "metadata": {
"collapsed": false "collapsed": false
}, },
"id": "cbd689b0f5617013" "source": [
"![session replay](https://github.com/AgentOps-AI/agentops/blob/main/docs/images/external/app_screenshots/session-overview.png?raw=true)"
]
}, },
{ {
"cell_type": "markdown", "cell_type": "markdown",
@ -236,70 +241,70 @@
"name": "stderr", "name": "stderr",
"output_type": "stream", "output_type": "stream",
"text": [ "text": [
"🖇 AgentOps: \u001B[34m\u001B[34mSession Replay: https://app.agentops.ai/drilldown?session_id=880c206b-751e-4c23-9313-8684537fc04d\u001B[0m\u001B[0m\n" "🖇 AgentOps: \u001b[34m\u001b[34mSession Replay: https://app.agentops.ai/drilldown?session_id=880c206b-751e-4c23-9313-8684537fc04d\u001b[0m\u001b[0m\n"
] ]
}, },
{ {
"name": "stdout", "name": "stdout",
"output_type": "stream", "output_type": "stream",
"text": [ "text": [
"\u001B[33mUser\u001B[0m (to Assistant):\n", "\u001b[33mUser\u001b[0m (to Assistant):\n",
"\n", "\n",
"What is (1423 - 123) / 3 + (32 + 23) * 5?\n", "What is (1423 - 123) / 3 + (32 + 23) * 5?\n",
"\n", "\n",
"--------------------------------------------------------------------------------\n", "--------------------------------------------------------------------------------\n",
"\u001B[31m\n", "\u001b[31m\n",
">>>>>>>> USING AUTO REPLY...\u001B[0m\n", ">>>>>>>> USING AUTO REPLY...\u001b[0m\n",
"\u001B[33mAssistant\u001B[0m (to User):\n", "\u001b[33mAssistant\u001b[0m (to User):\n",
"\n", "\n",
"\u001B[32m***** Suggested tool call (call_aINcGyo0Xkrh9g7buRuhyCz0): calculator *****\u001B[0m\n", "\u001b[32m***** Suggested tool call (call_aINcGyo0Xkrh9g7buRuhyCz0): calculator *****\u001b[0m\n",
"Arguments: \n", "Arguments: \n",
"{\n", "{\n",
" \"a\": 1423,\n", " \"a\": 1423,\n",
" \"b\": 123,\n", " \"b\": 123,\n",
" \"operator\": \"-\"\n", " \"operator\": \"-\"\n",
"}\n", "}\n",
"\u001B[32m***************************************************************************\u001B[0m\n", "\u001b[32m***************************************************************************\u001b[0m\n",
"\n", "\n",
"--------------------------------------------------------------------------------\n", "--------------------------------------------------------------------------------\n",
"\u001B[35m\n", "\u001b[35m\n",
">>>>>>>> EXECUTING FUNCTION calculator...\u001B[0m\n", ">>>>>>>> EXECUTING FUNCTION calculator...\u001b[0m\n",
"\u001B[33mUser\u001B[0m (to Assistant):\n", "\u001b[33mUser\u001b[0m (to Assistant):\n",
"\n", "\n",
"\u001B[33mUser\u001B[0m (to Assistant):\n", "\u001b[33mUser\u001b[0m (to Assistant):\n",
"\n", "\n",
"\u001B[32m***** Response from calling tool (call_aINcGyo0Xkrh9g7buRuhyCz0) *****\u001B[0m\n", "\u001b[32m***** Response from calling tool (call_aINcGyo0Xkrh9g7buRuhyCz0) *****\u001b[0m\n",
"1300\n", "1300\n",
"\u001B[32m**********************************************************************\u001B[0m\n", "\u001b[32m**********************************************************************\u001b[0m\n",
"\n", "\n",
"--------------------------------------------------------------------------------\n", "--------------------------------------------------------------------------------\n",
"\u001B[31m\n", "\u001b[31m\n",
">>>>>>>> USING AUTO REPLY...\u001B[0m\n", ">>>>>>>> USING AUTO REPLY...\u001b[0m\n",
"\u001B[33mAssistant\u001B[0m (to User):\n", "\u001b[33mAssistant\u001b[0m (to User):\n",
"\n", "\n",
"\u001B[32m***** Suggested tool call (call_prJGf8V0QVT7cbD91e0Fcxpb): calculator *****\u001B[0m\n", "\u001b[32m***** Suggested tool call (call_prJGf8V0QVT7cbD91e0Fcxpb): calculator *****\u001b[0m\n",
"Arguments: \n", "Arguments: \n",
"{\n", "{\n",
" \"a\": 1300,\n", " \"a\": 1300,\n",
" \"b\": 3,\n", " \"b\": 3,\n",
" \"operator\": \"/\"\n", " \"operator\": \"/\"\n",
"}\n", "}\n",
"\u001B[32m***************************************************************************\u001B[0m\n", "\u001b[32m***************************************************************************\u001b[0m\n",
"\n", "\n",
"--------------------------------------------------------------------------------\n", "--------------------------------------------------------------------------------\n",
"\u001B[35m\n", "\u001b[35m\n",
">>>>>>>> EXECUTING FUNCTION calculator...\u001B[0m\n", ">>>>>>>> EXECUTING FUNCTION calculator...\u001b[0m\n",
"\u001B[33mUser\u001B[0m (to Assistant):\n", "\u001b[33mUser\u001b[0m (to Assistant):\n",
"\n", "\n",
"\u001B[33mUser\u001B[0m (to Assistant):\n", "\u001b[33mUser\u001b[0m (to Assistant):\n",
"\n", "\n",
"\u001B[32m***** Response from calling tool (call_prJGf8V0QVT7cbD91e0Fcxpb) *****\u001B[0m\n", "\u001b[32m***** Response from calling tool (call_prJGf8V0QVT7cbD91e0Fcxpb) *****\u001b[0m\n",
"433\n", "433\n",
"\u001B[32m**********************************************************************\u001B[0m\n", "\u001b[32m**********************************************************************\u001b[0m\n",
"\n", "\n",
"--------------------------------------------------------------------------------\n", "--------------------------------------------------------------------------------\n",
"\u001B[31m\n", "\u001b[31m\n",
">>>>>>>> USING AUTO REPLY...\u001B[0m\n" ">>>>>>>> USING AUTO REPLY...\u001b[0m\n"
] ]
}, },
{ {
@ -316,94 +321,94 @@
"name": "stdout", "name": "stdout",
"output_type": "stream", "output_type": "stream",
"text": [ "text": [
"\u001B[33mAssistant\u001B[0m (to User):\n", "\u001b[33mAssistant\u001b[0m (to User):\n",
"\n", "\n",
"\u001B[32m***** Suggested tool call (call_CUIgHRsySLjayDKuUphI1TGm): calculator *****\u001B[0m\n", "\u001b[32m***** Suggested tool call (call_CUIgHRsySLjayDKuUphI1TGm): calculator *****\u001b[0m\n",
"Arguments: \n", "Arguments: \n",
"{\n", "{\n",
" \"a\": 32,\n", " \"a\": 32,\n",
" \"b\": 23,\n", " \"b\": 23,\n",
" \"operator\": \"+\"\n", " \"operator\": \"+\"\n",
"}\n", "}\n",
"\u001B[32m***************************************************************************\u001B[0m\n", "\u001b[32m***************************************************************************\u001b[0m\n",
"\n", "\n",
"--------------------------------------------------------------------------------\n", "--------------------------------------------------------------------------------\n",
"\u001B[35m\n", "\u001b[35m\n",
">>>>>>>> EXECUTING FUNCTION calculator...\u001B[0m\n", ">>>>>>>> EXECUTING FUNCTION calculator...\u001b[0m\n",
"\u001B[33mUser\u001B[0m (to Assistant):\n", "\u001b[33mUser\u001b[0m (to Assistant):\n",
"\n", "\n",
"\u001B[33mUser\u001B[0m (to Assistant):\n", "\u001b[33mUser\u001b[0m (to Assistant):\n",
"\n", "\n",
"\u001B[32m***** Response from calling tool (call_CUIgHRsySLjayDKuUphI1TGm) *****\u001B[0m\n", "\u001b[32m***** Response from calling tool (call_CUIgHRsySLjayDKuUphI1TGm) *****\u001b[0m\n",
"55\n", "55\n",
"\u001B[32m**********************************************************************\u001B[0m\n", "\u001b[32m**********************************************************************\u001b[0m\n",
"\n", "\n",
"--------------------------------------------------------------------------------\n", "--------------------------------------------------------------------------------\n",
"\u001B[31m\n", "\u001b[31m\n",
">>>>>>>> USING AUTO REPLY...\u001B[0m\n", ">>>>>>>> USING AUTO REPLY...\u001b[0m\n",
"\u001B[33mAssistant\u001B[0m (to User):\n", "\u001b[33mAssistant\u001b[0m (to User):\n",
"\n", "\n",
"\u001B[32m***** Suggested tool call (call_L7pGtBLUf9V0MPL90BASyesr): calculator *****\u001B[0m\n", "\u001b[32m***** Suggested tool call (call_L7pGtBLUf9V0MPL90BASyesr): calculator *****\u001b[0m\n",
"Arguments: \n", "Arguments: \n",
"{\n", "{\n",
" \"a\": 55,\n", " \"a\": 55,\n",
" \"b\": 5,\n", " \"b\": 5,\n",
" \"operator\": \"*\"\n", " \"operator\": \"*\"\n",
"}\n", "}\n",
"\u001B[32m***************************************************************************\u001B[0m\n", "\u001b[32m***************************************************************************\u001b[0m\n",
"\n", "\n",
"--------------------------------------------------------------------------------\n", "--------------------------------------------------------------------------------\n",
"\u001B[35m\n", "\u001b[35m\n",
">>>>>>>> EXECUTING FUNCTION calculator...\u001B[0m\n", ">>>>>>>> EXECUTING FUNCTION calculator...\u001b[0m\n",
"\u001B[33mUser\u001B[0m (to Assistant):\n", "\u001b[33mUser\u001b[0m (to Assistant):\n",
"\n", "\n",
"\u001B[33mUser\u001B[0m (to Assistant):\n", "\u001b[33mUser\u001b[0m (to Assistant):\n",
"\n", "\n",
"\u001B[32m***** Response from calling tool (call_L7pGtBLUf9V0MPL90BASyesr) *****\u001B[0m\n", "\u001b[32m***** Response from calling tool (call_L7pGtBLUf9V0MPL90BASyesr) *****\u001b[0m\n",
"275\n", "275\n",
"\u001B[32m**********************************************************************\u001B[0m\n", "\u001b[32m**********************************************************************\u001b[0m\n",
"\n", "\n",
"--------------------------------------------------------------------------------\n", "--------------------------------------------------------------------------------\n",
"\u001B[31m\n", "\u001b[31m\n",
">>>>>>>> USING AUTO REPLY...\u001B[0m\n", ">>>>>>>> USING AUTO REPLY...\u001b[0m\n",
"\u001B[33mAssistant\u001B[0m (to User):\n", "\u001b[33mAssistant\u001b[0m (to User):\n",
"\n", "\n",
"\u001B[32m***** Suggested tool call (call_Ygo6p4XfcxRjkYBflhG3UVv6): calculator *****\u001B[0m\n", "\u001b[32m***** Suggested tool call (call_Ygo6p4XfcxRjkYBflhG3UVv6): calculator *****\u001b[0m\n",
"Arguments: \n", "Arguments: \n",
"{\n", "{\n",
" \"a\": 433,\n", " \"a\": 433,\n",
" \"b\": 275,\n", " \"b\": 275,\n",
" \"operator\": \"+\"\n", " \"operator\": \"+\"\n",
"}\n", "}\n",
"\u001B[32m***************************************************************************\u001B[0m\n", "\u001b[32m***************************************************************************\u001b[0m\n",
"\n", "\n",
"--------------------------------------------------------------------------------\n", "--------------------------------------------------------------------------------\n",
"\u001B[35m\n", "\u001b[35m\n",
">>>>>>>> EXECUTING FUNCTION calculator...\u001B[0m\n", ">>>>>>>> EXECUTING FUNCTION calculator...\u001b[0m\n",
"\u001B[33mUser\u001B[0m (to Assistant):\n", "\u001b[33mUser\u001b[0m (to Assistant):\n",
"\n", "\n",
"\u001B[33mUser\u001B[0m (to Assistant):\n", "\u001b[33mUser\u001b[0m (to Assistant):\n",
"\n", "\n",
"\u001B[32m***** Response from calling tool (call_Ygo6p4XfcxRjkYBflhG3UVv6) *****\u001B[0m\n", "\u001b[32m***** Response from calling tool (call_Ygo6p4XfcxRjkYBflhG3UVv6) *****\u001b[0m\n",
"708\n", "708\n",
"\u001B[32m**********************************************************************\u001B[0m\n", "\u001b[32m**********************************************************************\u001b[0m\n",
"\n", "\n",
"--------------------------------------------------------------------------------\n", "--------------------------------------------------------------------------------\n",
"\u001B[31m\n", "\u001b[31m\n",
">>>>>>>> USING AUTO REPLY...\u001B[0m\n", ">>>>>>>> USING AUTO REPLY...\u001b[0m\n",
"\u001B[33mAssistant\u001B[0m (to User):\n", "\u001b[33mAssistant\u001b[0m (to User):\n",
"\n", "\n",
"The result of the calculation is 708.\n", "The result of the calculation is 708.\n",
"\n", "\n",
"--------------------------------------------------------------------------------\n", "--------------------------------------------------------------------------------\n",
"\u001B[33mUser\u001B[0m (to Assistant):\n", "\u001b[33mUser\u001b[0m (to Assistant):\n",
"\n", "\n",
"\n", "\n",
"\n", "\n",
"--------------------------------------------------------------------------------\n", "--------------------------------------------------------------------------------\n",
"\u001B[31m\n", "\u001b[31m\n",
">>>>>>>> USING AUTO REPLY...\u001B[0m\n", ">>>>>>>> USING AUTO REPLY...\u001b[0m\n",
"\u001B[33mAssistant\u001B[0m (to User):\n", "\u001b[33mAssistant\u001b[0m (to User):\n",
"\n", "\n",
"TERMINATE\n", "TERMINATE\n",
"\n", "\n",
@ -415,7 +420,7 @@
"output_type": "stream", "output_type": "stream",
"text": [ "text": [
"🖇 AgentOps: This run's cost $0.001800\n", "🖇 AgentOps: This run's cost $0.001800\n",
"🖇 AgentOps: \u001B[34m\u001B[34mSession Replay: https://app.agentops.ai/drilldown?session_id=880c206b-751e-4c23-9313-8684537fc04d\u001B[0m\u001B[0m\n" "🖇 AgentOps: \u001b[34m\u001b[34mSession Replay: https://app.agentops.ai/drilldown?session_id=880c206b-751e-4c23-9313-8684537fc04d\u001b[0m\u001b[0m\n"
] ]
} }
], ],
@ -474,7 +479,7 @@
" description=\"A simple calculator\", # A description of the tool.\n", " description=\"A simple calculator\", # A description of the tool.\n",
")\n", ")\n",
"\n", "\n",
"# Let the assistant start the conversation. It will end when the user types exit.\n", "# Let the assistant start the conversation. It will end when the user types \"exit\".\n",
"user_proxy.initiate_chat(assistant, message=\"What is (1423 - 123) / 3 + (32 + 23) * 5?\")\n", "user_proxy.initiate_chat(assistant, message=\"What is (1423 - 123) / 3 + (32 + 23) * 5?\")\n",
"\n", "\n",
"agentops.end_session(\"Success\")" "agentops.end_session(\"Success\")"
@ -493,13 +498,13 @@
}, },
{ {
"cell_type": "markdown", "cell_type": "markdown",
"source": [ "id": "a922a52ab5fce31",
"![Session Drilldown](https://github.com/AgentOps-AI/agentops/assets/14807319/561d59f3-c441-4066-914b-f6cfe32a598c)"
],
"metadata": { "metadata": {
"collapsed": false "collapsed": false
}, },
"id": "a922a52ab5fce31" "source": [
"![Session Drilldown](https://github.com/AgentOps-AI/agentops/blob/main/docs/images/external/app_screenshots/session-replay.png?raw=true)"
]
} }
], ],
"metadata": { "metadata": {

28
website/docs/ecosystem/agentops.md
View File

@ -1,29 +1,37 @@
# AgentOps 🖇️ # Agent Monitoring and Debugging with AgentOps
![logo](https://raw.githubusercontent.com/AgentOps-AI/agentops/35d5682866921a9e28d8ef66ae3c3b3d92d8fa6b/img/logo.png) <img src="https://github.com/AgentOps-AI/agentops/blob/main/docs/images/external/logo/banner-badge.png?raw=true" style="width: 40%;" alt="AgentOps logo"/>
[AgentOps](https://agentops.ai/?=autogen) provides session replays, metrics, and monitoring for agents. [AgentOps](https://agentops.ai/?=autogen) provides session replays, metrics, and monitoring for AI agents.
At a high level, AgentOps gives you the ability to monitor LLM calls, costs, latency, agent failures, multi-agent interactions, tool usage, session-wide statistics, and more. For more info, check out the [AgentOps Repo](https://github.com/AgentOps-AI/agentops). At a high level, AgentOps gives you the ability to monitor LLM calls, costs, latency, agent failures, multi-agent interactions, tool usage, session-wide statistics, and more. For more info, check out the [AgentOps Repo](https://github.com/AgentOps-AI/agentops).
| | |
| ------------------------------------- | ------------------------------------------------------------- |
| 📊 **Replay Analytics and Debugging** | Step-by-step agent execution graphs |
| 💸 **LLM Cost Management** | Track spend with LLM foundation model providers |
| 🧪 **Agent Benchmarking** | Test your agents against 1,000+ evals |
| 🔐 **Compliance and Security** | Detect common prompt injection and data exfiltration exploits |
| 🤝 **Framework Integrations** | Native Integrations with CrewAI, AutoGen, & LangChain |
<details open> <details open>
<summary>Agent Dashboard</summary> <summary><b><u>Agent Dashboard</u></b></summary>
<a href="https://app.agentops.ai?ref=gh"> <a href="https://app.agentops.ai?ref=gh">
<img src="https://github.com/AgentOps-AI/agentops/assets/14807319/158e082a-9a7d-49b7-9b41-51a49a1f7d3d" style="width: 90%;" alt="Agent Dashboard"/> <img src="https://github.com/AgentOps-AI/agentops/blob/main/docs/images/external/app_screenshots/overview.png?raw=true" style="width: 70%;" alt="Agent Dashboard"/>
</a> </a>
</details> </details>
<details> <details>
<summary>Session Analytics</summary> <summary><b><u>Session Analytics</u></b></summary>
<a href="https://app.agentops.ai?ref=gh"> <a href="https://app.agentops.ai?ref=gh">
<img src="https://github.com/AgentOps-AI/agentops/assets/14807319/d7228019-1488-40d3-852f-a61e998658ad" style="width: 90%;" alt="Session Analytics"/> <img src="https://github.com/AgentOps-AI/agentops/blob/main/docs/images/external/app_screenshots/session-overview.png?raw=true" style="width: 70%;" alt="Session Analytics"/>
</a> </a>
</details> </details>
<details> <details>
<summary>Session Replays</summary> <summary><b><u>Session Replays</u></b></summary>
<a href="https://app.agentops.ai?ref=gh"> <a href="https://app.agentops.ai?ref=gh">
<img src="https://github.com/AgentOps-AI/agentops/assets/14807319/561d59f3-c441-4066-914b-f6cfe32a598c" style="width: 90%;" alt="Session Replays"/> <img src="https://github.com/AgentOps-AI/agentops/blob/main/docs/images/external/app_screenshots/session-replay.png?raw=true" style="width: 70%;" alt="Session Replays"/>
</a> </a>
</details> </details>
@ -38,7 +46,7 @@ pip install agentops
``` ```
2. **Create an API Key:** 2. **Create an API Key:**
Create a user API key here: [Create API Key](https://app.agentops.ai/account) Create a user API key here: [Create API Key](https://app.agentops.ai/settings/projects)
3. **Configure Your Environment:** 3. **Configure Your Environment:**
Add your API key to your environment variables Add your API key to your environment variables

59
website/docs/topics/llm-observability.md
View File

@ -1,42 +1,37 @@
# LLM Observability # Agent Observability
AutoGen supports advanced LLM observability and monitoring through built-in logging and partner providers. AutoGen supports advanced LLM agent observability and monitoring through built-in logging and partner providers.
## What is LLM Observability ## AutoGen Observability Integrations
AI agent observability is the ability to monitor, measure, and understand the internal states and behaviors of AI agent systems.
Observability is crucial for ensuring transparency, reliability, and accountability in your agent systems. ### Built-In Logging
AutoGen's SQLite and File Logger - [Tutorial Notebook](/docs/notebooks/agentchat_logging)
### Full-Service Partner Integrations
AutoGen partners with [AgentOps](https://agentops.ai) to provide multi-agent tracking, metrics, and monitoring - [Tutorial Notebook](/docs/notebooks/agentchat_agentops)
## Development ## What is Observability?
Observability provides developers with the necessary insights to understand and improve the internal workings of their agents. Observability is necessary for maintaining reliability, tracking costs, and ensuring AI safety.
### Agent Development in Terminal is Limited **Without observability tools, developers face significant hurdles:**
- Lose track of what your agents did in between executions
- Parsing through terminal output searching for LLM completions
- Printing “tool called”
### Agent Development Dashboards Enable More - Tracking agent activities across sessions becomes a complex, error-prone task.
- Visual dashboard so you can see what your agents did in human-readable format - Manually sifting through verbose terminal outputs to understand LLM interactions is inefficient.
- LLM calls are magically recorded - prompt, completion, timestamps for each - with one line of code - Pinpointing the exact moments of tool invocations is often like finding a needle in a haystack.
- Agents and their events (including tool calls) are recorded with one more line of code
- Errors are magically associated to its causal event
- Record any other events to your session with two more lines of code
- Tons of other useful data if you’re developing with supported agent frameworks: SDK version
## Compliance
Observability and monitoring is critical to ensure AI agent systems adhere to laws and regulations in industries like finance and healthcare, preventing violations such as data breaches and privacy issues. **Key Features of Observability Dashboards:**
- Human-readable overview analytics and replays of agent activities.
- LLM cost, prompt, completion, timestamp, and metadata tracking for performance monitoring.
- Tool invocation, events, and agent-to-agent interactions for workflow monitoring.
- Error flagging and notifications for faster debugging.
- Access to a wealth of data for developers using supported agent frameworks, such as environments, SDK versions, and more.
- Insights into AI decision-making, allowing organizations to explain outcomes and build trust with stakeholders. ### Compliance
- Helps detect anomalies and unintended behaviors early, mitigating operational, financial, and reputational risks.
- Ensures compliance with data privacy regulations, preventing unauthorized access and misuse of sensitive information.
- Quick identification and response to compliance violations, supporting incident analysis and prevention.
## Available Observability Integrations Observability is not just a development convenience—it's a compliance necessity, especially in regulated industries:
- It offers insights into AI decision-making processes, fostering trust and transparency.
### Logging - Anomalies and unintended behaviors are detected promptly, reducing various risks.
- Autogen SQLite and File Logger - [Tutorial](/docs/notebooks/agentchat_logging) - Ensuring adherence to data privacy regulations, thereby safeguarding sensitive information.
- Compliance violations are quickly identified and addressed, enhancing incident management.
### Full-Service Partners
Autogen is currently partnered with [AgentOps](https://agentops.ai) for seamless observability integration.
[Learn how to install AgentOps](/docs/notebooks/agentchat_agentops)