Add OTel GenAI traces:
- `create_agent`
- `invoke_agnet`
- `execute_tool`
Introduces context manager helpers to create these traces. The helpers
also serve as instrumentation points for other instrumentation
libraries.
Resolves#6644
## Why are these changes needed?
This PR introduces a new `OpenAIAgent` implementation that uses the
[OpenAI Response
API](https://platform.openai.com/docs/guides/responses-vs-chat-completions)
as its backend. The OpenAI Assistant API will be deprecated in 2026, and
the Response API is its successor. This change ensures our codebase is
future-proof and aligned with OpenAI’s latest platform direction.
### Motivation
- **Deprecation Notice:** The OpenAI Assistant API will be deprecated in
2026.
- **Future-Proofing:** The Response API is the recommended replacement
and offers improved capabilities for stateful, multi-turn, and
tool-augmented conversations.
- **AgentChat Compatibility:** The new agent is designed to conform to
the behavior and expectations of `AssistantAgent` in AgentChat, but is
implemented directly on top of the OpenAI Response API.
### Key Changes
- **New Agent:** Adds `OpenAIAgent`, a stateful agent that interacts
with the OpenAI Response API.
- **Stateful Design:** The agent maintains conversation state, tool
usage, and other metadata as required by the Response API.
- **AssistantAgent Parity:** The new agent matches the interface and
behavior of `AssistantAgent` in AgentChat, ensuring a smooth migration
path.
- **Direct OpenAI Integration:** Uses the official `openai` Python
library for all API interactions.
- **Extensible:** Designed to support future enhancements, such as
advanced tool use, function calling, and multi-modal capabilities.
### Migration Path
- Existing users of the Assistant API should migrate to the new
`OpenAIAgent` to ensure long-term compatibility.
- Documentation and examples will be updated to reflect the new agent
and its usage patterns.
### References
- [OpenAI: Responses vs. Chat
Completions](https://platform.openai.com/docs/guides/responses-vs-chat-completions)
- [OpenAI Deprecation
Notice](https://platform.openai.com/docs/guides/responses-vs-chat-completions#deprecation-timeline)
---
## Related issue number
Closes#6032
## Checks
- [ ] I've included any doc changes needed for
<https://microsoft.github.io/autogen/>. See
<https://github.com/microsoft/autogen/blob/main/CONTRIBUTING.md> to
build and test documentation locally.
- [X] I've added tests (if relevant) corresponding to the changes
introduced in this PR.
- [X] I've made sure all auto checks have passed.
Co-authored-by: Griffin Bassman <griffinbassman@gmail.com>
<!-- Thank you for your contribution! Please review
https://microsoft.github.io/autogen/docs/Contribute before opening a
pull request. -->
<!-- Please add a reviewer to the assignee section when you create a PR.
If you don't have the access to it, we will shortly find a reviewer and
assign them to your PR. -->
There have been updates to the azure ai agent foundry sdk
(azure-ai-project). This PR updates the autogen `AzureAIAgent` which
wraps the azure ai agent. A list of some changes
- Update docstring samples to use `endpoint` (instead of connection
string previously)
- Update imports and arguments e.g, from `azure.ai.agents` etc
- Add a guide in ext docs showing Bing Search Grounding tool example.
<img width="1423" alt="image"
src="https://github.com/user-attachments/assets/0b7c8fa6-8aa5-4c20-831b-b525ac8243b7"
/>
## Why are these changes needed?
<!-- Please give a short summary of the change and the problem this
solves. -->
## Related issue number
Closes#6601
<!-- For example: "Closes #1234" -->
## Checks
- [ ] I've included any doc changes needed for
<https://microsoft.github.io/autogen/>. See
<https://github.com/microsoft/autogen/blob/main/CONTRIBUTING.md> to
build and test documentation locally.
- [ ] I've added tests (if relevant) corresponding to the changes
introduced in this PR.
- [ ] I've made sure all auto checks have passed.
Adding support for Bing grounding citations to the AzureAIAgent.
<!-- Please add a reviewer to the assignee section when you create a PR.
If you don't have the access to it, we will shortly find a reviewer and
assign them to your PR. -->
## Why are these changes needed?
<!-- Please give a short summary of the change and the problem this
solves. -->
## Related issue number
<!-- For example: "Closes #1234" -->
## Checks
- [X] I've included any doc changes needed for
<https://microsoft.github.io/autogen/>. See
<https://github.com/microsoft/autogen/blob/main/CONTRIBUTING.md> to
build and test documentation locally.
- [X] I've added tests (if relevant) corresponding to the changes
introduced in this PR.
- [X] I've made sure all auto checks have passed.
---------
Co-authored-by: Eric Zhu <ekzhu@users.noreply.github.com>
Co-authored-by: Dheeraj Bandaru <BandaruDheeraj@users.noreply.github.com>
- Added the support Azure AI Agent. The new agent is named AzureAIAgent.
- The agent supports Bing search, file search, and Azure search tools.
- Added a Jupiter notebook to demonstrate the usage of the AzureAIAgent.
## What's missing?
- AzureAIAgent support only text message responses
- Parallel execution for the custom functions.
## Related issue number
[5545](https://github.com/microsoft/autogen/issues/5545#event-16626859772)
---------
Co-authored-by: Eric Zhu <ekzhu@users.noreply.github.com>
Rename the `ChatMessage` and `AgentEvent` base classes to `BaseChatMessage` and `BaseAgentEvent`.
Bring back the `ChatMessage` and `AgentEvent` as union of built-in concrete types to avoid breaking existing applications that depends on Pydantic serialization.
Why?
Many existing code uses containers like this:
```python
class AppMessage(BaseModel):
name: str
message: ChatMessage
# Serialization is this:
m = AppMessage(...)
m.model_dump_json()
# Fields like HandoffMessage.target will be lost because it is now treated as a base class without content or target fields.
```
The assumption on `ChatMessage` or `AgentEvent` to be a union of concrete types could be in many existing code bases. So this PR brings back the union types, while keep method type hints such as those on `on_messages` to use the `BaseChatMessage` and `BaseAgentEvent` base classes for flexibility.
This PR refactored `AgentEvent` and `ChatMessage` union types to
abstract base classes. This allows for user-defined message types that
subclass one of the base classes to be used in AgentChat.
To support a unified interface for working with the messages, the base
classes added abstract methods for:
- Convert content to string
- Convert content to a `UserMessage` for model client
- Convert content for rendering in console.
- Dump into a dictionary
- Load and create a new instance from a dictionary
This way, all agents such as `AssistantAgent` and `SocietyOfMindAgent`
can utilize the unified interface to work with any built-in and
user-defined message type.
This PR also introduces a new message type, `StructuredMessage` for
AgentChat (Resolves#5131), which is a generic type that requires a
user-specified content type.
You can create a `StructuredMessage` as follow:
```python
class MessageType(BaseModel):
data: str
references: List[str]
message = StructuredMessage[MessageType](content=MessageType(data="data", references=["a", "b"]), source="user")
# message.content is of type `MessageType`.
```
This PR addresses the receving side of this message type. To produce
this message type from `AssistantAgent`, the work continue in #5934.
Added unit tests to verify this message type works with agents and
teams.
<!-- Thank you for your contribution! Please review
https://microsoft.github.io/autogen/docs/Contribute before opening a
pull request. -->
<!-- Please add a reviewer to the assignee section when you create a PR.
If you don't have the access to it, we will shortly find a reviewer and
assign them to your PR. -->
## Why are these changes needed?
Add utf encoding to file reading.
Without this, a default system encoding will be used. On Windows
machines this can default to any local encoding causing errors.
```python
with open(
os.path.join(os.path.abspath(os.path.dirname(__file__)), "page_script.js"), "rt", encoding="utf-8"
) as fh:
```
<!-- Please give a short summary of the change and the problem this
solves. -->
## Related issue number
<!-- For example: "Closes #1234" -->
Closes#6093
## Checks
- [ ] I've included any doc changes needed for
<https://microsoft.github.io/autogen/>. See
<https://github.com/microsoft/autogen/blob/main/CONTRIBUTING.md> to
build and test documentation locally.
- [ ] I've added tests (if relevant) corresponding to the changes
introduced in this PR.
- [ ] I've made sure all auto checks have passed.
Optionally limit what files and folders FileSurfer can access
(constraining it to a subtree of the FS).
This is not a replacement for Docker sandboxing, but can be used in
conjunction with sandboxing to help prevent FileSurfer from accessing
sensitive files.
Fixes issues like the following trace:
```
packages/autogen_ext/agents/file_surfer/_markdown_file_browser.py", line 39, in __init__
self.set_path(self._base_path)
File "/home/hmozannar/webby/.venv/lib/python3.12/site-packages/autogen_ext/agents/file_surfer/_markdown_file_browser.py", line 67, in set_path
self._open_path(path)
File "/home/hmozannar/webby/.venv/lib/python3.12/site-packages/autogen_ext/agents/file_surfer/_markdown_file_browser.py", line 210, in _open_path
io.StringIO(self._fetch_local_dir(path)), file_extension=".txt"
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/hmozannar/webby/.venv/lib/python3.12/site-packages/autogen_ext/agents/file_surfer/_markdown_file_browser.py", line 248, in _fetch_local_dir
mtime = datetime.datetime.fromtimestamp(os.path.getmtime(full_path)).strftime("%Y-%m-%d %H:%M")
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "<frozen genericpath>", line 67, in getmtime
PermissionError: [Errno 13] Permission denied: '/home/hmozannar/webby/autogen-studio/frontend/readme.txt'
```
This pull request introduces a new feature to the `FileSurfer` agent and
`MarkdownFileBrowser` by adding support for specifying a base path for
file browsing.
*
`python/packages/autogen-ext/src/autogen_ext/agents/file_surfer/_file_surfer.py`:
* Added `base_path` parameter to `FileSurfer` class and its
initialization method, with a default value of the current working
directory (`os.getcwd()`).
[[1]](diffhunk://#diff-084847b5e64c659c9aff0bd2d05bbcd0fff2c819a4b91bbe65fa0566054c0972R58)
[[2]](diffhunk://#diff-084847b5e64c659c9aff0bd2d05bbcd0fff2c819a4b91bbe65fa0566054c0972R80-R85)
* Updated `MarkdownFileBrowser` initialization within `FileSurfer` to
use the `base_path` parameter.
*
`python/packages/autogen-ext/src/autogen_ext/agents/file_surfer/_markdown_file_browser.py`:
* Added `base_path` parameter to `MarkdownFileBrowser` class and its
initialization method, with a default value of the current working
directory (`os.getcwd()`).
* Updated `MarkdownFileBrowser` to use the `base_path` for setting the
initial path and returning the current page path.
Fixes#4821 by adding a `close()` method to all clients.
Additionally:
* The m1 CLI is updated to close the client before exiting.
* The playwrightcontroller is updated to suppress some other unrelated
chatty warnings (e.g,, produced by markitdown when encountering
conversions that require external utilities)
<!-- Thank you for your contribution! Please review
https://microsoft.github.io/autogen/docs/Contribute before opening a
pull request. -->
<!-- Please add a reviewer to the assignee section when you create a PR.
If you don't have the access to it, we will shortly find a reviewer and
assign them to your PR. -->
## Why are these changes needed?
<!-- Please give a short summary of the change and the problem this
solves. -->
The PR introduces two changes.
The first change is adding a name attribute to
`FunctionExecutionResult`. The motivation is that semantic kernel
requires it for their function result interface and it seemed like a
easy modification as `FunctionExecutionResult` is always created in the
context of a `FunctionCall` which will contain the name. I'm unsure if
there was a motivation to keep it out but this change makes it easier to
trace which tool the result refers to and also increases api
compatibility with SK.
The second change is an update to how messages are mapped from autogen
to semantic kernel, which includes an update/fix in the processing of
function results.
## Related issue number
<!-- For example: "Closes #1234" -->
Related to #5675 but wont fix the underlying issue of anthropic
requiring tools during AssistantAgent reflection.
## Checks
- [ ] I've included any doc changes needed for
<https://microsoft.github.io/autogen/>. See
<https://github.com/microsoft/autogen/blob/main/CONTRIBUTING.md> to
build and test documentation locally.
- [ ] I've added tests (if relevant) corresponding to the changes
introduced in this PR.
- [ ] I've made sure all auto checks have passed.
---------
Co-authored-by: Leonardo Pinheiro <lpinheiro@microsoft.com>
<!-- Thank you for your contribution! Please review
https://microsoft.github.io/autogen/docs/Contribute before opening a
pull request. -->
<!-- Please add a reviewer to the assignee section when you create a PR.
If you don't have the access to it, we will shortly find a reviewer and
assign them to your PR. -->
## Why are these changes needed?
Make FileSurfer and CodeExecAgent Declarative.
These agent presents are used as part of magentic one and having them
declarative is a precursor to their use in AGS.
<!-- Please give a short summary of the change and the problem this
solves. -->
## Related issue number
<!-- For example: "Closes #1234" -->
Closes#5607
## Checks
- [ ] I've included any doc changes needed for
<https://microsoft.github.io/autogen/>. See
<https://github.com/microsoft/autogen/blob/main/CONTRIBUTING.md> to
build and test documentation locally.
- [ ] I've added tests (if relevant) corresponding to the changes
introduced in this PR.
- [ ] I've made sure all auto checks have passed.
A series of changes to the
`python/packages/autogen-ext/src/autogen_ext/agents/web_surfer/_multimodal_web_surfer.py`
file have been made to better support smaller models.
This includes changes to the prompts, state descriptions, and ordering
of messages.
Regression tasks with OpenAI models shows no change in GAIA scores,
while scores for Llama are significantly improved.
This PR fixes:
A prompting bug when no control had focus.
Awkward prompt phrasing.
Renamed page_down to scroll_down to better match other prompting and
agent descriptions.
Some agent descriptions were split over multiple lines in the M1
orchestrator. This PR ensures that each description appears on one, and
only one, line. This makes it easier for smaller models to understand.
<!-- Thank you for your contribution! Please review
https://microsoft.github.io/autogen/docs/Contribute before opening a
pull request. -->
<!-- Please add a reviewer to the assignee section when you create a PR.
If you don't have the access to it, we will shortly find a reviewer and
assign them to your PR. -->
## Why are these changes needed?
<!-- Please give a short summary of the change and the problem this
solves. -->
To allow serialization of OAI Assistant Agent.
## Related issue number
<!-- For example: "Closes #1234" -->
Closes#5130
## Checks
- [ ] I've included any doc changes needed for
https://microsoft.github.io/autogen/. See
https://microsoft.github.io/autogen/docs/Contribute#documentation to
build and test documentation locally.
- [ ] I've added tests (if relevant) corresponding to the changes
introduced in this PR.
- [ ] I've made sure all auto checks have passed.
This PR adds a method that approximately extracts the text visible in
the viewport of the web browser (as opposed to always printing the first
50 lines, or relying entirely on OCR).
Removed the TERMINATE clause from the system prompt since M1 handles
termination via the Orchestrator, and it is just ignored.
Removed the clause about saving to a particular file name, since tmp_
files are created in the current CodeExecutors.
<!-- Thank you for your contribution! Please review
https://microsoft.github.io/autogen/docs/Contribute before opening a
pull request. -->
<!-- Please add a reviewer to the assignee section when you create a PR.
If you don't have the access to it, we will shortly find a reviewer and
assign them to your PR. -->
## Why are these changes needed?
Make AssistantAgent and Handoff use BaseTool.
This ensures that they can be made declarative/serialized
<!-- Please give a short summary of the change and the problem this
solves. -->
## Related issue number
<!-- For example: "Closes #1234" -->
## Checks
- [ ] I've included any doc changes needed for
https://microsoft.github.io/autogen/. See
https://microsoft.github.io/autogen/docs/Contribute#documentation to
build and test documentation locally.
- [ ] I've added tests (if relevant) corresponding to the changes
introduced in this PR.
- [ ] I've made sure all auto checks have passed.
* v1, make assistant agent declarative
* make head tail context declarative
* update and formatting
* update assistant, format updates
* make websurfer declarative
* update formatting
* move declarative docs to advanced section
* remove tools until implemented
* minor updates to termination conditions
* update docs
* Minor fix for incorrect description of teams
* Add installation instructions
---------
Co-authored-by: Jack Gerrits <jackgerrits@users.noreply.github.com>
* Add MagenticOne API
* Add CodeExecutorAgent to MagenticOne for enhanced task execution
* Refactor MagenticOne class to inherit from MagenticOneGroupChat and streamline initialization
* Enhance MagenticOne class documentation with detailed usage examples and initialization instructions
* Refactor MagenticOne module structure and update import paths
* Remove unused imports
* Add documentation for MagenticOne module and remove redundant initialization comments
* Enhance MagenticOne class with human-in-the-loop mode and update examples
* Update MagenticOne class documentation with safety precautions and architecture details
* Run poe format
* Add blog post reference to MagenticOne class documentation
* change default of websurfer use_ocr to false because of refusals
* Refactor MagenticOne class to use ChatCompletionClient instead of OpenAIChatCompletionClient
* Add client capability validation to MagenticOne initialization
* Poe format
* Refactor imports in MagenticOne class for clarity and organization
* Add stacklevel parameter to warning in client capability validation
* Update README to recommend using Magentic-One API for improved integration
* Add create_args property to OpenAIChatCompletionClient for better access to initialization arguments
* Enhance client capability validation in MagenticOne to ensure compatibility with OpenAI GPT-4o model
* Refactor client capability validation in MagenticOne for improved clarity
* Update magentic_one.py
Co-authored-by: Eric Zhu <ekzhu@users.noreply.github.com>
* Remove create_args property from OpenAIChatCompletionClient and update validation logic in MagenticOne to directly access _create_args
* Refactor documentation in MagenticOne for improved readability and consistency
* Refactor client capability validation in MagenticOne to remove unnecessary model check for GPT-4o
* Add MagenticOne CLI (#4788)
* Add MagenticOne CLI script for task execution with OpenAI GPT-4o integration
* Fix argument parsing in MagenticOne CLI to require a single task input
* Add docstring to main function in MagenticOne CLI for improved usage clarity
* Fix example usage in docstring of MagenticOne CLI for correct argument order
* Refactor argument parsing in MagenticOne CLI for improved clarity and consistency
* Add type hints to run_task function in MagenticOne CLI
* Add type hint for main function in MagenticOne CLI
* Remove type ignore from main function call in MagenticOne CLI
---------
Co-authored-by: Hussein Mozannar <hmozannar@microsoft.com>
Co-authored-by: Eric Zhu <ekzhu@users.noreply.github.com>
