Add the following additional configuration options to
DockerCommandLineCodeExectutor:
- **extra_volumes** (Optional[Dict[str, Dict[str, str]]], optional): A
dictionary of extra volumes (beyond the work_dir) to mount to the
container. Defaults to None.
- **extra_hosts** (Optional[Dict[str, str]], optional): A dictionary of
host mappings to add to the container. (See Docker docs on extra_hosts)
Defaults to None.
- **init_command** (Optional[str], optional): A shell command to run
before each shell operation execution. Defaults to None.
## Why are these changes needed?
See linked issue below.
In summary: Enable the agents to:
- work with a richer set of sys admin tools on top of code execution
- add support for a 'project' directory the agents can interact on
that's accessible by bash tools and custom scripts
## Related issue number
Closes#5363
## Checks
- [x] I've included any doc changes needed for
https://microsoft.github.io/autogen/. See
https://microsoft.github.io/autogen/docs/Contribute#documentation to
build and test documentation locally.
- [x] I've added tests (if relevant) corresponding to the changes
introduced in this PR.
- [x] I've made sure all auto checks have passed.
This PR improves documentation on custom agents
- Shows example on how to create a custom agent that directly uses a
model client. In this case an example of a GeminiAssistantAgent that
directly uses the Gemini SDK model client.
- Shows that that CustomAgent can be easily added to any agentchat team
- Shows how the same CustomAgent can be made declarative by inheriting
the Component interface and implementing the required methods.
Closes#5450
## Why are these changes needed?
These changes are needed because currently there's no generic way to add
`tools` to autogen studio workflows using the existing DSL and schema
other than inline python.
This API will be quite verbose, and lacks a discovery mechanism, but it
unlocks a lot of programmatic use-cases.
## Related issue number
https://github.com/microsoft/autogen/issues/5170
Co-authored-by: Victor Dibia <victordibia@microsoft.com>
Co-authored-by: Eric Zhu <ekzhu@users.noreply.github.com>
<!-- Thank you for your contribution! Please review
https://microsoft.github.io/autogen/docs/Contribute before opening a
pull request. -->
<!-- Please add a reviewer to the assignee section when you create a PR.
If you don't have the access to it, we will shortly find a reviewer and
assign them to your PR. -->
## Why are these changes needed?
<!-- Please give a short summary of the change and the problem this
solves. -->
The current implementation tries to recreate the metadata but it does it
in an incomplete way. This PR uses SK built-in kernel from function
decorator to infer the callable from the `run_json` and makes better use
of the pydantic schemas for the input and output to infer the schema of
the kernel function.
## Related issue number
<!-- For example: "Closes #1234" -->
Closes#5458
## Checks
- [ ] I've included any doc changes needed for
https://microsoft.github.io/autogen/. See
https://microsoft.github.io/autogen/docs/Contribute#documentation to
build and test documentation locally.
- [ ] I've added tests (if relevant) corresponding to the changes
introduced in this PR.
- [ ] I've made sure all auto checks have passed.
---------
Co-authored-by: Leonardo Pinheiro <lpinheiro@microsoft.com>
Co-authored-by: Eric Zhu <ekzhu@users.noreply.github.com>
<!-- Thank you for your contribution! Please review
https://microsoft.github.io/autogen/docs/Contribute before opening a
pull request. -->
<!-- Please add a reviewer to the assignee section when you create a PR.
If you don't have the access to it, we will shortly find a reviewer and
assign them to your PR. -->
## Why are these changes needed?
<!-- Please give a short summary of the change and the problem this
solves. -->
The current stream processing of SK model adapter returns on the first
function call chunk but this behavior is incorrect end ends up returning
with an incomplete function call. The observed behavior is that the
function name and arguments are split into different chunks and this
update correctly processes the chunks in this way.
## Related issue number
<!-- For example: "Closes #1234" -->
Fixes the reply in #5420
## Checks
- [ ] I've included any doc changes needed for
https://microsoft.github.io/autogen/. See
https://microsoft.github.io/autogen/docs/Contribute#documentation to
build and test documentation locally.
- [ ] I've added tests (if relevant) corresponding to the changes
introduced in this PR.
- [ ] I've made sure all auto checks have passed.
---------
Co-authored-by: Leonardo Pinheiro <lpinheiro@microsoft.com>
<!-- Thank you for your contribution! Please review
https://microsoft.github.io/autogen/docs/Contribute before opening a
pull request. -->
<!-- Please add a reviewer to the assignee section when you create a PR.
If you don't have the access to it, we will shortly find a reviewer and
assign them to your PR. -->
## Why are these changes needed?
<!-- Please give a short summary of the change and the problem this
solves. -->
Semantic kernel prepends the plugin name to the tool name when passing
the tools to model clients and this is causing a mismatch between tool
names in SK and the AssistantAgent. Since plugin names are optional, we
have opted to remove it.
## Related issue number
<!-- For example: "Closes #1234" -->
Closes#5420
## Checks
- [ ] I've included any doc changes needed for
https://microsoft.github.io/autogen/. See
https://microsoft.github.io/autogen/docs/Contribute#documentation to
build and test documentation locally.
- [ ] I've added tests (if relevant) corresponding to the changes
introduced in this PR.
- [ ] I've made sure all auto checks have passed.
---------
Co-authored-by: Leonardo Pinheiro <lpinheiro@microsoft.com>
Presently MagenticOne and the m1 CLI use the LocalCommandLineExecutor
(presumably copied from the agbench code, which already runs in Docker).
This pr defaults m1 to Docker, and adds a code_executor parameter to
MagenticOne, which defaults to local for now to maintain backward
compatibility -- but this behavior is immediately deprecated.
A series of changes to the
`python/packages/autogen-ext/src/autogen_ext/agents/web_surfer/_multimodal_web_surfer.py`
file have been made to better support smaller models.
This includes changes to the prompts, state descriptions, and ordering
of messages.
Regression tasks with OpenAI models shows no change in GAIA scores,
while scores for Llama are significantly improved.
This PR fixes:
A prompting bug when no control had focus.
Awkward prompt phrasing.
Renamed page_down to scroll_down to better match other prompting and
agent descriptions.
Some agent descriptions were split over multiple lines in the M1
orchestrator. This PR ensures that each description appears on one, and
only one, line. This makes it easier for smaller models to understand.
<!-- Thank you for your contribution! Please review
https://microsoft.github.io/autogen/docs/Contribute before opening a
pull request. -->
<!-- Please add a reviewer to the assignee section when you create a PR.
If you don't have the access to it, we will shortly find a reviewer and
assign them to your PR. -->
## Why are these changes needed?
<!-- Please give a short summary of the change and the problem this
solves. -->
To allow serialization of OAI Assistant Agent.
## Related issue number
<!-- For example: "Closes #1234" -->
Closes#5130
## Checks
- [ ] I've included any doc changes needed for
https://microsoft.github.io/autogen/. See
https://microsoft.github.io/autogen/docs/Contribute#documentation to
build and test documentation locally.
- [ ] I've added tests (if relevant) corresponding to the changes
introduced in this PR.
- [ ] I've made sure all auto checks have passed.
<!-- Thank you for your contribution! Please review
https://microsoft.github.io/autogen/docs/Contribute before opening a
pull request. -->
<!-- Please add a reviewer to the assignee section when you create a PR.
If you don't have the access to it, we will shortly find a reviewer and
assign them to your PR. -->
## Why are these changes needed?
Fix bug where the `model_info` field is not serialized for the
`OpenAIChatCompletionClient` class. This was because the `_raw_config`
field was based on a version of the args that had been sanitized
(model_info removed). We need the full model info field for non-openai
models
```python
from autogen_ext.agents.web_surfer import MultimodalWebSurfer
from autogen_ext.models.openai import OpenAIChatCompletionClient
from autogen_core.models import ModelInfo
mistral_vllm_model = OpenAIChatCompletionClient(
model="TheBloke/Mistral-7B-Instruct-v0.2-GGUF",
base_url="http://localhost:1234/v1",
api_key="empty",
model_info=ModelInfo(vision=False, function_calling=True, json_output=False, family="unkown"),
)
(mistral_vllm_model.dump_component().model_dump_json())
```
Before
```
{
"provider": "autogen_ext.models.openai.OpenAIChatCompletionClient",
"component_type": "model",
"version": 1,
"component_version": 1,
"description": "Chat completion client for OpenAI hosted models.",
"label": "OpenAIChatCompletionClient",
"config": {
"model": "TheBloke/Mistral-7B-Instruct-v0.2-GGUF",
"api_key": "empty",
"base_url": "http://localhost:1234/v1"
}
}
```
After
```
{
"provider": "autogen_ext.models.openai.OpenAIChatCompletionClient",
"component_type": "model",
"version": 1,
"component_version": 1,
"description": "Chat completion client for OpenAI hosted models.",
"label": "OpenAIChatCompletionClient",
"config": {
"model": "TheBloke/Mistral-7B-Instruct-v0.2-GGUF",
"api_key": "empty",
"model_info": {
"vision": false,
"function_calling": true,
"json_output": false,
"family": "unkown"
},
"base_url": "http://localhost:1234/v1"
}
}
```
<!-- Please give a short summary of the change and the problem this
solves. -->
## Related issue number
<!-- For example: "Closes #1234" -->
## Checks
- [ ] I've included any doc changes needed for
https://microsoft.github.io/autogen/. See
https://microsoft.github.io/autogen/docs/Contribute#documentation to
build and test documentation locally.
- [ ] I've added tests (if relevant) corresponding to the changes
introduced in this PR.
- [ ] I've made sure all auto checks have passed.
## Why are these changes needed?
This pull request introduces the 'o3' model family and adds support for
the 'o3-mini' model.
---------
Co-authored-by: Eric Zhu <ekzhu@users.noreply.github.com>
<!-- Thank you for your contribution! Please review
https://microsoft.github.io/autogen/docs/Contribute before opening a
pull request. -->
<!-- Please add a reviewer to the assignee section when you create a PR.
If you don't have the access to it, we will shortly find a reviewer and
assign them to your PR. -->
## Why are these changes needed?
<!-- Please give a short summary of the change and the problem this
solves. -->
We are seeing this issue more often now, probably related to the load on
the API servers. Hence this PR:
1. Demotes the the `max_consecutive_empty_chunk_tolerance` parameter
from function to inline threshold
2. Change exception to a one time warning
## Related issue number
<!-- For example: "Closes #1234" -->
## Checks
- [x] I've included any doc changes needed for
https://microsoft.github.io/autogen/. See
https://microsoft.github.io/autogen/docs/Contribute#documentation to
build and test documentation locally.
- [x] I've added tests (if relevant) corresponding to the changes
introduced in this PR.
- [ ] I've made sure all auto checks have passed.
---------
Signed-off-by: Mohammad Mazraeh <mazraeh.mohammad@gmail.com>
Co-authored-by: Eric Zhu <ekzhu@users.noreply.github.com>
This PR adds a method that approximately extracts the text visible in
the viewport of the web browser (as opposed to always printing the first
50 lines, or relying entirely on OCR).
Removed the TERMINATE clause from the system prompt since M1 handles
termination via the Orchestrator, and it is just ignored.
Removed the clause about saving to a particular file name, since tmp_
files are created in the current CodeExecutors.
Resolves#3983
* introduce `model_client_stream` parameter in `AssistantAgent` to
enable token-level streaming output.
* introduce `ModelClientStreamingChunkEvent` as a type of `AgentEvent`
to pass the streaming chunks to the application via `run_stream` and
`on_messages_stream`. Although this will not affect the inner messages
list in the final `Response` or `TaskResult`.
* handle this new message type in `Console`.
<!-- Thank you for your contribution! Please review
https://microsoft.github.io/autogen/docs/Contribute before opening a
pull request. -->
<!-- Please add a reviewer to the assignee section when you create a PR.
If you don't have the access to it, we will shortly find a reviewer and
assign them to your PR. -->
## Why are these changes needed?
Make AssistantAgent and Handoff use BaseTool.
This ensures that they can be made declarative/serialized
<!-- Please give a short summary of the change and the problem this
solves. -->
## Related issue number
<!-- For example: "Closes #1234" -->
## Checks
- [ ] I've included any doc changes needed for
https://microsoft.github.io/autogen/. See
https://microsoft.github.io/autogen/docs/Contribute#documentation to
build and test documentation locally.
- [ ] I've added tests (if relevant) corresponding to the changes
introduced in this PR.
- [ ] I've made sure all auto checks have passed.
* Rebase to latest main branch
* Moved _azure module to azure
* Validate extra_create_args in and json response
* Added Support for Github Models
* Added normalize_name and assert_valid name
* Added Tests for AzureAIChatCompletionClient
* WIP: Azure AI Client
* Added: object-level usage data
* Added: doc string
* Added: check existing response_format value
* Added: _validate_config and _create_client
* lint
* merge dependencies
* add tests for img and function calling
* support actual tests through env vars
* address mypy errors
* doc example fix
* fmt
* fix doc fmt
* Update python/packages/autogen-ext/src/autogen_ext/models/azure/_azure_ai_client.py
---------
Co-authored-by: Rohan Thacker <thackerrohan4@gmail.com>
Co-authored-by: Eric Zhu <ekzhu@users.noreply.github.com>
Co-authored-by: Leonardo Pinheiro <lpinheiro@microsoft.com>
