Eric Zhu
483532180a
Improvements to agbench ( #5776 )
...
1. Add host network support in Docker and remove unused requirements
from argument check.
2. Use Pandas to simplify summary statistic calculations.
3. Add running time to summary statistics
```
Using tabulation method defined in '/home/ekzhu/autogen/python/packages/agbench/benchmarks/HumanEval/Scripts/custom_tabulate.py'
Task Id Trial 0 Success Trial 0 Time
-- ------------ ----------------- --------------
0 HumanEval_0 True 3
1 HumanEval_1 False 15
2 HumanEval_2 True 2
3 HumanEval_3 True 11
4 HumanEval_4 True 4
5 HumanEval_5 True 2
6 HumanEval_6 False 18
7 HumanEval_7 True 2
8 HumanEval_8 True 2
9 HumanEval_9 True 12
10 HumanEval_10 False 11
11 HumanEval_11 True 2
12 HumanEval_12 True 3
13 HumanEval_13 True 1
14 HumanEval_14 True 4
15 HumanEval_15 True 1
16 HumanEval_16 True 2
17 HumanEval_17 False 76
18 HumanEval_18 True 4
19 HumanEval_19 True 3
20 HumanEval_20 True 5
21 HumanEval_21 True 3
22 HumanEval_22 True 1
23 HumanEval_23 True 2
24 HumanEval_24 nan
Summary Statistics
Successes Failures Missing Total Average Success Rate Average Time Total Time
------- ----------- ---------- --------- ------- ---------------------- -------------- ------------
Trial 0 20 4 1 25 0.8 7.875 189
CAUTION: 'autogenbench tabulate' is in early preview and is not thoroughly tested.
Please do not cite values from these calculations in academic work without first inspecting and verifying the results in the run logs yourself.
```
Now the default tabulate output looks like this
---------
Co-authored-by: Ryan Sweet <rysweet@microsoft.com>
2025-03-16 09:13:12 -07:00
afourney
af5dcc7fdf
Significant updates to agbench. ( #5313 )
...
- Updated HumanEval template to use AgentChat
- Update templates to use config.yaml for model and other configuration
- Read environment from ENV.yaml (ENV.json still supported but
deprecated)
- Temporarily removed WebArena and AssistantBench. Neither had viable
Templates after `autogen_magentic_one` was removed. Templates need to be
update to AgentChat (in a future PR, but this PR is getting big enough
already)
2025-02-07 18:01:44 +00:00
afourney
088a50faa5
Remove old autogen_magentic_one package. ( #5305 )
...
This PR removes the older `autogen_magentic_one` package, and directs
people to use the new AgentChat implementation.
Hopefully this eases confusion.
---------
Co-authored-by: Jack Gerrits <jack@jackgerrits.com>
Co-authored-by: Eric Zhu <ekzhu@users.noreply.github.com>
2025-01-31 15:14:40 -08:00
Jack Gerrits
538f39497b
Replace create_completion_client_from_env with component config ( #4928 )
...
* Replace create_completion_client_from_env with component config
* json load
2025-01-08 14:33:28 +00:00
Jack Gerrits
87011ae01b
Migrate model context and models modules out of components ( #4613 )
...
* Move model context out of components
* move models out of components
* rename docs file
2024-12-09 10:00:08 -08:00
Eric Zhu
8dac072658
Update references in docs ( #4590 )
...
* Update agent doc
* Remove outdated doc
* Update references
* Update readme
* Update readme
2024-12-06 01:59:28 -08:00
Eric Zhu
fa550c2c36
fix docs ( #4589 )
...
* fix doc on distributed runtime
* Fix references
* Update references
* Fix import paths in user guide notebooks for code executor components
2024-12-06 01:23:05 -08:00
Jack Gerrits
2b878763f8
Move grpc runtimes to ext, flatten application ( #4553 )
...
* Move grpc runtimes to ext, flatten application
* rename to grpc
* fmt
2024-12-04 16:23:20 -08:00
Jack Gerrits
3022369eeb
Flatten core base and components ( #4513 )
...
* Flatten core base and components
* remove extra files
* dont export from deprecated locations
* format
* fmt
2024-12-03 17:00:44 -08:00
Leonardo Pinheiro
38f62e1609
migrate models ( #3848 )
...
* migrate models
* Update python/packages/autogen-agentchat/src/autogen_agentchat/agents/_tool_use_assistant_agent.py
Co-authored-by: Eric Zhu <ekzhu@users.noreply.github.com>
* refactor missing imports
* ignore type check errors
* Update python/packages/autogen-ext/src/autogen_ext/models/_openai/_model_info.py
Co-authored-by: Eric Zhu <ekzhu@users.noreply.github.com>
* update packages index page
---------
Co-authored-by: Leonardo Pinheiro <lpinheiro@microsoft.com>
Co-authored-by: Eric Zhu <ekzhu@users.noreply.github.com>
2024-10-22 11:40:41 -04:00
Hussein Mozannar
e11d84b996
Adding Benchmarks to agbench ( #3803 )
...
* Move from tomllib to tomli
* added example code for magentic-one + code comments
* adding benchmarks temporarily
* add license for datasets
* revert changes to magentic-one
* change license location
---------
Co-authored-by: Ryan Sweet <rysweet@microsoft.com>
2024-10-18 06:33:33 +02:00
