yujunjun/autogen
1
0
Fork 0
mirror of https://github.com/microsoft/autogen.git synced 2025-06-26 22:30:10 +00:00
Code Issues Packages Projects Releases Wiki Activity
autogen/python/packages/agbench/benchmarks
History
Eric Zhu 483532180a
Improvements to agbench (#5776) 
1. Add host network support in Docker and remove unused requirements
from argument check.
2. Use Pandas to simplify summary statistic calculations. 
3. Add running time to summary statistics

```
Using tabulation method defined in '/home/ekzhu/autogen/python/packages/agbench/benchmarks/HumanEval/Scripts/custom_tabulate.py'

    Task Id       Trial 0 Success      Trial 0 Time
--  ------------  -----------------  --------------
 0  HumanEval_0   True                            3
 1  HumanEval_1   False                          15
 2  HumanEval_2   True                            2
 3  HumanEval_3   True                           11
 4  HumanEval_4   True                            4
 5  HumanEval_5   True                            2
 6  HumanEval_6   False                          18
 7  HumanEval_7   True                            2
 8  HumanEval_8   True                            2
 9  HumanEval_9   True                           12
10  HumanEval_10  False                          11
11  HumanEval_11  True                            2
12  HumanEval_12  True                            3
13  HumanEval_13  True                            1
14  HumanEval_14  True                            4
15  HumanEval_15  True                            1
16  HumanEval_16  True                            2
17  HumanEval_17  False                          76
18  HumanEval_18  True                            4
19  HumanEval_19  True                            3
20  HumanEval_20  True                            5
21  HumanEval_21  True                            3
22  HumanEval_22  True                            1
23  HumanEval_23  True                            2
24  HumanEval_24                                nan

Summary Statistics

           Successes    Failures    Missing    Total    Average Success Rate    Average Time    Total Time
-------  -----------  ----------  ---------  -------  ----------------------  --------------  ------------
Trial 0           20           4          1       25                     0.8           7.875           189

CAUTION: 'autogenbench tabulate' is in early preview and is not thoroughly tested.
Please do not cite values from these calculations in academic work without first inspecting and verifying the results in the run logs yourself.

```

Now the default tabulate output looks like this

---------

Co-authored-by: Ryan Sweet <rysweet@microsoft.com>
2025-03-16 09:13:12 -07:00
..
GAIA
Significant updates to agbench. (#5313)
2025-02-07 18:01:44 +00:00
HumanEval
Improvements to agbench (#5776)
2025-03-16 09:13:12 -07:00
.gitignore
Adding Benchmarks to agbench (#3803)
2024-10-18 06:33:33 +02:00
process_logs.py
Adding Benchmarks to agbench (#3803)
2024-10-18 06:33:33 +02:00
README.md
Adding Benchmarks to agbench (#3803)
2024-10-18 06:33:33 +02:00

README.md

Benchmarking Agents

This directory provides ability to benchmarks agents (e.g., built using Autogen) using AgBench. Use the instructions below to prepare your environment for benchmarking. Once done, proceed to relevant benchmarks directory (e.g., benchmarks/GAIA) for further scenario-specific instructions.

Setup on WSL

  1. Install Docker Desktop. After installation, restart is needed, then open Docker Desktop, in Settings, Ressources, WSL Integration, Enable integration with additional distros – Ubuntu

  2. Clone autogen and export AUTOGEN_REPO_BASE. This environment variable enables the Docker containers to use the correct version agents.

    git clone git@github.com:microsoft/autogen.git
export AUTOGEN_REPO_BASE=<path_to_autogen>

  3. Install agbench. AgBench is currently a tool in the Autogen repo.

    cd autogen/python/packages/agbench
pip install -e .