90 Commits

Author SHA1 Message Date
Leyang
b10a7751d4
fix(report): show highlight elment when cache hit (#723)
* fix(report): show highlight elment when cache hit

* fix(core): replay animation

---------

Co-authored-by: yutao <yutao.tao@bytedance.com>
2025-05-16 22:28:29 +08:00
yuyutaotao
07614961c8
fix(visualizer): the style of error state in report (#641)
* fix(visualizer): the style of error state in report

* fix(core): fix ci
2025-04-25 12:00:48 +08:00
Leyang
974ebd34ed
fix(visualizer): scroll for result container (#640)
* fix(visualizer): scroll for result container

* feat(visualizer): add font weight to title in player component

* feat(visualizer): initialize subtitle for the first task in animation script generation
2025-04-25 11:46:42 +08:00
Leyang
ca644d8914
feat(core): allow custom midscene_run dir (#631)
* feat(core): support custom midscene_run dir

* feat(report): add search functionality to PlaywrightCaseSelector component

* refactor(shared): simplify base directory resolution and remove unused environment variable

* feat(shared): integrate shared environment variables across multiple packages

* refactor(shared): update base directory resolution to use dynamic midscene_run directory

* fix(puppeteer): increase screenshot timeout from 3s to 10s for improved reliability
2025-04-24 22:54:52 +08:00
yuyutaotao
ecefd8b0fa
fix(report): reduce context size in report file (#626)
* fix(core): reduce context size in report file

* chore(core): fix lint

* chore(core): resolve conflict

---------

Co-authored-by: zhouxiao.shaw <zhouxiao.shaw@bytedance.com>
2025-04-24 18:28:45 +08:00
Leyang
38c50a23c3
feat: support android device button (#567) 2025-04-22 23:55:42 +08:00
Leyang
fdbdb7d426
style(chrome-extension): opt chrome logo icon (#614) 2025-04-22 15:39:17 +08:00
Leyang
e1b8fa45a5
fix(android): update styles and transitions in playground components (#596)
* fix(android): update styles and transitions in playground components

* fix(android): optimize the style of the input box and the automatic scrolling function

* fix(android): remove max-width constraint from player component styles
2025-04-21 14:10:12 +08:00
Leyang
2c66902677
fix(core): player title max width (#595) 2025-04-18 19:16:17 +08:00
Leyang
76eb4e1417
fix(report): ai tap in report (#594)
* fix: ai tap in report

* chore(playwright): update lockfile

* fix(core): lint

---------

Co-authored-by: yutao <yutao.tao@bytedance.com>
2025-04-18 18:42:47 +08:00
Leyang
cf621a564e
fix(visualizer): update type references and improve tooltip handling in components (#589) 2025-04-18 15:18:03 +08:00
yuyutaotao
7e43a29d02
chore: update extension style (#585)
---------

Co-authored-by: zhouxiao.shaw <zhouxiao.shaw@bytedance.com>
2025-04-18 12:53:33 +08:00
Leyang
bcfcb3c4d0
fix: style of chrome (#586) 2025-04-18 11:33:20 +08:00
Leyang
7596da2ab7
feat: support tap in prompt (#584) 2025-04-18 10:27:56 +08:00
Leyang
b76211bd5d
feat: android playground (#542)
* refactor: android api

* refactor: enhance Android agent to accept options for device connection

* fix: type error

* fix: click after clearInput

* fix: click before clearInput

* feat: android playground

* feat: support npx package name

* feat: android playground joint

* fix: git ignore conflicts

* feat: ensure adb server is running before initializing adb client

* fix: deps consistency

* ci: add android playground

* feat: integrate shared constants and improve server configuration in android playground

* feat: android playground style

* feat: style opt

* feat: add @rsbuild/plugin-svgr dependency and improve URI handling in adb

* feat: remove unused water flow scripts and update comments to English

* feat: download report file

* feat: standalone android playground

* feat: use dynamic import

* feat: migrate CSS to LESS and remove unused styles in chrome extension and report

* feat: enhance Android playground with ScrcpyPlayer ref integration and device management improvements

* feat: optimize styles and layout in Android playground and visualizer components

* chore: add bin back

* chore: update build script to exclude documentation generation

* feat: add not ready message to PlaygroundResult for improved user guidance

* feat: add error handling for screenshot capture in Android page

* docs: update readme

* feat: add PNG validation for screenshot buffer in Android page

* feat: enhance UI components with improved styling and tooltips in ScrcpyPlayer and PromptInput

* docs: update uri parameter description in integrate-with-android documentation and improve uri handling in launch function

* style: update primary color to #2B83FF across multiple components and adjust margin in App.less

* refactor: replace userConfig with globalConfig for environment configuration management and update related functions

* feat: integrate server validation logic in App, AdbDevice, and ScrcpyPlayer components for improved connection handling

* style: enhance player component layout with overflow handling and margin adjustments

* style: refine player component layout with flex adjustments and improved spacing

* feat: add midscene model name display and improve layout in EnvConfig component

* feat: integrate ShinyText component for enhanced loading progress display in PlaygroundResult

* test: add test for isValidPNGImageBuffer

* style: remove background color from App.less and adjust AI config override behavior in env.ts

---------

Co-authored-by: yutao <yutao.tao@bytedance.com>
2025-04-17 17:44:11 +08:00
Leyang
258dba3011
fix: dry run (#566) 2025-04-15 17:46:28 +08:00
Zhou Xiao
814f99ca48
refactor(visualizer): optimize extension logic (#525) 2025-04-02 19:12:52 +08:00
Zhou Xiao
5ef9202c79
refactor(report): optimize visualizer common logic (#519)
* chore: delete unless visualizer logic

* chore: delete unless code

* chore: delete unless code

* chore: delete unless code

* chore: delete unless common logic

* chore: merge main branch

* chore: update lock file
2025-04-01 17:25:40 +08:00
yuyutaotao
adc3fd9e0d
feat: use aiTap(prompt, opt) style interface (#512) 2025-04-01 10:41:31 +08:00
Zhou Xiao
d1b232586b
chore: delete unless visualizer logic (#514) 2025-03-31 19:51:43 +08:00
Zhou Xiao
16d6efa1f0
refactor(report): make the report page an app type to improve the development experience (#513) 2025-03-31 18:46:35 +08:00
yuyutaotao
649aeceb43
feat: enable search area for locate (#473)
* feat: enable search area for locate

* fix: update evaluation

* fix: build error

* fix: ci

* fix: locator

* feat: show searchArea in report

* chore: add yaml support for aiTap

* feat: update status tip

* fix: #473 (#484)

* chore: optimize unit test list

---------

Co-authored-by: zhouxiao.shaw <zhouxiao.shaw@bytedance.com>
2025-03-24 09:50:27 +08:00
Zhou Xiao
47cb015c90
refactor(chrome-devtool): extract the chrome-devtool logic into an application, support local development debugging, and add contribution guidelines. (#476)
* chore: add chrome devtools app

* chore: resolve import error

* chore: support visualizer css

* add build logic

* chore: add build extension zip file script

* chore: migrate part of chrome extension content to app

* chore: delete unless file

* chore: optimize chrome devtool build script

* chore: optimize chrome devtool build script

* fix: resolve bridge mode test issues

* chore: optimize chrome devtool build script

* chore: optimize chrome devtool build script

* chore: optimize chrome devtool build script

* chore: update chrome devtools build process

* chore: optimize chrome devtool build script

* chore: optimize chrome devtool build script

* chore: optimize chrome devtool build script

* chore: optimize chrome devtool build script
2025-03-19 15:22:17 +08:00
yuyutaotao
8e1ba565d0
feat: optimize locator (#456)
---------

Co-authored-by: zhouxiao.shaw <zhouxiao.shaw@bytedance.com>
2025-03-17 19:19:54 +08:00
yuyutaotao
55d02961f7
fix: add an field for CoT in qwen (#453)
* fix: add an field for CoT in qwen

* chore: update prompt

* fix: scroll in puppeteer

* fix: multiple action in qwen

* fix: multiple action in qwen

* fix: multiple action in qwen

* chore: update prompt

* chore: change fields order to enhance the CoT
2025-03-11 22:43:59 +08:00
Zhou Xiao
5d63ef9151
refactor: switch bundle type to bundleless (#437) 2025-03-07 17:20:18 +08:00
yuyutaotao
6a08dd7151
chore: rename core package (#431)
* chore: rename core package

* fix: lint

* fix: ci

* fix: ci
2025-02-27 22:47:06 +08:00
yuyutaotao
a6ffbd07b7
fix: over planning for Qwen in page with form (#429) 2025-02-27 16:44:01 +08:00
yuyutaotao
8be082e308
feat(chrome-devtool): allow longer connection of chrome bridge (#416) 2025-02-26 14:20:27 +08:00
yuyutaotao
59ce2d0140
feat: locate by coord (#383)
---------

Co-authored-by: zhouxiao.shaw <zhouxiao.shaw@bytedance.com>
2025-02-21 09:56:09 +08:00
Zhou Xiao
f661a1882e
fix(report): resolve no locate generate error (#392) 2025-02-15 12:00:57 +08:00
Zhou Xiao
edf0871fea
feat(browser): Add the forceSameTabNavigation configuration to prevent AI from opening new pages during operations, thus avoiding task interruptions. (#389)
* feat(tab-control): enhance the configuration to limit AI from opening new tabs during operations, preventing failures.

* chore: optimize evaluate error

* chore: resolve navigation error

* fix(browser): add forceSameTabNavigation config toe limit open new tab

* chore: upgrade vitest version

* fix: typo (#390)

---------

Co-authored-by: yuyutaotao <167746126+yuyutaotao@users.noreply.github.com>
2025-02-14 21:54:47 +08:00
yuyutaotao
3aa1b33955
feat: use jpeg as default image format (#301)
---------

Co-authored-by: zhouxiao.shaw <zhouxiao.shaw@bytedance.com>
2025-01-20 20:02:49 +08:00
yuyutaotao
a114e707d1
feat: show pointer position in chrome extension (#286)
---------

Co-authored-by: zhouxiao.shaw <zhouxiao.shaw@bytedance.com>
2025-01-17 18:19:22 +08:00
yuyutaotao
fb2b9d123f
feat(chrome-devtool): add 'stop' button in extension (#281)
* feat: add 'stop' to playground

* feat: make extension stopable

---------

Co-authored-by: zhouxiao.shaw <zhouxiao.shaw@bytedance.com>
2025-01-15 19:18:26 +08:00
Zhou xiao
beb74f12e8
chore: optimize devtool tab connect (#279) 2025-01-14 21:31:33 +08:00
yuyutaotao
74774c6588
fix: memory leak in report file (#275)
* fix: memory leak in report file

* chore: remove unused console.log

* chore: set default value of TRACKING_ACTIVE_TAB_KEY into true

---------

Co-authored-by: zhouxiao.shaw <zhouxiao.shaw@bytedance.com>
2025-01-14 17:48:39 +08:00
Zhou xiao
cb52da2ffd
chore: Avoid obscuring the entire page content while optimizing the execution process of DevTools (#276) 2025-01-14 17:05:56 +08:00
Zhou xiao
0cf9a71940
feat(chrome-devtool): water flow animation on the target page (#271)
* feat(chrome-devtool): support water flow animation

* chore: optimize debugger cause window resize

* chore: optimize dynamic script inject

* chore: optimize report error
2025-01-14 14:13:50 +08:00
yuyutaotao
857c411d67
feat: allow tracking newly-opened tabs in Chrome extension (#272) 2025-01-14 11:22:20 +08:00
Zhou xiao
2b18ed55de
feat(ai-model): support vlm (#262)
* feat(ai-model): support plan to target

* chore: modify test

* chore: modify test

* chore: fix env config

* chore: unify the action logic

* chore: optimize type hint

* chore: optimize type hint

* chore: fix type hint

* chore: fix build type error

* chore: support open new tab

* feat: support auto complete element

* chore: add sleep event

* chore: add ai cost time

* chore: optimize prompt

* chore: optimize report prompt

* chore: optimize vlm name

* chore: fix command

* chore: optimize error handle and use check debugger list replace last tab id

* chore: fix chrome debugger attach logic

* chore: fix unit test
2025-01-13 14:32:17 +08:00
yuyutaotao
88fd4ed52a
fix: a more friendly error code for extension (#260) 2025-01-10 12:45:10 +08:00
yuyutaotao
8479bcb652
feat: bridge mode - phase 2 (#257)
* feat: allow auto reconnect in bridge mode after disconnect

* feat: show agent progress in extension

* feat: allow using bridge mode in yaml

* doc: add doc for bridge mode

* doc: add doc for bridge mode

* docs: update screenshot
2025-01-10 10:55:41 +08:00
yuyutaotao
ae49685348
feat: add bridge mode for extension (#228) 2025-01-07 11:10:28 +08:00
yuyutaotao
f9f4718ee6
feat: use our new version of logo (#229)
* feat: use our new version of logo

* fix: lint

* feat: update extension screenshot

* feat: update video cover

* feat: readme
2025-01-01 09:09:09 +08:00
yuyutaotao
e3481dc119
fix: show error message when aiAction fails in extension (#210) 2024-12-26 10:19:16 +08:00
yuyutaotao
05dad23bfd
chore: print yaml file in chrome extension (#205) 2024-12-25 20:23:12 +08:00
yuyutaotao
523adab12f
feat: support the if-statement in planning prompt (#184) 2024-12-19 10:44:08 +08:00
yuyutaotao
537a5c43d0
feat: add a real-time progress indicator for playground (#177) 2024-12-16 15:04:21 +08:00
yuyutaotao
082e347da9
feat: replanning before locating (#167) 2024-12-08 20:12:17 +08:00