458 Commits

Author SHA1 Message Date
yutao
54268eb08d docs(core): instruction for override config 2025-04-21 16:59:14 +08:00
yuyutaotao
86bd961542
docs: update docs (#574)
* docs: release android automation

* chore: update docs

* chore: update readme

* docs: append docs for doubao-1.5-ui-tars

---------

Co-authored-by: zhouxiao.shaw <zhouxiao.shaw@bytedance.com>

* chore(docs): update doubao docs

* chore(docs): merge docs for doubao

* docs(android): update

* docs(llm): upgrade open source model

---------

Co-authored-by: 晓 <zhouxiao.shaw@bytedance.com>
2025-04-21 14:18:55 +08:00
Leyang
69aeadb9a1
ci(workflow): dynamically obtain the scope of apps and packages (#602) 2025-04-21 14:16:56 +08:00
Leyang
e1b8fa45a5
fix(android): update styles and transitions in playground components (#596)
* fix(android): update styles and transitions in playground components

* fix(android): optimize the style of the input box and the automatic scrolling function

* fix(android): remove max-width constraint from player component styles
2025-04-21 14:10:12 +08:00
yuyutaotao
138864e6b1
fix(core): keep context in log file when locate fails (#597) 2025-04-21 13:56:30 +08:00
yuyutaotao
10efa40c8b
fix(cli): keep window after cli running (#598) 2025-04-21 13:54:58 +08:00
df77c335fe
docs(llm): update ui-tars 1.5 link (#601) 2025-04-19 16:15:01 +08:00
01f46a202f
refactor(android): improve logic for copying Android playground (#600) 2025-04-19 15:59:11 +08:00
daf308b1d0
fix(mcp): resolve mcp server log error (#599)
* fix(mcp): resolve mcp server log error

* chore(shared): delete unless code

* chore(workflow): fix lint error
2025-04-19 12:00:39 +08:00
github-actions[bot]
0a3bc1e230 release: v0.15.0 v0.15.0 2025-04-18 11:26:30 +00:00
Leyang
2c66902677
fix(core): player title max width (#595) 2025-04-18 19:16:17 +08:00
Leyang
76eb4e1417
fix(report): ai tap in report (#594)
* fix: ai tap in report

* chore(playwright): update lockfile

* fix(core): lint

---------

Co-authored-by: yutao <yutao.tao@bytedance.com>
2025-04-18 18:42:47 +08:00
Leyang
758fa44d71
fix(yaml): handle null case for android configuration in parseYamlScript function (#593)
* fix(yaml): handle null case for android configuration in parseYamlScript function

* fix(cli): fix yaml parser

---------

Co-authored-by: yutao <yutao.tao@bytedance.com>
2025-04-18 17:38:31 +08:00
Leyang
06ce3c889b
android playground compatible with windows (#592)
* fix(playground): simplify class property definitions and add root redirect for static file serving

* fix(scrcpy-player): update empty state rendering logic to handle server validity more accurately
2025-04-18 17:03:00 +08:00
Leyang
8fc7d74d0a
fix(core): race condition when initing env config (#590)
* fix(env): refactor globalConfig handling and update AI config access methods

* fix(page): add debug logging for adb command execution

* refactor(ai): initialize debug configuration and improve logging setup
2025-04-18 16:06:41 +08:00
Leyang
cf621a564e
fix(visualizer): update type references and improve tooltip handling in components (#589) 2025-04-18 15:18:03 +08:00
yuyutaotao
a6ac78aee3
chore(core): update prompt for section locator (#588) 2025-04-18 13:35:58 +08:00
yuyutaotao
7e43a29d02
chore: update extension style (#585)
---------

Co-authored-by: zhouxiao.shaw <zhouxiao.shaw@bytedance.com>
2025-04-18 12:53:33 +08:00
Zhou Xiao
bc5b6e2121
ci(workflow): add commit msg check (#587)
* ci(workflow): add commit check

* docs(workflow): add commit check to CONTRIBUTING

* chore(workflow): optimize lint error
2025-04-18 12:50:52 +08:00
Leyang
bcfcb3c4d0
fix: style of chrome (#586) 2025-04-18 11:33:20 +08:00
Zhou Xiao
2cc12dad12
feat(mcp): provide midscene mcp server (#562) 2025-04-18 10:47:48 +08:00
Leyang
7596da2ab7
feat: support tap in prompt (#584) 2025-04-18 10:27:56 +08:00
Leyang
6e7cf23c61
fix: esm for @yume-chan/async (#583) 2025-04-18 10:08:41 +08:00
yuyutaotao
ad457a33a8
feat(yaml): allow running javascript in yaml (#555)
* feat: allow running javascript in yaml

* feat: change the output dir

* fix: CI

* docs: update docs about evaluate javascript

* chore: merge main

* chore: merge main
2025-04-18 09:58:51 +08:00
Leyang
1d49e981d0
fix: compatible with node 20 (#581)
* fix: compatible with node 20(https://github.com/yume-chan/async/pull/44)

* fix: compatible with node 18 due ipv6
2025-04-17 22:56:02 +08:00
Leyang
b76211bd5d
feat: android playground (#542)
* refactor: android api

* refactor: enhance Android agent to accept options for device connection

* fix: type error

* fix: click after clearInput

* fix: click before clearInput

* feat: android playground

* feat: support npx package name

* feat: android playground joint

* fix: git ignore conflicts

* feat: ensure adb server is running before initializing adb client

* fix: deps consistency

* ci: add android playground

* feat: integrate shared constants and improve server configuration in android playground

* feat: android playground style

* feat: style opt

* feat: add @rsbuild/plugin-svgr dependency and improve URI handling in adb

* feat: remove unused water flow scripts and update comments to English

* feat: download report file

* feat: standalone android playground

* feat: use dynamic import

* feat: migrate CSS to LESS and remove unused styles in chrome extension and report

* feat: enhance Android playground with ScrcpyPlayer ref integration and device management improvements

* feat: optimize styles and layout in Android playground and visualizer components

* chore: add bin back

* chore: update build script to exclude documentation generation

* feat: add not ready message to PlaygroundResult for improved user guidance

* feat: add error handling for screenshot capture in Android page

* docs: update readme

* feat: add PNG validation for screenshot buffer in Android page

* feat: enhance UI components with improved styling and tooltips in ScrcpyPlayer and PromptInput

* docs: update uri parameter description in integrate-with-android documentation and improve uri handling in launch function

* style: update primary color to #2B83FF across multiple components and adjust margin in App.less

* refactor: replace userConfig with globalConfig for environment configuration management and update related functions

* feat: integrate server validation logic in App, AdbDevice, and ScrcpyPlayer components for improved connection handling

* style: enhance player component layout with overflow handling and margin adjustments

* style: refine player component layout with flex adjustments and improved spacing

* feat: add midscene model name display and improve layout in EnvConfig component

* feat: integrate ShinyText component for enhanced loading progress display in PlaygroundResult

* test: add test for isValidPNGImageBuffer

* style: remove background color from App.less and adjust AI config override behavior in env.ts

---------

Co-authored-by: yutao <yutao.tao@bytedance.com>
2025-04-17 17:44:11 +08:00
yuyutaotao
a1b5a54d89
fix(report): do not call mkdir in browser (#577)
* fix: log dir

* fix: log dir

* fix: gitignore config
2025-04-17 15:09:59 +08:00
yuyutaotao
824be26c85
fix: use tmpdir as a fallback for log file (#575)
---------

Co-authored-by: zhouxiao.shaw <zhouxiao.shaw@bytedance.com>
2025-04-17 10:54:19 +08:00
yuyutaotao
dd233ef25b
chore: adapting new model (#570)
* chore: adapting new model

* feat: adapt bbox format

* chore: adapt ui-tars
---------

Co-authored-by: zhouxiao.shaw <zhouxiao.shaw@bytedance.com>
2025-04-17 10:53:19 +08:00
yuyutaotao
445381efef
feat(core): show warning for oversized image (#564)
---------

Co-authored-by: zhouxiao.shaw <zhouxiao.shaw@bytedance.com>
2025-04-17 10:51:58 +08:00
yuyutaotao
c777c139e4
fix: puppeteer export (#569) 2025-04-17 10:50:16 +08:00
Leyang
80205b4625
feat: add uitars as vl mode (#565)
* feat: add uitars as vl mode

* feat: add uitars as vl mode

* refactor: simplify model description logic in insight utilities

* chore: warn for conflict configs

---------

Co-authored-by: yutao <yutao.tao@bytedance.com>
2025-04-16 11:12:56 +08:00
Leyang
258dba3011
fix: dry run (#566) 2025-04-15 17:46:28 +08:00
Leyang
3793e91899
docs: add android playground docs (#563)
* docs: add android playground docs

* chore: update pic

* docs: distinct android and web

* chore: update docs

* chore: update docs

---------

Co-authored-by: yutao <yutao.tao@bytedance.com>
2025-04-15 11:31:12 +08:00
yuyutaotao
edf48da198
docs: update caching docs (#559)
* docs: update caching docs

* docs: update caching docs

* docs: update docs

* chore: update docs
2025-04-14 14:50:48 +08:00
yuyutaotao
027283aaac
feat: disable http warning in yaml mode (#556) 2025-04-11 10:04:05 +08:00
Leyang
0ca9fda7ae
feat: android yaml support (#551)
* feat: android yaml support

* feat: enhance ScriptPlayer to support web and android environments

* chore: update error message

* refactor: unify environment interfaces for YAML scripts

---------

Co-authored-by: yutao <yutao.tao@bytedance.com>
2025-04-11 09:36:41 +08:00
Leyang
570c2d7294
fix: rethrow error with android faq when adb error occured (#554) 2025-04-10 20:51:56 +08:00
github-actions[bot]
1bd873b360 release: v0.14.3 v0.14.3 2025-04-10 09:10:39 +00:00
yuyutaotao
9be848b651
feat: allow insecure https in yaml (#553) 2025-04-10 16:57:46 +08:00
yuyutaotao
66a5d16d37
chore: update evaluation fixture (#552) 2025-04-10 16:54:21 +08:00
Leyang
202f54ca78
docs: add docs for android (#546)
* docs: add docs for android

* chore: update docs

* docs: add extra api for android

* chore: update docs

---------

Co-authored-by: yutao <yutao.tao@bytedance.com>
2025-04-09 17:04:56 +08:00
Leyang
fbaa314520
feat(android): implement custom method to retrieve screen size and simplify size handling (#544)
* feat(android): implement custom method to retrieve screen size and simplify size handling

* fix: use debugPage

* feat: print device info
2025-04-09 10:54:39 +08:00
Leyang
0bbe416151
fix(android): verify vl-model mode in AndroidAgent constructor (#547)
* fix(android): verify vl-model mode in AndroidAgent constructor

* fix: error message

---------

Co-authored-by: yutao <yutao.tao@bytedance.com>
2025-04-09 10:48:53 +08:00
github-actions[bot]
4d77207c0f release: v0.14.2 v0.14.2 2025-04-09 02:30:47 +00:00
yuyutaotao
732f605144
fix(llm): coords offset in vl locator (#545) 2025-04-08 17:56:15 +08:00
Zhou Xiao
8582f86793
fix(playwright): Prevent the test from ending due to consistent timeout of screenshots (#541) 2025-04-07 16:02:55 +08:00
lsustc
bd0bf46e9a
fix click event should move the mouse to the click area first (#535)
* fix click event should move the mouse to the click area first

* fix reusing the existing function
2025-04-06 21:53:21 +08:00
Zhou Xiao
c4d4eb7cdf
workflow(issue): add issue template (#540) 2025-04-05 10:44:35 +08:00
Zhou Xiao
2c3c540cb1
fix(playwright): make test.describe work correctly with ai fixtures (#537)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
2025-04-04 16:14:04 +08:00