138 Commits

Author SHA1 Message Date
Leyang
bf9b4e06e7
feat: migrate android code to @midscene/android (#505) 2025-03-26 18:59:50 +08:00
github-actions[bot]
00de5d6478 release: v0.13.1 2025-03-25 14:53:44 +00:00
Leyang
934a1e2b5d
use adb instead of appium (#483)
* feat: enable search area for locate

* fix: update evaluation

* fix: locator

* feat: show searchArea in report

* chore: add yaml support for aiTap

* feat: use adb instead appium

* feat: Adds debugging information and reconstructs input text capabilities

* feat: refactoring Android related functions and adding android modules

* feat: update the image scaling algorithm, adjust the Android page class to support device scaling, and remove test files that are no longer needed

* feat: adjust the Android page class to support device scaling, and remove test files that are no longer needed

* feat: use appium-adb instead of bare command

* fix: update entry for @midscene/android

* feat: optimize the screenshot processing logic, add a backup mechanism when screenshots fail, and update test cases to accommodate new features

* fix: rethrow error

* feat: add Android debug configuration options and update documentation

* chore: fix code style in #483 (#492)

* fix: remove try for error handle by outside

---------

Co-authored-by: yutao <yutao.tao@bytedance.com>
Co-authored-by: linyibing <linyibing@bytedance.com>
Co-authored-by: yuyutaotao <167746126+yuyutaotao@users.noreply.github.com>
2025-03-25 22:45:05 +08:00
github-actions[bot]
291f8e1a4b release: v0.13.0 2025-03-24 12:32:05 +00:00
yuyutaotao
649aeceb43
feat: enable search area for locate (#473)
* feat: enable search area for locate

* fix: update evaluation

* fix: build error

* fix: ci

* fix: locator

* feat: show searchArea in report

* chore: add yaml support for aiTap

* feat: update status tip

* fix: #473 (#484)

* chore: optimize unit test list

---------

Co-authored-by: zhouxiao.shaw <zhouxiao.shaw@bytedance.com>
2025-03-24 09:50:27 +08:00
github-actions[bot]
df16339e08 release: v0.12.8 2025-03-19 08:16:01 +00:00
Zhou Xiao
47cb015c90
refactor(chrome-devtool): extract the chrome-devtool logic into an application, support local development debugging, and add contribution guidelines. (#476)
* chore: add chrome devtools app

* chore: resolve import error

* chore: support visualizer css

* add build logic

* chore: add build extension zip file script

* chore: migrate part of chrome extension content to app

* chore: delete unless file

* chore: optimize chrome devtool build script

* chore: optimize chrome devtool build script

* fix: resolve bridge mode test issues

* chore: optimize chrome devtool build script

* chore: optimize chrome devtool build script

* chore: optimize chrome devtool build script

* chore: update chrome devtools build process

* chore: optimize chrome devtool build script

* chore: optimize chrome devtool build script

* chore: optimize chrome devtool build script

* chore: optimize chrome devtool build script
2025-03-19 15:22:17 +08:00
yuyutaotao
8e1ba565d0
feat: optimize locator (#456)
---------

Co-authored-by: zhouxiao.shaw <zhouxiao.shaw@bytedance.com>
2025-03-17 19:19:54 +08:00
github-actions[bot]
4b9736b4cd release: v0.12.7 2025-03-12 15:59:54 +00:00
yuyutaotao
220a8c3171
fix: package shared as cjs (#463)
* fix: package shared as cjs

* fix: package shared as cjs

* fix: cjs export

* fix: makes web-integration a pure cjs
2025-03-12 23:17:11 +08:00
github-actions[bot]
b4d2181a44 release: v0.12.6 2025-03-12 06:11:58 +00:00
github-actions[bot]
9dad7f3e7f release: v0.12.5 2025-03-10 08:56:47 +00:00
yuyutaotao
212e4e3725
fix: planning prompt (#448)
* feat: add more case for llm planning

* fix: ai e2e

* chore: use debug to print log

* chore: fix error in gpt mode
2025-03-10 16:50:43 +08:00
Zhou Xiao
d128745e31
fix(esm): resolve cli can't load esm module (#445)
* fix(esm): resolve cli can't load esm module

* chore: resolve deps error

* chore: resolve deps error

* chore: resolve deps error

* chore: resolve deps error

* chore: resolve deps error

* chore: resolve deps error

* chore: resolve deps error

* chore: resolve deps error

* chore: resolve deps error

* chore: resolve deps error

* chore: resolve deps error

* chore: resolve deps error
2025-03-09 21:50:20 +08:00
Zhou Xiao
5d63ef9151
refactor: switch bundle type to bundleless (#437) 2025-03-07 17:20:18 +08:00
github-actions[bot]
0d5f88a456 release: v0.12.4 2025-03-03 06:26:00 +00:00
github-actions[bot]
cbb850bb01 release: v0.12.3 2025-02-28 03:04:37 +00:00
github-actions[bot]
e35f277935 release: v0.12.2 2025-02-27 10:38:20 +00:00
github-actions[bot]
bb56bbc248 release: v0.12.1 2025-02-26 09:15:45 +00:00
github-actions[bot]
9f1e491aa8 release: v0.12.0 2025-02-21 08:49:04 +00:00
yuyutaotao
bbe9874e78
fix: coord offset of qwen model (#407)
---------

Co-authored-by: zhouxiao.shaw <zhouxiao.shaw@bytedance.com>
2025-02-21 10:30:20 +08:00
yuyutaotao
59ce2d0140
feat: locate by coord (#383)
---------

Co-authored-by: zhouxiao.shaw <zhouxiao.shaw@bytedance.com>
2025-02-21 09:56:09 +08:00
github-actions[bot]
9dbb4b6b04 release: v0.11.3 2025-02-17 08:10:50 +00:00
github-actions[bot]
dd2163a0d1 release: v0.11.2 2025-02-15 02:22:04 +00:00
Zhou Xiao
edf0871fea
feat(browser): Add the forceSameTabNavigation configuration to prevent AI from opening new pages during operations, thus avoiding task interruptions. (#389)
* feat(tab-control): enhance the configuration to limit AI from opening new tabs during operations, preventing failures.

* chore: optimize evaluate error

* chore: resolve navigation error

* fix(browser): add forceSameTabNavigation config toe limit open new tab

* chore: upgrade vitest version

* fix: typo (#390)

---------

Co-authored-by: yuyutaotao <167746126+yuyutaotao@users.noreply.github.com>
2025-02-14 21:54:47 +08:00
github-actions[bot]
f7c583cdf5 release: v0.11.1 2025-02-13 06:50:24 +00:00
yuyutaotao
2f2400dffa
fix: correctly collect elements in absolute container (#373) 2025-02-10 20:51:43 +08:00
github-actions[bot]
d1e2e2b560 release: v0.11.0 2025-02-10 10:35:13 +00:00
yuyutaotao
2a28472fa5
feat: use different color for annotations (#366) 2025-02-10 16:36:12 +08:00
Zhou Xiao
bdff171da6
fix(ui-tars): resolve page down and page up event error (#370) 2025-02-10 16:35:03 +08:00
yuyutaotao
9d5f2fbcac
feat(web-extract): extract web content as a tree (#337)
* feat: extract web content as a tree

* chore: update test data

* chore: update test data

* feat: update answer of evaluation

* chore: update test cases

* chore: remove focusing on cases

* fix: ci

* fix: put rect in html tree

* fix: CI

* fix: AI test

* fix: lint

* fix: CI

* fix: static-page compatibility

* fix: CI

* fix: map by markerId

* fix: llm planning prompt

* chore: update hash length

* chore: ignore writing dump file

* fix: lint

* fix: ci snapshot

* chore: snapshot tree in web extractor

* chore: export tree utils in core

* chore: export tree utils in core

* fix: CI

* fix: update test case and evaluation

* chore: remove unused file

* refactor(extract): modify dependencies (#358)

* refactor(extract): modify dependencies

* chore: modify files config

* chore: add indexId as key for map

---------

Co-authored-by: Zhou Xiao <zhouxiao.shaw@bytedance.com>
2025-02-07 14:55:52 +08:00
github-actions[bot]
e86021535a release: v0.10.5 2025-01-31 03:07:52 +00:00
github-actions[bot]
6c908842f9 release: v0.10.4 2025-01-30 06:18:36 +00:00
github-actions[bot]
8eacc354a7 release: v0.10.3 2025-01-27 08:21:33 +00:00
Zhou Xiao
e3eb6f6fbc
fix: resolve shared exports error (#335) 2025-01-27 16:16:42 +08:00
github-actions[bot]
e6fc69eb81 release: v0.10.2 2025-01-26 12:39:17 +00:00
Zhou Xiao
9c88186540
feat(ui-tars): enhance the UI-TARS keyboard event handling and optimize parser logic (#330) 2025-01-26 20:34:56 +08:00
github-actions[bot]
0f92941f54 release: v0.10.1 2025-01-23 07:15:56 +00:00
yuyutaotao
3aa1b33955
feat: use jpeg as default image format (#301)
---------

Co-authored-by: zhouxiao.shaw <zhouxiao.shaw@bytedance.com>
2025-01-20 20:02:49 +08:00
github-actions[bot]
9b86067ea6 release: v0.10.0 2025-01-17 10:31:04 +00:00
github-actions[bot]
d580bd02fe release: v0.9.2 2025-01-15 03:22:10 +00:00
Zhou xiao
2b18ed55de
feat(ai-model): support vlm (#262)
* feat(ai-model): support plan to target

* chore: modify test

* chore: modify test

* chore: fix env config

* chore: unify the action logic

* chore: optimize type hint

* chore: optimize type hint

* chore: fix type hint

* chore: fix build type error

* chore: support open new tab

* feat: support auto complete element

* chore: add sleep event

* chore: add ai cost time

* chore: optimize prompt

* chore: optimize report prompt

* chore: optimize vlm name

* chore: fix command

* chore: optimize error handle and use check debugger list replace last tab id

* chore: fix chrome debugger attach logic

* chore: fix unit test
2025-01-13 14:32:17 +08:00
github-actions[bot]
d92bfa8987 release: v0.9.1 2025-01-11 05:17:05 +00:00
github-actions[bot]
bc9542d8df release: v0.9.0 2025-01-09 07:06:20 +00:00
github-actions[bot]
de92f3fc13 release: v0.8.18 2025-01-07 11:53:17 +00:00
github-actions[bot]
4fd930340c release: v0.8.17 2025-01-07 03:54:08 +00:00
Zhou xiao
691eb6ef0a
feat(ai-model): support Image positioning and integrate langchain (#230)
* feat: add point img logic

* feat: migrate prompt to langchain

* chore: delete unless log

* chore: optimize test case

* chore: fix lint error

* chore: delete httpAgent logic

* chore: delete unless fn

* chore: fix some comment

* chore: fix ci error

* chore: delete unless fn

* chore: update prompt

* chore: delete unless language
2025-01-02 21:23:30 +08:00
github-actions[bot]
1e7c9caf1a release: v0.8.16 2025-01-02 02:38:32 +00:00
yuyutaotao
8d83debd13
fix: add some default value for tmp dir #231 (#233) 2025-01-02 10:18:44 +08:00
github-actions[bot]
85cb87f1ec release: v0.8.15 2025-01-01 13:15:36 +00:00