54 Commits

Author SHA1 Message Date
yuyutaotao
999e52ebc1
feat: update prompt for qwen (#491) 2025-03-24 19:47:19 +08:00
yuyutaotao
649aeceb43
feat: enable search area for locate (#473)
* feat: enable search area for locate

* fix: update evaluation

* fix: build error

* fix: ci

* fix: locator

* feat: show searchArea in report

* chore: add yaml support for aiTap

* feat: update status tip

* fix: #473 (#484)

* chore: optimize unit test list

---------

Co-authored-by: zhouxiao.shaw <zhouxiao.shaw@bytedance.com>
2025-03-24 09:50:27 +08:00
yuyutaotao
8e1ba565d0
feat: optimize locator (#456)
---------

Co-authored-by: zhouxiao.shaw <zhouxiao.shaw@bytedance.com>
2025-03-17 19:19:54 +08:00
Zhou Xiao
5d63ef9151
refactor: switch bundle type to bundleless (#437) 2025-03-07 17:20:18 +08:00
yuyutaotao
a6ffbd07b7
fix: over planning for Qwen in page with form (#429) 2025-02-27 16:44:01 +08:00
yuyutaotao
724a2c7230
fix(core): the prompt of qwen (#427)
* fix: fix the  prompt of qwen

* docs: update debugging config

* fix: sleep in qwen

---------

Co-authored-by: zhouxiao.shaw <zhouxiao.shaw@bytedance.com>
2025-02-26 15:04:28 +08:00
Zhaopeng XUAN
111cf25d31
feat(core/ui-tars): support aiAssert() & aiWaitFor for UI-Tars model (#412)
* Issue-41 Support aiAssert() for ui-tars model

---------
Co-authored-by: zhouxiao.shaw <zhouxiao.shaw@bytedance.com>
2025-02-25 14:49:14 +08:00
yuyutaotao
3f438f6c0c
fix: ci of qwen model (#410) 2025-02-21 16:15:51 +08:00
yuyutaotao
59ce2d0140
feat: locate by coord (#383)
---------

Co-authored-by: zhouxiao.shaw <zhouxiao.shaw@bytedance.com>
2025-02-21 09:56:09 +08:00
Zhou Xiao
f661a1882e
fix(report): resolve no locate generate error (#392) 2025-02-15 12:00:57 +08:00
Zhou Xiao
8e69f8d151
fix(keybord): resolve selectAll/Copy/Paste event (#384)
* fix(keybord): resolve selectAll/Copy/Paste event

* chore: fix e2e test

* chore: upgrade deps

* chore: add keybord commands  ref
2025-02-13 15:53:53 +08:00
Zhou Xiao
195eaad903
feat(cache): supports ui-tars model caching capability (#361) 2025-02-08 13:05:17 +08:00
yuyutaotao
9d5f2fbcac
feat(web-extract): extract web content as a tree (#337)
* feat: extract web content as a tree

* chore: update test data

* chore: update test data

* feat: update answer of evaluation

* chore: update test cases

* chore: remove focusing on cases

* fix: ci

* fix: put rect in html tree

* fix: CI

* fix: AI test

* fix: lint

* fix: CI

* fix: static-page compatibility

* fix: CI

* fix: map by markerId

* fix: llm planning prompt

* chore: update hash length

* chore: ignore writing dump file

* fix: lint

* fix: ci snapshot

* chore: snapshot tree in web extractor

* chore: export tree utils in core

* chore: export tree utils in core

* fix: CI

* fix: update test case and evaluation

* chore: remove unused file

* refactor(extract): modify dependencies (#358)

* refactor(extract): modify dependencies

* chore: modify files config

* chore: add indexId as key for map

---------

Co-authored-by: Zhou Xiao <zhouxiao.shaw@bytedance.com>
2025-02-07 14:55:52 +08:00
Zhou Xiao
9c88186540
feat(ui-tars): enhance the UI-TARS keyboard event handling and optimize parser logic (#330) 2025-01-26 20:34:56 +08:00
Zhou Xiao
839dc6c799
feat(browser-event): support drag event (#321) 2025-01-26 12:00:55 +08:00
Zhou xiao
4c82eff526
chore: fix e2e test (#316)
* chore: fix e2e test

* chore: ignore todo test

* chore: fix e2e test

* chore: upgrade gpt api version

* chore: fix e2e test

* chore: fix e2e test

* chore: fix direction

* chore: fix direction

* chore: split test

* chore: split test

* chore: modify input info

* fix: scroll param in ci

* fix: lint

* fix: ai test

* fix: ai test

---------

Co-authored-by: yutao <yutao.tao@bytedance.com>
2025-01-24 15:02:50 +08:00
yuyutaotao
fb2b9d123f
feat(chrome-devtool): add 'stop' button in extension (#281)
* feat: add 'stop' to playground

* feat: make extension stopable

---------

Co-authored-by: zhouxiao.shaw <zhouxiao.shaw@bytedance.com>
2025-01-15 19:18:26 +08:00
yuyutaotao
857c411d67
feat: allow tracking newly-opened tabs in Chrome extension (#272) 2025-01-14 11:22:20 +08:00
Zhou xiao
56845b75ba
chore: fix planning typo (#269) 2025-01-13 14:52:15 +08:00
Zhou xiao
2b18ed55de
feat(ai-model): support vlm (#262)
* feat(ai-model): support plan to target

* chore: modify test

* chore: modify test

* chore: fix env config

* chore: unify the action logic

* chore: optimize type hint

* chore: optimize type hint

* chore: fix type hint

* chore: fix build type error

* chore: support open new tab

* feat: support auto complete element

* chore: add sleep event

* chore: add ai cost time

* chore: optimize prompt

* chore: optimize report prompt

* chore: optimize vlm name

* chore: fix command

* chore: optimize error handle and use check debugger list replace last tab id

* chore: fix chrome debugger attach logic

* chore: fix unit test
2025-01-13 14:32:17 +08:00
yuyutaotao
8479bcb652
feat: bridge mode - phase 2 (#257)
* feat: allow auto reconnect in bridge mode after disconnect

* feat: show agent progress in extension

* feat: allow using bridge mode in yaml

* doc: add doc for bridge mode

* doc: add doc for bridge mode

* docs: update screenshot
2025-01-10 10:55:41 +08:00
Zhou xiao
691eb6ef0a
feat(ai-model): support Image positioning and integrate langchain (#230)
* feat: add point img logic

* feat: migrate prompt to langchain

* chore: delete unless log

* chore: optimize test case

* chore: fix lint error

* chore: delete httpAgent logic

* chore: delete unless fn

* chore: fix some comment

* chore: fix ci error

* chore: delete unless fn

* chore: update prompt

* chore: delete unless language
2025-01-02 21:23:30 +08:00
yuyutaotao
3c9418c264
fix: use some default param in scroll (#232)
* fix: use some default param in scroll

* fix: update param description
2025-01-02 10:19:22 +08:00
yuyutaotao
21b3574280
fix: fix scroll action in chrome extension (#194) 2024-12-23 12:11:10 +08:00
yuyutaotao
f3d46b504d
feat: invoke anthropic SDK to call Claude (#197)
* feat: invoke anthropic SDK

* chore: set response format for extract

* fix: do not throw if waitUntilNetworkIdle failed in aiAction

* fix: timeout config for Puppeteer

* chore: add instruction for connectivity test
2024-12-23 12:03:05 +08:00
yuyutaotao
523adab12f
feat: support the if-statement in planning prompt (#184) 2024-12-19 10:44:08 +08:00
yuyutaotao
537a5c43d0
feat: add a real-time progress indicator for playground (#177) 2024-12-16 15:04:21 +08:00
yuyutaotao
082e347da9
feat: replanning before locating (#167) 2024-12-08 20:12:17 +08:00
yuyutaotao
e7a58b3b24
faet: support yaml style scripts (#165) 2024-11-25 16:05:01 +08:00
Zhou xiao
08d46becf1
feat(cache): optimize cache logic id and export overrideAIConfig (#148)
* chore(cache): optimize cache logic id and export overrideAIConfig

* chore: use cache logic
2024-11-07 21:06:01 +08:00
yuyutaotao
198172dc4e
feat: optimize the speed of screenshot in browser (#144)
* feat: optimize the speed of screenshot in browser

* feat: remove unnecessary context call

* fix: CI
2024-11-05 14:28:16 +08:00
yuyutaotao
9e1eef5cfd
feat: Build a chrome extension for playground (#140) 2024-11-05 11:49:21 +08:00
Zhou xiao
adb9b58879
feat(ai-model): add claude computer ablity (#136)
* Add new changes

* Add computer test results and update AI evaluation tests

* chore: Update build outputs and configurations

* feat(ai-model): support claude computer ability use position replace element id

* feat: generate add and commit

* feat: implement computer ability test for Claude

* chore: fix build Lose

* chore: Add and commit changes

* chore: optimize ai position

* chore: optimize ai position

* Add AI evaluation results and update tests

* chore: optimize ai test

* chore: add and commit changes

* chore: optimize ai test content

* chore: fix test case

* chore: fix e2e test
2024-10-31 18:18:31 +08:00
yuyutaotao
c288baa448
feat: make playground working in the browser (#135)
---------

Co-authored-by: zhouxiao.shaw <zhouxiao.shaw@bytedance.com>
2024-10-28 11:04:40 +08:00
Zhou xiao
3b24df1ec0
workflow(ci): fix ai test error (#113)
* chore: update element inspector prompt and schema

* test: update snapshots for planning-input tests
2024-10-12 16:01:07 +08:00
yuyutaotao
f9dc0f698e
feat(ai-model): merge ai planning and insight call to accelerate the aiAction (#97)
---------

Co-authored-by: zhouxiao.shaw <zhouxiao.shaw@bytedance.com>

* feat(ai-model): optimize AI model for element inspection

* feat(ai-model): optimize AI model and add quick answer functionality

---------

Co-authored-by: yuyutaotao <167746126+yuyutaotao@users.noreply.github.com>

* feat(ai-model): implement quick answer functionality for element inspection

---------

Co-authored-by: zhouxiao.shaw <zhouxiao.shaw@bytedance.com>
2024-10-12 12:09:25 +08:00
Leyang
43bf165ddc
define appium export in package.json & encapsulated screenshot picture path generation (#95)
* feat: define appium export in package.json

* feat: encapsulated screenshot picture path generation
2024-09-23 10:57:19 +08:00
Zhou xiao
d481ea4201
feat(model): support azure open ai (#90)
* feat(model): support open ai azure methods

* chore: fix e2e test

* chore: add OPENAI_USE_AZURE env config

* docs: add openai azure env
2024-09-10 14:29:01 +08:00
Zhou xiao
eafa5bfa20
feat(cache): The cache is generalized to support puppeteers and mobile terminals (#85)
* feat(cache): The cache is generalized to support Puppeteers and mobile terminals

* chore: update cache test

* chore: update cache test

* chore: update cache test

* docs: update cache doc

* chore: update ai test command

* chore: update ai test command

* chore: update ai test command

* chore: optimize cache logic

* chore: update get dir path logic

* chore: update get dir path logic
2024-09-06 17:19:35 +08:00
Leyang
cfa92b3980
feat(app): supports control of iOS and Android devices through appium (#82)
Co-authored-by: zhouxiao.shaw <zhouxiao.shaw@bytedance.com>
Co-authored-by: linyibing <linyibing@bytedance.com>
2024-09-05 20:05:19 +08:00
Zhou xiao
84ee7130b8
feat(web): input supports additions, deletions, and changes (#78)
* feat(web): input supports additions, deletions, and changes

* chore: optimize unit test content
2024-08-28 19:31:59 +08:00
yuyutaotao
f8fdf60570
fix(web-extract): fix the extractor may fail if there is no <body/> (#76) 2024-08-28 19:21:32 +08:00
Zhou xiao
e071adfa70
chore: optimize code style (#77) 2024-08-28 14:59:25 +08:00
Zhou xiao
d2a5dbecba
refactor(shared): migrate sharp to jimp and migrate common img logci to shared lib (#74)
* fix(web): fix sharp deps

* chore: optimize sharp deps

* refactor(extract): migrate sharp to jimp

* refactor: migrate img common logic to shared lib

* chore: merge main branch

* chore: merge main branch

* chore: merge main branch

* chore: delete unless code

* chore: optimize code

* chore: optimize ai test branch trigger method

* chore: optimize ai test branch trigger method

* chore: optimize trigger method
2024-08-26 18:50:33 +08:00
yuyutaotao
6553da1906
feat: add 'aiWaitFor' (#61)
* feat: add

* feat: add

* feat: add  for playwright

* feat: add docs for 'aiWaitFor'

* feat: update docs for report

* feat: add 'wait-for' param in cli
2024-08-21 14:43:35 +08:00
yuyutaotao
af768796df
fix: Cli output, extractor, docs, visualizer (#48)
* fix: the output for --help

* fix: filter little elements

* feat: update executor and style

* feat: update extractor and docs
2024-08-10 07:57:15 +08:00
Zhou xiao
959473fbbb
feat(ai-model): support coze platfrom's model (#50)
* chore: optimize prompt structure

* feat(ai-model): add coze platform

* feat(ai-model): add coze platform

* chore: add unit test

* feat(ai-model): add extract unit test

* chore: add unit test

* chore: add unit test

* chore: optimize plan prompt

* chore: optimize plan prompt

* chore: fix lint error

* chore: fix lint error

* chore: fix unit test

* chore: ignore cache file
2024-08-09 21:37:41 +08:00
yuyutaotao
41058a015f
feat: implement the 'sleep' action (#41) 2024-08-07 20:03:13 +08:00
yuyutaotao
7edc2be46d
feat: implement .aiAssert, update some docs (#38)
* feat: implement .aiAssert, update some docs

* fix: lint

* fix: ci

* feat: update quick-start
2024-08-06 10:00:25 +08:00
Zhou xiao
e1ae31ac17
workflow(ci): optimize biome lint (#31)
* workflow(ci): format code

* workflow(ci): format code

* workflow(ci): format code

* workflow(ci): format code

* workflow(ci): format code
2024-08-04 08:28:19 +08:00