35 Commits

Author SHA1 Message Date
Leyang
7b190ae4ee
fix(web-integration): sanitize content for reportHtmlContent (#745)
* fix(web-integration): sanitize content for reportHtmlContent

* fix(workflow): version

* fix(web-integration): recover after retrieve

* fix(web-integration): test

* chore(core): fix lint

* fix(report): remove ! for content

* chore(core): add test case

* chore(core): add test case

---------

Co-authored-by: yutao <yutao.tao@bytedance.com>
2025-05-22 07:20:32 +08:00
yuyutaotao
08466cac1e
feat(core): add element describer (#750) 2025-05-21 21:05:47 +08:00
yuyutaotao
610e7979bf
fix(core): error in report file when planning failed (#747)
* fix(core): error in report file when planning failed

* chore(core): add testcase for aiQuery
2025-05-21 20:36:22 +08:00
Leyang
4718066bb1
feat(web-integration): add support for new agent method(aiBoolean, aiString, aiNumber, aiLocate) (#658)
* feat(web-integration): add support for new agent method(aiBoolean, aiString, aiNumber, aiLocate)

* fix(core, web-integration): update data extraction logic and refine return types for agent methods

* feat(site): add detailed descriptions and examples

* fix(web-integration): remove unused properties from aiLocate return type and update UI titles

* fix(core): enhance agent methods for better extraction and querying

* fix(core): update test scripts and refine data extraction prompts

* fix(core): refine return type in ExecutionTaskApply and remove unused import in inspect

* fix(core): update data extraction logic and enhance type handling in Insight class

* fix(web-integration): update import path for puppeteerAgentForTarget in player test file

* fix(core): tests fix

* fix(core): fix tests

* fix(shared): enhance imageInfo tests by validating individual properties and updating snapshots

* fix(site): remove waitForNavigationTimeout from Playwright integration examples

* fix(site): update timeout configuration examples in FAQ for clarity

* test(web-integration): remove unused client-extractor test and related snapshots

* chore(core): fix query implementation

* fix(report): improve data handling in DetailSide component and enhance output display

* fix(core): refine data extraction prompts and improve error handling in PageTaskExecutor

* fix(core): update data extraction prompt for clarity and type validation

---------

Co-authored-by: yutao <yutao.tao@bytedance.com>
2025-04-29 13:36:49 +08:00
Leyang
ca644d8914
feat(core): allow custom midscene_run dir (#631)
* feat(core): support custom midscene_run dir

* feat(report): add search functionality to PlaywrightCaseSelector component

* refactor(shared): simplify base directory resolution and remove unused environment variable

* feat(shared): integrate shared environment variables across multiple packages

* refactor(shared): update base directory resolution to use dynamic midscene_run directory

* fix(puppeteer): increase screenshot timeout from 3s to 10s for improved reliability
2025-04-24 22:54:52 +08:00
yuyutaotao
0c0675ba4b
feat: update deepThink interface (#517)
* feat: update deepThink interface

* fix: CI

* chore: update error message

* feat: update search area config

* fix: deepthink ui

---------

Co-authored-by: Zhou Xiao <zhouxiao.shaw@bytedance.com>
2025-04-02 19:26:56 +08:00
yuyutaotao
adc3fd9e0d
feat: use aiTap(prompt, opt) style interface (#512) 2025-04-01 10:41:31 +08:00
yuyutaotao
649aeceb43
feat: enable search area for locate (#473)
* feat: enable search area for locate

* fix: update evaluation

* fix: build error

* fix: ci

* fix: locator

* feat: show searchArea in report

* chore: add yaml support for aiTap

* feat: update status tip

* fix: #473 (#484)

* chore: optimize unit test list

---------

Co-authored-by: zhouxiao.shaw <zhouxiao.shaw@bytedance.com>
2025-03-24 09:50:27 +08:00
yuyutaotao
8e1ba565d0
feat: optimize locator (#456)
---------

Co-authored-by: zhouxiao.shaw <zhouxiao.shaw@bytedance.com>
2025-03-17 19:19:54 +08:00
yuyutaotao
2c5ea87131
fix: ai test (#460)
* fix: ai test

* fix: ci test

* fix: evaluation test

* fix: test

* fix: test

* fix: ai test

* fix: ai test
2025-03-12 13:49:50 +08:00
yuyutaotao
d0a3d86e22
docs: update instruction for bridge mode (#428)
* docs: update instruction for bridge mode

* fix: ai test
2025-02-26 17:06:14 +08:00
Zhou Xiao
1eefad4422
docs: optimize agent api doc (#415)
* docs: optimize agent api doc

* docs: optimize runyaml link

* docs: optimize prompt
2025-02-24 14:29:17 +08:00
yuyutaotao
3f438f6c0c
fix: ci of qwen model (#410) 2025-02-21 16:15:51 +08:00
yuyutaotao
59ce2d0140
feat: locate by coord (#383)
---------

Co-authored-by: zhouxiao.shaw <zhouxiao.shaw@bytedance.com>
2025-02-21 09:56:09 +08:00
Zhou Xiao
edf0871fea
feat(browser): Add the forceSameTabNavigation configuration to prevent AI from opening new pages during operations, thus avoiding task interruptions. (#389)
* feat(tab-control): enhance the configuration to limit AI from opening new tabs during operations, preventing failures.

* chore: optimize evaluate error

* chore: resolve navigation error

* fix(browser): add forceSameTabNavigation config toe limit open new tab

* chore: upgrade vitest version

* fix: typo (#390)

---------

Co-authored-by: yuyutaotao <167746126+yuyutaotao@users.noreply.github.com>
2025-02-14 21:54:47 +08:00
yuyutaotao
9d5f2fbcac
feat(web-extract): extract web content as a tree (#337)
* feat: extract web content as a tree

* chore: update test data

* chore: update test data

* feat: update answer of evaluation

* chore: update test cases

* chore: remove focusing on cases

* fix: ci

* fix: put rect in html tree

* fix: CI

* fix: AI test

* fix: lint

* fix: CI

* fix: static-page compatibility

* fix: CI

* fix: map by markerId

* fix: llm planning prompt

* chore: update hash length

* chore: ignore writing dump file

* fix: lint

* fix: ci snapshot

* chore: snapshot tree in web extractor

* chore: export tree utils in core

* chore: export tree utils in core

* fix: CI

* fix: update test case and evaluation

* chore: remove unused file

* refactor(extract): modify dependencies (#358)

* refactor(extract): modify dependencies

* chore: modify files config

* chore: add indexId as key for map

---------

Co-authored-by: Zhou Xiao <zhouxiao.shaw@bytedance.com>
2025-02-07 14:55:52 +08:00
Zhou Xiao
f8744e441a
fix(prompt): resolve the llm-planning format error (#341) 2025-01-30 14:14:14 +08:00
yuyutaotao
6d924e267f
feat(puppeteer): allow tracking new tab in puppeteer agent (#310) 2025-01-26 16:49:32 +08:00
Zhou xiao
4c82eff526
chore: fix e2e test (#316)
* chore: fix e2e test

* chore: ignore todo test

* chore: fix e2e test

* chore: upgrade gpt api version

* chore: fix e2e test

* chore: fix e2e test

* chore: fix direction

* chore: fix direction

* chore: split test

* chore: split test

* chore: modify input info

* fix: scroll param in ci

* fix: lint

* fix: ai test

* fix: ai test

---------

Co-authored-by: yutao <yutao.tao@bytedance.com>
2025-01-24 15:02:50 +08:00
yuyutaotao
ed45cfab6c
fix(action): keyboard and div extracting (#304)
* fix: clearInput in browser

* fix: input issue on mac

* fix: playwright input

* fix: issue when collecting fixed item
---------

Co-authored-by: zhouxiao.shaw <zhouxiao.shaw@bytedance.com>

* fix: extract fixed style content

* fix: extract fixed style content

* fix: customize window size in yaml
2025-01-21 19:21:09 +08:00
yuyutaotao
8479bcb652
feat: bridge mode - phase 2 (#257)
* feat: allow auto reconnect in bridge mode after disconnect

* feat: show agent progress in extension

* feat: allow using bridge mode in yaml

* doc: add doc for bridge mode

* doc: add doc for bridge mode

* docs: update screenshot
2025-01-10 10:55:41 +08:00
yuyutaotao
4a82e9bda9
feat: let max_tokens configurable (#212)
* feat: let max_tokens configurable

* fix: update ci test case
2024-12-26 13:24:21 +08:00
yuyutaotao
523adab12f
feat: support the if-statement in planning prompt (#184) 2024-12-19 10:44:08 +08:00
yuyutaotao
537a5c43d0
feat: add a real-time progress indicator for playground (#177) 2024-12-16 15:04:21 +08:00
yuyutaotao
082e347da9
feat: replanning before locating (#167) 2024-12-08 20:12:17 +08:00
yuyutaotao
4282e7bf07
feat: lower the cost of page description (#159) 2024-11-20 16:00:34 +08:00
Zhou xiao
08d46becf1
feat(cache): optimize cache logic id and export overrideAIConfig (#148)
* chore(cache): optimize cache logic id and export overrideAIConfig

* chore: use cache logic
2024-11-07 21:06:01 +08:00
yuyutaotao
f8507efd50
chore: some minor bug fixes (#146)
* feat: always use latest process.env as fallback

---------

Co-authored-by: zhouxiao.shaw <zhouxiao.shaw@bytedance.com>
2024-11-07 11:29:50 +08:00
Zhou xiao
1eb4c8de9f
workflow(ci): fix ci error (#142) 2024-11-04 17:14:14 +08:00
yuyutaotao
ee521e31e0
fix: fix the Player style when the locator is failed (#129)
* fix: fix the Player style when the locator is failed

* fix: add marker image into UIContext
2024-10-17 10:44:30 +08:00
yuyutaotao
f9dc0f698e
feat(ai-model): merge ai planning and insight call to accelerate the aiAction (#97)
---------

Co-authored-by: zhouxiao.shaw <zhouxiao.shaw@bytedance.com>

* feat(ai-model): optimize AI model for element inspection

* feat(ai-model): optimize AI model and add quick answer functionality

---------

Co-authored-by: yuyutaotao <167746126+yuyutaotao@users.noreply.github.com>

* feat(ai-model): implement quick answer functionality for element inspection

---------

Co-authored-by: zhouxiao.shaw <zhouxiao.shaw@bytedance.com>
2024-10-12 12:09:25 +08:00
Zhou xiao
10757a8ba1
refacotr(ai-model): optimize model evalution method (#98) 2024-09-29 17:16:07 +08:00
yuyutaotao
7d9792c651
fix(ci): fix build error for Cli, remove used package (#89) 2024-09-09 18:07:22 +08:00
yuyutaotao
30dc7a3c22
chore(web): merge the Page implementation of Puppeteer and Playwright (#87) 2024-09-09 15:57:36 +08:00
Zhou xiao
eafa5bfa20
feat(cache): The cache is generalized to support puppeteers and mobile terminals (#85)
* feat(cache): The cache is generalized to support Puppeteers and mobile terminals

* chore: update cache test

* chore: update cache test

* chore: update cache test

* docs: update cache doc

* chore: update ai test command

* chore: update ai test command

* chore: update ai test command

* chore: optimize cache logic

* chore: update get dir path logic

* chore: update get dir path logic
2024-09-06 17:19:35 +08:00