149 Commits

Author SHA1 Message Date
yuyutaotao
08466cac1e
feat(core): add element describer (#750) 2025-05-21 21:05:47 +08:00
Thulof Qu
1a4eabd577
docs(common): fix broken relative links (#746) 2025-05-21 19:24:11 +08:00
yuyutaotao
1589d10f35
docs(core): update blog (#735)
* docs(core): update blog

* docs(blog): update workflow blog

* docs(site): add details and pic

* docs(site): update pic

* docs(site): update pic 2

---------

Co-authored-by: quanruzhuoxiu <quanruzhuoxiu@gmail.com>
2025-05-21 16:49:26 +08:00
Leyang
5a1a3ba18a
feat(web-integration): support disable cache for a single api call (#740)
* feat(web-integration): support disable cache for a single api call

* feat(workflow): version mismatch

* feat(web-integration): cache rename to cacheable

* feat(web-integration): add cacheable option to multiple API methods and update caching documentation

* docs(site): update cacheable option descriptions to reference caching feature documentation

* docs(core): update caching doc

---------

Co-authored-by: yutao <yutao.tao@bytedance.com>
2025-05-21 16:46:20 +08:00
89fd77b96a
docs(llm): add doubao-1.5-vl-pro model (#719)
* docs(llm): add doubao-1.5-vl-pro model

* docs(site): move model comparison to choose a model doc

* docs(core): update doubao model

---------

Co-authored-by: yutao <yutao.tao@bytedance.com>
2025-05-19 15:56:28 +08:00
Wang Fanghua
c407921449
fix(site): add aiScroll import for playwright example docs. (#726) 2025-05-19 10:45:34 +08:00
yuyutaotao
b261ed7f2a
feat(web): use xpath and yaml as cache (#711)
* feat(web-integration): use xpath for cache instead of id

* feat(web-integration): enhance TaskCache to support xpaths for cache matching and add new test cases

* feat(web-integration): add debug log for unknown page types in TaskCache

* feat(web-integration): update caching logic and cache hit conditions for Plan and Locate tasks

* chore(core): update debug log

* feat(web-integration): update rspress.config and enhance TaskCache structure with new properties

* feat(web-integration): recalculate id when hit cache

* fix(web-integration): update mock implementation in task-cache test to use evaluate method

* feat(web-integration): enhance element caching by adding XPath support and improving cache hit logic

* chore(core): lint

* feat(web-integration): improve XPath handling in web-extractor

* test(web-integration): fix tests

* feat(core, web-integration): add attributes to LocateResultElement and enhance element handling

* fix(core): lint

* feat(web-integration): add midsceneVersion to TaskCache and update cache validation logic

* fix(core): test

* fix(web-integration): update cache validation logic to prevent reading outdated midscene cache files

* feat(web-integration): enhance TaskCache to track used cache items and improve cache retrieval logic

* fix(core): xpath logic (#710)

* feat(core): resue context for locate

* feat(core): build yamlFlow from aiAction

* feat(core): refine task-cache

* feat(core): update cache

* feat(core): refine task-cache

* feat(core): refine task-cache

* feat(core): remove unused checkElementExistsByXPath

* feat(core): use yaml file as cache

* chore(core): fix lint

* chore(core): print warning for previous cache

* refactor(core): remove quickAnswer references and improve element matching logic

* fix(core): update import path for buildYamlFlowFromPlans

* chore(web-integration): update output image and skip task error test

* fix(web-integration): update test snapshots to handle beta versions

* fix(web-integration): adjust test snapshots for version consistency

* fix(web-integration): track original cache length and adjust matching logic in tests

* fix(web-integration): update test URLs to reflect new target site and enable previously skipped test

* chore(core): update cache docs

* fix(core): test

* feat(core): try to match element from plan

* fix(web-integration): cache id stable when retry in palywright

* fix(web-integration): typo

* style(web-integration): lint

* fix(web-integration): stable cacheid in tests

* fix(web-integration): cache id

---------

Co-authored-by: quanruzhuoxiu <quanruzhuoxiu@gmail.com>
2025-05-16 17:16:56 +08:00
Leyang
ab4fcfe220
docs(site): blog: Key to Automation Stability: Programming Practices Using Structured API (#709)
* docs(site): blog: Key to Automation Stability: Programming Practices Using Structured API

* docs(core): update blog

* docs(site): add blog post on optimizing automation code with structured API

---------

Co-authored-by: zhouxiao.shaw <zhouxiao.shaw@bytedance.com>
Co-authored-by: yutao <yutao.tao@bytedance.com>
2025-05-16 14:20:11 +08:00
f99582ad66
docs(llm): add llm.txt (#694)
* docs(llm): add llm.txt

* docs(llm): delete unless code

* docs: modify sidebar
2025-05-08 17:36:38 +08:00
iyaozhen
cd05fb3a98
docs(site): fix volcengine version USE_VLM_UI_TARS value and modify 火山云 to 火山引擎 (#692)
Co-authored-by: yaozhen.00 <yaozhen.00@bytedance.com>
2025-05-07 19:54:23 +08:00
Leyang
8661c97dcf
docs(site): update timeout parameters in API to include options for disabling timeouts (#686)
* docs(site): update timeout parameters in API to include options for disabling timeouts

* docs(core): update docs for timeout

---------

Co-authored-by: yutao <yutao.tao@bytedance.com>
2025-05-06 11:31:06 +08:00
yuyutaotao
c1bc73c78b
feat(android): customize adb path (#684)
* feat(shared): add custom adb path
feat(android): add custom adb path

* feat(android): add docs for custom adb path

---------

Co-authored-by: HBLADEH <1012582116@qq.com>
2025-04-30 17:16:38 +08:00
yuyutaotao
6a8ccbbde4
fix(core): id conflict in quick answer (#679)
* fix(core): id conflict in quick answer

* fix(core): lint

* docs(core): update docs for qwen

* fix(core): test case

* refactor(core): refactor elementById args

* refactor(core): refactor elementById args

---------

Co-authored-by: zhouxiao.shaw <zhouxiao.shaw@bytedance.com>
2025-04-30 16:51:31 +08:00
yuyutaotao
b8f29e8e66
fix(core): use unified config for doubao-ui-tars model (#678) 2025-04-29 21:39:58 +08:00
yuyutaotao
2c13c082e5
docs(core): update docs for aiQuery (#673) 2025-04-29 19:57:51 +08:00
yuyutaotao
3506d8f1e3
docs(core): update profiling config (#677) 2025-04-29 19:57:30 +08:00
Leyang
4718066bb1
feat(web-integration): add support for new agent method(aiBoolean, aiString, aiNumber, aiLocate) (#658)
* feat(web-integration): add support for new agent method(aiBoolean, aiString, aiNumber, aiLocate)

* fix(core, web-integration): update data extraction logic and refine return types for agent methods

* feat(site): add detailed descriptions and examples

* fix(web-integration): remove unused properties from aiLocate return type and update UI titles

* fix(core): enhance agent methods for better extraction and querying

* fix(core): update test scripts and refine data extraction prompts

* fix(core): refine return type in ExecutionTaskApply and remove unused import in inspect

* fix(core): update data extraction logic and enhance type handling in Insight class

* fix(web-integration): update import path for puppeteerAgentForTarget in player test file

* fix(core): tests fix

* fix(core): fix tests

* fix(shared): enhance imageInfo tests by validating individual properties and updating snapshots

* fix(site): remove waitForNavigationTimeout from Playwright integration examples

* fix(site): update timeout configuration examples in FAQ for clarity

* test(web-integration): remove unused client-extractor test and related snapshots

* chore(core): fix query implementation

* fix(report): improve data handling in DetailSide component and enhance output display

* fix(core): refine data extraction prompts and improve error handling in PageTaskExecutor

* fix(core): update data extraction prompt for clarity and type validation

---------

Co-authored-by: yutao <yutao.tao@bytedance.com>
2025-04-29 13:36:49 +08:00
cd358bdac3
docs(mcp): optimize mcp showcase and add how to get report (#662)
* docs(mcp): optimize mcp showcase and add how to get report

* docs: optimize docs
2025-04-28 14:18:09 +08:00
yuyutaotao
58c5cc0989
docs(core): update docs for doubao vl model (#661) 2025-04-28 13:33:56 +08:00
yuyutaotao
5fb208a08c
feat(core): adapt UI tars 1.5 (#616)
* feat(core): adapt ui-tars 1.5

* chore(core): adaptr ui-tars-1.5

* chore(core): fix lint

* fix(core): env building issue

* fix(core): update import for uiTarsModelVersion from shared env

* feat(core): ui-tars hotkey event

* chore(core): move @ui-tars/action-parser to devDependencies

* fix(core): adapting new model
2025-04-28 08:42:43 +08:00
d02452d879
docs(mcp): add mcp doc (#656) 2025-04-27 22:35:00 +08:00
Leyang
ca644d8914
feat(core): allow custom midscene_run dir (#631)
* feat(core): support custom midscene_run dir

* feat(report): add search functionality to PlaywrightCaseSelector component

* refactor(shared): simplify base directory resolution and remove unused environment variable

* feat(shared): integrate shared environment variables across multiple packages

* refactor(shared): update base directory resolution to use dynamic midscene_run directory

* fix(puppeteer): increase screenshot timeout from 3s to 10s for improved reliability
2025-04-24 22:54:52 +08:00
Leyang
f85cd6cd1b
feat(playwright): enhance report filename generation with unique identifiers and support split mode (#629)
* feat(playwright): enhance report filename generation with unique identifiers and support split mode

* fix(playwright): update reporter type from "single" to "merged" and adjust related configurations

* chore(playwright): improve filename sanitization in reporter

* feat(web-integration): add function to replace illegal path characters

* feat(web-integration): update path sanitization function to replace spaces and illegal characters
2025-04-24 22:42:08 +08:00
yuyutaotao
ce7929bbbc
docs(android): update doc for quick start (#632) 2025-04-24 18:01:41 +08:00
Leyang
03a597e022
feat(web-integration): enhance timeout configurations and logging for network idle and navigation (#624)
* feat(web-integration): enhance timeout configurations and logging for network idle and navigation

* fix(web-integration): refine timeout warning messages and remove unnecessary test files

* feat(site): add network timeout customization details and additional parameters for Puppeteer

* fix(site): update default timeout values and enhance customization options for network idle in YAML

* fix(site): remove redundant timeout customization details in FAQ documentation

* fix(web-integration): enhance Playwright agent to support network idle functionality

* docs(playwright): update config docs

* docs(playwright): update config docs

* fix(web-integration): refactor network idle handling in Playwright agent

---------

Co-authored-by: yutao <yutao.tao@bytedance.com>
2025-04-24 10:28:26 +08:00
Leyang
fdbdb7d426
style(chrome-extension): opt chrome logo icon (#614) 2025-04-22 15:39:17 +08:00
yuyutaotao
e7f185d7b4
feat(core): adapt for gemini-2.5-pro (#605)
* feat(core): introduce gemini as vl

* feat(core): introduce gemini

* docs(core): update gemini docs

* docs(core): docs for gemini

---------

Co-authored-by: zhouxiao.shaw <zhouxiao.shaw@bytedance.com>

* docs(core): update quick start for models

* chore(core): update docs
2025-04-22 10:26:08 +08:00
yuyutaotao
0406c598d2
docs(core): update model config doc (#613)
* docs(core): update model config doc

* docs(core): update model-chosen doc
2025-04-22 10:19:50 +08:00
yuyutaotao
27634e0c39
docs(android): enhance automation documentation for Lynx compatibility (#611) 2025-04-21 23:25:13 +08:00
Leyang
2589a9c4ca
docs(android): update android docs (#607)
* docs: release android automation

* chore(docs): update doubao docs

* chore(docs): merge docs for doubao

* docs(android): update

* docs(site): add more android case

* docs(site): update slogan and authors

* docs(site): android yaml

* docs(core): instruction for override config

* docs(core): update readme

* Update README.md

* docs(core): update readme

* docs(core): update readme

* docs(core): update readme

* docs(core): update readme

* docs(core): update README and blog for Android automation support

* docs(core): update android playground doc

* docs(core): enhance Android integration documentation with setup instructions

* docs(core): update android playground doc

* docs(core): update Android integration documentation and add setup instructions

* docs(core): update bridge mode title

* docs(core): update yaml docs

* docs(site): chore update

* docs(site): update YAML documentation with setup instructions and clarify parameters

* docs(core): update instructions

* chore: update docs

* chore: update bridge mode docs

* docs(site): translate to zh

* docs(site): translate error

* docs(site): remove unnecessary code block in YAML automation documentation

* docs(core): update blog

* docs(core): update instructions

* docs(core): update instructions

---------

Co-authored-by: yutao <yutao.tao@bytedance.com>
Co-authored-by: yuyutaotao <167746126+yuyutaotao@users.noreply.github.com>
2025-04-21 20:51:17 +08:00
yuyutaotao
86bd961542
docs: update docs (#574)
* docs: release android automation

* chore: update docs

* chore: update readme

* docs: append docs for doubao-1.5-ui-tars

---------

Co-authored-by: zhouxiao.shaw <zhouxiao.shaw@bytedance.com>

* chore(docs): update doubao docs

* chore(docs): merge docs for doubao

* docs(android): update

* docs(llm): upgrade open source model

---------

Co-authored-by: 晓 <zhouxiao.shaw@bytedance.com>
2025-04-21 14:18:55 +08:00
yuyutaotao
ad457a33a8
feat(yaml): allow running javascript in yaml (#555)
* feat: allow running javascript in yaml

* feat: change the output dir

* fix: CI

* docs: update docs about evaluate javascript

* chore: merge main

* chore: merge main
2025-04-18 09:58:51 +08:00
Leyang
b76211bd5d
feat: android playground (#542)
* refactor: android api

* refactor: enhance Android agent to accept options for device connection

* fix: type error

* fix: click after clearInput

* fix: click before clearInput

* feat: android playground

* feat: support npx package name

* feat: android playground joint

* fix: git ignore conflicts

* feat: ensure adb server is running before initializing adb client

* fix: deps consistency

* ci: add android playground

* feat: integrate shared constants and improve server configuration in android playground

* feat: android playground style

* feat: style opt

* feat: add @rsbuild/plugin-svgr dependency and improve URI handling in adb

* feat: remove unused water flow scripts and update comments to English

* feat: download report file

* feat: standalone android playground

* feat: use dynamic import

* feat: migrate CSS to LESS and remove unused styles in chrome extension and report

* feat: enhance Android playground with ScrcpyPlayer ref integration and device management improvements

* feat: optimize styles and layout in Android playground and visualizer components

* chore: add bin back

* chore: update build script to exclude documentation generation

* feat: add not ready message to PlaygroundResult for improved user guidance

* feat: add error handling for screenshot capture in Android page

* docs: update readme

* feat: add PNG validation for screenshot buffer in Android page

* feat: enhance UI components with improved styling and tooltips in ScrcpyPlayer and PromptInput

* docs: update uri parameter description in integrate-with-android documentation and improve uri handling in launch function

* style: update primary color to #2B83FF across multiple components and adjust margin in App.less

* refactor: replace userConfig with globalConfig for environment configuration management and update related functions

* feat: integrate server validation logic in App, AdbDevice, and ScrcpyPlayer components for improved connection handling

* style: enhance player component layout with overflow handling and margin adjustments

* style: refine player component layout with flex adjustments and improved spacing

* feat: add midscene model name display and improve layout in EnvConfig component

* feat: integrate ShinyText component for enhanced loading progress display in PlaygroundResult

* test: add test for isValidPNGImageBuffer

* style: remove background color from App.less and adjust AI config override behavior in env.ts

---------

Co-authored-by: yutao <yutao.tao@bytedance.com>
2025-04-17 17:44:11 +08:00
Leyang
3793e91899
docs: add android playground docs (#563)
* docs: add android playground docs

* chore: update pic

* docs: distinct android and web

* chore: update docs

* chore: update docs

---------

Co-authored-by: yutao <yutao.tao@bytedance.com>
2025-04-15 11:31:12 +08:00
yuyutaotao
edf48da198
docs: update caching docs (#559)
* docs: update caching docs

* docs: update caching docs

* docs: update docs

* chore: update docs
2025-04-14 14:50:48 +08:00
Leyang
570c2d7294
fix: rethrow error with android faq when adb error occured (#554) 2025-04-10 20:51:56 +08:00
yuyutaotao
9be848b651
feat: allow insecure https in yaml (#553) 2025-04-10 16:57:46 +08:00
Leyang
202f54ca78
docs: add docs for android (#546)
* docs: add docs for android

* chore: update docs

* docs: add extra api for android

* chore: update docs

---------

Co-authored-by: yutao <yutao.tao@bytedance.com>
2025-04-09 17:04:56 +08:00
yuyutaotao
5e7f82c357
docs: update blog (#530) 2025-04-03 14:36:45 +08:00
yuyutaotao
767ffcd873
docs: blog for instant action and deep think (#527) 2025-04-03 14:26:30 +08:00
yuyutaotao
bcdf90b997
feat: add context for aiAction (#528) 2025-04-02 20:34:23 +08:00
yuyutaotao
cf4acf30d9
feat: print stats in debug (#515)
Co-authored-by: Zhou Xiao <zhouxiao.shaw@bytedance.com>
2025-04-01 17:20:37 +08:00
yuyutaotao
adc3fd9e0d
feat: use aiTap(prompt, opt) style interface (#512) 2025-04-01 10:41:31 +08:00
neverland
dcf6296dc2
docs: lazy loading report.gif (#503) 2025-03-26 11:40:04 +08:00
yuyutaotao
a15fa3d47f
docs: updat faq (#501) 2025-03-26 10:34:57 +08:00
Leyang
934a1e2b5d
use adb instead of appium (#483)
* feat: enable search area for locate

* fix: update evaluation

* fix: locator

* feat: show searchArea in report

* chore: add yaml support for aiTap

* feat: use adb instead appium

* feat: Adds debugging information and reconstructs input text capabilities

* feat: refactoring Android related functions and adding android modules

* feat: update the image scaling algorithm, adjust the Android page class to support device scaling, and remove test files that are no longer needed

* feat: adjust the Android page class to support device scaling, and remove test files that are no longer needed

* feat: use appium-adb instead of bare command

* fix: update entry for @midscene/android

* feat: optimize the screenshot processing logic, add a backup mechanism when screenshots fail, and update test cases to accommodate new features

* fix: rethrow error

* feat: add Android debug configuration options and update documentation

* chore: fix code style in #483 (#492)

* fix: remove try for error handle by outside

---------

Co-authored-by: yutao <yutao.tao@bytedance.com>
Co-authored-by: linyibing <linyibing@bytedance.com>
Co-authored-by: yuyutaotao <167746126+yuyutaotao@users.noreply.github.com>
2025-03-25 22:45:05 +08:00
yuyutaotao
f5049f556d
docs: docs for new API (#499) 2025-03-25 17:16:07 +08:00
Zhou Xiao
45ddfa5ee6
chore: use cdn replace netlify (#496) 2025-03-25 11:10:23 +08:00
Zhou Xiao
36d47e4aef
feat(playwright): add aiTap/aiInput and other AI actions for Playwright integration (#489)
* feat(playwright): add aiTap/aiInput and other AI actions for Playwright integration

* chore: update doc

---------

Co-authored-by: yutao <yutao.tao@bytedance.com>
2025-03-24 19:30:53 +08:00
yuyutaotao
8e1ba565d0
feat: optimize locator (#456)
---------

Co-authored-by: zhouxiao.shaw <zhouxiao.shaw@bytedance.com>
2025-03-17 19:19:54 +08:00