27 Commits

Author SHA1 Message Date
zhouxiao.shaw
cfa45ac11d feat(chrome-extension): align React and TypeScript versions across the project
- Downgrade React and React DOM to 18.3.1 in all packages for consistency
- Update @types/react and @types/react-dom to 18.3.1
- Ensure TypeScript version is consistently set to ^5.8.3
- Adjust dependencies in package.json files for various apps and packages
- Enhance Chrome extension functionality with safe API wrappers for better error handling
- Implement checks for Chrome extension environment in recording components
2025-05-29 16:15:57 +08:00
zhouxiao.shaw
84fb598233 fix(core): align dependency versions across workspace
- Update React to ^19.1.0 and React DOM to >=19.1.0 in all packages
- Update @types/react to ^19.1.4 and @types/react-dom to ^19.1.3
- Update TypeScript to ^5.8.3 across all packages
- Update dayjs to use caret version ^1.11.11
- Align all rsbuild and rslib plugin versions
- Fix dependency version inconsistencies to pass CI checks
2025-05-28 12:31:58 +08:00
Leyang
7b190ae4ee
fix(web-integration): sanitize content for reportHtmlContent (#745)
* fix(web-integration): sanitize content for reportHtmlContent

* fix(workflow): version

* fix(web-integration): recover after retrieve

* fix(web-integration): test

* chore(core): fix lint

* fix(report): remove ! for content

* chore(core): add test case

* chore(core): add test case

---------

Co-authored-by: yutao <yutao.tao@bytedance.com>
2025-05-22 07:20:32 +08:00
yuyutaotao
08466cac1e
feat(core): add element describer (#750) 2025-05-21 21:05:47 +08:00
yuyutaotao
610e7979bf
fix(core): error in report file when planning failed (#747)
* fix(core): error in report file when planning failed

* chore(core): add testcase for aiQuery
2025-05-21 20:36:22 +08:00
Leyang
b10a7751d4
fix(report): show highlight elment when cache hit (#723)
* fix(report): show highlight elment when cache hit

* fix(core): replay animation

---------

Co-authored-by: yutao <yutao.tao@bytedance.com>
2025-05-16 22:28:29 +08:00
yuyutaotao
b261ed7f2a
feat(web): use xpath and yaml as cache (#711)
* feat(web-integration): use xpath for cache instead of id

* feat(web-integration): enhance TaskCache to support xpaths for cache matching and add new test cases

* feat(web-integration): add debug log for unknown page types in TaskCache

* feat(web-integration): update caching logic and cache hit conditions for Plan and Locate tasks

* chore(core): update debug log

* feat(web-integration): update rspress.config and enhance TaskCache structure with new properties

* feat(web-integration): recalculate id when hit cache

* fix(web-integration): update mock implementation in task-cache test to use evaluate method

* feat(web-integration): enhance element caching by adding XPath support and improving cache hit logic

* chore(core): lint

* feat(web-integration): improve XPath handling in web-extractor

* test(web-integration): fix tests

* feat(core, web-integration): add attributes to LocateResultElement and enhance element handling

* fix(core): lint

* feat(web-integration): add midsceneVersion to TaskCache and update cache validation logic

* fix(core): test

* fix(web-integration): update cache validation logic to prevent reading outdated midscene cache files

* feat(web-integration): enhance TaskCache to track used cache items and improve cache retrieval logic

* fix(core): xpath logic (#710)

* feat(core): resue context for locate

* feat(core): build yamlFlow from aiAction

* feat(core): refine task-cache

* feat(core): update cache

* feat(core): refine task-cache

* feat(core): refine task-cache

* feat(core): remove unused checkElementExistsByXPath

* feat(core): use yaml file as cache

* chore(core): fix lint

* chore(core): print warning for previous cache

* refactor(core): remove quickAnswer references and improve element matching logic

* fix(core): update import path for buildYamlFlowFromPlans

* chore(web-integration): update output image and skip task error test

* fix(web-integration): update test snapshots to handle beta versions

* fix(web-integration): adjust test snapshots for version consistency

* fix(web-integration): track original cache length and adjust matching logic in tests

* fix(web-integration): update test URLs to reflect new target site and enable previously skipped test

* chore(core): update cache docs

* fix(core): test

* feat(core): try to match element from plan

* fix(web-integration): cache id stable when retry in palywright

* fix(web-integration): typo

* style(web-integration): lint

* fix(web-integration): stable cacheid in tests

* fix(web-integration): cache id

---------

Co-authored-by: quanruzhuoxiu <quanruzhuoxiu@gmail.com>
2025-05-16 17:16:56 +08:00
yuyutaotao
73188fa43f
fix(core): bundle template content into core package (#699)
* fix(core): bundle template content into core package

* fix(core): lint

* docs: add bundler mode error msg

* fix(core): ci

---------

Co-authored-by: zhouxiao.shaw <zhouxiao.shaw@bytedance.com>
2025-05-09 11:41:16 +08:00
yuyutaotao
6a8ccbbde4
fix(core): id conflict in quick answer (#679)
* fix(core): id conflict in quick answer

* fix(core): lint

* docs(core): update docs for qwen

* fix(core): test case

* refactor(core): refactor elementById args

* refactor(core): refactor elementById args

---------

Co-authored-by: zhouxiao.shaw <zhouxiao.shaw@bytedance.com>
2025-04-30 16:51:31 +08:00
Leyang
9c2abf7729
feat(core): yaml support aiBoolean, aiNumber, aiString, aiLocate (#674) 2025-04-29 15:46:48 +08:00
Leyang
4718066bb1
feat(web-integration): add support for new agent method(aiBoolean, aiString, aiNumber, aiLocate) (#658)
* feat(web-integration): add support for new agent method(aiBoolean, aiString, aiNumber, aiLocate)

* fix(core, web-integration): update data extraction logic and refine return types for agent methods

* feat(site): add detailed descriptions and examples

* fix(web-integration): remove unused properties from aiLocate return type and update UI titles

* fix(core): enhance agent methods for better extraction and querying

* fix(core): update test scripts and refine data extraction prompts

* fix(core): refine return type in ExecutionTaskApply and remove unused import in inspect

* fix(core): update data extraction logic and enhance type handling in Insight class

* fix(web-integration): update import path for puppeteerAgentForTarget in player test file

* fix(core): tests fix

* fix(core): fix tests

* fix(shared): enhance imageInfo tests by validating individual properties and updating snapshots

* fix(site): remove waitForNavigationTimeout from Playwright integration examples

* fix(site): update timeout configuration examples in FAQ for clarity

* test(web-integration): remove unused client-extractor test and related snapshots

* chore(core): fix query implementation

* fix(report): improve data handling in DetailSide component and enhance output display

* fix(core): refine data extraction prompts and improve error handling in PageTaskExecutor

* fix(core): update data extraction prompt for clarity and type validation

---------

Co-authored-by: yutao <yutao.tao@bytedance.com>
2025-04-29 13:36:49 +08:00
Leyang
e93bc20cf1
feat(core): use append instead string join (#651)
* feat(utils): optimize reportHTMLContent to handle large data with Buffer and newlines

* feat(report): enhance test data handling and filtering in PlaywrightCaseSelector

* feat(report): enhance reportHTMLContent to support file writing and improved dump data handling

* fix(core): correct file writing logic and improve dump data handling in reportHTMLContent

* feat(core): add tests for reportHTMLContent

* feat(core): add performance test for handling multiple large reports in utils
2025-04-28 14:51:19 +08:00
yuyutaotao
07614961c8
fix(visualizer): the style of error state in report (#641)
* fix(visualizer): the style of error state in report

* fix(core): fix ci
2025-04-25 12:00:48 +08:00
Leyang
ca644d8914
feat(core): allow custom midscene_run dir (#631)
* feat(core): support custom midscene_run dir

* feat(report): add search functionality to PlaywrightCaseSelector component

* refactor(shared): simplify base directory resolution and remove unused environment variable

* feat(shared): integrate shared environment variables across multiple packages

* refactor(shared): update base directory resolution to use dynamic midscene_run directory

* fix(puppeteer): increase screenshot timeout from 3s to 10s for improved reliability
2025-04-24 22:54:52 +08:00
48ee2211ac
chore(workflow): add report miss polyfill deps (#634) 2025-04-24 18:41:54 +08:00
yuyutaotao
ecefd8b0fa
fix(report): reduce context size in report file (#626)
* fix(core): reduce context size in report file

* chore(core): fix lint

* chore(core): resolve conflict

---------

Co-authored-by: zhouxiao.shaw <zhouxiao.shaw@bytedance.com>
2025-04-24 18:28:45 +08:00
Leyang
76eb4e1417
fix(report): ai tap in report (#594)
* fix: ai tap in report

* chore(playwright): update lockfile

* fix(core): lint

---------

Co-authored-by: yutao <yutao.tao@bytedance.com>
2025-04-18 18:42:47 +08:00
Leyang
cf621a564e
fix(visualizer): update type references and improve tooltip handling in components (#589) 2025-04-18 15:18:03 +08:00
Leyang
7596da2ab7
feat: support tap in prompt (#584) 2025-04-18 10:27:56 +08:00
Leyang
b76211bd5d
feat: android playground (#542)
* refactor: android api

* refactor: enhance Android agent to accept options for device connection

* fix: type error

* fix: click after clearInput

* fix: click before clearInput

* feat: android playground

* feat: support npx package name

* feat: android playground joint

* fix: git ignore conflicts

* feat: ensure adb server is running before initializing adb client

* fix: deps consistency

* ci: add android playground

* feat: integrate shared constants and improve server configuration in android playground

* feat: android playground style

* feat: style opt

* feat: add @rsbuild/plugin-svgr dependency and improve URI handling in adb

* feat: remove unused water flow scripts and update comments to English

* feat: download report file

* feat: standalone android playground

* feat: use dynamic import

* feat: migrate CSS to LESS and remove unused styles in chrome extension and report

* feat: enhance Android playground with ScrcpyPlayer ref integration and device management improvements

* feat: optimize styles and layout in Android playground and visualizer components

* chore: add bin back

* chore: update build script to exclude documentation generation

* feat: add not ready message to PlaygroundResult for improved user guidance

* feat: add error handling for screenshot capture in Android page

* docs: update readme

* feat: add PNG validation for screenshot buffer in Android page

* feat: enhance UI components with improved styling and tooltips in ScrcpyPlayer and PromptInput

* docs: update uri parameter description in integrate-with-android documentation and improve uri handling in launch function

* style: update primary color to #2B83FF across multiple components and adjust margin in App.less

* refactor: replace userConfig with globalConfig for environment configuration management and update related functions

* feat: integrate server validation logic in App, AdbDevice, and ScrcpyPlayer components for improved connection handling

* style: enhance player component layout with overflow handling and margin adjustments

* style: refine player component layout with flex adjustments and improved spacing

* feat: add midscene model name display and improve layout in EnvConfig component

* feat: integrate ShinyText component for enhanced loading progress display in PlaygroundResult

* test: add test for isValidPNGImageBuffer

* style: remove background color from App.less and adjust AI config override behavior in env.ts

---------

Co-authored-by: yutao <yutao.tao@bytedance.com>
2025-04-17 17:44:11 +08:00
yuyutaotao
0c0675ba4b
feat: update deepThink interface (#517)
* feat: update deepThink interface

* fix: CI

* chore: update error message

* feat: update search area config

* fix: deepthink ui

---------

Co-authored-by: Zhou Xiao <zhouxiao.shaw@bytedance.com>
2025-04-02 19:26:56 +08:00
Zhou Xiao
814f99ca48
refactor(visualizer): optimize extension logic (#525) 2025-04-02 19:12:52 +08:00
Zhou Xiao
5ef9202c79
refactor(report): optimize visualizer common logic (#519)
* chore: delete unless visualizer logic

* chore: delete unless code

* chore: delete unless code

* chore: delete unless code

* chore: delete unless common logic

* chore: merge main branch

* chore: update lock file
2025-04-01 17:25:40 +08:00
Zhou Xiao
ca28ae0b29
fix(report): resolve report script error (#516) 2025-04-01 15:03:42 +08:00
Zhou Xiao
d1b232586b
chore: delete unless visualizer logic (#514) 2025-03-31 19:51:43 +08:00
Zhou Xiao
16d6efa1f0
refactor(report): make the report page an app type to improve the development experience (#513) 2025-03-31 18:46:35 +08:00
Zhou Xiao
0e4b94fc9d
chore(report): init report project (#511) 2025-03-31 11:34:38 +08:00