138 Commits

Author SHA1 Message Date
zhouxiao.shaw
4de7fd81f7 feat(record): improve session management and code quality
- Optimize session creation to use timestamp naming by default without requiring modal input
     - Improve UI layout to display titles and descriptions more effectively in both list and
     detail views
     - Apply code formatting improvements across the codebase for better readability
     - Fix linting issues in scripts and component files
2025-06-01 11:30:06 +08:00
zhouxiao.shaw
a0f798f6aa feat(chrome-extension): enhance event recording with new features and dependencies
- Add @rsbuild/plugin-node-polyfill to support Node.js polyfills in the project
- Update event recording logic to include viewport dimensions and element positioning
- Improve screenshot handling by integrating new properties in RecordedEvent interface
- Enhance RecordTimeline component to display element positioning and dimensions
- Optimize event handling and UI updates for better user experience
- Clean up and refactor code for improved readability and maintainability
2025-05-29 20:18:00 +08:00
zhouxiao.shaw
84fb598233 fix(core): align dependency versions across workspace
- Update React to ^19.1.0 and React DOM to >=19.1.0 in all packages
- Update @types/react to ^19.1.4 and @types/react-dom to ^19.1.3
- Update TypeScript to ^5.8.3 across all packages
- Update dayjs to use caret version ^1.11.11
- Align all rsbuild and rslib plugin versions
- Fix dependency version inconsistencies to pass CI checks
2025-05-28 12:31:58 +08:00
Leyang
7b190ae4ee
fix(web-integration): sanitize content for reportHtmlContent (#745)
* fix(web-integration): sanitize content for reportHtmlContent

* fix(workflow): version

* fix(web-integration): recover after retrieve

* fix(web-integration): test

* chore(core): fix lint

* fix(report): remove ! for content

* chore(core): add test case

* chore(core): add test case

---------

Co-authored-by: yutao <yutao.tao@bytedance.com>
2025-05-22 07:20:32 +08:00
yuyutaotao
08466cac1e
feat(core): add element describer (#750) 2025-05-21 21:05:47 +08:00
Leyang
5a1a3ba18a
feat(web-integration): support disable cache for a single api call (#740)
* feat(web-integration): support disable cache for a single api call

* feat(workflow): version mismatch

* feat(web-integration): cache rename to cacheable

* feat(web-integration): add cacheable option to multiple API methods and update caching documentation

* docs(site): update cacheable option descriptions to reference caching feature documentation

* docs(core): update caching doc

---------

Co-authored-by: yutao <yutao.tao@bytedance.com>
2025-05-21 16:46:20 +08:00
github-actions[bot]
049d9f6565 release: v0.17.1 2025-05-20 09:11:17 +00:00
bac9051d2d
feat(evaluation): add screenspot v2 evaluation (#737)
* feat(evaluation): add screenspot v2 evaluation

* style(evaluation): format files array in package.json
2025-05-20 15:52:03 +08:00
Leyang
fb2580616c
fix cache bugs(scroll instant, text node info, cache hit condition) (#732)
* fix(web-integration): cache hit when element.id exist and scroll element instantly

* fix(web-integration): use parent xpath for text node

* fix(web-integration): only scroll in to view when element is not completly visible

* fix(shared): distinct text node

* test(web-integration): getElementInfoByXpath

* test(web-integration): rename desc

* test(web-integration): fix

* test(web-integration): snapshot attributes only

* test(web-integration): fix test
2025-05-20 13:19:32 +08:00
github-actions[bot]
39ddc57df3 release: v0.17.0 2025-05-16 14:32:01 +00:00
yuyutaotao
b261ed7f2a
feat(web): use xpath and yaml as cache (#711)
* feat(web-integration): use xpath for cache instead of id

* feat(web-integration): enhance TaskCache to support xpaths for cache matching and add new test cases

* feat(web-integration): add debug log for unknown page types in TaskCache

* feat(web-integration): update caching logic and cache hit conditions for Plan and Locate tasks

* chore(core): update debug log

* feat(web-integration): update rspress.config and enhance TaskCache structure with new properties

* feat(web-integration): recalculate id when hit cache

* fix(web-integration): update mock implementation in task-cache test to use evaluate method

* feat(web-integration): enhance element caching by adding XPath support and improving cache hit logic

* chore(core): lint

* feat(web-integration): improve XPath handling in web-extractor

* test(web-integration): fix tests

* feat(core, web-integration): add attributes to LocateResultElement and enhance element handling

* fix(core): lint

* feat(web-integration): add midsceneVersion to TaskCache and update cache validation logic

* fix(core): test

* fix(web-integration): update cache validation logic to prevent reading outdated midscene cache files

* feat(web-integration): enhance TaskCache to track used cache items and improve cache retrieval logic

* fix(core): xpath logic (#710)

* feat(core): resue context for locate

* feat(core): build yamlFlow from aiAction

* feat(core): refine task-cache

* feat(core): update cache

* feat(core): refine task-cache

* feat(core): refine task-cache

* feat(core): remove unused checkElementExistsByXPath

* feat(core): use yaml file as cache

* chore(core): fix lint

* chore(core): print warning for previous cache

* refactor(core): remove quickAnswer references and improve element matching logic

* fix(core): update import path for buildYamlFlowFromPlans

* chore(web-integration): update output image and skip task error test

* fix(web-integration): update test snapshots to handle beta versions

* fix(web-integration): adjust test snapshots for version consistency

* fix(web-integration): track original cache length and adjust matching logic in tests

* fix(web-integration): update test URLs to reflect new target site and enable previously skipped test

* chore(core): update cache docs

* fix(core): test

* feat(core): try to match element from plan

* fix(web-integration): cache id stable when retry in palywright

* fix(web-integration): typo

* style(web-integration): lint

* fix(web-integration): stable cacheid in tests

* fix(web-integration): cache id

---------

Co-authored-by: quanruzhuoxiu <quanruzhuoxiu@gmail.com>
2025-05-16 17:16:56 +08:00
Leyang
b9ff80a0db
implement repeat function for scrolling until actions (#713)
* feat(android): implement repeat function for scrolling until actions

* fix(shared): fix potential error in getAIConfig by ensuring trim is called correctly

* feat(android): update scrolling behavior with adjustable duration and added sleep

* feat(android): refine scrolling durations with new constants for fast and normal scroll
2025-05-14 18:28:10 +08:00
github-actions[bot]
571ec07829 release: v0.16.10 2025-05-09 09:00:22 +00:00
yuyutaotao
34e15c7247
fix(core): empty report issue (#702)
* fix(core): empty report issue

* fix(core): empty report issue
2025-05-09 16:51:30 +08:00
github-actions[bot]
7f982f7cb7 release: v0.16.9 2025-05-09 03:44:51 +00:00
yutao
6224154bdc Merge branch 'main' of https://github.com/neewbee/midscene into neewbee-main 2025-05-09 10:55:12 +08:00
github-actions[bot]
c5a88931f0 release: v0.16.8 2025-04-30 14:31:14 +00:00
yuyutaotao
c1bc73c78b
feat(android): customize adb path (#684)
* feat(shared): add custom adb path
feat(android): add custom adb path

* feat(android): add docs for custom adb path

---------

Co-authored-by: HBLADEH <1012582116@qq.com>
2025-04-30 17:16:38 +08:00
yuyutaotao
b8f29e8e66
fix(core): use unified config for doubao-ui-tars model (#678) 2025-04-29 21:39:58 +08:00
github-actions[bot]
7ce8225369 release: v0.16.7 2025-04-29 08:36:53 +00:00
github-actions[bot]
aa2e767b79 release: v0.16.6 2025-04-29 05:54:48 +00:00
Leyang
4718066bb1
feat(web-integration): add support for new agent method(aiBoolean, aiString, aiNumber, aiLocate) (#658)
* feat(web-integration): add support for new agent method(aiBoolean, aiString, aiNumber, aiLocate)

* fix(core, web-integration): update data extraction logic and refine return types for agent methods

* feat(site): add detailed descriptions and examples

* fix(web-integration): remove unused properties from aiLocate return type and update UI titles

* fix(core): enhance agent methods for better extraction and querying

* fix(core): update test scripts and refine data extraction prompts

* fix(core): refine return type in ExecutionTaskApply and remove unused import in inspect

* fix(core): update data extraction logic and enhance type handling in Insight class

* fix(web-integration): update import path for puppeteerAgentForTarget in player test file

* fix(core): tests fix

* fix(core): fix tests

* fix(shared): enhance imageInfo tests by validating individual properties and updating snapshots

* fix(site): remove waitForNavigationTimeout from Playwright integration examples

* fix(site): update timeout configuration examples in FAQ for clarity

* test(web-integration): remove unused client-extractor test and related snapshots

* chore(core): fix query implementation

* fix(report): improve data handling in DetailSide component and enhance output display

* fix(core): refine data extraction prompts and improve error handling in PageTaskExecutor

* fix(core): update data extraction prompt for clarity and type validation

---------

Co-authored-by: yutao <yutao.tao@bytedance.com>
2025-04-29 13:36:49 +08:00
github-actions[bot]
9c5d22a57e release: v0.16.5 2025-04-29 05:17:39 +00:00
github-actions[bot]
061ff92c69 release: v0.16.4 2025-04-29 04:25:49 +00:00
github-actions[bot]
4f41e99ddc release: v0.16.3 2025-04-28 15:14:58 +00:00
github-actions[bot]
c4112adb51 release: v0.16.2 2025-04-28 06:22:02 +00:00
github-actions[bot]
33d2733c77 release: v0.16.1 2025-04-28 02:58:31 +00:00
neewee
5d96f60853 feat(core): support HTTP proxy and reorder dependencies
Added HTTP proxy support with `https-proxy-agent` and cleaned up dependency order.
2025-04-28 10:04:23 +08:00
yuyutaotao
5fb208a08c
feat(core): adapt UI tars 1.5 (#616)
* feat(core): adapt ui-tars 1.5

* chore(core): adaptr ui-tars-1.5

* chore(core): fix lint

* fix(core): env building issue

* fix(core): update import for uiTarsModelVersion from shared env

* feat(core): ui-tars hotkey event

* chore(core): move @ui-tars/action-parser to devDependencies

* fix(core): adapting new model
2025-04-28 08:42:43 +08:00
github-actions[bot]
8acbb6a32f release: v0.16.0 2025-04-28 00:26:25 +00:00
github-actions[bot]
ee3821f7f9 release: v0.15.5 2025-04-26 07:42:06 +00:00
github-actions[bot]
767b3e3a8c release: v0.15.4 2025-04-25 09:18:44 +00:00
Leyang
ca644d8914
feat(core): allow custom midscene_run dir (#631)
* feat(core): support custom midscene_run dir

* feat(report): add search functionality to PlaywrightCaseSelector component

* refactor(shared): simplify base directory resolution and remove unused environment variable

* feat(shared): integrate shared environment variables across multiple packages

* refactor(shared): update base directory resolution to use dynamic midscene_run directory

* fix(puppeteer): increase screenshot timeout from 3s to 10s for improved reliability
2025-04-24 22:54:52 +08:00
yuyutaotao
ecefd8b0fa
fix(report): reduce context size in report file (#626)
* fix(core): reduce context size in report file

* chore(core): fix lint

* chore(core): resolve conflict

---------

Co-authored-by: zhouxiao.shaw <zhouxiao.shaw@bytedance.com>
2025-04-24 18:28:45 +08:00
github-actions[bot]
bd104a8afa release: v0.15.3 2025-04-24 08:04:27 +00:00
Leyang
03a597e022
feat(web-integration): enhance timeout configurations and logging for network idle and navigation (#624)
* feat(web-integration): enhance timeout configurations and logging for network idle and navigation

* fix(web-integration): refine timeout warning messages and remove unnecessary test files

* feat(site): add network timeout customization details and additional parameters for Puppeteer

* fix(site): update default timeout values and enhance customization options for network idle in YAML

* fix(site): remove redundant timeout customization details in FAQ documentation

* fix(web-integration): enhance Playwright agent to support network idle functionality

* docs(playwright): update config docs

* docs(playwright): update config docs

* fix(web-integration): refactor network idle handling in Playwright agent

---------

Co-authored-by: yutao <yutao.tao@bytedance.com>
2025-04-24 10:28:26 +08:00
github-actions[bot]
ec3fd040d4 release: v0.15.2 2025-04-23 01:51:12 +00:00
github-actions[bot]
385f3eb639 release: v0.15.1 2025-04-21 12:13:32 +00:00
daf308b1d0
fix(mcp): resolve mcp server log error (#599)
* fix(mcp): resolve mcp server log error

* chore(shared): delete unless code

* chore(workflow): fix lint error
2025-04-19 12:00:39 +08:00
github-actions[bot]
0a3bc1e230 release: v0.15.0 2025-04-18 11:26:30 +00:00
yuyutaotao
ad457a33a8
feat(yaml): allow running javascript in yaml (#555)
* feat: allow running javascript in yaml

* feat: change the output dir

* fix: CI

* docs: update docs about evaluate javascript

* chore: merge main

* chore: merge main
2025-04-18 09:58:51 +08:00
Leyang
b76211bd5d
feat: android playground (#542)
* refactor: android api

* refactor: enhance Android agent to accept options for device connection

* fix: type error

* fix: click after clearInput

* fix: click before clearInput

* feat: android playground

* feat: support npx package name

* feat: android playground joint

* fix: git ignore conflicts

* feat: ensure adb server is running before initializing adb client

* fix: deps consistency

* ci: add android playground

* feat: integrate shared constants and improve server configuration in android playground

* feat: android playground style

* feat: style opt

* feat: add @rsbuild/plugin-svgr dependency and improve URI handling in adb

* feat: remove unused water flow scripts and update comments to English

* feat: download report file

* feat: standalone android playground

* feat: use dynamic import

* feat: migrate CSS to LESS and remove unused styles in chrome extension and report

* feat: enhance Android playground with ScrcpyPlayer ref integration and device management improvements

* feat: optimize styles and layout in Android playground and visualizer components

* chore: add bin back

* chore: update build script to exclude documentation generation

* feat: add not ready message to PlaygroundResult for improved user guidance

* feat: add error handling for screenshot capture in Android page

* docs: update readme

* feat: add PNG validation for screenshot buffer in Android page

* feat: enhance UI components with improved styling and tooltips in ScrcpyPlayer and PromptInput

* docs: update uri parameter description in integrate-with-android documentation and improve uri handling in launch function

* style: update primary color to #2B83FF across multiple components and adjust margin in App.less

* refactor: replace userConfig with globalConfig for environment configuration management and update related functions

* feat: integrate server validation logic in App, AdbDevice, and ScrcpyPlayer components for improved connection handling

* style: enhance player component layout with overflow handling and margin adjustments

* style: refine player component layout with flex adjustments and improved spacing

* feat: add midscene model name display and improve layout in EnvConfig component

* feat: integrate ShinyText component for enhanced loading progress display in PlaygroundResult

* test: add test for isValidPNGImageBuffer

* style: remove background color from App.less and adjust AI config override behavior in env.ts

---------

Co-authored-by: yutao <yutao.tao@bytedance.com>
2025-04-17 17:44:11 +08:00
yuyutaotao
a1b5a54d89
fix(report): do not call mkdir in browser (#577)
* fix: log dir

* fix: log dir

* fix: gitignore config
2025-04-17 15:09:59 +08:00
yuyutaotao
824be26c85
fix: use tmpdir as a fallback for log file (#575)
---------

Co-authored-by: zhouxiao.shaw <zhouxiao.shaw@bytedance.com>
2025-04-17 10:54:19 +08:00
github-actions[bot]
1bd873b360 release: v0.14.3 2025-04-10 09:10:39 +00:00
github-actions[bot]
4d77207c0f release: v0.14.2 2025-04-09 02:30:47 +00:00
github-actions[bot]
92e4d7ed61 release: v0.14.1 2025-04-03 08:40:18 +00:00
yuyutaotao
c6cd10ebb2
fix: filename (#534) 2025-04-03 16:37:00 +08:00
github-actions[bot]
00dc2f9f50 release: v0.14.0 2025-04-03 06:01:06 +00:00
Zhou Xiao
6468bb0206
refactor(logger): use log file output instead of command line output in Node (#509)
* chore: add element detail info

* chore: add element detail info

* chore: add element detail info

* chore: optimize logger logic

* chore: optimize logger logic
2025-03-31 19:22:39 +08:00