268 Commits

Author SHA1 Message Date
zhouxiao.shaw
cc4bb0172a refactor(event-recorder-bridge): remove idle screenshot logging and event send stats logging 2025-06-22 09:41:57 +08:00
zhouxiao.shaw
b29a854bc4 refactor(useRecordingControl): simplify navigation handling and remove event buffering logic 2025-06-21 22:40:05 +08:00
zhouxiao.shaw
b3190a6628 refactor(event-recorder-bridge): streamline screenshot handling and improve event sending logic 2025-06-21 22:13:38 +08:00
zhouxiao.shaw
79e002bd9c refactor(event-recorder-bridge): enhance idle screenshot management and page change monitoring 2025-06-21 22:13:19 +08:00
zhouxiao.shaw
c7dec93a7b refactor(export-controls): update progress tracking for element description generation 2025-06-21 15:49:04 +08:00
zhouxiao.shaw
81e740193f refactor(useRecordingControl): specify type for service worker connection 2025-06-21 15:43:45 +08:00
zhouxiao.shaw
6cbf764f75 refactor(recorder): improve navigation detection and enhance recording state management 2025-06-21 15:42:28 +08:00
zhouxiao.shaw
ae30f6bc18 refactor(recorder): remove unnecessary logging and adjust debounce timing 2025-06-21 15:05:42 +08:00
zhouxiao.shaw
46d456b773 refactor(export-controls): remove redundant event description generation 2025-06-20 12:00:00 +08:00
zhouxiao.shaw
faa618e45e refactor(recorder): streamline event processing and enhance element detection logic 2025-06-20 11:30:45 +08:00
zhouxiao.shaw
e20b29b292 refactor(recorder): restore recorder component and enhance event logging 2025-06-19 18:03:00 +08:00
zhouxiao.shaw
dd6fa50eb1 chore: hidden recorder 2025-06-19 17:33:10 +08:00
zhouxiao.shaw
cd217649ab chore: fix lint error 2025-06-19 17:32:45 +08:00
zhouxiao.shaw
63bbd6a108 refactor(recorder): ensure complete generation of element descriptions and clean up session handling 2025-06-19 16:07:02 +08:00
zhouxiao.shaw
cd6f4c0ff8 refactor(recorder): optimize event handling and improve UI element rendering 2025-06-19 15:51:58 +08:00
Leyang
09c6aaaa8b
feat(web-integration): support user expected xpath option for locate methods (#844)
* feat(web-integration): support user expected xpath option for locate methods

* docs(site): xpath option for api

* fix(ci): nx config

* docs(site): update docs
2025-06-19 10:55:14 +08:00
Leyang
bee1c40637
feat(web-integration): support aiAsk for agent (#841)
* feat(web-integration): support aiAsk for agent

* docs(core): update docs

---------

Co-authored-by: yutao <yutao.tao@bytedance.com>
2025-06-19 10:55:07 +08:00
Leyang
2965e3cf10
docs(site): 0.19 changelog (#842) 2025-06-18 16:56:37 +08:00
780b7f0b9c
refactor(recorder): replace localStorage with IndexedDB for recording data storage and optimize navigation record (#831)
* refactor(recorder): use indexdeb replace localStorage

* chore: optimize note

* chore: optimize recorder title

* chore: delete unless file

* chore: optimize indexdb storage logic

* feat(recorder): support refresh page and continue record event

* chore: optimize recorder logic

* chore: optimize checkbox and password type

* feat(recorder): support refresh page and continue record event

* chore: optimize refresh or open new tab click event

* chore: resolve restart session

* chore: optimize refresh or open new tab click event

* feat(recorder): enhance recording control and session management

* chore: optimize refresh or open new tab click event

* chore: remove commented-out recorder section from popup

* chore: fix lint error

* chore: fix lint error
2025-06-18 12:10:33 +08:00
github-actions[bot]
bcfc2fcc29 release: v0.19.1 2025-06-17 05:23:12 +00:00
Leyang
5f79ef8c6b
feat(android): support ime strategy for android device (#839)
* feat(android): support ime strategy for android device

* docs(site): update
2025-06-17 09:33:03 +08:00
Leyang
5e9eb72924
fix(web-integration): reserve the visible nodes of invisible elements (#840) 2025-06-16 20:38:22 +08:00
Leyang
50203ede36
fix(core): only replace script tag (#829)
* fix(core): only replace script tag

* test(web-integration): add more test
2025-06-16 12:18:36 +08:00
65fc4ead23
docs(site): fix code block formatting in playwright integration guide (#830) 2025-06-13 15:55:08 +08:00
github-actions[bot]
95e24f98ee release: v0.19.0 2025-06-13 03:32:14 +00:00
Leyang
32f503f69e
fix(shared): reduce null element node (#827)
* fix(shared): reduce null element node

* feat(web-integration): delete content and locator, support visible-only for domIncluded

* feat(web-integration): remove logScreenshot documentation and update tests to use treeToList for content extraction

* feat(web-integration): add trimContextByViewport function to filter execution tasks by visible elements

* release: v0.18.1

* fix(workflow): update default release version to 'prepatch'

* fix(web-integration): node detect empty

* release: v0.18.2

* fix(evaluation): restore generateExtractData

* fix(web-integration): reporter export

---------

Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
2025-06-13 11:25:31 +08:00
Leyang
199ab17504
feat(cli): support dotenv-override argv for cli (#819)
* feat(cli): support dotenv-override argv for cli

* feat(cli): support dotenv-debug argv for cli

* docs(core): update doc of dotenv

---------

Co-authored-by: yutao <yutao.tao@bytedance.com>
2025-06-11 17:09:17 +08:00
Leyang
a84100e7d8
feat(mcp, shared, visualizer): add license (#818)
* feat(mcp, shared, visualizer): add license

* docs(core): update playwright reporter doc

---------

Co-authored-by: yutao <yutao.tao@bytedance.com>
2025-06-11 17:09:05 +08:00
Leyang
3510a8c736
feat(web-integration): support unstableLogContent api for agent (#826)
* feat(web-integration): support unstableLogContent api for agent

* fix(web-integration): use sync mehtod

* docs(core): update API doc

* fix(web-integration): playwright

* docs(site): update

---------

Co-authored-by: yutao <yutao.tao@bytedance.com>
2025-06-11 16:50:11 +08:00
c9f9a48c70
feat(recorder): add record ability to chrome extension (#769)
* chore: add record example

* chore: delete unless file

* chore: optimize common logic

* chore: optimize record logic

* chore: support horizontal scroll

* chore: optimize label input logic

* chore: delete unless file

* chore: modify record log content

* chore: add record lib

* chore: optimize record iife logic

* chore: delete unless config

* chore: modify comment

* feat(chrome-extension): add recording functionality

- Add new record UI components (record.tsx, record.less)
- Integrate recording capabilities into popup extension
- Update worker script to handle recording events
- Add content script wrapper for recording injection
- Remove EventRecorder from record-form app (moved to extension)
- Update manifest.json permissions for recording
- Update dependencies for recording functionality
- Enhance store with recording state management

* feat(chrome-extension): improve recording stability with page refresh handling

- Add tab update monitoring to automatically stop recording on page refresh/navigation
- Implement content script injection check with ping mechanism
- Enhance error handling with specific messages for Chrome internal pages
- Improve recording stop reliability when content script becomes unavailable
- Add graceful handling for missing content script during stop operation

* fix(core): align dependency versions across workspace

- Update React to ^19.1.0 and React DOM to >=19.1.0 in all packages
- Update @types/react to ^19.1.4 and @types/react-dom to ^19.1.3
- Update TypeScript to ^5.8.3 across all packages
- Update dayjs to use caret version ^1.11.11
- Align all rsbuild and rslib plugin versions
- Fix dependency version inconsistencies to pass CI checks

* refactor(chrome-extension): rename content-script-wrapper to event-recorder-bridge

- Renamed content-script-wrapper.js to event-recorder-bridge.js
- Updated references in rsbuild.config.ts and record.tsx
- Updated internal comments to match new name
- New name better describes the file's bridge functionality

* feat(chrome-extension): enhance recording session management and UI

- Introduced RecordingSession interface for better session handling
- Implemented session persistence with localStorage for recording sessions
- Added UI components for creating, editing, and deleting recording sessions
- Enhanced event handling to load and display events from selected sessions
- Improved user feedback with success messages for session actions
- Updated styles for session management and recording status indicators

* feat(chrome-extension): enhance recording UI and session management

- Simplified the Record component structure by separating list and detail views for better user experience.
- Improved styling for the recording sessions list and detail views, ensuring better responsiveness and usability.
- Added functionality to switch between list and detail views for recording sessions.
- Updated session management to allow for viewing, editing, and deleting sessions with improved user feedback.
- Enhanced event handling and display for recorded events within the detail view.

* feat(record): add screenshot capture for recorded events

- Add screenshot capture before and after each recorded event
- Implement screenshot capture in chrome extension service worker
- Add screenshot fields to RecordedEvent interface
- Create UI components for displaying screenshot thumbnails in timeline
- Add hover effects and popover for screenshot preview
- Remove unused AGENTS.md file

* feat(chrome-extension): align React and TypeScript versions across the project

- Downgrade React and React DOM to 18.3.1 in all packages for consistency
- Update @types/react and @types/react-dom to 18.3.1
- Ensure TypeScript version is consistently set to ^5.8.3
- Adjust dependencies in package.json files for various apps and packages
- Enhance Chrome extension functionality with safe API wrappers for better error handling
- Implement checks for Chrome extension environment in recording components

* feat(chrome-extension): enhance event recording with new features and dependencies

- Add @rsbuild/plugin-node-polyfill to support Node.js polyfills in the project
- Update event recording logic to include viewport dimensions and element positioning
- Improve screenshot handling by integrating new properties in RecordedEvent interface
- Enhance RecordTimeline component to display element positioning and dimensions
- Optimize event handling and UI updates for better user experience
- Clean up and refactor code for improved readability and maintainability

* feat(record): enhance event data with coordinates in event recorder

- Add x and y coordinates to the event data sent to the Chrome extension
- Update RecordTimeline component to utilize new coordinate properties for element positioning
- Ensure compatibility with existing event handling logic

* chore: optimize select box logic

* chore: delete unless config

* feat(record): optimize AI description generation to be asynchronous and non-blocking

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>

* feat(record): implement smooth scroll with enhanced animations and performance

- Add requestAnimationFrame-based smooth scrolling for better performance
- Implement container-level auto-scroll to bottom when new events arrive
- Extract event optimization logic to separate eventOptimizer utility module
- Add smooth scroll CSS properties and touch optimization for mobile
- Enhance hover animations with cubic-bezier easing and GPU acceleration
- Add slideInUp and fadeInScale keyframe animations for timeline items
- Improve scrollbar styling with transition effects
- Optimize will-change properties for better rendering performance
- Add overscroll-behavior to prevent scroll propagation

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>

* chore(workflow): add CLAUDE.md with project documentation and development guide

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>

* feat(record): optimize session creation to auto-start recording with timestamp name

* feat(record): optimize session creation to use timestamp naming by default

- Modified session creation to auto-create with timestamp name without showing modal
- Improved record UI to display session titles more effectively
- Reorganized description display in both list and detail views

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>

* feat(record): improve session management and code quality

     - Optimize session creation to use timestamp naming by default without requiring modal input
     - Improve UI layout to display titles and descriptions more effectively in both list and
     detail views
     - Apply code formatting improvements across the codebase for better readability
     - Fix linting issues in scripts and component files

* feat(record): implement caching for element descriptions and screenshots to enhance performance

* feat(record): refactor stopRecording logic and add visibility change handling for extension popup

* feat(record): auto-stop recording when leaving detail view

* feat(record): improve recording state management and handle asynchronous stop actions

* chore: upgrade lock file

* feat(record): add persistence for recording state in local storage

* feat(record): add modular structure with components and hooks for improved organization

* feat(record): implement AI-generated titles and descriptions for recording sessions

* feat(record): enhance recording session management with improved layout and AI title generation

* feat(record): enhance recording control hook with session update callback

* fix: update coding standards and improve code formatting across multiple files

* feat(record): add Playwright test generation feature and export controls

* feat(record): enhance AI description generation with caching and callback handling

* feat(record): replace RecordedEvent with ChromeRecordedEvent for improved event handling

* feat(record): improve debug logging and event handling in recording sessions

* feat(record): enhance event optimization and logging, add diagnosis feature for recording chain

* feat(record): update RecordDetail and RecordList components, improve diagnostic function and type imports

* feat(record): enhance tab monitoring with event listeners for activation and updates

* feat(record): sort sessions by updated time in RecordList component

* feat(record): implement session ID handling and cleanup for previous recordings

* feat(eventOptimizer): improve element rectangle handling and callback updates in description generation

* feat(record): enhance AI title and description generation logic, improve event handling and logging

* feat(record): implement debounce mechanism for AI description generation and enhance event rectangle validation

* feat(record): enhance AI description generation with improved caching and debouncing logic, and add session ID dependency in cleanup

* feat(record): update AI title and description generation logic to prevent overwriting existing session names and descriptions, and improve event handling in scroll events

* refactor(record): clean up code formatting and improve logging consistency across components

* docs(record): update CLAUDE.md with detailed implementation and optimization strategies for the recording system

* feat(record): enhance event details display with JSON copy functionality and improved cache key generation

* chore(commitlint): increase maximum header length to 300 characters

* chore(commitlint): disable scope-enum rule enforcement

* feat(record): implement unified logging system and replace console logs with structured logger

* feat(record): add hash ID generation for recorded events and clean up elementRect handling

* feat(record): add before and after screenshot functionality to event timeline

* feat(record): remove event-recorder-bridge script and update build configuration

* feat(record): update screenshot handling in RecordTimeline component to display before and after images

* refactor(record): clean up code formatting and remove unnecessary lines

* feat(build): enhance build configuration with caching and input/output settings

* feat(record): pause active recording sessions when creating a new session

* feat(record): enhance Playwright export controls with live event handling and improved description checks

* refactor(record): streamline configuration and improve logging clarity

* feat(record): update YAML generation to follow Midscene.js conventions with enhanced action mapping and descriptive naming

* feat(record): simplify YAML generation for Midscene.js with clearer structure and updated action mapping

* feat(record): enhance YAML export functionality with improved event handling and loading state

* feat(record): add Playwright and YAML test generation utilities with shared types and export controls

* feat(record): replace PlaywrightExportControls with a unified ExportControls component

* feat(record): integrate Playwright test generation from core package and update language handling in UI components

* feat(record): add @midscene/record dependency and integrate YAML test generation from core package

* feat(record): enhance event handling by saving current recording events during session updates and cleanup

* feat(record): enhance event recording by adding final navigation event and improving error handling in event recorder bridge

* feat(record): improve event handling by conditionally clearing events during recording start

* feat(record): optimize event handling by implementing debouncing and refining fallback descriptions

* feat(record): enhance test generation by extracting navigation and viewport info for improved options

* feat(record): implement event persistence with local storage for recording sessions

* chore: clean up code formatting and improve readability across multiple files

* feat(record): add dropdown menu for code generation and streamline state management

* feat(record): remove YAML export functionality from ExportControls and related components

* feat(record): add ProgressModal component for enhanced recording feedback

* feat(record): enhance ExportControls and ProgressModal with improved UI and confetti animation

* feat(record): simplify ExportControls and RecordDetail components by removing unused functionality and improving session details display

* feat(record): improve error handling and logging in content script checks and enhance tab URL validation

* feat(record): add generatedCode field to RecordingSession and improve error handling in content script checks

* feat(record): add canvas-confetti for celebratory effects in ProgressModal and update dependencies

* feat(record): refactor RecordDetail to use sessionId and improve session retrieval with error handling

* feat(progress-modal): adjust confetti display duration and update completion timing in ProgressModal

* chore: upgrade lock file

* refactor(snapshot): simplify element description rules for clarity and brevity

* feat(record): initialize optimizedEvents state as empty array and refactor logging methods for improved clarity

* refactor(export-controls): streamline event description generation and update progress step order for clarity

* feat(progress-modal): add ShinyText component for enhanced step title display during loading

* refactor(export-controls): consolidate test generation logic into a common function for improved maintainability

* refactor(export-controls, eventOptimizer): improve code readability and consistency by formatting and simplifying logic

* chore: delete unless config

* chore: add ignore file

* chore: add ignore file

* refactor: migrate from 'record' to 'recorder' package and consolidate related files

* refactor: remove playwright example file and integrate example code into prompts and YAML generation

* chore: hidden recorder

* chore: delete unless file

* chore: delete unless file

* refactor: migrate from 'record-form' to 'recorder-form' and update related configurations

* refactor: remove shiny-text component and update imports in related files

* refactor: enhance ProgressModal styling for better alignment and readability

* refactor: update finalEvents assignment in ExportControls for session title generation

* refactor: comment out recorder section in PlaygroundPopup for future review

* chore(ci): fix lint error

---------

Co-authored-by: Claude <noreply@anthropic.com>
2025-06-11 16:40:21 +08:00
Leyang
aeab83c777
docs: add previous changelog (#816)
* docs(site): add previous changelog

* docs(site): add recently changelog

* docs(site): summary for previous changlog
2025-06-09 17:44:00 +08:00
Leyang
7ced918451
docs(site): add changelog for 0.18 (#815) 2025-06-09 11:55:25 +08:00
github-actions[bot]
67590d3273 release: v0.18.0 2025-06-09 03:04:12 +00:00
Leyang
37ae71241e
feat(web-integration): support logScreenshot api for agent (#813)
* feat(web-integration): support logScreenshot api for agent

* docs(core): update api description

* docs(site): fix error

* feat(web-integration): title is also optional

* docs(site): fix

---------

Co-authored-by: yutao <yutao.tao@bytedance.com>
2025-06-06 18:58:49 +08:00
Leyang
f3fcbea17b
feat(visualizer): support download video for player (#812)
* feat(visualizer): support download video for palyer

* fix(core): video reporter

---------

Co-authored-by: yutao <yutao.tao@bytedance.com>
2025-06-06 17:44:50 +08:00
Leyang
738251481c
docs(site): update gemini example (#811) 2025-06-06 13:37:36 +08:00
Leyang
a3fa0e23eb
support android auto hide keyborad for aiInput (#808)
* fix(android): modify automatically closing the keyboard after typed text (#793)

* fix(android): modify automatically closing the keyboard after typed text

* fix(android): rename to autoDismissKeyboard

* feat(android): support autoDismissKeyboard option for aiInput

* docs(core): update doc

* refactor(android): agent no need support autoDismissKeyboard parameter

* refactor(android): build parameter in build plan

* chore(core): update android docs

---------

Co-authored-by: welkang <welkang@qq.com>
Co-authored-by: yutao <yutao.tao@bytedance.com>
2025-06-06 10:56:05 +08:00
Leyang
b17b3b3ded
docs(site): changelog for 0.17.4 (#807)
* docs(site): changelog for 0.17.4

* docs(core): update change log

* docs(core): update change log

* docs(site): update sidebar title

---------

Co-authored-by: yutao <yutao.tao@bytedance.com>
2025-06-05 13:04:15 +08:00
Leyang
548a9801b8
feat: support customizing adb host (#801)
* feat(android): support custom host and port of adb

* feat(android): add options for custom ADB path, host, and port in AndroidDevice constructor

* docs(android): custom remote host and port for adb
2025-06-04 14:37:41 +08:00
github-actions[bot]
e8db6dc56a release: v0.17.5 2025-06-03 08:21:59 +00:00
github-actions[bot]
14b9200236 release: v0.17.4 2025-05-30 06:31:25 +00:00
Leyang
fd48cbd8c9
docs(core): add query right click and query from html docs (#787)
* docs(site): add right click docs

* docs(site): add ai query opts docs

* fix(web-integration): test

* docs(core): update api doc

---------

Co-authored-by: yutao <yutao.tao@bytedance.com>
2025-05-30 14:25:53 +08:00
yuyutaotao
ddc60d0416
docs(core): update blog of workflow (#776) 2025-05-28 22:20:09 +08:00
Leyang
8f4e722239
feat(core): support error message for ai assert in yaml (#774) 2025-05-27 19:15:25 +08:00
yuyutaotao
5451c27db4
fix(core): fix planning test (#773)
* fix(core): fix planning test

* fix(report): report insight view

* chore(core): rename workflow

* fix(core): report replay scripts

* fix(core): test case

* docs(core): update api doc for ui-tars

* chore(core): fix test case
2025-05-27 19:13:15 +08:00
github-actions[bot]
0dd38c2537 release: v0.17.3 2025-05-26 06:36:57 +00:00
yuyutaotao
8709830180
fix(core): error in showing query result in playground (#766)
* fix(core): error in showing query result in playground

* fix(core): error in showing query result in playground
2025-05-26 10:33:25 +08:00
Leyang
8c8e5d236a
fix(web-integration): add cacheable option to YAML actions and pass the cacheable parameter (#764) 2025-05-26 10:27:48 +08:00
Leyang
d07c6a78e3
feat(web-integration): support full xpath for cache (#754)
* feat(web-integration): support full xpath for cache

* refactor(web-integration): simplify getElementXPath to improve clarity and performance

* feat(web-integration): add support for 'A' node type and related extraction logic

* feat(web-integration): merge children rects of button

* feat(web-integration): enhance caching documentation and tests

* test(web-integration): update snapshot

* docs(core): update caching

---------

Co-authored-by: yutao <yutao.tao@bytedance.com>
2025-05-23 17:51:04 +08:00
github-actions[bot]
3b4671b0be release: v0.17.2 2025-05-23 06:18:49 +00:00