feat: bridge mode - phase 2 (#257)

* feat: allow auto reconnect in bridge mode after disconnect

* feat: show agent progress in extension

* feat: allow using bridge mode in yaml

* doc: add doc for bridge mode

* doc: add doc for bridge mode

* docs: update screenshot

README.md

@@ -41,12 +41,14 @@ Midscene.js is an AI-powered automation SDK can control the page, perform assert
 ## Resources 📄
 
 * [Home Page: https://midscenejs.com](https://midscenejs.com/)
-* [Quick Experience By Chrome Extension](https://midscenejs.com/quick-experience.html)
+* [Quick Experience By Chrome Extension](https://midscenejs.com/quick-experience.html), this is where you should get started 
+* Integration
+  * [Automate with Scripts in YAML](https://midscenejs.com/automate-with-scripts-in-yaml.html), use this if you prefer to write YAML file instead of code
+  * [Bridge Mode by Chrome Extension](https://midscenejs.com/bridge-mode-by-chrome-extension.html), use this to control the desktop Chrome by scripts
+  * [Integrate with Puppeteer](https://midscenejs.com/integrate-with-puppeteer.html)
+  * [Integrate with Playwright](https://midscenejs.com/integrate-with-playwright.html)
 * [API Reference](https://midscenejs.com/api.html)
-* [Automate with Scripts in YAML](https://midscenejs.com/automate-with-scripts-in-yaml.html)
-* [Integrate with Puppeteer](https://midscenejs.com/integrate-with-puppeteer.html)
-* [Integrate with Playwright](https://midscenejs.com/integrate-with-playwright.html)
-* [Customize Model and Provider](https://midscenejs.com/model-provider.html)
+* [Customize Model and Provider](https://midscenejs.com/model-provider.html), see how to use your own model and provider
 
 ## Community
 

README.zh.md

@@ -39,11 +39,13 @@ Midscene.js 是一个由 AI 驱动的自动化 SDK，能够使用自然语言对
 ## 资源 📄
 
 * [官网首页: https://midscenejs.com](https://midscenejs.com/zh)
-* [使用 Chrome 插件体验](https://midscenejs.com/zh/quick-experience.html)
+* [使用 Chrome 插件体验](https://midscenejs.com/zh/quick-experience.html)，请从这里开始体验 Midscene 
+* 集成方案
+  * [使用 YAML 格式的自动化脚本](https://midscenejs.com/zh/automate-with-scripts-in-yaml.html), 如果你更喜欢写 YAML 文件而不是代码
+  * [使用 Chrome 插件桥接模式（Bridge Mode）](https://midscenejs.com/zh/bridge-mode-by-chrome-extension.html), 使用 Midscene 来控制桌面端 Chrome 
+  * [集成到 Puppeteer](https://midscenejs.com/zh/integrate-with-puppeteer.html)
+  * [集成到 Playwright](https://midscenejs.com/zh/integrate-with-playwright.html)
 * [API 文档](https://midscenejs.com/zh/api.html)
-* [使用 YAML 格式的自动化脚本](https://midscenejs.com/zh/automate-with-scripts-in-yaml.html)
-* [集成到 Puppeteer](https://midscenejs.com/zh/integrate-with-puppeteer.html)
-* [集成到 Playwright](https://midscenejs.com/zh/integrate-with-playwright.html)
 * [自定义模型和服务商](https://midscenejs.com/zh/model-provider.html)
 
 ## 社区

apps/site/docs/en/automate-with-scripts-in-yaml.mdx

@@ -144,6 +144,9 @@ target:
   
   # string, the path to save the aiQuery result, optional
   output: <path-to-output-file>
+
+  # string, the bridge mode to use, optional, default is 'currentTab', can be 'newTabWithUrl' or 'currentTab'
+  bridgeMode: <mode>
 ```
 
 The `tasks` part is an array indicates the tasks to do. Remember to write a `-` before each item which means an array item.
@@ -178,6 +181,11 @@ tasks:
       # ...
 ```
 
+## Use bridge mode
+By using bridge mode, you can utilize YAML scripts to automate the web browser on your desktop. This is particularly useful if you want to reuse cookies, plugins, and page states, or if you want to manually interact with automation scripts.
+
+See [Bridge Mode by Chrome Extension](./bridge-mode-by-chrome-extension.html) for more details.
+
 ## FAQ
 
 **How to get cookies in JSON format from Chrome?**

apps/site/docs/en/bridge-mode-by-chrome-extension.mdx

@@ -86,7 +86,38 @@ Destroy the connection.
 
 ## Use bridge mode in yaml-script
 
-We are still building this, and it will be ready soon.
+[Yaml scripts](./automate-with-scripts-in-yaml) is a way for developers to write automation scripts in yaml format, which is easy to read and write comparing to javascript.
+
+To use bridge mode in yaml script, set the `bridgeMode` property in the `target` section. If you want to use the current tab, set it to `currentTab`, otherwise set it to `newTabWithUrl`.
+
+For example, the following script will open a new tab by Chrome extension bridge: 
+
+```diff
+target:
+  url: https://www.bing.com
++ bridgeMode: newTabWithUrl
+tasks:
+```
+
+Run the script:
+
+```bash
+midscene ./bing.yaml
+```
+
+Remember to start the chrome extension and click 'Allow connection' button after the script is running.
+
+### Unsupported options
+
+In bridge mode, these options will be ignored (they will follow your desktop browser's settings):
+- userAgent
+- viewportWidth
+- viewportHeight
+- viewportScale
+- waitForNetworkIdle
+- cookie
+
+
 
 
 

apps/site/docs/en/faq.md

@@ -12,7 +12,8 @@ There are some limitations with Midscene. We are still working on them.
 
 1. The interaction types are limited to only tap, type, keyboard press, and scroll.
 2. LLM is not 100% stable. Even GPT-4o can't return the right answer all the time. Following the [Prompting Tips](./prompting-tips) will help improve stability.
-3. Since we use JavaScript to retrieve items from the page, the elements inside the iframe cannot be accessed.
+3. Since we use JavaScript to retrieve elements from the page, the elements inside the iframe cannot be accessed.
+4. Do not use Midscene to bypass CAPTCHA. Some LLM services are set to decline requests that involve CAPTCHA-solving (e.g., OpenAI), while the DOM of some CAPTCHA pages is not accessible by regular web scraping methods. Therefore, using Midscene to bypass CAPTCHA is not a reliable method.
 
 ## Can I use a model other than `gpt-4o`?
 

apps/site/docs/en/index.mdx

@@ -59,9 +59,9 @@ Also, there are several ways to integrate Midscene into your code project:
 
 Midscene will provide a visual report after each run. With this report, you can review the animated replay and view the details of each step in the process. What's more, there is a playground in the report file for you to adjust your prompt without re-running all your scripts.
 
-![](/report.gif)
-
-![](/playground.png)
+<p align="center">
+  <img src="/report.gif" alt="visualized report" />
+</p>
 
 ## Just you and model provider, no third-party services
 

apps/site/docs/zh/automate-with-scripts-in-yaml.mdx

@@ -178,6 +178,12 @@ tasks:
       # ...
 ```
 
+## 使用桥接模式
+
+通过使用桥接模式，你可以利用 YAML 脚本在已有的桌面浏览器上执行自动化。这对于需要复用 Cookies、插件和页面状态，或者需要手工与自动化脚本交互的情况非常有用。
+
+请参阅 [通过 Chrome 扩展桥接模式](./bridge-mode-by-chrome-extension.html) 了解更多详细信息。
+
 ## FAQ
 
 **如何从 Chrome 中获取 JSON 格式的 Cookies？**

apps/site/docs/zh/bridge-mode-by-chrome-extension.mdx

@@ -84,11 +84,37 @@ tsx demo-new-tab.ts
 
 销毁连接。
 
-## 在 YAML 脚本中使用桥接模式
-
-这个功能正在开发中，很快就会与你见面。
-
-
+## 在 YAML 自动化脚本中使用桥接模式
 
+[Yaml 格式的自动化脚本](./automate-with-scripts-in-yaml) 是 Midscene 提供给开发者的一种编写自动化脚本的方式。通过使用 yaml 格式，脚本会变得易于阅读和编写。
+
+在 Yaml 脚本中使用桥接模式时，需要配置 `target` 中的 `bridgeMode` 属性。如果想要使用当前标签页，设置为 `currentTab`，否则设置为 `newTabWithUrl`。
+
+例如，以下脚本将会通过 Chrome 插件的桥接模式打开一个新的标签页：
+
+```diff
+target:
+  url: https://www.bing.com
++ bridgeMode: newTabWithUrl
+tasks:
+```
+
+运行脚本：
+
+```bash
+midscene ./bing.yaml
+```
+
+在运行脚本后，记得要启动 Chrome 插件，并点击 'Allow connection' 按钮。
+
+### 不支持的选项
+
+在桥接模式下，以下选项将不会生效（它们将遵循桌面浏览器的设置）：
+- userAgent
+- viewportWidth
+- viewportHeight
+- viewportScale
+- waitForNetworkIdle
+- cookie
 
 

apps/site/docs/zh/faq.md

@@ -15,6 +15,7 @@ Midscene 存在一些局限性，我们仍在努力改进。
 1. 交互类型有限：目前仅支持点击、输入、键盘和滚动操作。
 2. 稳定性风险：即使是 GPT-4o 也无法确保 100% 返回正确答案。遵循 [编写提示词的技巧](./prompting-tips) 可以帮助提高 SDK 稳定性。
 3. 元素访问受限：由于我们使用 JavaScript 从页面提取元素，所以无法访问 iframe 内部的元素。
+4. 无法绕过验证码：有些 LLM 服务会拒绝涉及验证码解决的请求（例如 OpenAI），而有些验证码页面的 DOM 无法通过常规的网页抓取方法访问。因此，使用 Midscene 绕过验证码不是一个可靠的方法。
 
 ## 能否选用 `gpt-4o` 以外的其他模型？
 

apps/site/docs/zh/index.mdx

@@ -48,9 +48,9 @@ console.log("headphones in stock", items);
 
 此外，Midscene 报告里还集成了一个 Playground，用以在报告中重新运行 Prompt 并调试。
 
-![](/report.gif)
-
-![](/playground.png)
+<p align="center">
+  <img src="/report.gif" alt="visualized report" />
+</p>
 
 ## 直连模型端，无需三方服务
 

packages/cli/src/yaml-runner.ts

@@ -7,15 +7,14 @@ import {
   contextInfo,
   contextTaskListSummary,
   isTTY,
-  singleTaskInfo,
   spinnerInterval,
 } from './printer';
 import { TTYWindowRenderer } from './tty-renderer';
 
-import { assert } from 'node:console';
+import assert from 'node:assert';
 import type { FreeFn } from '@midscene/core';
+import { AgentOverChromeBridge } from '@midscene/web/bridge-mode';
 import { puppeteerAgentForTarget } from '@midscene/web/puppeteer';
-
 export const launchServer = async (
   dir: string,
 ): Promise<ReturnType<typeof createServer>> => {
@@ -48,30 +47,30 @@ export async function playYamlFiles(
       keepWindow: options?.keepWindow,
       testId: fileName,
     };
-    const player = new ScriptPlayer(
-      script,
-      async (target) => {
-        const freeFn: FreeFn[] = [];
+    const player = new ScriptPlayer(script, async (target) => {
+      const freeFn: FreeFn[] = [];
 
-        // launch local server if needed
-        let localServer: Awaited<ReturnType<typeof launchServer>> | undefined;
-        let urlToVisit: string | undefined;
-        assert(typeof target.url === 'string', 'url is required');
-        if (target.serve) {
-          localServer = await launchServer(target.serve);
-          const serverAddress = localServer.server.address();
-          freeFn.push({
-            name: 'local_server',
-            fn: () => localServer?.server.close(),
-          });
-          if (target.url.startsWith('/')) {
-            urlToVisit = `http://${serverAddress?.address}:${serverAddress?.port}${target.url}`;
-          } else {
-            urlToVisit = `http://${serverAddress?.address}:${serverAddress?.port}/${target.url}`;
-          }
-          target.url = urlToVisit;
+      // launch local server if needed
+      let localServer: Awaited<ReturnType<typeof launchServer>> | undefined;
+      let urlToVisit: string | undefined;
+      if (target.serve) {
+        assert(typeof target.url === 'string', 'url is required in serve mode');
+        localServer = await launchServer(target.serve);
+        const serverAddress = localServer.server.address();
+        freeFn.push({
+          name: 'local_server',
+          fn: () => localServer?.server.close(),
+        });
+        if (target.url.startsWith('/')) {
+          urlToVisit = `http://${serverAddress?.address}:${serverAddress?.port}${target.url}`;
+        } else {
+          urlToVisit = `http://${serverAddress?.address}:${serverAddress?.port}/${target.url}`;
         }
+        target.url = urlToVisit;
+      }
 
+      // puppeteer
+      if (!target.bridgeMode) {
         const { agent, freeFn: newFreeFn } = await puppeteerAgentForTarget(
           target,
           preference,
@@ -79,14 +78,46 @@ export async function playYamlFiles(
         freeFn.push(...newFreeFn);
 
         return { agent, freeFn };
-      },
-      (taskStatus) => {
-        if (!isTTY) {
-          const { nameText } = singleTaskInfo(taskStatus);
-          // console.log(`${taskStatus.status} - ${nameText}`);
+      }
+
+      // bridge mode
+      assert(
+        target.bridgeMode === 'newTabWithUrl' ||
+          target.bridgeMode === 'currentTab',
+        `bridgeMode config value must be either "newTabWithUrl" or "currentTab", but got ${target.bridgeMode}`,
+      );
+
+      if (
+        target.userAgent ||
+        target.viewportWidth ||
+        target.viewportHeight ||
+        target.viewportScale ||
+        target.waitForNetworkIdle ||
+        target.cookie
+      ) {
+        console.warn(
+          'puppeteer options (userAgent, viewportWidth, viewportHeight, viewportScale, waitForNetworkIdle, cookie) are not supported in bridge mode, will be ignored',
+        );
+      }
+
+      const agent = new AgentOverChromeBridge();
+      if (target.bridgeMode === 'newTabWithUrl') {
+        await agent.connectNewTabWithUrl(target.url);
+      } else {
+        if (target.url) {
+          console.warn('url will be ignored in bridge mode with "currentTab"');
         }
-      },
-    );
+        await agent.connectCurrentTab();
+      }
+      freeFn.push({
+        name: 'destroy_agent_over_chrome_bridge',
+        fn: () => agent.destroy(),
+      });
+      return {
+        agent,
+        freeFn,
+      };
+    });
     fileContextList.push({ file, player });
   }
 

packages/cli/tests/midscene_scripts_bridge/current_tab/check_content.yaml

@@ -0,0 +1,6 @@
+target:
+  bridgeMode: currentTab
+tasks:
+  - name: check page content
+    flow:
+      - aiAssert: this is a web page

packages/cli/tests/midscene_scripts_bridge/new_tab/bing.yaml

@@ -0,0 +1,13 @@
+target:
+  url: https://www.bing.com
+  bridgeMode: newTabWithUrl
+tasks:
+  - name: search weather
+    flow:
+      - sleep: 5000
+      - ai: input 'weather today' in input box, click search button
+      - sleep: 5000
+
+  - name: check result
+    flow:
+      - aiAssert: the result shows the weather info

packages/cli/tests/midscene_scripts_bridge/new_tab/local.yml

@@ -0,0 +1,8 @@
+target:
+  serve: ./tests/server_root
+  url: index.html
+  bridgeMode: newTabWithUrl
+tasks:
+  - name: check title
+    flow:
+      - aiAssert: the content title is "My App"

packages/midscene/src/types.ts

@@ -197,6 +197,8 @@ export type InsightAssertionResponse = AIAssertionResponse;
  * agent
  */
 
+export type OnTaskStartTip = (tip: string) => Promise<void> | void;
+
 export interface AgentWaitForOpt {
   checkIntervalMs?: number;
   timeoutMs?: number;

packages/midscene/src/yaml.d.ts

@@ -10,17 +10,22 @@ export interface MidsceneYamlTask {
 }
 
 export interface MidsceneYamlScriptEnv {
+  serve?: string;
   url: string;
+
+  // puppeteer only
   userAgent?: string;
   viewportWidth?: number;
   viewportHeight?: number;
   viewportScale?: number;
-  serve?: string;
   waitForNetworkIdle?: {
     timeout?: number; // ms, 30000 for default, set to 0 to disable
     continueOnNetworkIdleError?: boolean; // should continue if failed to wait for network idle, true for default
   };
   cookie?: string;
+
+  // bridge mode only
+  bridgeMode?: 'newTabWithUrl' | 'currentTab';
   output?: string;
 }
 

packages/visualizer/src/extension/bridge.tsx

@@ -17,12 +17,13 @@ enum BridgeStatus {
   Connected = 'connected',
 }
 
-const connectTimeout = 30 * 1000;
+const connectTimeout = 60 * 1000;
 const connectRetryInterval = 300;
 export default function Bridge() {
   const activeBridgePageRef = useRef<ChromeExtensionPageBrowserSide | null>(
     null,
   );
+  const allowAutoConnectionRef = useRef(false);
 
   const [bridgeStatus, setBridgeStatus] = useState<BridgeStatus>(
     BridgeStatus.Closed,
@@ -59,6 +60,7 @@ export default function Bridge() {
 
   const stopListeningFlag = useRef(false);
   const stopListening = () => {
+    allowAutoConnectionRef.current = false;
     stopListeningFlag.current = true;
   };
 
@@ -68,20 +70,28 @@ export default function Bridge() {
       throw new Error('There is already a connection, cannot start a new one');
     }
     const startTime = Date.now();
-    setBridgeLog([]);
     setBridgeAgentStatus('');
     appendBridgeLog('Listening for connection...');
     setBridgeStatus(BridgeStatus.OpenForConnection);
     stopListeningFlag.current = false;
 
+    let noConnectionTip = 'No connection found within timeout';
+    console.log('startConnection');
     while (Date.now() - startTime < timeout) {
       try {
         if (stopListeningFlag.current) {
+          noConnectionTip = 'Listening stopped by user';
           break;
         }
         const activeBridgePage = new ChromeExtensionPageBrowserSide(
           () => {
+            console.log('stopConnection');
             stopConnection();
+            if (allowAutoConnectionRef.current) {
+              setTimeout(() => {
+                startConnection();
+              }, connectRetryInterval);
+            }
           },
           (message, type) => {
             appendBridgeLog(message);
@@ -102,7 +112,7 @@ export default function Bridge() {
     }
 
     setBridgeStatus(BridgeStatus.Closed);
-    appendBridgeLog('No connection found within timeout');
+    appendBridgeLog(noConnectionTip);
   };
 
   let statusElement: any;
@@ -116,7 +126,13 @@ export default function Bridge() {
       </span>
     );
     statusBtn = (
-      <Button type="primary" onClick={() => startConnection()}>
+      <Button
+        type="primary"
+        onClick={() => {
+          allowAutoConnectionRef.current = true;
+          startConnection();
+        }}
+      >
         Allow connection
       </Button>
     );
@@ -147,10 +163,28 @@ export default function Bridge() {
         </span>
       </span>
     );
-    statusBtn = <Button onClick={stopConnection}>Stop</Button>;
+    statusBtn = (
+      <Button
+        onClick={() => {
+          allowAutoConnectionRef.current = false;
+          stopConnection();
+        }}
+      >
+        Stop
+      </Button>
+    );
   } else {
     statusElement = <span>Unknown Status - {bridgeStatus}</span>;
-    statusBtn = <Button onClick={stopConnection}>Stop</Button>;
+    statusBtn = (
+      <Button
+        onClick={() => {
+          allowAutoConnectionRef.current = false;
+          stopConnection();
+        }}
+      >
+        Stop
+      </Button>
+    );
   }
 
   const logs = [...bridgeLog].reverse().map((log, index) => {
@@ -175,7 +209,11 @@ export default function Bridge() {
         In Bridge Mode, you can control this browser by the Midscene SDK running
         in the local terminal. This is useful for interacting both through
         scripts and manually, or to reuse cookies.{' '}
-        <a href="https://www.midscenejs.com/bridge-mode-by-chrome-extension">
+        <a
+          href="https://www.midscenejs.com/bridge-mode-by-chrome-extension"
+          target="_blank"
+          rel="noreferrer"
+        >
           More about bridge mode
         </a>
       </p>

packages/visualizer/src/extension/popup.tsx

@@ -53,21 +53,6 @@ const shotAndOpenPlayground = async (
   });
 };
 
-// {
-//   /* <p>
-//             To keep the current page context, you can also{' '}
-//             <Button
-//               onClick={handleSendToPlayground}
-//               loading={loading}
-//               type="link"
-//               size="small"
-//               icon={<SendOutlined />}
-//             >
-//               send to fullscreen playground
-//             </Button>
-//           </p> */
-// }
-
 function PlaygroundPopup() {
   const [loading, setLoading] = useState(false);
   const extensionVersion = getExtensionVersion();

packages/web-integration/src/bridge-mode/agent-cli-side.ts

@@ -92,7 +92,11 @@ export const getBridgePageInCliSide = (): ChromeExtensionPageCliSide => {
 export class AgentOverChromeBridge extends PageAgent<ChromeExtensionPageCliSide> {
   constructor() {
     const page = getBridgePageInCliSide();
-    super(page, {});
+    super(page, {
+      onTaskStartTip: (tip: string) => {
+        this.page.showStatusMessage(tip);
+      },
+    });
   }
 
   async connectNewTabWithUrl(url: string) {
@@ -109,11 +113,6 @@ export class AgentOverChromeBridge extends PageAgent<ChromeExtensionPageCliSide>
         'the `options` parameter of aiAction is not supported in cli side',
       );
     }
-    return await super.aiAction(prompt, {
-      onTaskStart: (task) => {
-        const tip = `${typeStr(task)} - ${paramStr(task)}`;
-        this.page.showStatusMessage(tip);
-      },
-    });
+    return await super.aiAction(prompt);
   }
 }

packages/web-integration/src/bridge-mode/io-server.ts

@@ -60,9 +60,10 @@ export class BridgeServer {
         this.connectionTipTimer && clearTimeout(this.connectionTipTimer);
         this.connectionTipTimer = null;
         if (this.socket) {
-          console.log('server already connected, refusing new connection');
           socket.emit(BridgeEvent.Refused);
-          reject(new Error('server already connected by another client'));
+          return reject(
+            new Error('server already connected by another client'),
+          );
         }
 
         try {
@@ -71,10 +72,7 @@ export class BridgeServer {
 
           const clientVersion = socket.handshake.query.version;
           console.log(
-            'Bridge connected, cli-side version:',
-            __VERSION__,
-            ', browser-side version:',
-            clientVersion,
+            `Bridge connected, cli-side version v${__VERSION__}, browser-side version v${clientVersion}`,
           );
 
           socket.on(BridgeEvent.CallResponse, (params: BridgeCallResponse) => {
@@ -88,7 +86,12 @@ export class BridgeServer {
           socket.on('disconnect', (reason: string) => {
             this.connectionLost = true;
             this.connectionLostReason = reason;
-            this.onDisconnect?.(reason);
+
+            try {
+              this.io?.close();
+            } catch (e) {
+              // ignore
+            }
 
             // flush all pending calls as error
             for (const id in this.calls) {
@@ -103,6 +106,8 @@ export class BridgeServer {
                 );
               }
             }
+
+            this.onDisconnect?.(reason);
           });
 
           setTimeout(() => {

packages/web-integration/src/bridge-mode/page-browser-side.ts

@@ -83,7 +83,7 @@ export class ChromeExtensionPageBrowserSide extends ChromeExtensionProxyPage {
     );
     await this.bridgeClient.connect();
     this.onLogMessage(
-      `Bridge connected, cli-side version ${this.bridgeClient.serverVersion}, browser-side version: ${__VERSION__}`,
+      `Bridge connected, cli-side version v${this.bridgeClient.serverVersion}, browser-side version v${__VERSION__}`,
       'log',
     );
   }

packages/web-integration/src/common/agent.ts

@@ -3,10 +3,11 @@ import {
   type AgentAssertOpt,
   type AgentWaitForOpt,
   type ExecutionDump,
-  type ExecutionTaskProgressOptions,
+  type ExecutionTask,
   type GroupedActionDump,
   Insight,
   type InsightAction,
+  type OnTaskStartTip,
 } from '@midscene/core';
 import { NodeType } from '@midscene/shared/constants';
 
@@ -19,6 +20,7 @@ import {
 import { PageTaskExecutor } from '../common/tasks';
 import { WebElementInfo } from '../web-element';
 import type { AiTaskCache } from './task-cache';
+import { paramStr, typeStr } from './ui-utils';
 import { printReportMsg, reportFileName } from './utils';
 import { type WebUIContext, parseContextFromWebPage } from './utils';
 
@@ -32,6 +34,7 @@ export interface PageAgentOpt {
   generateReport?: boolean;
   /* if auto print report msg, default true */
   autoPrintReportMsg?: boolean;
+  onTaskStartTip?: OnTaskStartTip;
 }
 
 export class PageAgent<PageType extends WebPage = WebPage> {
@@ -142,8 +145,17 @@ export class PageAgent<PageType extends WebPage = WebPage> {
     }
   }
 
-  async aiAction(taskPrompt: string, options?: ExecutionTaskProgressOptions) {
-    const { executor } = await this.taskExecutor.action(taskPrompt, options);
+  private async callbackOnTaskStartTip(task: ExecutionTask) {
+    if (this.opts.onTaskStartTip) {
+      const tip = `${typeStr(task)} - ${paramStr(task)}`;
+      await this.opts.onTaskStartTip(tip);
+    }
+  }
+
+  async aiAction(taskPrompt: string) {
+    const { executor } = await this.taskExecutor.action(taskPrompt, {
+      onTaskStart: this.callbackOnTaskStartTip.bind(this),
+    });
     this.appendExecutionDump(executor.dump());
     this.writeOutActionDumps();
 
@@ -154,7 +166,9 @@ export class PageAgent<PageType extends WebPage = WebPage> {
   }
 
   async aiQuery(demand: any) {
-    const { output, executor } = await this.taskExecutor.query(demand);
+    const { output, executor } = await this.taskExecutor.query(demand, {
+      onTaskStart: this.callbackOnTaskStartTip.bind(this),
+    });
     this.appendExecutionDump(executor.dump());
     this.writeOutActionDumps();
 
@@ -166,7 +180,9 @@ export class PageAgent<PageType extends WebPage = WebPage> {
   }
 
   async aiAssert(assertion: string, msg?: string, opt?: AgentAssertOpt) {
-    const { output, executor } = await this.taskExecutor.assert(assertion);
+    const { output, executor } = await this.taskExecutor.assert(assertion, {
+      onTaskStart: this.callbackOnTaskStartTip.bind(this),
+    });
     this.appendExecutionDump(executor.dump());
     this.writeOutActionDumps();
 

packages/web-integration/src/common/tasks.ts

@@ -619,10 +619,15 @@ export class PageTaskExecutor {
     };
   }
 
-  async query(demand: InsightExtractParam): Promise<ExecutionResult> {
+  async query(
+    demand: InsightExtractParam,
+    options?: ExecutionTaskProgressOptions,
+  ): Promise<ExecutionResult> {
     const description =
       typeof demand === 'string' ? demand : JSON.stringify(demand);
-    const taskExecutor = new Executor(description);
+    const taskExecutor = new Executor(description, undefined, undefined, {
+      onTaskStart: options?.onTaskStart,
+    });
     const queryTask: ExecutionTaskInsightQueryApply = {
       type: 'Insight',
       subType: 'Query',
@@ -654,9 +659,12 @@ export class PageTaskExecutor {
 
   async assert(
     assertion: string,
+    options?: ExecutionTaskProgressOptions,
   ): Promise<ExecutionResult<InsightAssertionResponse>> {
     const description = `assert: ${assertion}`;
-    const taskExecutor = new Executor(description);
+    const taskExecutor = new Executor(description, undefined, undefined, {
+      onTaskStart: options?.onTaskStart,
+    });
     const assertionPlan: PlanningAction<PlanningActionParamAssert> = {
       type: 'Assert',
       param: {

packages/web-integration/src/yaml/player.ts

@@ -95,8 +95,6 @@ export class ScriptPlayer {
   async playTask(taskStatus: ScriptPlayerTaskStatus, agent: PageAgent) {
     const { flow } = taskStatus;
     assert(flow, 'missing flow in task');
-    const notifyTaskStatusChange =
-      this.notifyCurrentTaskStatusChange.bind(this);
 
     for (const flowItemIndex in flow) {
       const currentStep = Number.parseInt(flowItemIndex, 10);
@@ -113,17 +111,7 @@ export class ScriptPlayer {
           typeof prompt === 'string',
           'prompt for aiAction must be a string',
         );
-        await agent.aiAction(prompt, {
-          onTaskStart(task) {
-            const tip = `${typeStr(task)} - ${paramStr(task)}`;
-            const actionItem = flowItem as MidsceneYamlFlowItemAIAction;
-            actionItem.aiActionProgressTips =
-              actionItem.aiActionProgressTips || [];
-            actionItem.aiActionProgressTips.push(tip);
-
-            notifyTaskStatusChange();
-          },
-        });
+        await agent.aiAction(prompt);
       } else if ((flowItem as MidsceneYamlFlowItemAIAssert).aiAssert) {
         const assertTask = flowItem as MidsceneYamlFlowItemAIAssert;
         const prompt = assertTask.aiAssert;
@@ -226,10 +214,14 @@ export class ScriptPlayer {
     }
 
     // free the resources
-    freeFn.forEach((fn) => {
+    for (const fn of freeFn) {
       try {
-        fn.fn();
-      } catch (e) {}
-    });
+        // console.log('freeing', fn.name);
+        await fn.fn();
+        // console.log('freed', fn.name);
+      } catch (e) {
+        // console.error('error freeing', fn.name, e);
+      }
+    }
   }
 }

packages/web-integration/src/yaml/utils.ts

@@ -22,10 +22,6 @@ export function parseYamlScript(
     typeof obj.target === 'object',
     `property "target" must be an object${pathTip}`,
   );
-  assert(
-    typeof obj.target.url === 'string',
-    `property "target.url" must be provided in yaml script: ${pathTip}`,
-  );
   assert(obj.tasks, `property "tasks" is required in yaml script${pathTip}`);
   assert(
     Array.isArray(obj.tasks),

packages/web-integration/tests/ai/web/puppeteer/showcase.test.ts

@@ -10,18 +10,17 @@ describe(
       const { originPage, reset } = await launchPage(
         'https://www.saucedemo.com/',
       );
+      const onTaskStartTip = vi.fn();
       const mid = new PuppeteerAgent(originPage, {
         cacheId: 'puppeteer(Sauce Demo by Swag Lab)',
+        onTaskStartTip,
       });
 
-      const onTaskStart = vi.fn();
-
       await mid.aiAction(
         'type "standard_user" in user name input, type "secret_sauce" in password, click "Login", sleep 1s',
-        { onTaskStart: onTaskStart as any },
       );
 
-      expect(onTaskStart.mock.calls.length).toBeGreaterThan(1);
+      expect(onTaskStartTip.mock.calls.length).toBeGreaterThan(1);
 
       await expect(async () => {
         await mid.aiWaitFor('there is a cookie prompt in the UI', {

packages/web-integration/tests/unit-test/bridge/io.test.ts

@@ -180,4 +180,62 @@ describe('bridge-io', () => {
     server.close();
     client.disconnect();
   });
+
+  it('callback error after client disconnect', async () => {
+    const port = testPort++;
+    const server = new BridgeServer(port);
+    server.listen();
+
+    const client = new BridgeClient(
+      `ws://localhost:${port}`,
+      (method, args) => {
+        return new Promise((resolve, reject) => {
+          setTimeout(() => {
+            resolve('ok');
+          }, 100 * 1000);
+        });
+      },
+    );
+    await client.connect();
+
+    const callPromise = server.call('test', ['a', 'b']);
+
+    // sleep 2s
+    await new Promise((resolve) => setTimeout(resolve, 2 * 1000));
+
+    await client.disconnect();
+    await expect(callPromise).rejects.toThrow(/Connection lost/);
+  });
+
+  it('multiple server', async () => {
+    const commonPort = testPort++;
+    const server1 = new BridgeServer(commonPort);
+    server1.listen();
+
+    const client = new BridgeClient(
+      `ws://localhost:${commonPort}`,
+      (method, args) => {
+        return Promise.resolve('ok');
+      },
+    );
+    await client.connect();
+    await new Promise((resolve) => setTimeout(resolve, 1000));
+    await client.disconnect();
+    // server port should be closed at this time
+    await new Promise((resolve) => setTimeout(resolve, 1000));
+
+    const server2 = new BridgeServer(commonPort);
+    server2.listen();
+
+    const client2 = new BridgeClient(
+      `ws://localhost:${commonPort}`,
+      (method, args) => {
+        return Promise.resolve('ok2');
+      },
+    );
+    await client2.connect();
+
+    const res = await server2.call('test', ['a', 'b']);
+    expect(res).toEqual('ok2');
+  });
 });