midscene/apps/site/docs/zh/integrate-with-android.mdx

216 lines
5.9 KiB
Plaintext
Raw Permalink Normal View History

import PrepareAndroid from './common/prepare-android.mdx';
import SetupEnv from './common/setup-env.mdx';
# 与 Android(adb) 集成
在使用 adb 连接 Android 设备后,你可以使用 Midscene javascript SDK 来控制 Android 设备。
import { PackageManagerTabs } from '@theme';
:::info 样例项目
使用 javascript SDK 控制 Android 设备:[https://github.com/web-infra-dev/midscene-example/blob/main/android/javascript-sdk-demo](https://github.com/web-infra-dev/midscene-example/blob/main/android/javascript-sdk-demo)
与 Vitest 集成和测试:[https://github.com/web-infra-dev/midscene-example/tree/main/android/vitest-demo](https://github.com/web-infra-dev/midscene-example/tree/main/android/vitest-demo)
:::
<PrepareAndroid />
<SetupEnv />
## 第一步:安装依赖
<PackageManagerTabs command="install @midscene/android --save-dev" />
## 第二步:编写脚本
这里以使用安卓浏览器搜索耳机为例。(当然,你也可以使用设备上的其他任何应用)
编写下方代码,保存为 `./demo.ts`
```typescript title="./demo.ts"
import { AndroidAgent, AndroidDevice, getConnectedDevices } from '@midscene/android';
const sleep = (ms) => new Promise((r) => setTimeout(r, ms));
Promise.resolve(
(async () => {
const devices = await getConnectedDevices();
const page = new AndroidDevice(devices[0].udid);
// 👀 初始化 Midscene agent
const agent = new AndroidAgent(page,{
aiActionContext:
'如果出现位置、权限、用户协议等弹窗,点击同意。如果出现登录页面,关闭它。',
});
await page.connect();
// 👀 打开浏览器并导航到 ebay.com请确保当前页面有浏览器 App 喔)
await agent.aiAction('open browser and navigate to ebay.com');
await sleep(5000);
// 👀 输入关键词,执行搜索
await agent.aiAction('在搜索框输入 "Headphones" ,敲回车');
// 👀 等待加载完成
await agent.aiWaitFor("页面中至少有一个耳机商品");
// 或者你也可以使用一个普通的 sleep:
// await sleep(5000);
// 👀 理解页面内容,提取数据
const items = await agent.aiQuery(
"{itemTitle: string, price: Number}[], 找到列表里的商品标题和价格"
);
console.log("耳机商品信息", items);
// 👀 用 AI 断言
await agent.aiAssert("界面左侧有类目筛选功能");
})()
);
```
## 第三步:运行
使用 `tsx` 来运行
```bash
# run
npx tsx demo.ts
```
稍等片刻,你会看到如下输出:
```log
[
{
itemTitle: 'JBL Tour Pro 2 - True wireless Noise Cancelling earbuds with Smart Charging Case',
price: 551.21
},
{
itemTitle: 'Soundcore Space One无线耳机40H ANC播放时间2XStronger语音还原',
price: 543.94
}
]
```
## 第四步:查看运行报告
当上面的命令执行成功后,会在控制台输出:`Midscene - report file updated: /path/to/report/some_id.html` 通过浏览器打开该文件即可看到报告。
## Android Agent 上的更多接口
除了 [API 参考](./API) 中的通用 Agent 接口AndroidAgent 还提供了一些其他接口:
### `agent.launch()`
启动一个网页或原生页面。
* 类型
```typescript
function launch(uri: string): Promise<void>;
```
* 参数:
feat: android playground (#542) * refactor: android api * refactor: enhance Android agent to accept options for device connection * fix: type error * fix: click after clearInput * fix: click before clearInput * feat: android playground * feat: support npx package name * feat: android playground joint * fix: git ignore conflicts * feat: ensure adb server is running before initializing adb client * fix: deps consistency * ci: add android playground * feat: integrate shared constants and improve server configuration in android playground * feat: android playground style * feat: style opt * feat: add @rsbuild/plugin-svgr dependency and improve URI handling in adb * feat: remove unused water flow scripts and update comments to English * feat: download report file * feat: standalone android playground * feat: use dynamic import * feat: migrate CSS to LESS and remove unused styles in chrome extension and report * feat: enhance Android playground with ScrcpyPlayer ref integration and device management improvements * feat: optimize styles and layout in Android playground and visualizer components * chore: add bin back * chore: update build script to exclude documentation generation * feat: add not ready message to PlaygroundResult for improved user guidance * feat: add error handling for screenshot capture in Android page * docs: update readme * feat: add PNG validation for screenshot buffer in Android page * feat: enhance UI components with improved styling and tooltips in ScrcpyPlayer and PromptInput * docs: update uri parameter description in integrate-with-android documentation and improve uri handling in launch function * style: update primary color to #2B83FF across multiple components and adjust margin in App.less * refactor: replace userConfig with globalConfig for environment configuration management and update related functions * feat: integrate server validation logic in App, AdbDevice, and ScrcpyPlayer components for improved connection handling * style: enhance player component layout with overflow handling and margin adjustments * style: refine player component layout with flex adjustments and improved spacing * feat: add midscene model name display and improve layout in EnvConfig component * feat: integrate ShinyText component for enhanced loading progress display in PlaygroundResult * test: add test for isValidPNGImageBuffer * style: remove background color from App.less and adjust AI config override behavior in env.ts --------- Co-authored-by: yutao <yutao.tao@bytedance.com>
2025-04-17 17:44:11 +08:00
* `uri: string` - 要打开的 uri可以是网页 url 或原生 app 的 package name 或 activity name如果存在 activity name则以 / 分隔例如com.android.settings/.Settings
* 返回值:
* `Promise<void>`
* 示例:
```typescript
import { AndroidAgent, AndroidDevice } from '@midscene/android';
const page = new AndroidDevice('s4ey59');
const agent = new AndroidAgent(page);
await agent.launch('https://www.ebay.com'); // 打开网页
await agent.launch('com.android.settings'); // 打开系统设置 app(package name)
await agent.launch('com.android.settings/.Settings'); // 打开系统设置 app(package name) 的 .Settings(activity name) 页面
```
### `agentFromAdbDevice()`
从已连接的 adb 设备中,创建一个 AndroidAgent。
* 类型
```typescript
function agentFromAdbDevice(deviceId?: string, opts?: PageAgentOpt): Promise<AndroidAgent>;
```
* 参数:
* `deviceId?: string` - 可选参数,要连接的 adb 设备 id如果未传入则使用第一个连接的设备
* `opts?: PageAgentOpt` - 可选参数,用于初始化 AndroidAgent 的配置,参考 [构造器](./API)
* 返回值:
* `Promise<AndroidAgent>` 返回一个 AndroidAgent 实例
* 示例:
```typescript
import { agentFromAdbDevice } from '@midscene/android';
const agent = await agentFromAdbDevice('s4ey59'); // 传入 deviceId
const agent = await agentFromAdbDevice(); // 不传入 deviceId则使用第一个连接的设备
```
### `getConnectedDevices()`
获取所有连接的 Android 设备。
* 类型
```typescript
function getConnectedDevices(): Promise<Device[]>;
interface Device {
/**
* The device udid.
*/
udid: string;
/**
* Current device state, as it is visible in
* _adb devices -l_ output.
*/
state: string;
port?: number;
}
```
* 返回值:
* `Promise<Device[]>` 返回一个 Device 数组
* 示例:
```typescript
import { agentFromAdbDevice, getConnectedDevices } from '@midscene/android'
const devices = await getConnectedDevices();
console.log(devices);
const agent = await agentFromAdbDevice(devices[0].udid);
```
## 更多
* 更多 Agent 上的 API 接口请参考 [API 参考](./API)。
* 更多关于提示词的技巧请参考 [提示词技巧](./prompting-tips)
## FAQ
### 为什么我连接了设备,但是通过 adb 仍然无法控制?
请检查是否在系统设置的开发者选项中如果存在『USB 调试(安全设置)』,也需要开启。
<p align="center">
<img src="/android-usb-debug.png" alt="android usb debug" width="400"/>
</p>
### 如何使用自定义的 adb 路径?
你可以使用 `MIDSCENE_ADB_PATH` 环境变量来指定 adb 可执行文件的路径。
```bash
export MIDSCENE_ADB_PATH=/path/to/adb
```