---
url: /android-api-reference.md
---

# API reference (Android)

Use this doc when you need to customize Midscene's Android automation or review Android-only constructor options. For shared parameters (reporting, hooks, caching, etc.), see the platform-agnostic [API reference (Common)](./api).

## Action Space

`AndroidDevice` uses the following action space; the Midscene Agent can use these actions while planning tasks:

- `Tap` — Tap an element.
- `DoubleClick` — Double-tap an element.
- `Input` — Enter text with `replace`/`append`/`clear` modes and optional `autoDismissKeyboard`.
- `Scroll` — Scroll from an element or screen center in any direction, with helpers to reach the top, bottom, left, or right.
- `DragAndDrop` — Drag from one element to another.
- `KeyboardPress` — Press a specified key.
- `LongPress` — Long-press a target element with optional duration.
- `PullGesture` — Pull up or down (e.g., to refresh) with optional distance and duration.
- `ClearInput` — Clear the contents of an input field.
- `Launch` — Open a web URL or `package/.Activity` string.
- `RunAdbShell` — Execute raw `adb shell` commands.
- `AndroidBackButton` — Trigger the system back action.
- `AndroidHomeButton` — Return to the home screen.
- `AndroidRecentAppsButton` — Open the multitasking/recent apps view.

## AndroidDevice {#androiddevice}

Create a connection to an adb-managed device that an AndroidAgent can drive.

### Import

```ts
import { AndroidDevice, getConnectedDevices } from '@midscene/android';
```

### Constructor

```ts
const device = new AndroidDevice(deviceId, {
  // device options...
});
```

### Device options

- `deviceId: string` — Value returned by `adb devices` or `getConnectedDevices()`.
- `autoDismissKeyboard?: boolean` — Automatically hide the keyboard after input. Default `true`.
- `keyboardDismissStrategy?: 'esc-first' | 'back-first'` — Order for dismissing keyboards. Default `'esc-first'`.
- `androidAdbPath?: string` — Custom path to the adb executable.
- `remoteAdbHost?: string` / `remoteAdbPort?: number` — Point to a remote adb server.
- `imeStrategy?: 'always-yadb' | 'yadb-for-non-ascii'` — Choose when to invoke [yadb](https://github.com/ysbing/YADB) for text input. Default `'yadb-for-non-ascii'`.
  - `'yadb-for-non-ascii'` (default) — Uses yadb for Unicode characters (including Latin Unicode like ö, é, ñ), Chinese, Japanese, and format specifiers (like %s, %d). Pure ASCII text uses the faster native `adb input text`.
  - `'always-yadb'` — Always uses yadb for all text input, providing maximum compatibility but slightly slower for pure ASCII text.
- `displayId?: number` — Target a specific virtual display if the device mirrors multiple displays.
- `screenshotResizeScale?: number` — Downscale screenshots before sending them to the model. Defaults to `1 / devicePixelRatio`.
- `alwaysRefreshScreenInfo?: boolean` — Re-query rotation and screen size every step. Default `false`.
- `scrcpyConfig?: object` — Scrcpy high-performance screenshot configuration, disabled by default. See [Scrcpy Screenshot Mode](#scrcpy) below.

### Scrcpy Screenshot Mode {#scrcpy}

By default, Midscene captures screenshots via `adb shell screencap`, which takes ~500–2000ms per call. Enabling Scrcpy mode streams H.264 video from the device and captures frames in real time, reducing screenshot latency to approximately **100–200ms**.

**How to enable:**

```ts
const device = new AndroidDevice(deviceId, {
  scrcpyConfig: {
    enabled: true,
  },
});
```

**Optional parameters:**

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `enabled` | `boolean` | `false` | Enable Scrcpy screenshots |
| `maxSize` | `number` | `0` | Max video dimension (width or height). `0` = no scaling |
| `videoBitRate` | `number` | `2000000` | H.264 encoding bitrate (bps) |
| `idleTimeoutMs` | `number` | `30000` | Auto-disconnect after idle (ms). Set to `0` to disable |

:::tip
Scrcpy mode automatically falls back to ADB screenshots if the connection fails. No extra error handling is needed.
:::

### Usage notes

- Discover devices with `getConnectedDevices()`; the `udid` matches `adb devices`.
- Supports remote adb via `remoteAdbHost/remoteAdbPort`; set `androidAdbPath` if adb is not on PATH.
- Use `screenshotResizeScale` to cut latency on high-DPI devices.

### Examples

#### Quick start

```ts
import { AndroidAgent, AndroidDevice, getConnectedDevices } from '@midscene/android';

const [first] = await getConnectedDevices();
const device = new AndroidDevice(first.udid);
await device.connect();

const agent = new AndroidAgent(device, {
  aiActionContext: 'If a permissions dialog appears, accept it.',
});

await agent.launch('https://www.ebay.com');
await agent.aiAct('search "Headphones" and wait for results');
const items = await agent.aiQuery(
  '{itemTitle: string, price: number}[], find item in list and corresponding price',
);
console.log(items);
```

#### Launch native packages

```ts
await agent.launch('com.android.settings/.Settings');
await agent.back();
await agent.home();
```

## AndroidAgent {#androidagent}

Wire Midscene's AI planner to an AndroidDevice for UI automation.

### Import

```ts
import { AndroidAgent } from '@midscene/android';
```

### Constructor

```ts
const agent = new AndroidAgent(device, {
  // common agent options...
});
```

### Android-specific options

- `customActions?: DeviceAction[]` — Extend planning with actions defined via `defineAction`.
- `appNameMapping?: Record<string, string>` &mdash; Map friendly app names to package names. When you pass an app name to `launch(target)`, the agent will look up the package name in this mapping. If no mapping is found, it will attempt to launch `target` as-is. User-provided mappings take precedence over default mappings.
- All other fields match [API constructors](./api#common-parameters): `generateReport`, `reportFileName`, `aiActionContext`, `modelConfig`, `cacheId`, `createOpenAIClient`, `onTaskStartTip`, and more.

### Usage notes

:::info

- Use one agent per device connection.
- Android-only helpers such as `launch` and `runAdbShell` are also exposed in YAML scripts. See [Android platform-specific actions](./automate-with-scripts-in-yaml#the-android-part).
- For shared interaction methods, see [API reference (Common)](./api#interaction-methods).

:::

### Android-specific methods

#### `agent.launch()`

Launch a web URL or native Android activity/package.

```ts
function launch(target: string): Promise<void>;
```

- `target: string` &mdash; Can be a web URL, a string in `package/.Activity` format (e.g., `com.android.settings/.Settings`), an app package name, or an app name. If you pass an app name and it exists in `appNameMapping`, it will be automatically resolved to the mapped package name; otherwise, `target` will be launched as-is.

#### `agent.runAdbShell()`

Run a raw `adb shell` command through the connected device.

```ts
function runAdbShell(command: string): Promise<string>;
```

- `command: string` &mdash; Command passed verbatim to `adb shell`.

```ts
const result = await agent.runAdbShell('dumpsys battery');
console.log(result);
```

#### Navigation helpers

- `agent.back(): Promise<void>` &mdash; Trigger the Android system Back action.
- `agent.home(): Promise<void>` &mdash; Return to the launcher.
- `agent.recentApps(): Promise<void>` &mdash; Open the Recents/Overview screen.

### Helper utilities

#### `agentFromAdbDevice()`

Create an `AndroidAgent` from any connected adb device.

```ts
function agentFromAdbDevice(
  deviceId?: string,
  opts?: PageAgentOpt & AndroidDeviceOpt,
): Promise<AndroidAgent>;
```

- `deviceId?: string` &mdash; Connect to a specific device; omitted means “first available”.
- `opts?: PageAgentOpt & AndroidDeviceOpt` &mdash; Combine agent options with [AndroidDevice](#androiddevice) settings.

#### `getConnectedDevices()`

Enumerate adb devices Midscene can drive.

```ts
function getConnectedDevices(): Promise<Array<{
  udid: string;
  state: string;
  port?: number;
}>>;
```

### See also

- [Android getting started](./android-getting-started) for setup and scripting steps.



---
url: /android-getting-started.md
---




import { PackageManagerTabs } from '@theme';

# Android Getting Started

This guide walks you through everything required to automate an Android device with Midscene: connect a real phone over adb, configure model credentials, try the no-code Playground, and run your first JavaScript script.

:::info Demo Projects

Control Android devices with JavaScript: [https://github.com/web-infra-dev/midscene-example/blob/main/android/javascript-sdk-demo](https://github.com/web-infra-dev/midscene-example/blob/main/android/javascript-sdk-demo)

Integrate Vitest for testing: [https://github.com/web-infra-dev/midscene-example/tree/main/android/vitest-demo](https://github.com/web-infra-dev/midscene-example/tree/main/android/vitest-demo)

:::

## Set up API keys for model

Set your model configs into the environment variables. You may refer to [Model strategy](../model-strategy) for more details.

```bash
export MIDSCENE_MODEL_BASE_URL="https://replace-with-your-model-service-url/v1"
export MIDSCENE_MODEL_API_KEY="replace-with-your-api-key"
export MIDSCENE_MODEL_NAME="replace-with-your-model-name"
export MIDSCENE_MODEL_FAMILY="replace-with-your-model-family"
```

For more configuration details, please refer to [Model strategy](../model-strategy) and [Model configuration](../model-config).


## Prepare your Android device

Before scripting, confirm adb can talk to your device and the device trusts your machine.

### Install adb and set `ANDROID_HOME`

- Install via [Android Studio](https://developer.android.com/studio) or the [command-line tools](https://developer.android.com/studio#command-line-tools-only)
- Verify installation:

```bash
adb --version
```

Example output indicates success:

```log
Android Debug Bridge version 1.0.41
Version 34.0.4-10411341
Installed as /usr/local/bin//adb
Running on Darwin 24.3.0 (arm64)
```

- Set `ANDROID_HOME` as documented in [Android environment variables](https://developer.android.com/tools/variables), then confirm:

```bash
echo $ANDROID_HOME
```

Any non-empty output means it is configured:

```log
/Users/your_username/Library/Android/sdk
```

### Enable USB debugging and verify the device

In the system settings developer options, enable **USB debugging** (and **USB debugging (Security settings)** if present), then connect the device via USB.

<p align="center">
  <img src="/android-usb-debug-en.png" alt="android usb debug" width="400"/>
</p>

Verify the connection:

```bash
adb devices -l
```

Example success output:

```log
List of devices attached
s4ey59	device usb:34603008X product:cezanne model:M2006J device:cezan transport_id:3
```

## Try Playground (no code)

Playground is the fastest way to validate the connection and observe AI-driven steps without writing code. It shares the same core as `@midscene/android`, so anything that works here will behave the same once scripted.

<video src="https://lf3-static.bytednsdoc.com/obj/eden-cn/nupipfups/Midscene/1.0-showcases/booking2.mp4" controls/>

1. Launch the Playground CLI:

```bash
npx --yes @midscene/android-playground
```

2. Click the gear icon in the Playground window, then paste your API key configuration. Refer back to [Model configuration](./model-config) if you still need credentials.

![](/android-set-env.png)

## Start experiencing

After configuration, you can start using Midscene right away. It provides several key operation tabs:

- **Act**: interact with the page. This is Auto Planning, corresponding to `aiAct`. For example:
```
Type “Midscene” in the search box, run the search, and open the first result
```

```
Fill out the registration form and make sure every field passes validation
```

- **Query**: extract JSON data from the interface, corresponding to `aiQuery`.

Similar methods include `aiBoolean()`, `aiNumber()`, and `aiString()` for directly extracting booleans, numbers, and strings.

```
Extract the user ID from the page and return JSON data in the { id: string } structure
```

- **Assert**: understand the page and assert; if the condition is not met, throw an error, corresponding to `aiAssert`.

```
There is a login button on the page, with a user agreement link below it
```

- **Tap**: click on an element. This is Instant Action, corresponding to `aiTap`.
```
Click the login button
```

> For the difference between Auto Planning and Instant Action, see the [API](../api.mdx) document.


## Integration with Midscene Agent

Once Playground works, move to a repeatable script with the JavaScript SDK.

### Step 1. Install dependencies

<PackageManagerTabs command="install @midscene/android --save-dev" />

### Step 2. Write scripts

Save the following code as `./demo.ts`. It opens the browser on the device, searches eBay, and asserts the result list.

```typescript title="./demo.ts"
import {
  AndroidAgent,
  AndroidDevice,
  getConnectedDevices,
} from '@midscene/android';

const sleep = (ms) => new Promise((r) => setTimeout(r, ms));
Promise.resolve(
  (async () => {
    const devices = await getConnectedDevices();
    const device = new AndroidDevice(devices[0].udid);

    const agent = new AndroidAgent(device, {
      aiActionContext:
        'If any location, permission, user agreement, etc. popup, click agree. If login page pops up, close it.',
    });
    await device.connect();

    await agent.aiAct('open browser and navigate to ebay.com');
    await sleep(5000);
    await agent.aiAct('type "Headphones" in search box, hit Enter');
    await agent.aiWaitFor('There is at least one headphone product');

    const items = await agent.aiQuery(
      '{itemTitle: string, price: Number}[], find item in list and corresponding price',
    );
    console.log('headphones in stock', items);

    await agent.aiAssert('There is a category filter on the left');
  })(),
);
```

### Step 3. Run

```bash
npx tsx demo.ts
```

### Step 4: View the report

Successful runs print `Midscene - report file updated: /path/to/report/some_id.html`. Open the generated HTML file in a browser to replay every interaction, query, and assertion.

## Advanced

Use this section when you need to customize device behavior, wire Midscene into your framework, or troubleshoot adb issues. For detailed constructor parameters, jump to the [API reference(Android)](./android-api-reference).

### Extend Midscene on Android

Use `defineAction()` for custom gestures and pass them through `customActions`. Midscene will append them to the planner so AI can call your domain-specific action names.

```typescript
import { getMidsceneLocationSchema, z } from '@midscene/core';
import { defineAction } from '@midscene/core/device';
import { AndroidAgent, AndroidDevice, getConnectedDevices } from '@midscene/android';

const ContinuousClick = defineAction({
  name: 'continuousClick',
  description: 'Click the same target repeatedly',
  paramSchema: z.object({
    locate: getMidsceneLocationSchema(),
    count: z.number().int().positive().describe('How many times to click'),
  }),
  async call(param) {
    const { locate, count } = param;
    console.log('click target center', locate.center);
    console.log('click count', count);
  },
});

const devices = await getConnectedDevices();
const device = new AndroidDevice(devices[0].udid);
await device.connect();

const agent = new AndroidAgent(device, {
  customActions: [ContinuousClick],
});

await agent.aiAct('click the red button five times');
```

See [Integrate with any interface](./integrate-with-any-interface#define-a-custom-action) for a deeper explanation of custom actions and action schemas.

## More

- For every Agent method, check the [API reference (Common)](./api#interaction-methods).
- For the Android API reference, see [Android Agent API](./android-api-reference).
- Demo projects
  - Android JavaScript SDK demo: [https://github.com/web-infra-dev/midscene-example/blob/main/android/javascript-sdk-demo](https://github.com/web-infra-dev/midscene-example/blob/main/android/javascript-sdk-demo)
  - Android + Vitest demo: [https://github.com/web-infra-dev/midscene-example/tree/main/android/vitest-demo](https://github.com/web-infra-dev/midscene-example/tree/main/android/vitest-demo)

### Complete example (Vitest + AndroidAgent)

```typescript
import {
  AndroidAgent,
  AndroidDevice,
  getConnectedDevices,
} from '@midscene/android';
import type { TestStatus } from '@midscene/core';
import { ReportMergingTool } from '@midscene/core/report';
import { sleep } from '@midscene/core/utils';
import type ADB from 'appium-adb';
import {
  afterAll,
  afterEach,
  beforeAll,
  beforeEach,
  describe,
  it,
} from 'vitest';

describe('Android Settings Test', () => {
  let page: AndroidDevice;
  let adb: ADB;
  let agent: AndroidAgent;
  let startTime: number;
  let itTestStatus: TestStatus = 'passed';
  const reportMergingTool = new ReportMergingTool();

  beforeAll(async () => {
    const devices = await getConnectedDevices();
    page = new AndroidDevice(devices[0].udid);
    adb = await page.getAdb();
  });

  beforeEach((ctx) => {
    startTime = performance.now();
    agent = new AndroidAgent(page, {
      groupName: ctx.task.name,
    });
  });

  afterEach((ctx) => {
    if (ctx.task.result?.state === 'pass') {
      itTestStatus = 'passed';
    } else if (ctx.task.result?.state === 'skip') {
      itTestStatus = 'skipped';
    } else if (ctx.task.result?.errors?.[0].message.includes('timed out')) {
      itTestStatus = 'timedOut';
    } else {
      itTestStatus = 'failed';
    }
    reportMergingTool.append({
      reportFilePath: agent.reportFile as string,
      reportAttributes: {
        testId: `${ctx.task.name}`,
        testTitle: `${ctx.task.name}`,
        testDescription: 'description',
        testDuration: (Date.now() - ctx.task.result?.startTime!) | 0,
        testStatus: itTestStatus,
      },
    });
  });

  afterAll(() => {
    reportMergingTool.mergeReports('my-android-setting-test-report');
  });

  it('toggle wlan', async () => {
    await adb.shell('input keyevent KEYCODE_HOME');
    await sleep(1000);
    await adb.shell('am start -n com.android.settings/.Settings');
    await sleep(1000);
    await agent.aiAct('find and enter WLAN setting');
    await agent.aiAct(
      'toggle WLAN status *once*, if WLAN is off pls turn it on, otherwise turn it off.',
    );
  });

  it('toggle bluetooth', async () => {
    await adb.shell('input keyevent KEYCODE_HOME');
    await sleep(1000);
    await adb.shell('am start -n com.android.settings/.Settings');
    await sleep(1000);
    await agent.aiAct('find and enter bluetooth setting');
    await agent.aiAct(
      'toggle bluetooth status *once*, if bluetooth is off pls turn it on, otherwise turn it off.',
    );
  });
});
```

:::tip

Merged reports are stored inside `midscene_run/report` by default. Override the directory with `MIDSCENE_RUN_DIR` when running in CI.

:::

## FAQ

### Why can't I control the device even though I've connected it?

A common error is:

```
Error:
Exception occurred while executing 'tap':
java.lang.SecurityException: Injecting input events requires the caller (or the source of the instrumentation, if any) to have the INJECT_EVENTS permission.
```

Make sure USB debugging is enabled and the device is unlocked in developer options.

<p align="center">
  <img src="/android-usb-debug-en.png" alt="android usb debug" width="400" />
</p>

### How do I use a custom adb path or remote adb server?

Set the environment variables first:

```bash
export MIDSCENE_ADB_PATH=/path/to/adb
export MIDSCENE_ADB_REMOTE_HOST=192.168.1.100
export MIDSCENE_ADB_REMOTE_PORT=5037
```

You can also provide the same information via the constructor:

```typescript
const device = new AndroidDevice('s4ey59', {
  androidAdbPath: '/path/to/adb',
  remoteAdbHost: '192.168.1.100',
  remoteAdbPort: 5037,
});
```



---
url: /android-introduction.md
---




# Android Automation Support

Midscene can drive adb tools to support Android automation.

By adapting a visual model solution, the automation process works with any app tech stack—whether built with Native, Flutter, React Native, or Lynx. Developers only need to focus on the final experience when debugging UI automation scripts.

The Android UI automation solution comes with all the features of Midscene:

- Supports zero-code trial using Playground.
- Supports JavaScript SDK.
- Supports automation scripts in YAML format and command-line tools.
- Supports HTML reports to replay all operation paths.

## Showcases

**Prompt** : Open the Booking App, search for a hotel in Tokyo for four adults on Christmas, with a score of 8 or above.

<video src="https://lf3-static.bytednsdoc.com/obj/eden-cn/nupipfups/Midscene/1.0-showcases/booking2.mp4" poster="https://lf3-static.bytednsdoc.com/obj/eden-cn/nupipfups/Midscene/1.0-showcases/booking.png" height="300" controls />

View the full report of this task: [report.html](https://lf3-static.bytednsdoc.com/obj/eden-cn/nupipfups/Midscene/1.0-showcases/booking.html)


See more showcases: [showcases](./showcases.mdx)

## Try Midscene Playground on Android

With Midscene.js playground, you can experience Android automation capabilities without writing any code.

<video src="https://lf3-static.bytednsdoc.com/obj/eden-cn/nupipfups/Midscene/1.0-showcases/booking2.mp4" controls/>

See [Getting Started](./android-getting-started#try-playground-no-code) to learn how to launch the Playground.

## Next Steps

* [Getting Started](./android-getting-started)
* [Use JavaScript SDK](./android-getting-started)
* [Use YAML format automation scripts](./automate-with-scripts-in-yaml) and [command-line tools](./automate-with-scripts-in-yaml)



---
url: /api.md
---

# API reference (Common)

## Constructors

Midscene ships multiple agents tuned for specific automation environments. Every constructor takes the target page/device plus a shared options bag (reporting, caching, AI configuration, hooks), and then layers on platform-only switches such as navigation guards in browsers or ADB wiring on Android. Use the sections below to review the import path and platform-specific parameters for each agent:

- In Puppeteer, use [PuppeteerAgent](./web-api-reference#puppeteer-agent)
- In Playwright, use [PlaywrightAgent](./web-api-reference#playwright-agent)
- In Bridge Mode, use [AgentOverChromeBridge](./web-api-reference#chrome-bridge-agent)
- On Android, use [Android API reference](./android-api-reference)
- On iOS, use [iOS API reference](./ios-api-reference)
- For GUI agents integrating with your own interface, refer to [Custom Interface Agent](./integrate-with-any-interface)

### Parameters

All agents share these base options:

- `generateReport: boolean`: If true, a report file will be generated. (Default: true)
- `reportFileName: string`: The name of the report file. (Default: generated by midscene)
- `autoPrintReportMsg: boolean`: If true, report messages will be printed. (Default: true)
- `cacheId: string | undefined`: If provided, this cacheId will be used to save or match the cache. (Default: undefined, means cache feature is disabled)
- `aiActContext: string`: Some background knowledge that should be sent to the AI model when calling `agent.aiAct()`, like 'close the cookie consent dialog first if it exists' (Default: undefined). Previously exposed as `aiActionContext`; the legacy name is still accepted for backward compatibility.
- `replanningCycleLimit: number`: The maximum number of `aiAct` replanning cycles. Default is 20 (40 for UI-TARS models). Prefer setting this via the agent option; reading `MIDSCENE_REPLANNING_CYCLE_LIMIT` is only for backward compatibility.
- `waitAfterAction: number`: Wait time in milliseconds after each action execution. This allows the UI to settle and stabilize before the next action. Default is 300ms.
- `onTaskStartTip: (tip: string) => void | Promise<void>`: Optional hook that fires before each execution task begins with a human-readable summary of the task (Default: undefined)
- `outputFormat: 'single-html' | 'html-and-external-assets'`: Controls how the report is generated. `'single-html'` (default) embeds all screenshots as base64 in a single HTML file. `'html-and-external-assets'` saves screenshots as separate PNG files in a subdirectory, useful when report files become too large. **Note**: When using `'html-and-external-assets'`, the report must be accessed via an HTTP server or CDN URL - it cannot be opened directly using the `file://` protocol. This is because browser CORS (Cross-Origin Resource Sharing) restrictions prevent loading local images from relative paths when using the file protocol. For local testing, start a simple HTTP server in the report directory. Navigate to the report directory and run one of these commands:
  - Using Node.js: `npx serve`
  - Using Python: `python -m http.server` or `python3 -m http.server`
  Then access the report via `http://localhost:3000` (or the port shown in the terminal).
- `screenshotShrinkFactor: number`: Controls the scaling ratio of screenshots to reduce the image size sent to the AI model, thereby reducing token consumption. The default value is 1 (no scaling). If set to 2, the width and height of the screenshot will be halved, and the area will be reduced to a quarter of the original. You can adjust this value based on your actual situation to find the best balance between image clarity and token consumption.
  - For mobile devices, setting `screenshotShrinkFactor` to 2 can reduce token consumption while maintaining clarity, but it is not recommended to set it higher than 3, as this may cause the image to be too blurry and affect the AI model's understanding.
  - For web pages, if the content is complex or contains a lot of details, it is not recommended to set `screenshotShrinkFactor` to avoid overly blurry screenshots. Additionally, if you want higher clarity for web page screenshots, you can configure Puppeteer or Playwright's `deviceScaleFactor` to 2, which will allow Puppeteer or Playwright to render the page as if it were a high-definition screen.

### Custom model configuration

Use `modelConfig: Record<string, string | number>` to configure models directly in code instead of environment variables.

> If `modelConfig` is provided at agent initialization, **all system environment variables for model config are ignored**. Only the values in this object are used.
> The keys/values you can set here are the same ones listed in [Model configuration](./model-config). You can also see explanation in [Model strategy](./model-strategy).

**Basic Example (single model for all intents):**
```typescript
const agent = new PuppeteerAgent(page, {
  modelConfig: {
    MIDSCENE_MODEL_NAME: 'qwen3-vl-plus',
    MIDSCENE_MODEL_BASE_URL: 'https://dashscope.aliyuncs.com/compatible-mode/v1',
    MIDSCENE_MODEL_API_KEY: 'sk-...',
    MIDSCENE_MODEL_FAMILY: 'qwen3-vl'
  }
});
```

**Configure different models for different task types (via intent-specific keys):**
```typescript
const agent = new PuppeteerAgent(page, {
  modelConfig: {
    // default
    MIDSCENE_MODEL_NAME: 'qwen3-vl-plus',
    MIDSCENE_MODEL_API_KEY: 'sk-default-key',
    MIDSCENE_MODEL_BASE_URL: '.....',
    MIDSCENE_MODEL_FAMILY: 'qwen3-vl',

    // planning intent
    MIDSCENE_PLANNING_MODEL_NAME: 'gpt-5.1',
    MIDSCENE_PLANNING_MODEL_API_KEY: 'sk-planning-key',
    MIDSCENE_PLANNING_MODEL_BASE_URL: '...',
    
    // insight intent
    MIDSCENE_INSIGHT_MODEL_NAME: 'qwen-vl-plus',
    MIDSCENE_INSIGHT_MODEL_API_KEY: 'sk-insight-key'
  }
});
```

### Custom OpenAI client

`createOpenAIClient: (openai, options) => Promise<OpenAI | undefined>` lets you wrap the OpenAI client instance for integrating observability tools (such as LangSmith, LangFuse) or applying custom middleware.

**Parameter Description:**
- `openai: OpenAI` - The base OpenAI client instance created by Midscene with all necessary configurations (API key, base URL, proxy, etc.)
- `options: Record<string, unknown>` - OpenAI initialization options, including:
  - `baseURL?: string` - API endpoint URL
  - `apiKey?: string` - API key
  - `dangerouslyAllowBrowser: boolean` - Always true in Midscene
  - Other OpenAI configuration options

**Return Value:**
- Return the wrapped OpenAI client instance, or `undefined` to use the original instance

**Example (LangSmith Integration):**
```typescript
import { wrapOpenAI } from 'langsmith/wrappers';

const agent = new PuppeteerAgent(page, {
  createOpenAIClient: async (openai, options) => {
    // Wrap with LangSmith for planning tasks
    if (options.baseURL?.includes('planning')) {
      return wrapOpenAI(openai, {
        metadata: { task: 'planning' }
      });
    }

    // Return the original client for other tasks
    return openai;
  }
});
```

**Note:** For LangSmith and Langfuse integration, we recommend using the environment-variable approach documented in [Model configuration](./model-config#using-langsmith), which requires no `createOpenAIClient` code. If you provide a custom client wrapper function, it will override the auto-integration behavior from environment variables.

## Interaction methods

Below are the main APIs available for the various Agents in Midscene.

:::info Auto Planning v.s. Instant Action

In Midscene, you can choose to use either auto planning or instant action.

- `agent.ai()` is for Auto Planning: Midscene will automatically plan the steps and execute them. It's more smart and looks like more fashionable style for AI agents. But it may be slower and heavily rely on the quality of the AI model.
- `agent.aiTap()`, `agent.aiHover()`, `agent.aiInput()`, `agent.aiKeyboardPress()`, `agent.aiScroll()`, `agent.aiDoubleClick()`, `agent.aiRightClick()` are for Instant Action: Midscene will directly perform the specified action, while the AI model is responsible for basic tasks such as locating elements. It's faster and more reliable if you are certain about the action you want to perform.

:::

### `agent.aiAct()` or `agent.ai()`

This method allows you to perform a series of UI actions described in natural language. Midscene automatically plans the steps and executes them.

:::info Backward Compatibility

This method was previously named `aiAction()` in earlier versions. The current version supports both names for backward compatibility. We recommend using the new `aiAct()` method for code consistency.

:::

- Type

```typescript
function aiAct(
  prompt: string,
  options?: {
    cacheable?: boolean;
    deepThink?: 'unset' | true | false;
    deepLocate?: boolean;
    fileChooserAccept?: string | string[];
  },
): Promise<void>;
function ai(prompt: string): Promise<void>; // shorthand form
```

- Parameters:

  - `prompt: string` - A natural language description of the UI steps.
  - `options?: Object` - Optional, a configuration object containing:
    - `cacheable?: boolean` - Whether cacheable when enabling [caching feature](./caching.mdx). True by default.
    - `deepThink?: 'unset' | true | false` - Whether to enable deep thinking during planning when the model supports it (depends on MIDSCENE_MODEL_FAMILY). Default is `'unset'` (same as omitting) and follows the model provider's default strategy. When explicitly set to `true` or `false`, it overrides the `MIDSCENE_MODEL_REASONING_ENABLED` environment variable. [Learn more about deepThink](./model-strategy#about-the-deepthink-option-in-aiact).
    - `deepLocate?: boolean` - Whether to enable [Deep Locate](#deep-locate-deeplocate). False by default.
    - `fileChooserAccept?: string | string[]` - When a file chooser pops up, specify the file path(s) to accept. Can be a single file path or an array of paths. Only available in web pages (Playwright, Puppeteer).
      - **Note**: If the file input does not support multiple files (no `multiple` attribute) but multiple files are provided, an error will be thrown.
      - **Note**: If a file chooser is triggered but no `fileChooserAccept` parameter is provided, the file chooser will be ignored and the page can continue to operate normally.

- Return Value:

  - Returns a Promise that resolves to void when all steps are completed; if execution fails, an error is thrown.

- Examples:

```typescript
// Basic usage
await agent.aiAct(
  'Type "JavaScript" into the search box, then click the search button',
);

// Using the shorthand .ai form
await agent.ai(
  'Click the login button at the top of the page, then enter "test@example.com" in the username field',
);

// When using UI Agent models like ui-tars, you can try a more goal-driven prompt
await agent.aiAct('Post a Tweet "Hello World"');
```

:::tip

Under the hood, Midscene uses AI model to split the instruction into a series of steps (a.k.a. "Planning"). It then executes these steps sequentially. If Midscene determines that the actions cannot be performed, an error will be thrown.

For optimal results, please provide clear and detailed instructions for `agent.aiAct()`.

Related Documentation:

- [Model strategy](./model-strategy)

:::

### `agent.aiTap()`

Tap something.

- Type

```typescript
function aiTap(locate: string | Object, options?: Object): Promise<void>;
```

- Parameters:

  - `locate: string | Object` - A natural language description of the element to tap, or [prompting with images](#prompting-with-images).
  - `options?: Object` - Optional, a configuration object containing:
    - `deepLocate?: boolean` - Whether to enable [Deep Locate](#deep-locate-deeplocate). Previously named `deepThink`, now renamed to `deepLocate`. False by default.
    - `xpath?: string` - The xpath of the element to operate. If provided, Midscene will first use this xpath to locate the element before using the cache and the AI model. Empty by default.
    - `cacheable?: boolean` - Whether cacheable when enabling [caching feature](./caching.mdx). True by default.
    - `fileChooserAccept?: string | string[]` - When a file chooser pops up, specify the file path(s) to accept. Can be a single file path or an array of paths. Only available in web pages (Playwright, Puppeteer).
      - **Note**: If the file input does not support multiple files (no `multiple` attribute) but multiple files are provided, an error will be thrown.
      - **Note**: If a file chooser is triggered but no `fileChooserAccept` parameter is provided, the file chooser will be ignored and the page can continue to operate normally.

- Return Value:

  - Returns a `Promise<void>`

- Examples:

```typescript
await agent.aiTap('The login button at the top of the page');

// Use deepLocate feature to precisely locate the element
await agent.aiTap('The login button at the top of the page', {
  deepLocate: true,
});

// File upload: tap the upload button and select files
await agent.aiTap('Choose file button', { fileChooserAccept: ['./document.pdf'] });
await agent.aiTap('Upload images', { fileChooserAccept: ['./image1.jpg', './image2.png'] });
```

### `agent.aiHover()`

> Only available in web pages, not available in Android.

Move mouse over something.

- Type

```typescript
function aiHover(locate: string | Object, options?: Object): Promise<void>;
```

- Parameters:

  - `locate: string | Object` - A natural language description of the element to hover over, or [prompting with images](#prompting-with-images).
  - `options?: Object` - Optional, a configuration object containing:
    - `deepLocate?: boolean` - Whether to enable [Deep Locate](#deep-locate-deeplocate). Previously named `deepThink`, now renamed to `deepLocate`. False by default.
    - `xpath?: string` - The xpath of the element to operate. If provided, Midscene will first use this xpath to locate the element before using the cache and the AI model. Empty by default.
    - `cacheable?: boolean` - Whether cacheable when enabling [caching feature](./caching.mdx). True by default.

- Return Value:

  - Returns a `Promise<void>`

- Examples:

```typescript
await agent.aiHover('The version number of the current page');
```

### `agent.aiInput()`

Input text into something.

- Type

```typescript
// Recommended: locate first, then options with value
function aiInput(
  locate: string | Object,
  opt: {
    value: string | number;
    deepLocate?: boolean;
    xpath?: string;
    cacheable?: boolean;
    autoDismissKeyboard?: boolean;
    mode?: 'replace' | 'clear' | 'typeOnly';
  },
): Promise<void>;

// Backward compatible (legacy)
function aiInput(
  value: string | number,
  locate: string | Object,
  options?: Object,
): Promise<void>;
```

- Parameters:

  **Recommended usage:**
  - `locate: string | Object` - A natural language description of the element to input text into, or [prompting with images](#prompting-with-images).
  - `opt: Object` - Configuration object containing:
    - `value: string | number` - **Required**. The text content to input.
      - When `mode` is `'replace'`: The text will replace all existing content in the input field.
      - When `mode` is `'typeOnly'`: The text will be typed directly without clearing the field first.
      - When `mode` is `'clear'`: The text is ignored and the input field will be cleared.
    - `deepLocate?: boolean` - Whether to enable [Deep Locate](#deep-locate-deeplocate). Previously named `deepThink`, now renamed to `deepLocate`. False by default.
    - `xpath?: string` - The xpath of the element to operate. If provided, Midscene will first use this xpath to locate the element before using the cache and the AI model. Empty by default.
    - `cacheable?: boolean` - Whether cacheable when enabling [caching feature](./caching.mdx). True by default.
    - `autoDismissKeyboard?: boolean` - If true, the keyboard will be dismissed after input text, only available in Android/iOS. (Default: true)
    - `mode?: 'replace' | 'clear' | 'typeOnly'` - Input mode. (Default: 'replace')
      - `'replace'`: Clear the input field first, then input the text.
      - `'typeOnly'`: Type the value directly without clearing the field first.
      - `'clear'`: Clear the input field without entering new text.

  **Backward compatible usage (deprecated but still supported):**
  - `value: string | number` - The text content to input.
  - `locate: string | Object` - A natural language description of the element, or [prompting with images](#prompting-with-images).
  - `options?: Object` - Optional configuration object. The type is the same as the `opt` type in the recommended usage.

- Return Value:

  - Returns a `Promise<void>`

- Examples:

```typescript
// Recommended
await agent.aiInput('The search input box', { value: 'Hello World' });

// Backward compatible (not recommended)
await agent.aiInput('Hello World', 'The search input box');
```

:::note Signature Update

We recently updated the `aiInput` API signature to put the locate prompt as the first parameter, making the parameter order more intuitive. The legacy signature `aiInput(value, locate, options)` is still fully compatible, but new code should use the recommended signature.

:::

### `agent.aiKeyboardPress()`

Press a keyboard key.

- Type

```typescript
// Recommended: locate first, then options with keyName
function aiKeyboardPress(
  locate: string | Object,
  opt: {
    keyName: string;
    deepLocate?: boolean;
    xpath?: string;
    cacheable?: boolean;
  },
): Promise<void>;

// Backward compatible (legacy)
function aiKeyboardPress(
  key: string,
  locate?: string | Object,
  options?: Object,
): Promise<void>;
```

- Parameters:

  **Recommended usage:**
  - `locate: string | Object` - A natural language description of the element to press the key on, or [prompting with images](#prompting-with-images).
  - `opt: Object` - Configuration object containing:
    - `keyName: string` - **Required**. The web key to press, e.g. 'Enter', 'Tab', 'Escape', etc. Key Combination is not supported. Refer to the [complete list of supported key names in our source code](https://github.com/web-infra-dev/midscene/blob/main/packages/shared/src/us-keyboard-layout.ts) for available values.
    - `deepLocate?: boolean` - Whether to enable [Deep Locate](#deep-locate-deeplocate). Previously named `deepThink`, now renamed to `deepLocate`. False by default.
    - `xpath?: string` - The xpath of the element to operate. If provided, Midscene will first use this xpath to locate the element before using the cache and the AI model. Empty by default.
    - `cacheable?: boolean` - Whether cacheable when enabling [caching feature](./caching.mdx). True by default.

  **Backward compatible usage (deprecated but still supported):**
  - `key: string` - The web key to press, e.g. 'Enter', 'Tab', 'Escape', etc. Key Combination is not supported.
  - `locate?: string | Object` - Optional, a natural language description of the element, or [prompting with images](#prompting-with-images).
  - `options?: Object` - Optional configuration object. The type is the same as the `opt` type in the recommended usage.

- Return Value:

  - Returns a `Promise<void>`

- Examples:

```typescript
// Recommended
await agent.aiKeyboardPress('The search input box', { keyName: 'Enter' });

// Backward compatible (not recommended)
await agent.aiKeyboardPress('Enter', 'The search input box');
```

:::note Signature Update

We recently updated the `aiKeyboardPress` API signature to put the locate prompt as the first parameter, making the parameter order more intuitive. The legacy signature `aiKeyboardPress(key, locate, options)` is still fully compatible, but new code should use the recommended signature.

:::

### `agent.aiScroll()`

Scroll a page or an element.

- Type

```typescript
// Recommended: locate first, then options with scroll parameters
function aiScroll(
  locate: string | Object | undefined,
  opt: {
    scrollType?: 'singleAction' | 'scrollToBottom' | 'scrollToTop' | 'scrollToRight' | 'scrollToLeft';
    direction?: 'down' | 'up' | 'left' | 'right';
    distance?: number | null;
    deepLocate?: boolean;
    xpath?: string;
    cacheable?: boolean;
  },
): Promise<void>;

// Backward compatible (legacy)
function aiScroll(
  scrollParam: PlanningActionParamScroll,
  locate?: string | Object,
  options?: Object,
): Promise<void>;
```

- Parameters:

  **Recommended usage:**
  - `locate: string | Object | undefined` - A natural language description of the element to scroll on, or [prompting with images](#prompting-with-images). If not provided or undefined, Midscene will perform scroll on the current mouse position.
  - `opt: Object` - Configuration object containing:
    - `scrollType?: 'singleAction' | 'scrollToBottom' | 'scrollToTop' | 'scrollToRight' | 'scrollToLeft'` - The scroll behavior. Defaults to `singleAction`.
    - `direction?: 'down' | 'up' | 'right' | 'left'` - The direction to scroll. Defaults to `down`. Only effective when `scrollType` is `singleAction`. Whether it is Android or Web, the scrolling direction here all refers to which direction of the page's content will appear on the screen. For example, when the scrolling direction is `down`, the hidden content at the bottom of the page will gradually reveal itself from the bottom of the screen upwards.
    - `distance?: number | null` - Optional, the distance to scroll in px. Use `null` to allow Midscene to decide automatically.
    - `deepLocate?: boolean` - Whether to enable [Deep Locate](#deep-locate-deeplocate). Previously named `deepThink`, now renamed to `deepLocate`. False by default.
    - `xpath?: string` - The xpath of the element to operate. If provided, Midscene will first use this xpath to locate the element before using the cache and the AI model. Empty by default.
    - `cacheable?: boolean` - Whether cacheable when enabling [caching feature](./caching.mdx). True by default.

  **Backward compatible usage (deprecated but still supported):**
  - `scrollParam: PlanningActionParamScroll` - The scroll parameter (contains scrollType, direction, distance).
  - `locate?: string | Object` - Optional, a natural language description of the element, or [prompting with images](#prompting-with-images).
  - `options?: Object` - Optional configuration object. The type is the same as the `opt` type in the recommended usage.

- Return Value:

  - Returns a `Promise<void>`

- Examples:

```typescript
// Recommended
await agent.aiScroll('The form panel', {
  scrollType: 'singleAction',
  direction: 'up',
  distance: 100,
});

// Backward compatible (not recommended)
await agent.aiScroll(
  { scrollType: 'singleAction', direction: 'up', distance: 100 },
  'The form panel',
);
```

:::note Signature Update

We recently updated the `aiScroll` API signature to put the locate prompt as the first parameter, making the parameter order more intuitive. The legacy signature `aiScroll(scrollParam, locate, options)` is still fully compatible, but new code should use the recommended signature.

:::

### `agent.aiDoubleClick()`

Double-click on an element.

- Type

```typescript
function aiDoubleClick(locate: string | Object, options?: Object): Promise<void>;
```

- Parameters:

  - `locate: string | Object` - A natural language description of the element to double-click on, or [prompting with images](#prompting-with-images).
  - `options?: Object` - Optional, a configuration object containing:
    - `deepLocate?: boolean` - Whether to enable [Deep Locate](#deep-locate-deeplocate). Previously named `deepThink`, now renamed to `deepLocate`. False by default.
    - `xpath?: string` - The xpath of the element to operate. If provided, Midscene will first use this xpath to locate the element before using the cache and the AI model. Empty by default.
    - `cacheable?: boolean` - Whether cacheable when enabling [caching feature](./caching.mdx). True by default.

- Return Value:

  - Returns a `Promise<void>`

- Examples:

```typescript
await agent.aiDoubleClick('The file name at the top of the page');

// Use deepLocate feature to precisely locate the element
await agent.aiDoubleClick('The file name at the top of the page', {
  deepLocate: true,
});
```

### `agent.aiRightClick()`

> Only available in web pages, not available in Android.

Right-click on an element. Please note that Midscene cannot interact with the native context menu in browser after right-clicking. This interface is usually used for the element that listens to the right-click event by itself.

- Type

```typescript
function aiRightClick(locate: string, options?: Object): Promise<void>;
```

- Parameters:

  - `locate: string | Object` - A natural language description of the element to right-click on, or [prompting with images](#prompting-with-images).
  - `options?: Object` - Optional, a configuration object containing:
    - `deepLocate?: boolean` - Whether to enable [Deep Locate](#deep-locate-deeplocate). Previously named `deepThink`, now renamed to `deepLocate`. False by default.
    - `xpath?: string` - The xpath of the element to operate. If provided, Midscene will first use this xpath to locate the element before using the cache and the AI model. Empty by default.
    - `cacheable?: boolean` - Whether cacheable when enabling [caching feature](./caching.mdx). True by default.

- Return Value:

  - Returns a `Promise<void>`

- Examples:

```typescript
await agent.aiRightClick('The file name at the top of the page');

// Use deepLocate feature to precisely locate the element
await agent.aiRightClick('The file name at the top of the page', {
  deepLocate: true,
});
```

## Data extraction

### `agent.aiAsk()`

Ask the AI model any question about the current page. It returns the answer in string from the AI model.

- Type

```typescript
function aiAsk(prompt: string | Object, options?: Object): Promise<string>;
```

- Parameters:

  - `prompt: string | Object` - A natural language description of the question, or [prompting with images](#prompting-with-images).
  - `options?: Object` - Optional, a configuration object containing:
    - `domIncluded?: boolean | 'visible-only'` - Whether to send simplified DOM information to the model, usually used for extracting invisible attributes like image links. If set to `'visible-only'`, only the visible elements will be sent. False by default.
    - `screenshotIncluded?: boolean` - Whether to send screenshot to the model. True by default.

- Return Value:

  - Return a Promise. Return the answer from the AI model.

- Examples:

```typescript
const result = await agent.aiAsk('What should I do to test this page?');
console.log(result); // Output the answer from the AI model
```

Besides `aiAsk`, you can also use `aiQuery` to extract structured data from the UI.

### `agent.aiQuery()`

This method allows you to extract structured data from current page. Simply define the expected format (e.g., string, number, JSON, or an array) in the `dataDemand`, and Midscene will return a result that matches the format.

- Type

```typescript
function aiQuery<T>(dataDemand: string | Object, options?: Object): Promise<T>;
```

- Parameters:

  - `dataDemand: T`: A description of the expected data and its return format.
  - `options?: Object` - Optional, a configuration object containing:
    - `domIncluded?: boolean | 'visible-only'` - Whether to send simplified DOM information to the model, usually used for extracting invisible attributes like image links. If set to `'visible-only'`, only the visible elements will be sent. False by default.
    - `screenshotIncluded?: boolean` - Whether to send screenshot to the model. True by default.

- Return Value:

  - Returns any valid basic type, such as string, number, JSON, array, etc.
  - Just describe the format in `dataDemand`, and Midscene will return a matching result.

- Examples:

```typescript
const dataA = await agent.aiQuery({
  time: 'The date and time displayed in the top-left corner as a string',
  userInfo: 'User information in the format {name: string}',
  tableFields: 'An array of table field names, string[]',
  tableDataRecord:
    'Table records in the format {id: string, [fieldName]: string}[]',
});

// You can also describe the expected return format using a string:

// dataB will be an array of strings
const dataB = await agent.aiQuery('string[], list of task names');

// dataC will be an array of objects
const dataC = await agent.aiQuery(
  '{name: string, age: string}[], table data records',
);

// Use domIncluded feature to extract invisible attributes
const dataD = await agent.aiQuery(
  '{name: string, age: string, avatarUrl: string}[], table data records',
  { domIncluded: true },
);
```

### `agent.aiBoolean()`

Extract a boolean value from the UI.

- Type

```typescript
function aiBoolean(prompt: string | Object, options?: Object): Promise<boolean>;
```

- Parameters:
  - `prompt: string | Object` - A natural language description of the expected value, or [prompting with images](#prompting-with-images).
  - `options?: Object` - Optional, a configuration object containing:
    - `domIncluded?: boolean | 'visible-only'` - Whether to send simplified DOM information to the model, usually used for extracting invisible attributes like image links. If set to `'visible-only'`, only the visible elements will be sent. False by default.
    - `screenshotIncluded?: boolean` - Whether to send screenshot to the model. True by default.
- Return Value:

  - Returns a `Promise<boolean>` when AI returns a boolean value.

- Examples:

```typescript
const boolA = await agent.aiBoolean('Whether there is a login dialog');

// Use domIncluded feature to extract invisible attributes
const boolB = await agent.aiBoolean('Whether the login button has a link', {
  domIncluded: true,
});
```

### `agent.aiNumber()`

Extract a number value from the UI.

- Type

```typescript
function aiNumber(prompt: string | Object, options?: Object): Promise<number>;
```

- Parameters:
  - `prompt: string | Object` - A natural language description of the expected value, or [prompting with images](#prompting-with-images).
  - `options?: Object` - Optional, a configuration object containing:
    - `domIncluded?: boolean | 'visible-only'` - Whether to send simplified DOM information to the model, usually used for extracting invisible attributes like image links. If set to `'visible-only'`, only the visible elements will be sent. False by default.
    - `screenshotIncluded?: boolean` - Whether to send screenshot to the model. True by default.
- Return Value:

  - Returns a `Promise<number>` when AI returns a number value.

- Examples:

```typescript
const numberA = await agent.aiNumber('The remaining points of the account');

// Use domIncluded feature to extract invisible attributes
const numberB = await agent.aiNumber(
  'The value of the remaining points element',
  { domIncluded: true },
);
```

### `agent.aiString()`

Extract a string value from the UI.

- Type

```typescript
function aiString(prompt: string | Object, options?: Object): Promise<string>;
```

- Parameters:
  - `prompt: string | Object` - A natural language description of the expected value, or [prompting with images](#prompting-with-images).
  - `options?: Object` - Optional, a configuration object containing:
    - `domIncluded?: boolean | 'visible-only'` - Whether to send simplified DOM information to the model, usually used for extracting invisible attributes like image links. If set to `'visible-only'`, only the visible elements will be sent. False by default.
    - `screenshotIncluded?: boolean` - Whether to send screenshot to the model. True by default.
- Return Value:

  - Returns a `Promise<string>` when AI returns a string value.

- Examples:

```typescript
const stringA = await agent.aiString('The first item in the list');

// Use domIncluded feature to extract invisible attributes
const stringB = await agent.aiString('The link of the first item in the list', {
  domIncluded: true,
});
```

## More APIs

### `agent.aiAssert()`

Specify an assertion in natural language, and the AI determines whether the condition is true. If the assertion fails, the SDK throws an error that includes both the optional `errorMsg` and a detailed reason generated by the AI.

- Type

```typescript
function aiAssert(
  assertion: string | Object, 
  errorMsg?: string, 
  options?: Object
): Promise<void>;
```

- Parameters:

  - `assertion: string | Object` - The assertion described in natural language, or [prompting with images](#prompting-with-images).
  - `errorMsg?: string` - An optional error message to append if the assertion fails.
  - `options?: Object` - Optional, a configuration object containing:
    - `domIncluded?: boolean | 'visible-only'` - Whether to send simplified DOM information to the model, usually used for extracting invisible attributes like image links. If set to `'visible-only'`, only the visible elements will be sent. False by default.
    - `screenshotIncluded?: boolean` - Whether to send screenshot to the model. True by default.

- Return Value:

  - Returns a Promise that resolves to void if the assertion passes; if it fails, an error is thrown with `errorMsg` and additional AI-provided information.

- Example:

```typescript
await agent.aiAssert('The price of "Sauce Labs Onesie" is 7.99');
```

:::tip

Assertions are critical in test scripts. To reduce the risk of errors due to AI hallucination (e.g., missing an error), you can also combine `.aiQuery` with standard JavaScript assertions instead of using `.aiAssert`.

For example, you might replace the above code with:

```typescript
const items = await agent.aiQuery(
  '"{name: string, price: number}[], return product names and prices',
);
const onesieItem = items.find((item) => item.name === 'Sauce Labs Onesie');
expect(onesieItem).toBeTruthy();
expect(onesieItem.price).toBe(7.99);
```

:::

### `agent.aiLocate()`

Locate an element using natural language.

- Type

```typescript
function aiLocate(
  locate: string | Object,
  options?: Object,
): Promise<{
  rect: {
    left: number;
    top: number;
    width: number;
    height: number;
  };
  center: [number, number];
  dpr: number; // device pixel ratio
}>;
```

- Parameters:

  - `locate: string | Object` - A natural language description of the element to locate, or [prompting with images](#prompting-with-images).
  - `options?: Object` - Optional, a configuration object containing:
    - `deepLocate?: boolean` - Whether to enable [Deep Locate](#deep-locate-deeplocate). Previously named `deepThink`, now renamed to `deepLocate`. False by default.
    - `xpath?: string` - The xpath of the element to operate. If provided, Midscene will first use this xpath to locate the element before using the cache and the AI model. Empty by default.
    - `cacheable?: boolean` - Whether cacheable when enabling [caching feature](./caching.mdx). True by default.

- Return Value:

  - Returns a `Promise` when the element is located parsed as an locate info object.

- Examples:

```typescript
const locateInfo = await agent.aiLocate(
  'The login button at the top of the page',
);
console.log(locateInfo);
```

### `agent.aiWaitFor()`

Wait until a specified condition, described in natural language, becomes true. Considering the cost of AI calls, the check interval will not exceed the specified `checkIntervalMs`.

- Type

```typescript
function aiWaitFor(
  assertion: string,
  options?: {
    timeoutMs?: number;
    checkIntervalMs?: number;
  },
): Promise<void>;
```

- Parameters:

  - `assertion: string` - The condition described in natural language.
  - `options?: object` - An optional configuration object containing:
    - `timeoutMs?: number` - Maximum window in milliseconds for starting a new check (default: 15000). If the previous evaluation began before this window closes, Midscene keeps checking; otherwise it stops with a timeout.
    - `checkIntervalMs?: number` - Interval for checking in milliseconds (default: 3000).

- Return Value:

  - Returns a Promise that resolves to void if the condition is met; if not, an error is thrown when the timeout is reached.

- Examples:

```typescript
// Basic usage
await agent.aiWaitFor(
  'There is at least one headphone information displayed on the interface',
);

// Using custom options
await agent.aiWaitFor('The shopping cart icon shows a quantity of 2', {
  timeoutMs: 30000, // Wait for 30 seconds
  checkIntervalMs: 5000, // Check every 5 seconds
});
```

:::tip

Given the time consumption of AI services, `.aiWaitFor` might not be the most efficient method. Sometimes, using a simple sleep function may be a better alternative.

:::

### `agent.runYaml()`

Execute an automation script written in YAML. Only the `tasks` part of the script is parsed and executed, and it returns the results of all `.aiQuery` calls within the script.

- Type

```typescript
function runYaml(yamlScriptContent: string): Promise<{ result: any }>;
```

- Parameters:

  - `yamlScriptContent: string` - The YAML-formatted script content.

- Return Value:

  - Returns an object with a `result` property that includes the results of all `.aiQuery` calls.

- Example:

```typescript
const { result } = await agent.runYaml(`
tasks:
  - name: search weather
    flow:
      - ai: input 'weather today' in input box, click search button
      - sleep: 3000

  - name: query weather
    flow:
      - aiQuery: "the result shows the weather info, {description: string}"
`);
console.log(result);
```

:::tip

For more information about YAML scripts, please refer to [Automate with Scripts in YAML](./automate-with-scripts-in-yaml).

:::

### `agent.setAIActContext()`

Set the background knowledge that should be sent to the AI model when calling `agent.aiAct()` or `agent.ai()`. This will override the previous setting.

For instant action type APIs, like `aiTap()`, this setting will not take effect.

- Type

```typescript
function setAIActContext(aiActContext: string): void;
```

- Parameters:

  - `aiActContext: string` - The background knowledge that should be sent to the AI model. The deprecated `aiActionContext` name is still accepted.

- Example:

```typescript
await agent.setAIActContext('Close the cookie consent dialog first if it exists');
```

:::note

`agent.setAIActionContext()` is deprecated. Please use `agent.setAIActContext()` instead. The deprecated method remains as an alias for compatibility.

:::

### `agent.evaluateJavaScript()`

> Only available in web pages, not available in Android.

Evaluate a JavaScript expression in the web page context.

- Type

```typescript
function evaluateJavaScript(script: string): Promise<any>;
```

- Parameters:

  - `script: string` - The JavaScript expression to evaluate.

- Return Value:

  - Returns the result of the JavaScript expression.

- Example:

```typescript
const result = await agent.evaluateJavaScript('document.title');
console.log(result);
```

### `agent.recordToReport()`

Log the current screenshot with a description in the report file.

- Type

```typescript
function recordToReport(title?: string, options?: Object): Promise<void>;
```

- Parameters:

  - `title?: string` - Optional, the title of the screenshot, if not provided, the title will be 'untitled'.
  - `options?: Object` - Optional, a configuration object containing:
    - `content?: string` - The description of the screenshot.

- Return Value:

  - Returns a `Promise<void>`

- Examples:

```typescript
await agent.recordToReport('Login page', {
  content: 'User A',
});
```

### `agent.freezePageContext()`

Freeze the current page context, allowing all subsequent operations to reuse the same page snapshot without retrieving the page state repeatedly. This significantly improves performance when executing a large number of concurrent operations. 

Some notes:
* Usually, you do not need to use this method, unless you are certain that "context retrieval" is the bottleneck of your test script.
* You need to call `agent.unfreezePageContext()` in time to restore the real-time page state.
* Do not call this method in interaction operations, it will make the AI model unable to perceive the latest page state, causing confusing errors.

- Type

```typescript
function freezePageContext(): Promise<void>;
```

- Return Value:

  - `Promise<void>`

- Examples:

```typescript
// Freeze the page context
await agent.freezePageContext();

// Some queries...
const results = await Promise.all([
  await agent.aiQuery('Username input box value'),
  await agent.aiQuery('Password input box value'),
  await agent.aiLocate('Login button'),
]);
console.log(results);

// Unfreeze the page context, subsequent operations will use real-time page state
await agent.unfreezePageContext();
```

:::tip

In the report, operations using frozen context will display a 🧊 icon in the Insight tab.

:::

### `agent.unfreezePageContext()`

Unfreezes the page context, restoring the use of real-time page state.

- Type

```typescript
function unfreezePageContext(): Promise<void>;
```

- Return Value:

  - `Promise<void>`

### `agent._unstableLogContent()`

Retrieve the log content from the report file. The structure of the log object may change in future versions.

- Type

```typescript
function _unstableLogContent(): Object;
```

- Return Value:

  - Returns an object that contains the log content.

- Examples:

```typescript
const logContent = agent._unstableLogContent();
console.log(logContent);
```


## Deep Locate (deepLocate)

`deepLocate` is an optional parameter available on all APIs that require element location (`aiTap`, `aiHover`, `aiInput`, `aiKeyboardPress`, `aiScroll`, `aiDoubleClick`, `aiRightClick`, `aiLocate`, etc.).

When enabled, Midscene will call the AI model twice to precisely locate the element, which can improve accuracy. This is especially useful when the target element is small or hard to distinguish from its surroundings. For newer models (e.g. Qwen3 / Doubao 1.6 / Gemini 3), the gain is less noticeable in most cases, so enable it only when needed.

- **Default**: `false`
- **Parameter renamed**: Previously named `deepThink`, renamed to `deepLocate` since v1.5.1. The old `deepThink` parameter remains compatible.

```typescript
// Enable deepLocate to precisely locate hard-to-identify elements
await agent.aiTap('Shopping cart icon in the top right', { deepLocate: true });
```

:::note

`deepThink` in `aiAct()` and `deepLocate` here are completely different features:

- `deepLocate`: Controls the **precision of element location** by making Midscene run two locate passes to improve accuracy.
- `deepThink` in `aiAct()`: Controls the **reasoning strategy during planning** (toggles AI reasoning capability).

[See aiAct deepThink documentation](./model-strategy#about-the-deepthink-option-in-aiact)

:::

## Prompting with images

You can use images as supplements in the prompt to describe things that cannot be expressed in natural language.

When prompting with images, the format of the prompt parameters is as follows:

```javascript
{
  // Prompt text, in which images can be referred
  prompt: string,
  // The images referred in the prompt text
  images?: {
    // Image name, corresponding to the names referred in the prompt text
    name: string,
    // Image url, can be a local image path, Base64 string, or http link
    url: string
  }[]
  // When convertHttpImage2Base64 is true, the image links in the http format will be converted into Base64 encoding and sent to the LLM.
  // Which is applicable when the image links are not publicly accessible.
  convertHttpImage2Base64?: boolean
}
```

- Example 1: use images to inspect the tap position.

```javascript
await agent.aiTap({
  prompt: 'The specific logo',
  images: [
    {
      name: 'The specific logo',
      url: 'https://github.githubassets.com/assets/GitHub-Mark-ea2971cee799.png',
    },
  ],
});
```

- Example 2: use images to assert the page content.

```javascript
await agent.aiAssert({
  prompt: 'Whether there is a specific logo on the page.',
  images: [
    {
      name: 'The specific logo',
      url: 'https://github.githubassets.com/assets/GitHub-Mark-ea2971cee799.png',
    },
  ],
});
```

**Notes on Image Size**

When prompting with images, it is necessary to pay attention to the requirements of the AI model provider regarding the size and dimensions of the images. Images that are too large (such as exceeding 10M) or too small (such as being less than 10 pixels) may cause errors when the model is invoked. The specific restrictions should be based on the documentation provided by the AI model provider you are using.

## Properties

### `.reportFile`

The path to the report file.


## Report Merging Tool

When running multiple automation workflows, each agent generates its own report file. The `ReportMergingTool` can merge multiple automation reports into a single report for unified viewing and management.

### Use cases

- Running multiple workflows in a suite and need one consolidated report
- Cross-platform automation (for example, Web and Android) that needs a unified result
- CI/CD pipelines that require a summarized automation report

### `new ReportMergingTool()`

Create a report merging tool instance.

- Example:

```typescript
import { ReportMergingTool } from '@midscene/core/report';

const reportMergingTool = new ReportMergingTool();
```

### `.append()`

Add an automation report to the list to be merged, typically right after each workflow finishes.

- Type

```typescript
function append(reportInfo: ReportFileWithAttributes): void;
```

- Parameters:

  - `reportInfo: ReportFileWithAttributes` - Report information object containing:
    - `reportFilePath: string` - Path to the report file, usually `agent.reportFile`
    - `reportAttributes: object` - Report attributes
      - `testId: string` - Unique identifier for the automation workflow
      - `testTitle: string` - Automation workflow title
      - `testDescription: string` - Automation workflow description
      - `testDuration: number` - Automation execution duration (in milliseconds)
      - `testStatus: 'passed' | 'failed' | 'timedOut' | 'skipped' | 'interrupted'` - Automation status

- Return Value:

  - `void`

- Example:

```typescript
// Add a report in your test hooks
afterEach((ctx) => {
  let workflowStatus = 'passed';
  if (ctx.task.result?.state === 'fail') {
    workflowStatus = 'failed';
  }

  reportMergingTool.append({
    reportFilePath: agent.reportFile as string,
    reportAttributes: {
      testId: ctx.task.name,
      testTitle: ctx.task.name,
      testDescription: 'Automation workflow description',
      testDuration: Date.now() - startTime,
      testStatus: workflowStatus,
    },
  });
});
```

### `.mergeReports()`

Merge all added reports into a single HTML file.

- Type

```typescript
function mergeReports(
  reportFileName?: 'AUTO' | string,
  opts?: {
    rmOriginalReports?: boolean;
    overwrite?: boolean;
  },
): string | null;
```

- Parameters:

  - `reportFileName?: 'AUTO' | string` - Name of the merged report file
    - Defaults to `'AUTO'`, which will generate a file name automatically
    - You can also provide a custom name (no `.html` suffix needed)
  - `opts?: object` - Optional configuration object
    - `rmOriginalReports?: boolean` - Whether to delete the original report files after merging (default: `false`)
    - `overwrite?: boolean` - Whether to overwrite when the target file already exists (default: `false`)

- Return Value:

  - Returns the merged report file path when successful
  - Returns `null` if there are fewer than two reports to merge

- Examples:

```typescript
// Basic usage with an auto-generated file name
afterAll(() => {
  reportMergingTool.mergeReports();
});

// Specify a custom file name
afterAll(() => {
  reportMergingTool.mergeReports('my-automation-report');
});

// Merge and delete the original reports
afterAll(() => {
  reportMergingTool.mergeReports('my-automation-report', {
    rmOriginalReports: true,
  });
});

// Overwrite an existing report file
afterAll(() => {
  reportMergingTool.mergeReports('my-automation-report', {
    overwrite: true,
  });
});
```

### `.clear()`

Clear the list of reports to be merged. Use this if you need to reuse the same instance for multiple merge operations.

- Type

```typescript
function clear(): void;
```

- Return Value:

  - `void`

- Example:

```typescript
reportMergingTool.mergeReports('first-batch');
reportMergingTool.clear(); // Clear the list
// Continue adding new reports...
```

### Full example

Below is a complete example of using `ReportMergingTool` in a Vitest suite:

```typescript
import { describe, it, beforeEach, afterEach, afterAll } from 'vitest';
import { AndroidAgent, AndroidDevice } from '@midscene/android';
import { ReportMergingTool } from '@midscene/core/report';

describe('Android settings automation', () => {
  let device: AndroidDevice;
  let agent: AndroidAgent;
  let startTime: number;
  const reportMergingTool = new ReportMergingTool();

  beforeEach((ctx) => {
    startTime = performance.now();
    agent = new AndroidAgent(device, {
      groupName: ctx.task.name,
    });
  });

  afterEach((ctx) => {
    let workflowStatus = 'passed';
    if (ctx.task.result?.state === 'pass') {
      workflowStatus = 'passed';
    } else if (ctx.task.result?.state === 'skip') {
      workflowStatus = 'skipped';
    } else if (ctx.task.result?.errors?.[0]?.message.includes('timed out')) {
      workflowStatus = 'timedOut';
    } else {
      workflowStatus = 'failed';
    }

    // Add the report to the merge list
    reportMergingTool.append({
      reportFilePath: agent.reportFile as string,
      reportAttributes: {
        testId: ctx.task.name,
        testTitle: ctx.task.name,
        testDescription: 'Automation workflow description',
        testDuration: (Date.now() - ctx.task.result?.startTime!) | 0,
        testStatus: workflowStatus,
      },
    });
  });

  afterAll(() => {
    // Merge all automation reports
    reportMergingTool.mergeReports('android-settings-automation-report');
  });

  it('toggle WLAN', async () => {
    await agent.aiAct('Find and open WLAN settings');
    await agent.aiAct('Toggle WLAN once');
  });

  it('toggle Bluetooth', async () => {
    await agent.aiAct('Find and open Bluetooth settings');
    await agent.aiAct('Toggle Bluetooth once');
  });
});
```

:::tip

The merged report is saved under the `midscene_run/report` directory. Open the generated HTML file in your browser to review the workflows.

:::



---
url: /automate-with-scripts-in-yaml.md
---



# Automate with scripts in YAML

In most cases, developers write automation scripts just to perform some smoke tests, like checking for the appearance of certain content or verifying that a key user path is accessible. In such situations, maintaining a large test project is unnecessary.

Midscene offers a way to perform automation using `.yaml` files, which helps you focus on the script itself rather than the testing framework. This allows any team member to write automation scripts without needing to learn any API.

Here is an example. By reading its content, you should be able to understand how it works.

```yaml
web:
  url: https://www.bing.com

tasks:
  - name: Search for weather
    flow:
      - ai: Search for "today's weather"
      - sleep: 3000

  - name: Check results
    flow:
      - aiAssert: The results show weather information
```

:::info Sample Project

You can find a sample project that uses YAML scripts for automation here:

- [Web](https://github.com/web-infra-dev/midscene-example/tree/main/yaml-scripts-demo)
- [Android](https://github.com/web-infra-dev/midscene-example/tree/main/android/yaml-scripts-demo)
- [Computer (Mac/Windows/Linux)](https://github.com/web-infra-dev/midscene-example/tree/main/computer/yaml-scripts-demo)

:::

## Set up API keys for model

Set your model configs into the environment variables. You may refer to [Model strategy](../model-strategy) for more details.

```bash
export MIDSCENE_MODEL_BASE_URL="https://replace-with-your-model-service-url/v1"
export MIDSCENE_MODEL_API_KEY="replace-with-your-api-key"
export MIDSCENE_MODEL_NAME="replace-with-your-model-name"
export MIDSCENE_MODEL_FAMILY="replace-with-your-model-family"
```

For more configuration details, please refer to [Model strategy](../model-strategy) and [Model configuration](../model-config).


To execute YAML workflows from the command line, install the Midscene CLI. See [YAML script runner](./yaml-script-runner) for installation guidance, `.env` usage, and details on the `midscene` runner.

## Script file structure

Script files use YAML format to describe automation tasks. It defines the target to be manipulated (like a webpage or an Android app) and the series of steps to perform.

A standard `.yaml` script file includes a `web`, `android`, `ios`, or `computer` section to configure the environment, an optional `agent` section to configure AI agent behavior, and a `tasks` section to define the automation tasks.

```yaml
web:
  url: https://www.bing.com

# The tasks section defines the series of steps to be executed
tasks:
  - name: Search for weather
    flow:
      - ai: Search for "today's weather"
      - sleep: 3000
      - aiAssert: The results show weather information
```


### The `agent` part

The `agent` section configures AI agent behavior and test report options. All fields are optional.

```yaml
# AI agent configuration
agent:
  # Test identifier, used for reporting and cache identification, optional
  testId: <string>

  # Report group name, optional
  groupName: <string>

  # Report group description, optional
  groupDescription: <string>

  # Whether to generate test reports, optional, defaults to true
  generateReport: <boolean>

  # Whether to automatically print report messages, optional, defaults to true
  autoPrintReportMsg: <boolean>

  # Custom report file name, optional
  reportFileName: <string>

  # Maximum AI replanning cycle limit, optional, defaults to 20 (40 for UI-TARS model)
  replanningCycleLimit: <number>

  # Background knowledge to send to the AI model when calling aiAct, optional
  aiActContext: <string>
  # Legacy alias (aiActionContext) remains for backward compatibility, but avoid using it in new scripts

  # Cache configuration, optional
  cache:
    # Cache strategy, optional, values: 'read-only' | 'read-write' | 'write-only'
    strategy: <string>
    # Cache ID, required
    id: <string>
```

:::tip Agent Configuration Notes

- **Applicable environments**: Web, iOS, and Android environments all support `agent` configuration
- **testId priority**: CLI parameter > YAML agent.testId > filename
- **aiActContext**: Provides background knowledge to the AI model, like how to handle popups, business introduction, etc. A legacy alias remains for backward compatibility (see inline comment) but should not be used in new scripts.
- **Cache configuration**: For detailed usage, refer to the [Caching documentation](./caching.mdx)

:::

#### Usage example

```yaml
# agent configuration, applies to all environments
agent:
  testId: "checkout-test"
  groupName: "E2E Test Suite"
  groupDescription: "Complete checkout flow testing"
  generateReport: true
  autoPrintReportMsg: false
  reportFileName: "checkout-report"
  replanningCycleLimit: 30
  aiActContext: "If any popup appears, click agree. If login page appears, skip it."
  cache:
    id: "checkout-cache"
    strategy: "read-write"

# iOS environment configuration
ios:
  launch: https://www.bing.com
  wdaPort: 8100

# Or Android environment configuration
android:
  deviceId: s4ey59
  launch: https://www.bing.com

tasks:
  - name: Search for weather
    flow:
      - ai: Search for "today's weather"
      - aiAssert: The results show weather information
```


### The `web` part

```yaml
web:
  # The URL to visit, required. If `serve` is provided, provide the relative path.
  url: <url>

  # Serve a local path as a static server, optional.
  serve: <root-directory>

  # The browser user agent, optional.
  userAgent: <ua>

  # The browser viewport width, optional, defaults to 1280.
  viewportWidth: <width>

  # The browser viewport height, optional, defaults to 960.
  viewportHeight: <height>

  # The browser's device pixel ratio, optional, defaults to the system's value.
  deviceScaleFactor: <scale>

  # Path to a JSON format browser cookie file, optional.
  cookie: <path-to-cookie-file>

  # The strategy for waiting for network idle, optional.
  waitForNetworkIdle:
    # The timeout in milliseconds, optional, defaults to 2000ms.
    timeout: <ms>
    # Whether to continue on timeout, optional, defaults to true.
    continueOnNetworkIdleError: <boolean>

  # The path to the JSON file for outputting aiQuery/aiAssert results, optional.
  output: <path-to-output-file>

  # Whether to save log content to a JSON file, optional, defaults to `false`. If true, saves to `unstableLogContent.json`. If a string, saves to the specified path. The log content structure may change in the future.
  unstableLogContent: <boolean | path-to-unstable-log-file>

  # Whether to restrict page navigation to the current tab, optional, defaults to true.
  forceSameTabNavigation: <boolean>

  # The bridge mode, optional, defaults to false. Can be 'newTabWithUrl' or 'currentTab'. See below for more details.
  bridgeMode: false | 'newTabWithUrl' | 'currentTab'

  # Whether to close newly created tabs when the bridge disconnects, optional, defaults to false.
  closeNewTabsAfterDisconnect: <boolean>

  # Whether to ignore HTTPS certificate errors, optional, defaults to false.
  acceptInsecureCerts: <boolean>

  # Custom Chrome launch arguments (Puppeteer only, not supported in bridge mode), optional.
  # Use this to customize Chrome browser behavior, such as disabling third-party cookie blocking.
  # ⚠️ Security Warning: Some arguments (e.g., --no-sandbox, --disable-web-security) may reduce browser security.
  # Use only in controlled testing environments.
  chromeArgs:
    - '--disable-features=ThirdPartyCookiePhaseout'
    - '--disable-features=SameSiteByDefaultCookies'
    - '--window-size=1920,1080'
```

### The `android` part

```yaml
android:
  # The device ID, optional, defaults to the first connected device.
  deviceId: <device-id>

  # The launch URL, optional, defaults to the device's current page.
  launch: <url>

  # The path to the JSON file for outputting aiQuery/aiAssert results, optional.
  output: <path-to-output-file>

  # All other options supported by the AndroidDevice constructor
  # For example: androidAdbPath, remoteAdbHost, remoteAdbPort,
  # imeStrategy, displayId, autoDismissKeyboard, keyboardDismissStrategy,
  # screenshotResizeScale, alwaysRefreshScreenInfo, etc.
  # See the AndroidDevice constructor documentation for the complete list
```

:::tip View Complete Android Configuration Options

YAML scripts now support all configuration options from the `AndroidDevice` constructor. For the complete list of options, please refer to [AndroidDevice Constructor in Android Integration Documentation](./android-getting-started#androiddevice-constructor).

:::

#### Android Platform-Specific Actions

**`runAdbShell` - Execute ADB Shell Commands**

Execute ADB shell commands on Android devices.

```yaml
android:
  deviceId: 'test-device'

tasks:
  - name: Clear app data
    flow:
      - runAdbShell: 'pm clear com.example.app'

  - name: Get battery info
    flow:
      - runAdbShell: 'dumpsys battery'
```

**Common ADB Shell Commands:**
- `pm clear <package>` - Clear app data
- `dumpsys battery` - Get battery information
- `dumpsys window` - Get window information
- `settings get secure android_id` - Get device ID
- `input keyevent <keycode>` - Send key events

**`launch` - Launch App or URL**

Launch Android apps or open URLs.

```yaml
android:
  deviceId: 'test-device'

tasks:
  - name: Launch Settings app
    flow:
      - launch: com.android.settings

  - name: Open webpage
    flow:
      - launch: https://www.example.com
```

### The `ios` part

```yaml
ios:
  # WebDriverAgent port, optional, defaults to 8100.
  wdaPort: <port>

  # WebDriverAgent host address, optional, defaults to localhost.
  wdaHost: <host>

  # Whether to auto dismiss keyboard, optional, defaults to false.
  autoDismissKeyboard: <boolean>

  # Launch URL or app bundle ID, optional, defaults to the device's current page.
  launch: <url-or-bundle-id>

  # The path to the JSON file for outputting aiQuery/aiAssert results, optional.
  output: <path-to-output-file>

  # Whether to save log content to a JSON file, optional, defaults to `false`. If true, saves to `unstableLogContent.json`. If a string, saves to the specified path. The log content structure may change in the future.
  unstableLogContent: <boolean | path-to-unstable-log-file>

  # All other options supported by the IOSDevice constructor
  # See the IOSDevice constructor documentation for the complete list
```

:::tip View Complete iOS Configuration Options

YAML scripts now support all configuration options from the `IOSDevice` constructor. For the complete list of options, please refer to [IOSDevice constructor in the iOS API reference](./ios-api-reference#iosdevice-constructor).

:::

#### iOS Platform-Specific Actions

**`runWdaRequest` - Execute WebDriverAgent API Requests**

Execute WebDriverAgent API requests directly on iOS devices.

```yaml
ios:
  launch: 'com.apple.mobilesafari'

tasks:
  - name: Press home button via WDA
    flow:
      - runWdaRequest:
          method: POST
          endpoint: /session/test/wda/pressButton
          data:
            name: home

  - name: Get device information
    flow:
      - runWdaRequest:
          method: GET
          endpoint: /wda/device/info
```

**Parameters:**
- `method` (string, required): HTTP method (GET, POST, DELETE, etc.)
- `endpoint` (string, required): WebDriverAgent API endpoint
- `data` (any, optional): Request body data

**Common WebDriverAgent Endpoints:**
- `/wda/screen` - Get screen information
- `/wda/device/info` - Get device information
- `/session/{sessionId}/wda/pressButton` - Press hardware buttons
- `/session/{sessionId}/wda/apps/launch` - Launch apps
- `/session/{sessionId}/wda/apps/activate` - Activate apps

**`launch` - Launch App or URL**

Launch iOS apps or open URLs.

```yaml
ios:
  wdaPort: 8100

tasks:
  - name: Launch Settings app
    flow:
      - launch: com.apple.Preferences

  - name: Open webpage
    flow:
      - launch: https://www.example.com
```

### The `computer` part

The `computer` section is used for PC desktop automation. It allows you to control the desktop environment, including mouse movements, keyboard inputs, and screen queries.

```yaml
computer:
  # The display ID to use, optional, defaults to the primary display.
  displayId: <display-id>

  # The path to the JSON file for outputting aiQuery/aiAssert results, optional.
  output: <path-to-output-file>
```

#### Usage example

```yaml
computer: {}

tasks:
  - name: Open browser and search
    flow:
      - aiAct: press Cmd+Space
      - sleep: 500
      - aiAct: type "Safari" and press Enter
      - sleep: 2000
      - aiAct: press Cmd+L to focus address bar
      - aiAct: type "https://www.bing.com"
      - aiAct: press Enter
      - sleep: 3000
      - aiAct: type "weather today" in the search box and press Enter
      - aiAssert: The results show weather information
```

:::tip Platform Notes

The demo scripts above use macOS commands. For Windows, modify the scripts to use:
- `press Windows key` instead of `press Cmd+Space`
- `type "Chrome"` instead of `type "Safari"`
- `press Ctrl+L` instead of `press Cmd+L`

:::

### The `tasks` part

The `tasks` part is an array that defines the steps of the script. Remember to add a `-` before each step to indicate it's an array item.

The interfaces in the `flow` section are almost identical to the [API](./api.html), with some differences in parameter nesting levels.

```yaml
tasks:
  - name: <name>
    continueOnError: <boolean> # Optional, whether to continue to the next task on error, defaults to false.
    flow:
      # Auto Planning (.ai)
      # ----------------

      # Perform an interaction. `ai` is a shorthand for `aiAct`.
      - ai: <prompt>
        cacheable: <boolean> # Optional, whether to cache the result of this API call when the [caching feature](./caching.mdx) is enabled. Defaults to True.
        deepThink: <boolean> # Optional, enable deep thinking for planning when the model supports it (depends on MIDSCENE_MODEL_FAMILY). Default is undefined and follows the model provider's default strategy.

      # This usage is the same as `ai`.
      # Note: In earlier versions, this was also written as `aiAction`. The current version supports both names.
      - aiAct: <prompt>
        cacheable: <boolean> # Optional, whether to cache the result of this API call when the [caching feature](./caching.mdx) is enabled. Defaults to True.
        deepThink: <boolean> # Optional, enable deep thinking for planning when the model supports it (depends on MIDSCENE_MODEL_FAMILY). Default is undefined and follows the model provider's default strategy.

      # Instant Action (.aiTap, .aiHover, .aiInput, .aiKeyboardPress, .aiScroll)
      # ----------------

      # Tap an element described by a prompt.
      - aiTap: <prompt>
        deepThink: <boolean> # Optional, whether to use deepThink to precisely locate the element. Defaults to False.
        xpath: <xpath> # Optional, the xpath of the target element for the operation. If provided, Midscene will prioritize this xpath to find the element before using the cache and the AI model. Defaults to empty.
        cacheable: <boolean> # Optional, whether to cache the result of this API call when the [caching feature](./caching.mdx) is enabled. Defaults to True.

      # Hover over an element described by a prompt.
      - aiHover: <prompt>
        deepThink: <boolean> # Optional, whether to use deepThink to precisely locate the element. Defaults to False.
        xpath: <xpath> # Optional, the xpath of the target element for the operation. If provided, Midscene will prioritize this xpath to find the element before using the cache and the AI model. Defaults to empty.
        cacheable: <boolean> # Optional, whether to cache the result of this API call when the [caching feature](./caching.mdx) is enabled. Defaults to True.

      # Input text into an element described by a prompt.
      - aiInput: <prompt> # The element to input text into.
        value: <final text content of the input>
        deepThink: <boolean> # Optional, whether to use deepThink to precisely locate the element. Defaults to False.
        xpath: <xpath> # Optional, the xpath of the target element for the operation. If provided, Midscene will prioritize this xpath to find the element before using the cache and the AI model. Defaults to empty.
        cacheable: <boolean> # Optional, whether to cache the result of this API call when the [caching feature](./caching.mdx) is enabled. Defaults to True.

      # Press a key (e.g., Enter, Tab, Escape) on an element described by a prompt.
      - aiKeyboardPress: <prompt> # The element to press the key on.
        keyName: <key>
        deepThink: <boolean> # Optional, whether to use deepThink to precisely locate the element. Defaults to False.
        xpath: <xpath> # Optional, the xpath of the target element for the operation. If provided, Midscene will prioritize this xpath to find the element before using the cache and the AI model. Defaults to empty.
        cacheable: <boolean> # Optional, whether to cache the result of this API call when the [caching feature](./caching.mdx) is enabled. Defaults to True.

      # Scroll globally or on an element described by a prompt.
      - aiScroll: <prompt> # Optional, the element to scroll on.
        scrollType: 'singleAction' # or 'scrollToBottom' | 'scrollToTop' | 'scrollToRight' | 'scrollToLeft'. Defaults to 'singleAction'.
        direction: 'down' # or 'up' | 'left' | 'right'. Defaults to 'down'. Only effective when scrollType is singleAction.
        distance: <number> # Optional, the scroll distance in pixels. Use null to let Midscene decide automatically.
        deepThink: <boolean> # Optional, whether to use deepThink to precisely locate the element. Defaults to False.
        xpath: <xpath> # Optional, the xpath of the target element for the operation. If provided, Midscene will prioritize this xpath to find the element before using the cache and the AI model. Defaults to empty.
        cacheable: <boolean> # Optional, whether to cache the result of this API call when the [caching feature](./caching.mdx) is enabled. Defaults to True.

      # Log the current screenshot with a description in the report file.
      - recordToReport: <title> # Optional, the title of the screenshot. If not provided, the title will be 'untitled'.
        content: <content> # Optional, the description of the screenshot.

      # Data Extraction
      # ----------------

      # Perform a query that returns a JSON object.
      - aiQuery: <prompt> # Remember to describe the format of the result in the prompt.
        name: <name> # The key for the query result in the JSON output.

      # More APIs
      # ----------------

      # Wait for a condition to be met, with a timeout (in ms, optional, defaults to 30000).
      - aiWaitFor: <prompt>
        timeout: <ms>

      # Perform an assertion.
      - aiAssert: <prompt>
        errorMessage: <error-message> # Optional, the error message to print if the assertion fails.
        name: <name> # Optional, give the assertion a name, which will be used as a key in the JSON output.

      # Wait for a specified amount of time.
      - sleep: <ms>

      # Execute a piece of JavaScript code in the web page context.
      - javascript: <javascript>
        name: <name> # Optional, assign a name to the return value, which will be used as a key in the JSON output.

  - name: <name>
    flow:
      # ...
```

#### Prompting with images

For steps whose prompt accepts images, you can attach images to the prompt by setting the `images` field to an array of objects, each containing a `name` and a `url`. (see the [API reference](./api#prompting-with-images)), replace the string value with an object that contains:

- `prompt`: The text prompt.
- `images`: (Optional) The reference images used in the prompt. Each image needs a `name` and a `url`.
- `convertHttpImage2Base64`: (Optional) Converts HTTP image links to Base64 before sending them to the model, which is useful when the link is not publicly accessible.

Image URLs can be local paths, Base64 strings, or remote links. When using image links that cannot be accessed for the model, set `convertHttpImage2Base64: true` so Midscene will download the image and send the base64 string to the model.

For interactions like `aiTap`, `aiHover`, `aiDoubleClick`, `aiRightClick`, put the text and images in the `locate` field.

```yaml
tasks:
  - name: Verify branding
    flow:
      - aiHover:
          locate:
            prompt: Move the cursor to the region containing the GitHub logo.
            images:
              - name: GitHub logo
                url: https://github.githubassets.com/assets/GitHub-Mark-ea2971cee799.png
            convertHttpImage2Base64: true

      - aiTap:
          locate:
            prompt: Tap the region containing the GitHub logo.
            images:
              - name: GitHub logo
                url: https://github.githubassets.com/assets/GitHub-Mark-ea2971cee799.png
            convertHttpImage2Base64: true
```

For insight steps like `aiAsk`, `aiQuery`, `aiBoolean`, `aiNumber`, `aiString`, and `aiAssert`, you can set the `prompt` and `images` fields directly.

```yaml
tasks:
  - name: Verify branding
    flow:
      - aiAssert:
          prompt: Check whether the image appears on the page.
          images:
            - name: target logo
              url: https://github.githubassets.com/assets/GitHub-Mark-ea2971cee799.png
          convertHttpImage2Base64: true
```

## Notes

### `agent.runYaml` only parses the `tasks` field

When using the `agent.runYaml()` API, only the `tasks` field in the YAML file is parsed and executed.

At that point the agent has already been initialized in the JS script, so it cannot be reinitialized based on the `agent` configuration in the YAML file.



---
url: /awesome-midscene.md
---

# Awesome Midscene

A curated list of community projects that extend Midscene.js capabilities across different platforms and programming languages.

## Community projects

### iOS automation
- **[midscene-ios](https://github.com/lhuanyu/midscene-ios)** - iOS Mirror automation support for Midscene
  - Enables automated testing and interaction with iOS applications
  - Extends Midscene's cross-platform capabilities to Apple's mobile ecosystem

### PC automation
- **[midscene-pc](https://github.com/Mofangbao/midscene-pc)** - PC operation device for Windows, macOS, and Linux
  - Enables automated testing and interaction with desktop applications across all major platforms
  - Supports both local and remote operation capabilities
- **[midscene-pc-docker](https://github.com/Mofangbao/midscene-pc-docker)** - Docker container image with Midscene-PC server pre-installed
  - Based on Ubuntu 20 with GNOME desktop for maximum application compatibility
  - Includes built-in VNC service for browser-based desktop monitoring
  - Deploy automation client directly on standard servers with a single command

### Python SDK
- **[Midscene-Python](https://github.com/Python51888/Midscene-Python)** - Python SDK for Midscene automation
  - Brings Midscene's AI-powered automation capabilities to Python developers
  - Allows integration with existing Python testing and automation workflows

### Java SDK
- **[midscene-java](https://github.com/Master-Frank/midscene-java)** by @Master-Frank - Java SDK for Midscene automation
  - Offers a JVM-friendly way to script Midscene experiences similar to the Python SDK
  - Fits easily into existing Java automation or testing pipelines
- **[midscene-java](https://github.com/alstafeev/midscene-java)** by @alstafeev - Java SDK for Midscene automation
  - Provides a JVM-native interface for scripting Midscene
  - Integrates seamlessly into established Java testing frameworks and automation workflows

## Contributing

Have you created a project that extends Midscene.js? We'd love to feature it here!

To add your project to this list, please submit an issue to the [Midscene repository](https://github.com/web-infra-dev/midscene), and tell us your awesome midscene project.

## Criteria for inclusion

Projects featured in Awesome Midscene should:
- Extend or integrate with Midscene.js functionality
- Be actively maintained
- Have clear documentation and usage examples
- Provide value to the Midscene community

---

*Don't see your favorite platform or language supported yet? Consider creating a community project or contributing to existing ones!*



---
url: /blog-introducing-instant-actions-and-deep-think.md
---

# Introducing Instant Actions and Deep Think

From Midscene v0.14.0, we have introduced two new features: Instant Actions and Deep Think.

## Instant Actions - a more predictable way to perform actions

You may have already been familiar with our `.ai` interface. It's an auto-planning interface to interact with web pages. For example, when performing a search, you can do this:

```typescript
await agent.ai('type "Headphones" in search box, hit Enter');
```

Behind the scene, Midscene will call the LLM to plan the steps and execute them. You can see the report file to see the process. It's a very common way for AI agents to these kinds of tasks.

![](/blog/report-planning.png)

In the meantime, there are many testing engineers who want a faster way to perform actions. When using AI models with complex prompts, some of the LLM models may find it hard to plan the proper steps, or the coordinates of the elements may not be accurate. It could be frustrating for debugging the unpredictable process.

To solve this problem, we have introduced the `aiTap()`, `aiHover()`, `aiInput()`, `aiKeyboardPress()`, `aiScroll()` interfaces. They are call the **"instant actions"**. These interfaces will directly perform the specified action as the interface name suggests, while the AI model is responsible for the easier tasks such as locating elements. The whole process can be obviously faster and more reliable after using them.

For example, the search action above can be rewritten as:

```typescript
await agent.aiInput('Headphones', 'search-box');
await agent.aiKeyboardPress('Enter');
```

The typical workflow in the report file is like this, as you can see there is no planning process in the report file:

![](/blog/report-instant-action.png)

The scripts with instant actions seems a little bit redundant (or not 'ai-style'), but we believe these structured interfaces are a good way to save time debugging when the action is already clear.

## Deep Think - a more accurate way to locate elements

When using Midscene with some complex widgets, the LLM may find it hard to locate the target element. We have introduced a new option named `deepThink` to the instant actions.

The signature of the instant actions with `deepThink` is like this:

```typescript
await agent.aiTap('target', { deepThink: true });
```

`deepThink` is a strategy of locating elements. It will first find an area that contains the target element, then "focus" on this area to search the element again. By this way, the coordinates of the target element will be more accurate. 

Let's take the workflow editor page of Coze.com as an example. There are many customized icons on the sidebar. This is usually hard for LLMs to distinguish the target element from its surroundings.

![](/blog/coze-sidebar.png)

After using `deepThink` in instant actions, the yaml scripts will be like this (of course, you can also use the javascript interface):

```yaml
tasks:
  - name: edit input panel
    flow:
      - aiTap: the triangle icon on the left side of the text "Input"
        deepThink: true
      - aiTap: the first checkbox in the Input form
        deepThink: true
      - aiTap: the expand button on the second row of the Input form (on the right of the checkbox)
        deepThink: true
      - aiTap: the delete button on the second last row of the Input form
        deepThink: true
      - aiTap: the add button on the last row of the Input form （second button from the right）
        deepThink: true
```

By viewing the report file, you can see Midscene has found every target element in the area.

![](/blog/report-coze-deep-think.png)

Just like the example above, the highly-detailed prompt for `deepThink` is the key to keeping results stable.

`deepThink` is only available with the models that support visual grounding like qwen2.5-vl. If you are using LLM models like gpt-4o, it won't work.



---
url: /bridge-mode.md
---



# Bridge mode by Chrome extension

import { PackageManagerTabs } from '@theme';

The bridge mode in the Midscene Chrome extension is a tool that allows you to use local scripts to control the desktop version of Chrome. Your scripts can connect to either a new tab or the currently active tab.

Using the desktop version of Chrome allows you to reuse all cookies, plugins, page status, and everything else you want. You can work with automation scripts to complete your tasks. This mode is commonly referred to as 'man-in-the-loop' in the context of automation.

![bridge mode](/midscene-bridge-mode.png)

:::info Demo Project

check the demo project of bridge mode: [https://github.com/web-infra-dev/midscene-example/blob/main/bridge-mode-demo](https://github.com/web-infra-dev/midscene-example/blob/main/bridge-mode-demo)

:::

## Set up API keys for model

Set your model configs into the environment variables. You may refer to [Model strategy](../model-strategy) for more details.

```bash
export MIDSCENE_MODEL_BASE_URL="https://replace-with-your-model-service-url/v1"
export MIDSCENE_MODEL_API_KEY="replace-with-your-api-key"
export MIDSCENE_MODEL_NAME="replace-with-your-model-name"
export MIDSCENE_MODEL_FAMILY="replace-with-your-model-family"
```

For more configuration details, please refer to [Model strategy](../model-strategy) and [Model configuration](../model-config).


> In bridge mode, the AI models configs should be set in the Node.js side instead of the browser side.

## Get started

### Step 1. Install Midscene extension from Chrome web store

Install [Midscene extension from Chrome web store](https://chromewebstore.google.com/detail/midscene/gbldofcpkknbggpkmbdaefngejllnief)

### Step 2. Install dependencies

<PackageManagerTabs command="install @midscene/web tsx --save-dev" />

### Step 3. Write scripts

Write and save the following code as `./demo-new-tab.ts`.

```typescript
import { AgentOverChromeBridge } from "@midscene/web/bridge-mode";

const sleep = (ms) => new Promise((r) => setTimeout(r, ms));
Promise.resolve(
  (async () => {
    const agent = new AgentOverChromeBridge();

    // This will connect to a new tab on your desktop Chrome
    await agent.connectNewTabWithUrl("https://www.bing.com");

    // these are the same as normal Midscene agent
    await agent.ai('type "AI 101" and hit Enter');
    await sleep(3000);

    await agent.aiAssert("there are some search results");
    await agent.destroy();
  })()
);
```

### Step 4. Run the script

Run your scripts

```bash
tsx demo-new-tab.ts
```

After running the script, the Chrome extension will pop up a confirmation dialog asking whether to allow the connection. Click "Allow" to allow the current connection, or click "Always Allow" to automatically allow all future connection requests (can be reset in the Bridge Mode panel). You should then see a new tab opened in your desktop Chrome, controlled by your scripts.

<p align="center">
  <img src="/bridge_in_extension.png" alt="bridge in extension" width="400"/>
</p>

:::tip

The extension listens for connection requests in the background by default, no manual action is needed. The extension icon will display a status badge: yellow dot for listening, green dot for connected.

:::

## Use bridge mode in YAML script

[Yaml scripts](./automate-with-scripts-in-yaml) is a way for developers to write automation scripts in yaml format, which is easy to read and write comparing to javascript.

To use bridge mode in yaml script, set the `bridgeMode` property in the `web` section. If you want to use the current tab, set it to `currentTab`, otherwise set it to `newTabWithUrl`.

Set `closeNewTabsAfterDisconnect` to true if you want to close the newly created tabs when the bridge is destroyed. This is optional and the default value is false.

For example, the following script will open a new tab by Chrome extension bridge: 

```diff
web:
  url: https://www.bing.com
+ bridgeMode: newTabWithUrl
+ closeNewTabsAfterDisconnect: true
tasks:
```

Run the script:

```bash
midscene ./bing.yaml
```

After the script starts, click "Allow" in the confirmation dialog to proceed.

### Unsupported options

In bridge mode, these options will be ignored (they will follow your desktop browser's settings):
- userAgent
- viewportWidth
- viewportHeight
- deviceScaleFactor
- waitForNetworkIdle
- cookie

## Remote Access Configuration

By default, the Bridge Server only listens on `127.0.0.1`, allowing only local Chrome extension connections. If you need cross-machine communication (e.g., server on machine A, browser on machine B), you can enable remote access:

**Server Side (Node.js Script):**

```typescript
// Enable remote access (recommended)
const agent = new AgentOverChromeBridge({
  allowRemoteAccess: true  // Listen on 0.0.0.0:3766
});

// Or specify a specific network interface
const agent = new AgentOverChromeBridge({
  host: '192.168.1.100',  // Listen on specific IP
  port: 3766
});
```

**Client Side (Chrome Extension):**

1. Open the Chrome extension's Bridge Mode page
2. Fill in the server address in the "Bridge Server URL" input field
   - Local mode: `ws://localhost:3766` (default)
   - Remote mode: `ws://192.168.1.100:3766` (replace with your server IP)
3. Run your script and click "Allow" in the confirmation dialog

<p align="center">
  <img src="/bridge_remote_config.png" alt="bridge remote config" width="400"/>
</p>

:::warning Security Notice

When remote access is enabled, the Bridge Server will be exposed on the network. Please ensure:
- Only use in trusted network environments
- Use firewall to restrict access
- Do not use in public network environments to avoid security risks

:::

## FAQ

* Where should I config the model parameters (like `MIDSCENE_MODEL_API_KEY`), in the browser or in the terminal?

When using bridge mode, you should config the model parameters in the terminal. For more configuration details, please refer to [Model strategy](./model-strategy).

## More

- For every Agent method, check the [API reference](./api#interaction-methods).
- For the full Chrome Bridge API surface, see [API reference (Web)](./web-api-reference#chrome-bridge-agent).
- Demo project
  - Bridge mode demo: [https://github.com/web-infra-dev/midscene-example/blob/main/bridge-mode-demo](https://github.com/web-infra-dev/midscene-example/blob/main/bridge-mode-demo)



---
url: /caching.md
---

# Caching AI planning & locate

Midscene supports caching Plan steps and matched DOM element information to reduce AI model calls and greatly improve execution efficiency. Please note that DOM element cache is only supported for web automation tasks, and has [certain limitations](#limitations-of-xpath-in-caching-element-location).

**Effect**

With caching hit, time cost is significantly reduced. For example, in the following case, execution time was reduced from 51 seconds to 28 seconds.

* **before**

![](/cache/no-cache-time.png)

* **after**

![](/cache/use-cache-time.png)

## Cache files and storage

Midscene's caching mechanism is based on input stability and output reusability. When the same task instructions are repeatedly executed in similar page environments, Midscene will prioritize using cached results to avoid repeated AI model calls, significantly improving execution efficiency.

The core caching mechanisms include:
- **Task instruction caching**: For planning operations (such as `ai`, `aiAct`), Midscene uses the prompt instruction as the cache key to store the execution plan returned by AI
- **Element location caching**: For location operations (such as `aiLocate`, `aiTap`), the system uses the location prompt as the cache key to store element XPath information, and verifies whether the XPath is still valid on the next execution
- **Invalidation mechanism**: When cache becomes invalid, the system automatically falls back to AI model for re-analysis
- **Never cache query results**: The query results like `aiBoolean`, `aiQuery`, `aiAssert` will never be cached.

Cache contents will be saved in the `./midscene_run/cache` directory with the `.cache.yaml` as the extension name.

## Cache strategies

By configuring the `cache` option, you can enable caching for your agent.

### Disable cache

Configuration: `cache: false` or not configuring the `cache` option

Completely disable cache functionality, always call AI model for every operation. Suitable when you need real-time results or for debugging. By default, if you don't configure the `cache` option, caching is disabled.

```javascript
// Direct Agent creation
const agent = new PuppeteerAgent(page, {
  cache: false,
});
```

```yaml
# YAML configuration
agent:
  cache: false
```

### Read-write mode

Configuration: `cache: { id: "my-cache-id" }` or `cache: { strategy: "read-write", id: "my-cache-id" }`

Automatically read existing cache and update cache files during execution. The default value of `strategy` is `read-write`.

```javascript
// Direct Agent creation - explicit cache ID
const agent = new PuppeteerAgent(page, {
  cache: { id: "my-cache-id" },
});

// Explicitly specify strategy
const agent = new PuppeteerAgent(page, {
  cache: { strategy: "read-write", id: "my-cache-id" },
});
```

```yaml
# YAML configuration - explicit cache ID
agent:
  cache:
    id: "my-cache-test"

# Explicitly specify strategy
agent:
  cache:
    id: "my-cache-test"
    strategy: "read-write"
```

YAML mode also supports `cache: true` to automatically use the file name as the cache ID.

### Read-only, manual write

Configuration: `cache: { strategy: "read-only", id: "my-cache-id" }`

Only read cache, no automatic writing to cache files. Requires manual `agent.flushCache()` call to write cache files. Suitable for production environments to ensure cache consistency.

```javascript
// Direct Agent creation
const agent = new PuppeteerAgent(page, {
  cache: { strategy: "read-only", id: "my-cache-id" },
});

// Manual cache write required
await agent.flushCache();
```

```yaml
# YAML configuration
agent:
  cache:
    id: "my-cache-test"
    strategy: "read-only"
```

### Write-only mode

Configuration: `cache: { strategy: "write-only", id: "my-cache-id" }`

Only write to cache, do not read existing cache contents. Always call AI model for each execution and write results to cache file. Suitable for initially building cache or updating cache.

```javascript
// Direct Agent creation
const agent = new PuppeteerAgent(page, {
  cache: { strategy: "write-only", id: "my-cache-id" },
});
```

```yaml
# YAML configuration
agent:
  cache:
    id: "my-cache-test"
    strategy: "write-only"
```

### Compatibility mode (not recommended)

Configuration via `MIDSCENE_CACHE=1` environment variable with cacheId, equivalent to read-write mode.

```javascript
// Old way, requires MIDSCENE_CACHE=1 environment variable and cacheId
const agent = new PuppeteerAgent(originPage, {
  cacheId: 'puppeteer-swag-sab'
});
```

```bash
MIDSCENE_CACHE=1 tsx demo.ts
```

## Using Playwright AI Fixture from `@midscene/web/playwright`

When using `PlaywrightAiFixture` from `@midscene/web/playwright`, pass the same `cache` options to control caching behaviour.

### Disable cache

```typescript
// fixture.ts in sample code
export const test = base.extend<PlayWrightAiFixtureType>(
  PlaywrightAiFixture({
    cache: false,
  }),
);
```

### Read-write mode

```typescript
// fixture.ts in sample code
// Auto-generate cache ID from test metadata
export const test = base.extend<PlayWrightAiFixtureType>(
  PlaywrightAiFixture({
    cache: true,
  }),
);

// fixture.ts in sample code
// Explicitly provide cache ID
export const test = base.extend<PlayWrightAiFixtureType>(
  PlaywrightAiFixture({
    cache: { id: "my-fixture-cache" },
  }),
);
```

### Read-only, manual write

```typescript
// fixture.ts in sample code
export const test = base.extend<PlayWrightAiFixtureType>(
  PlaywrightAiFixture({
    cache: { strategy: "read-only", id: "readonly-cache" },
  }),
);
```

When you run the fixture in read-only mode you need to manually persist the cache after your test steps. Use the `agentForPage` helper provided by the fixture to fetch the underlying agent, then call `agent.flushCache()` at the point where you want to write the cache file:

```typescript
test.afterEach(async ({ page, agentForPage }, testInfo) => {
  // Only flush cache if the test passed
  if (testInfo.status === 'passed') {
    console.log('Test passed, flushing Midscene cache...');
    const agent = await agentForPage(page);
    await agent.flushCache();
  } else {
    console.log(`Test ${testInfo.status}, skipping Midscene cache flush.`);
  }
});

test('manual cache flush', async ({ agentForPage, page, aiTap, aiWaitFor }) => {
  const agent = await agentForPage(page);

  await aiTap('first highlighted link in the hero section');
  await aiWaitFor('the detail page loads completely');

  await agent.flushCache();
});
```

### Write-only mode

```typescript
// fixture.ts in sample code
export const test = base.extend<PlayWrightAiFixtureType>(
  PlaywrightAiFixture({
    cache: { strategy: "write-only", id: "write-only-cache" },
  }),
);
```

In write-only mode, each test will call the AI model and automatically write results to cache file without reading existing cache.

## Cache Cleaning

Midscene supports cleaning unused cache records when flushing cache to file, ensuring cache files stay lean and maintainable. This feature is **completely manual** and requires explicit call to `agent.flushCache({ cleanUnused: true })`.

### Manual Cleaning Mechanism

When calling `agent.flushCache({ cleanUnused: true })`, the system will:

1. **Keep used caches**: Cache records that were matched and used in the current run will be retained
2. **Keep new caches**: Cache records newly generated in the current run will be retained
3. **Remove unused caches**: Old cache records that were not accessed will be automatically deleted
4. **Write to file**: The cleaned cache will be written to file

### Usage

**Call in test afterEach:**

```javascript
describe('test suite', () => {
  let resetFn: () => Promise<void>;
  let agent: PuppeteerAgent;

  afterEach(async () => {
    // Clean cache and write to file
    if (agent) {
      await agent.flushCache({ cleanUnused: true });
    }

    // Then close the page
    if (resetFn) {
      await resetFn();
    }
  });

  it('test case', async () => {
    const { originPage, reset } = await launchPage('https://example.com/');
    resetFn = reset;
    agent = new PuppeteerAgent(originPage, {
      cache: { id: 'my-cache-id' },
    });

    // ... test logic
  });
});
```

**For Playwright AI Fixture users:**

```typescript
test.afterEach(async ({ page, agentForPage }) => {
  const agent = await agentForPage(page);
  await agent.flushCache({ cleanUnused: true });
});
```

### Cleanup Behavior by Mode

- **read-write mode**: Calling `flushCache({ cleanUnused: true })` will clean and write to file
- **read-only mode**: Calling `flushCache({ cleanUnused: true })` will also clean and write to file (manual flush overrides read-only restriction)
- **write-only mode**: No cleanup (does not read cache)

**Note**: If you don't pass `cleanUnused: true` parameter, `flushCache()` will only write to file without cleaning unused caches.

## FAQ

### No cache file is generated

Please ensure you have correctly configured caching:

1. **Direct Agent creation**: Set `cache: { id: "your-cache-id" }` in the constructor
2. **Playwright AI Fixture mode**: Set `cache: true` or `cache: { id: "your-cache-id" }` in fixture configuration
3. **YAML script mode**: Set `agent.cache.id` in the YAML file
4. **Read-only mode**: Ensure you called the `agent.flushCache()` method
5. **Legacy approach**: Set `cacheId` and enable `MIDSCENE_CACHE=1` environment variable

### How to check if the cache is hit?

You can view the report file. If the cache is hit, you will see the `cache` tip and the time cost is obviously reduced.

### Why the cache is missed on CI?

You need to commit the cache files to the repository in CI and recheck the cache hit conditions.

### Does it mean that AI services are no longer needed after using cache?

No. Caching is the way to accelerate the execution, but it's not a tool for ensuring long-term script stability. We have noticed many scenarios where the cache may miss when the DOM structure changes. AI services are still needed to reevaluate the task when the cache miss occurs.

### How to manually remove the cache?

You can remove the cache file in the `./midscene_run/cache` directory, or edit the contents in the cache file.

### How to disable the cache for a single API?

You can use the `cacheable` option to disable the cache for a single API.

Please refer to the documentation of the corresponding [API](./api.mdx) for details.

### Limitations of XPath in caching element location

Midscene uses [XPath](https://developer.mozilla.org/en-US/docs/Web/XML/XPath) to cache the element location. ⁠We are using a relatively strict strategy to prevent false matches. In these situations, the cache will not be accessed:

1. The text content of the new element at the same XPath is different from the cached element.
2. The DOM structure of the page is changed from the cached one.

Additionally, since element location caching relies on DOM structure, caching is not available in the following scenarios:

1. **Canvas elements**: Graphics content inside Canvas does not have DOM nodes and cannot be located via XPath.
2. **Cross-origin iframes**: Browser security policies restrict access to the internal DOM of cross-origin iframes.
3. **Shadow DOM (closed mode)**: Closed Shadow DOM cannot be accessed from outside.
4. **WebGL / Dynamic SVG content**: Dynamically generated graphics may not have a stable DOM structure.

When the cache is not hit or not available, the process will fall back to using AI services to find the element.

### Get more debug logs for caching

You can set the `DEBUG=midscene:cache:*` environment variable to get more debug logs for caching.



---
url: /changelog.md
---

# Changelog

## v1.5 - HarmonyOS Support

v1.5 adds HarmonyOS automation support, Qwen3.5 and doubao-seed 2.0 model support, along with multiple improvements to desktop automation, report system, Chrome extension, and more.

### HarmonyOS Automation Support

New `@midscene/harmony` package officially supports HarmonyOS platform automation. Midscene's automation capabilities now extend from Web, Android, iOS, and Desktop to the HarmonyOS ecosystem.

### Qwen3.5 & doubao-seed 2.0 Model Support

Added support for Qwen3.5 and doubao-seed 2.0 models, allowing developers to leverage newer models for better visual understanding.

### Generic Model Reasoning Configuration

New `MIDSCENE_MODEL_REASONING_EFFORT` environment variable provides a generic model reasoning effort configuration, enabling developers to uniformly control reasoning behavior across different models.

### Desktop Automation Improvements

- **Xvfb virtual display support**: Support Xvfb virtual display for headless Linux environments, enabling desktop automation on CI/CD servers without GUI
- **Connection health check**: Added health check during desktop automation connection for improved reliability
- **macOS input optimization**: All text input on macOS now uses clipboard to avoid IME issues
- **Mouse control failure detection**: Automatically detects mouse control failure and warns about admin privilege requirements
- **Stop execution optimization**: Checks destroyed state to abort screenshot operations promptly when stopping execution

### Screenshot & Display Optimization

- **Custom screenshot shrink**: Support custom screenshot shrink ratio to optimize performance while maintaining recognition accuracy
- **Android scalingRatio decoupling**: Decoupled scalingRatio from size() method for improved flexibility

### Report System Improvements

- **More detailed timing**: Finer-grained timing information in reports helps developers analyze performance bottlenecks more precisely
- **Directory mode support for mergeReports**: `mergeReports` now supports directory mode report files

### Chrome Extension Improvements

- **Always decline option**: Chrome extension adds "always decline" option with confirm race condition fix
- **Close Bridge server after CLI**: Bridge server automatically closes after CLI command completes, preventing lingering processes

### Bug Fixes

- Fixed `z.preprocess` handling in input mode schema for correct form rendering
- Fixed Android swipe parameter passing
- Fixed web size calculation
- Fixed `BASE_URL_FIX_SCRIPT` closing tag not recognized by HTML parser
- Fixed undefined page guard in PlaywrightAgent/PuppeteerAgent constructors

## v1.4 - Skills: Let AI Assistants Control Your Devices

v1.4 introduces Midscene Skills — a set of installable skill packs for AI assistants like Claude Code and OpenClaw, enabling them to directly control browsers, desktops, Android, and iOS devices. This release also includes a standalone desktop MCP service, independent CLI entry points for each platform package, enhanced AI planning, and more.

### Midscene Skills — Device Control Skills for AI Assistants

Midscene Skills is a set of skill packs that can be installed into AI assistants like Claude Code and OpenClaw. Once installed, AI assistants can control browsers, desktops, Android, and iOS devices using natural language.

Each platform package (`@midscene/android`, `@midscene/ios`, `@midscene/web`, etc.) now exposes an independent CLI entry point, which is the foundation that Skills is built upon.

**Supported Platforms:**
- Browser (Puppeteer headless mode)
- Chrome Bridge (user's own desktop Chrome)
- Desktop (macOS, Windows, Linux)
- Android (via ADB)
- iOS (via WebDriverAgent)

See: [Midscene Skills](https://github.com/web-infra-dev/midscene-skills)

### Standalone Desktop Automation MCP Package

New `@midscene/computer-mcp` package provides PC desktop automation as a standalone MCP service. Developers can use desktop automation capabilities directly in MCP-compatible tools like Cursor and Trae without additional integration.

See docs: [PC Desktop Automation](./computer-introduction)

### Chrome Extension MCP Background Connection

The Chrome extension now supports background Bridge mode MCP connection, exposing the desktop browser as an MCP tool to AI assistants, further expanding the MCP ecosystem.

### AI Planning Enhancements

- **New `deepLocate` option for `aiAct`**: Enable deep locating during action execution, improving element locating accuracy in complex interfaces
- **Swipe vs DragAndDrop semantic distinction**: The model can now more precisely distinguish between swipe and drag-and-drop operations, reducing gesture planning errors
- **LLM planning page navigation restrictions**: Prevents the model from generating unreasonable page navigation during planning, improving task execution stability
- **AppleScript keyboard input on macOS**: Improved keyboard input stability and compatibility in desktop automation
- **Cursor move action**: New cursor move action support

### YAML Scripts & File Upload Enhancements

- **YAML `aiTap` supports `fileChooserAccept`**: Handle file upload dialogs directly in YAML scripts
- **Directory upload support**: Web supports `webkitdirectory` type folder selection upload

### Chrome Extension Bridge Mode Caching

Bridge mode now supports caching, reusing existing AI planning results to reduce repeated calls and improve debugging efficiency.

### Android Improvements

- Optimized text input logic for improved input stability

### iOS Improvements

- **Playground live screen stream**: iOS Playground now features live screen preview for real-time device monitoring during debugging.

## v1.3 - PC Desktop Automation Support

v1.3 introduces brand new PC desktop automation capabilities, significantly improves Android screenshot performance, and brings multiple enhancements to report system and stability.

### New PC Desktop Automation Support

Midscene now supports PC desktop automation on Windows, macOS, and Linux, driving native keyboard and mouse controls. Whether it's Electron, Qt, WPF, or native desktop applications, they can all be automated through the visual model approach.

<video src="https://lf3-static.bytednsdoc.com/obj/eden-cn/nupipfups/Midscene/pc-twitter2.mp4" controls/>

**Core Capabilities:**
- **Mouse Operations**: click, double-click, right-click, mouse move, drag and drop
- **Keyboard Input**: text input, key combinations (Cmd/Ctrl/Alt/Shift)
- **Screenshots**: capture screenshots from any monitor
- **Multi-monitor Support**: operate across multiple displays simultaneously

**Usage Methods:**
- Zero-code trial via Computer Playground
- JavaScript SDK for scripting
- YAML automation scripts and CLI tools
- HTML report playback for all operation paths

See documentation: [PC Desktop Automation](./computer-introduction)

### Major Android Screenshot Performance Improvement

With Scrcpy screenshot mode enabled, screenshot time drops from 500–2000ms to **100–200ms**, significantly improving Android automation response speed. This is particularly useful for remote device debugging and high frame rate scenarios.

See documentation: [Scrcpy Screenshot Mode](./android-api-reference#scrcpy)

### Deep Thinking Mode Enhancement

The `aiAct` deep thinking (deepThink) mode now not only helps with element location but also optimizes overall task planning, achieving better execution results in complex forms and multi-step workflows.

### Report Experience Optimization

- **Timeline Collapse**: New collapse toggle button for easier viewing of long task flows
- **Time Unit Changed to Seconds**: More readable
- **Step Sync Highlighting**: Sidebar step highlighting syncs in real-time with player playback
- **Reduced Memory Usage**: Optimized report generation mechanism to effectively reduce runtime memory usage

### Mobile Platform Improvements

#### Android
- More stable special character and Unicode input
- More relaxed app package name matching for Launch action (ignores case and spaces)
- Auto-retry when screenshot anomalies occur on certain devices

#### iOS
- More relaxed Bundle ID matching (ignores case and spaces)

### Web Automation Improvements

- Fixed issue where Puppeteer could hang when taking screenshots of inactive tabs
- Fixed inaccurate window size in headed mode
- `shareBrowserContext` mode now supports preserving localStorage and sessionStorage
- Playwright multi-project configuration automatically distinguishes test cases by browser in reports
- Fixed `typeOnly` mode not working in YAML script input actions

### Other Improvements

- Image processing performance improved
- SVG icon cache issue fixed
- Playground now displays specific reasons for model configuration errors

## v1.2 - Zhipu AI Open-Source Model Support and File Upload Support

v1.2 introduces support for Zhipu AI open-source models, adds file upload functionality, and fixes several issues affecting user experience, making automated testing more reliable.

### New Zhipu AI Open-Source Model Support

#### Zhipu GLM-V Vision Model
- Zhipu GLM-V series models are open-source vision models launched by Zhipu AI, available in multiple parameter versions, supporting both cloud deployment and local deployment.
- See: [GLM-V Model Configuration](./model-common-config.mdx#glm-v)

#### Zhipu AutoGLM Mobile Automation Model
- Zhipu AutoGLM is an open-source mobile automation model launched by Zhipu AI. It can understand mobile screen content based on natural language instructions, and combined with intelligent planning capabilities, generate operation processes to complete user needs.
- See: [AutoGLM Model Configuration](./model-common-config.mdx#auto-glm)

### File upload feature

File upload is a common requirement in Web automation scenarios. v1.2 adds file upload capability for the web, supporting natural language operations for file input boxes, making form automation more complete.

See: [aiTap file upload](./api.mdx#agentaitap)

### Cache mechanism optimization

Fixed the issue where cache wasn't updated after DOM changes. When page DOM changes cause cache validation to fail, the system now automatically updates the cache, avoiding operation failures due to stale cache and improving automation script stability.

### Report and Playground improvements

#### Deep thinking tag optimization
- Fixed the issue where deepThink tags weren't displayed correctly in reports when using `.aiAct()` method with deep thinking. Now you can clearly see which operations used deep thinking capability in reports
- Improved the style of summary rows in reports for better readability

#### Playground stability improvements
- Fixed the issue where Playground didn't properly create agent instances in `getActionSpace` when using agentFactory mode, ensuring normal operation across various usage modes
- Optimized Playground output display to prevent overly long reportHTML content from affecting the interface

### Model configuration updates

Updated configuration parameters for Qwen model's deep thinking functionality to ensure compatibility with the latest model version.

## v1.1 - `aiAct` deep thinking and extensible MCP SDK

v1.1 optimizes model planning capabilities and MCP extensibility, making automation more stable in complex scenarios while providing more flexible solutions for enterprise MCP service deployments.

### `aiAct` can enable deep thinking (deepThink)

When deep thinking is enabled in `aiAct`, the model will interpret intent more thoroughly and optimize its planning results. This is suited for complex forms, multi-step flows, and similar scenarios. It improves accuracy but increases planning latency.

Currently supported: Qwen3-vl on Alibaba Cloud and Doubao-vision on Volcano Engine. See [Model strategy](./model-strategy) for details.

Example usage:

```typescript
await agent.aiAct('If the UI shows an "Add shipping address" button, expand the existing "Shipping address" list and select the last item', { deepThink: true });
```

### MCP extension and SDK exposure

Developers can use the MCP SDK exposed by Midscene to flexibly deploy a public MCP service. This capability applies to Agent instances on any platform.

Typical application scenarios:
- Run MCP in enterprise intranet to control private device pools
- Package Midscene capabilities as internal microservices for multiple teams
- Extend custom automation toolchains

See documentation: [MCP Services](./mcp)

### Chrome extension improvements
- Fixed potential event loss during recording, improving recording stability
- Optimized coordinate passing in `describeElement` for better element description accuracy

### CLI and configuration enhancements
- **File parameter support**: Fixed CLI issue where `--files` parameter wasn't properly handled when `--config` was specified; now they can be flexibly combined
- **Dynamic configuration**: Fixed Playground not reading the `MIDSCENE_REPLANNING_CYCLE_LIMIT` environment variable properly

### iOS Agent compatibility improvements
- Optimized `getWindowSize` method to automatically fall back to legacy endpoint when newer API is unavailable, improving compatibility with WebDriverAgent versions

### Report and Playground improvements
- Fixed issue where report wasn't properly initialized before accessing screen properties
- Fixed abnormal behavior of stop function in Playground
- Improved error handling during video export to avoid crashes caused by frame cancel

Thanks to contributors: @FriedRiceNoodles

## v1.0 - Midscene v1.0 is here!

Midscene v1.0 is here! Try it out today and see how it can help you automate your workflows.

### See our new [Showcases](./showcases) page for real-world examples

Register the GitHub form autonomously in a web browser and pass all field validations:

<video src="https://lf3-static.bytednsdoc.com/obj/eden-cn/nupipfups/Midscene/1.0-showcases/github2.mp4" height="300" controls />

Plus these real-world showcases:
* [iOS Automation - Meituan coffee order](./showcases#ios)
* [iOS Automation - Auto-like the first @midscene_ai tweet](./showcases#ios)
* [Android Automation - DCar: Xiaomi SU7 specs](./showcases#android)
* [Android Automation - Booking a hotel for Christmas](./showcases#android)
* [MCP Integration - Midscene MCP UI prepatch release](./showcases#mcp)

Some community developers have successfully built on Midscene's capability to [integrate with any interface](./integrate-with-any-interface), extending it with a robotic arm plus vision and voice models for in-vehicle large-screen testing scenarios. See the video below.

<video src="https://lf3-static.bytednsdoc.com/obj/eden-cn/vhaeh7vhabf/AI_Vision_Powered_Robotic_Arm.mp4" height="300" controls />

### 🚀 Pure vision path

Starting in v1.0, Midscene fully adopts a visual-understanding approach to deliver more stable UI automation. Visual models provide:

- **Stable results**: Leading vision models (Doubao Seed 1.6, Qwen3-VL, etc.) are reliable enough for most production needs
- **UI workflow planning**: Vision models generally excel at planning UI flows and can handle many complex task sequences
- **Works everywhere**: Automation no longer depends on the rendering stack—Android, iOS, desktop apps, or a browser `<canvas>`: if you can capture a screenshot, Midscene can interact with it
- **Developer-friendly**: Dropping selectors and DOM keeps prompts simpler; teammates unfamiliar with rendering tech can become productive quickly
- **Far fewer tokens**: Removing DOM extraction cuts token usage by about 80%, lowering cost and speeding up local runs
- **Open-source options**: Open-source vision models like Qwen3-VL 8B/30B perform well, enabling self-hosted deployments

See our updated [Model strategy](./model-strategy) for details.

### 🚀 Better results by multiple model combinations

Beyond the default interaction intent, Midscene defines Planning and Insight intents, and developers can enable dedicated models for each. For example, you can use a GPT model for planning while using the default Qwen3-VL model for element localization.

Multi-model combinations let you scale up handling of complex requirements as needed.

### 🚀 Runtime architecture optimization

* Reuse portions of context to reduce device-info calls and improve runtime performance
* Optimize Action Space combinations for Web and mobile environments to provide models with a more useful toolset

### 🚀 Report improvements

* Parameter view: mark interaction parameter locations, merge screenshot context, and identify model planning results faster
* Style updates: dark mode report rendering for better readability
* Token usage display: summarize token consumption by model to analyze cost across scenarios

### 🚀 MCP architecture redesign

We redefined Midscene MCP services around vision-driven UI operations. Each Action in the iOS / Android / Web Action Space is exposed as an MCP tool, providing atomic operations so developers can focus on higher-level Agents without worrying about UI operation details, while keeping success rates high. See [MCP architecture](./mcp).

### 🚀 Mobile automation

#### iOS improvements
- Added compatibility across WebDriverAgent 5.x–7.x
- Added WebDriver Clear API support to solve dynamic input field issues
- Improved device compatibility

#### Android improvements
- Added screenshot polling fallback to improve remote device stability
- Added automatic screen-orientation adaptation (displayId screenshots)
- Added YAML script support for `runAdbShell`

#### Cross-platform
- Expose system operation helpers on the Agent instance, including Home, Back, RecentApp, and more

### 🚧 API changes

Method renames (backward compatible)
- Renamed `aiAction()` → `aiAct()` (old method kept with deprecation warning)
- Renamed `logScreenshot()` → `recordToReport()` (old method kept with deprecation warning)

Environment variable renames (backward compatible)
- Renamed `OPENAI_API_KEY` → `MIDSCENE_MODEL_API_KEY` (new variable preferred, old variable as fallback)
- Renamed `OPENAI_BASE_URL` → `MIDSCENE_MODEL_BASE_URL` (new variable preferred, old variable as fallback)

### ⬆️ Upgrade to the latest version

Upgrade only the packages you use. For example:

`npm install @midscene/web@latest`
`npm install @midscene/android@latest`
`npm install @midscene/ios@latest`

If you use the CLI installed globally:

`npm i -g @midscene/cli`

## v0.30 - Cache management upgrade and mobile experience optimization

### More flexible cache strategy

v0.30 improves the cache system, allowing you to control cache behavior based on actual needs:

- **Multiple cache modes available**: Supports read-only, write-only, and read-write strategies. For example, use read-only mode in CI environments to reuse cache, and use write-only mode in local development to update cache
- **Automatic cleanup of unused cache**: Agent can automatically clean up unused cache records when destroyed, preventing cache files from accumulating
- **Simplified unified configuration**: Cache configuration parameters for CLI and Agent are now unified, no need to remember different configurations

### Report management convenience

- **Support for merging multiple reports**: In addition to playwright scenarios, all scenarios now support merging multiple automation execution reports into a single file for centralized viewing and sharing of test results

### Mobile automation optimization

#### iOS platform improvements
- **Real device support improvement**: Removed simctl check restriction, making iOS real device automation smoother
- **Auto-adapt device display**: Implemented automatic device pixel ratio detection, ensuring accurate element positioning on different iOS devices

#### Android platform enhancements
- **Flexible screenshot optimization**: Added `screenshotResizeRatio` option, allowing you to customize screenshot size while ensuring visual recognition accuracy, reducing network transmission and storage overhead
- **Screen info cache control**: Use `alwaysRefreshScreenInfo` option to control whether to fetch screen information each time, allowing cache reuse in stable environments to improve performance
- **Direct ADB command execution**: AndroidAgent added `runAdbCommand` method for convenient execution of custom device control commands

#### Cross-platform consistency
- **ClearInput support on all platforms**: Solves the problem of AI being unable to accurately plan clear input operations across platforms

### Feature enhancements

- **Failure classification**: CLI execution results can now distinguish between "skipped failures" and "actual failures", helping locate issue causes
- **aiInput append mode**: Added `append` option to append input while preserving existing content, suitable for editing scenarios
- **Chrome extension improvements**:
  - Popup mode preference saved to localStorage, remembering your choice on next open
  - Bridge mode supports auto-connect, reducing manual operations
  - Support for GPT-4o and non-visual language models

### Type safety improvements

- **Zod schema validation**: Introduced type checking for action parameters, detecting parameter errors during development to avoid runtime issues
- **Number type support**: Fixed `aiInput` support for number type values, making type handling more robust

### Bug fixes

- Fixed potential issues caused by Playwright circular dependencies
- Fixed issue where `aiWaitFor` as the first statement could not generate reports
- Improved video recorder delay logic to ensure the last frame is captured
- Optimized report display logic to view both error information and element positioning information simultaneously
- Fixed issue where `cacheable` option in `aiAction` subtasks was not properly passed

### Community

- Awesome Midscene section added [midscene-java](./awesome-midscene.md) community project

## v0.29 - iOS platform support added

### iOS platform support added
The biggest highlight of v0.29 is the official introduction of iOS platform support! Now you can connect and automate iOS devices through WebDriver, extending Midscene's powerful AI automation capabilities to the Apple ecosystem, details: [Support iOS automation](./ios-introduction).

### Qwen3-VL model adaptation

We've adapted the latest Qwen `Qwen3-VL` model, giving developers faster and more accurate visual understanding capabilities. See [Model strategy](./model-strategy).

### AI core capability enhancement

- **UI-TARS Model Performance Optimization**: Optimized aiAct planning, improved dialogue history management, and provided better context awareness capabilities
- **AI Assertion and Action Optimization**: We updated the prompt for `aiAssert` and optimized the internal implementation of `aiAct`, making AI-driven assertions and action execution more precise and reliable

### Reporting and debugging experience optimization
- **URL Parameter Playback Control**: To improve debugging experience, you can now directly control the default behavior of report playback through URL parameters

### Documentation
- Updated documentation deployment cache strategy to ensure users can access the latest documentation content in time

## v0.28 - Build your own GUI automation agent by integrating with your own interface (preview feature)

### Support for integration with any interface (preview feature)

v0.28 introduces the capability to integrate with your own interface. Define an interface controller class that conforms to the `AbstractInterface` definition, and you can get a fully-featured Midscene Agent.

The typical use case for this feature is to build a GUI automation Agent for your own interface, such as IoT devices, in-house applications, car displays, etc.!

Combined with the universal Playground architecture and SDK enhancement features, developers can conveniently debug custom devices.

For more information, please refer to [Integrate with Any Interface (Preview Feature)](./integrate-with-any-interface.mdx)

### Android platform optimization
- **Planning Cache Support**: Added planning cache functionality for Android platform, improving execution efficiency
- **Input Strategy Enhancement**: Optimized input clearing strategy based on IME settings, improving Android platform input experience
- **Scroll Calculation Improvement**: Optimized scroll endpoint calculation algorithm for Android platform

### Gesture operation extension
- **Double-Click Operation Support**: Added support for double-click actions
- **Long Press and Swipe Gestures**: Added support for long press and swipe gestures

### Core function enhancement
- **Agent Configuration Isolation**: Implemented model configuration isolation between different agents, avoiding configuration conflicts
- **Execution Option Extension**: Added useCache and replanningCycleLimit configuration options for Agent, providing more fine-grained control
- **YAML Script Support**: Support for running universal custom devices through YAML scripts, enhancing automation capabilities

### Bug fixes
- Fixed Qwen model search region size issues
- Optimized deepThink parameter handling and rectangle size calculation
- Resolved issues related to Playwright double-click operations
- Improved TEXT action type processing logic

### Documentation and community
- Added custom interface documentation to help developers better extend functionality
- Added [Awesome Midscene](./awesome-midscene.md) section in README to showcase community projects

## v0.27 - Core module refactoring, assertions and reports functionally enhanced

### Core module refactoring

Based on the introduction of [Rslib](https://github.com/web-infra-dev/rslib) in v0.26 to improve development experience and reduce contribution thresholds, v0.27 takes it a step further by refactoring the core modules on a large scale. This makes it extremely easy to extend new devices and add new AI operations, and we sincerely welcome community developers to contribute!

**Due to the wide scope of this refactoring, please feel free to report any issues you encounter after upgrading, and we will address them promptly.**

### API enhancement

- **`aiAssert` Functionally Enhanced**
  - New `name` field allows naming different assertion tasks, making it easier to identify and parse in JSON output results
  - New `domIncluded` and `screenshotIncluded` options allow flexible control over whether to send DOM snapshots and page screenshots to AI

### Chrome extension playground upgrade

- All Agent APIs can be directly debugged and run in the Playground! Interactive, extraction, and verification cover three major categories of methods, with visual operations and verification that boost your automation development efficiency! Come experience the truly versatile AI automation platform! 🚀

### Report function optimization
- **New Marking Layer Switch**: The report player has added a switch to hide the marking layer, allowing users to view the original page view without obstruction when playing back.

### Bug fixes
- Fixed the problem that `aiWaitFor` sometimes caused the report to not be generated
- Reduced memory consumption of Playwright plugin


## v0.26 - Toolchain fully integrated [Rslib](https://github.com/web-infra-dev/rslib), greatly improving development experience and reducing contribution threshold

### Web integration optimization
- Support freezing page context([freezePageContext](./api.mdx#agentfreezepagecontext)/[unfreezePageContext](./api.mdx#agentunfreezepagecontext)), so that all subsequent operations reuse the same page snapshot, avoiding repeated page status acquisition
- Add all agent APIs to Playwright fixture, simplify test script writing, and solve the problem of not generating reports when using agentForPage

### Android automation enhancement
- New keyboard hiding strategy([keyboardDismissStrategy](./android-getting-started#androiddevice-constructor)), allowing you to specify the way to automatically hide the keyboard

### Report function optimization
- Report content lazy parsing, solving the problem of report crash when the report is large
- Report player adds automatic zoom switch, making it easier to view the global perspective playback
- Support aiAssert / aiQuery tasks in report playback, to fully show the entire page change process
- Fix the problem that the sidebar status is not displayed as a failure icon when the assertion fails
- Fix the problem that the drop-down filter in the report cannot be switched

### Build and engineering
- Build tool migration to [Rslib](https://github.com/web-infra-dev/rslib) library development tool, improving build efficiency and development experience
- Full repository source code jump, making it easier for developers to view source code
- MCP npm package product volume optimization, from 56M to 30M, greatly improving loading speed

### Bug fixes
- CLI automatically opens headed mode when keepWindow is true
- Fix the implementation problem of getGlobalConfig, solve the problem of abnormal environment variable initialization
- Ensure that the mime-type in base64 encoding is correct
- Fix the return value type of aiAssert task

## v0.25 - Support using images as AI prompt input

### Core function enhancement
- New worker runtime support, support running in worker environment
- Support using images as AI prompt input, see [Prompting with images](./api.mdx#prompting-with-images)
- Image processing upgrade, using Photon & Sharp for efficient image cropping

### Web integration optimization
- Get XPath by coordinates, improve cache reproducibility
- Cache file moves plan module to the front, improving readability
- Chrome Recorder supports exporting all events to markdown documents
- agent supports specifying HTML report name, see [reportFileName](./api.mdx)

### Android automation enhancement
- Long press gesture support
- Pull-to-refresh support

### Bug fixes
- Use global config to handle environment variables, avoid issues caused by multiple packaging
- Manually construct error information when error object serialization fails
- Fix playwright report type dependency declaration order issue
- Fix MCP packaging issue

### Documentation AI-friendly
- [LLMs.txt](./llm-txt.mdx) is now available in both Chinese and English, making it easier for AI to understand
- Each document now has a copy-to-markdown button, making it easier to feed to AI

### Other function enhancement
- Chrome Recorder supports aiScroll function
- Refactor aiAssert to be consistent with aiBoolean

## v0.24 - MCP for Android automation

### MCP for Android automation
- You can now use Midscene MCP to automate Android apps, just like you use it for web apps. Read more: [MCP for Android Automation](./mcp#android-device-management)

### Optimization
- For Mac platform Puppeteer, a double input clearing mechanism has been added to ensure that the input box is cleared before input

### Development experience
- Simplify the way to build `htmlElement.js` to avoid report template build issues caused by circular dependencies
- Optimize development workflow, just use `npm run dev` to enter midscene project development


## v0.23 - New report style and YAML script ability enhancement

### Report system upgrade
#### New report style
- New report style design, providing clearer and more beautiful test result display
- Optimize report layout and visual effects, improve user reading experience
- Enhance report readability and information hierarchy structure

![](https://lf3-static.bytednsdoc.com/obj/eden-cn/nupipfups/Midscene/new%20report.png)

### YAML script ability enhancement

#### Support multiple YAML files batch execution
- New config mode support, support configure Yaml file running order, browser reuse strategy, parallelism
- Support getting JSON format running results

![](https://lf3-static.bytednsdoc.com/obj/eden-cn/nupipfups/Midscene/Tuji_20250722_161353.338.png)

### Test coverage enhancement

#### Android test enhancement
- New Android platform related test cases, improve code quality and stability
- Improve test coverage, ensure the reliability of Android features

## v0.22- Chrome extension recording function released

### Web integration enhancement

#### New recording function
- Chrome extension adds recording function, which can record user operations on the page and generate automation scripts
- Support recording click, input, scroll and other common operations, greatly reducing the threshold for writing automation scripts
- The recorded operations can be directly played back and debugged in the Playground

#### Upgrade to IndexedDB for storage
- Chrome extension's Playground and Bridge now use IndexedDB for data storage
- Compared to the previous storage scheme, it provides larger storage capacity and better performance
- Support storing more complex data structures, laying the foundation for future feature extensions

#### Customize replanning cycle limit
- Set the `MIDSCENE_REPLANNING_CYCLE_LIMIT` environment variable to customize the maximum number of re-planning cycles allowed when executing operations (aiAct).
- The default value is 10. When the AI needs to re-plan more than this limit, an error will be thrown and suggest splitting the task.
- Provide more flexible task execution control, adapting to different automation scenarios

```bash
export MIDSCENE_REPLANNING_CYCLE_LIMIT=10 # default value is 10
```

### Android interaction optimization

#### New screenshot path generation
- Generate a unique file path for each screenshot to avoid file overwrite issues
- Improve stability in concurrent test scenarios

## v0.21 - Chrome extension UI upgrade

### Web integration enhancement

#### New chat-style user interface

- New chat-style user interface design for better user experience

<video src="https://lf3-static.bytednsdoc.com/obj/eden-cn/nupipfups/Midscene/recording_2025-07-07_08-16-16.mp4" controls/>

#### Flexible timeout configuration

- Supports overriding timeout settings from test fixture, providing more flexible timeout control
- Applicable scenarios: Different test cases require different timeout settings

#### Unified Puppeteer and Playwright configuration

- New `waitForNavigationTimeout` and `waitForNetworkIdleTimeout` parameters for Playwright
- Unified timeout options configuration for Puppeteer and Playwright, providing consistent API experience, reducing learning costs

#### New data export callback mechanism

- New `agent.onDumpUpdate` callback function, can get real-time notification when data is exported
- Refactored the post-task processing flow to ensure the correct execution of asynchronous operations
- Applicable scenarios: Monitoring or processing exported data

### Android interaction optimization

#### Input experience improvement

- Changed click input to slide operation, improving interaction response and stability
- Reduced operation failures caused by inaccurate clicks

## v0.20 - Support for assigning XPath to locate elements

### Web integration enhancement

#### New `aiAsk` method

- Allows direct questioning of the AI model to obtain string-formatted answers for the current page.
- Applicable scenarios: Tasks requiring AI reasoning such as Q&A on page content and information extraction.
- Example:

```typescript
await agent.aiAsk('any question')
```

#### Support for passing XPath to locate elements

- Location priority: Specified XPath > Cache > AI model location.
- Applicable scenarios: When the XPath of an element is known and the AI model location needs to be skipped.
- Example:

```typescript
await agent.aiTap('submit button', { xpath: '//button[@id="submit"]' })
```

### Android improvement

#### Playground tasks can be cancelled

- Supports interrupting ongoing automation tasks to improve debugging efficiency.

#### Enhanced `aiLocate` API

- Returns the Device Pixel Ratio, which is commonly used to calculate the real coordinates of elements.

### Report generation optimization

Improve report generation mechanism, from batch storage to single append, effectively reducing memory usage and avoiding memory overflow when the number of test cases is large.


## v0.19 - Support for getting complete execution process data

### New API for getting Midscene execution process data

Add the `_unstableLogContent` API to the agent. Get the execution process data of Midscene, including the time of each step, the AI tokens consumed, and the screenshot.

The report is generated based on this data, which means you can customize your own report using this data.

Read more: [API documentation](./api.mdx#agent_unstablelogcontent)

### CLI support for adjusting Midscene env variable priority

By default, `dotenv` does not override the global environment variables in the `.env` file. If you want to override, you can use the `--dotenv-override` option.

Read more: [Use YAML-based Automation Scripts](./automate-with-scripts-in-yaml.mdx#use-env-file-to-override-global-environment-variables)

### Reduce report file size

Reduce the size of the generated report by trimming redundant data, significantly reducing the report file size for complex pages. The typical report file size for complex pages has been reduced from 47.6M to 15.6M!

## v0.18 - Enhanced reporting features

🚀 Midscene has another update! It makes your testing and automation processes even more powerful:

### Custom node in report

* Add the `recordToReport` API to the agent. Take a screenshot of the current page as a report node, and support setting the node title and description to make the automated testing process more intuitive. Applicable for capturing screenshots of key steps, error status capture, UI validation, etc.

![](/blog/recordToReport-api.png)

* Example:

```javascript
test('login github', async ({ ai, aiAssert, aiInput, recordToReport }) => {
  if (CACHE_TIME_OUT) {
    test.setTimeout(200 * 1000);
  }
  await ai('Click the "Sign in" button');
  await aiInput('quanru', 'username');
  await aiInput('123456', 'password');

  // log by your own
  await recordToReport('Login page', {
    content: 'Username is quanru, password is 123456',
  });

  await ai('Click the "Sign in" button');
  await aiAssert('Login success');
});
```

### Support for downloading reports as videos

* Support direct video download from the report player, just by clicking the download button on the player interface.

![](/blog/export-video.png)

* Applicable scenarios: Share test results, archive reproduction steps, and demonstrate problem reproduction.

### More Android configurations exposed

* Optimize input interactions in Android apps and allow connecting to remote Android devices

  * `autoDismissKeyboard?: boolean` - Optional parameter. Whether to automatically dismiss the keyboard after entering text. The default value is true.

  * `androidAdbPath?: string` - Optional parameter. Used to specify the path of the adb executable file.

  * `remoteAdbHost?: string` - Optional parameter. Used to specify the remote adb host.

  * `remoteAdbPort?: number` - Optional parameter. Used to specify the remote adb port.

* Examples:

```typescript
await agent.aiInput('Search Box', 'Test Content', { autoDismissKeyboard: true })
```

```typescript
const agent = await agentFromAdbDevice('s4ey59', {
    autoDismissKeyboard: false, // Optional parameter. Whether to automatically dismiss the keyboard after entering text. The default value is true.
    androidAdbPath: '/usr/bin/adb', // Optional parameter. Used to specify the path of the adb executable file.
    remoteAdbHost: '192.168.10.1', // Optional parameter. Used to specify the remote adb host.
    remoteAdbPort: '5037' // Optional parameter. Used to specify the remote adb port.
})
```

Upgrade now to experience these powerful new features!

* [Custom Report Node API documentation](/en/api.mdx#log-screenshot)

* [API documentation for more Android configuration items](/en/android-getting-started.mdx#androiddevice-constructor)


## v0.17 - Let AI see the DOM of the page

### Data query API enhanced

To meet more automation and data extraction scenarios, the following APIs have been enhanced with the `options` parameter, supporting more flexible DOM information and screenshots:

- `agent.aiQuery(dataDemand, options)`
- `agent.aiBoolean(prompt, options)`
- `agent.aiNumber(prompt, options)`
- `agent.aiString(prompt, options)`

#### New `options` parameter

- `domIncluded`: Whether to pass the simplified DOM information to AI model, default is off. This is useful for extracting attributes that are not visible on the page, like image links.
- `screenshotIncluded`: Whether to pass the screenshot to AI model, default is on.

#### Code example

```typescript
// Extract all contact information (including hidden avatarUrl attributes)
const contactsData = await agent.aiQuery(
  "{name: string, id: number, company: string, department: string, avatarUrl: string}[], extract all contact information including hidden avatarUrl attributes",
  { domIncluded: true }
);

// Check if the id attribute of the first contact is 1
const isId1 = await agent.aiBoolean(
  "Is the first contact's id is 1?",
  { domIncluded: true }
);

// Get the ID of the first contact (hidden attribute)
const firstContactId = await agent.aiNumber("First contact's id?", { domIncluded: true });

// Get the avatar URL of the first contact (invisible attribute on the page)
const avatarUrl = await agent.aiString(
  "What is the Avatar URL of the first contact?",
  { domIncluded: true }
);
```

### New right-click ability

Have you ever encountered a scenario where you need to automate a right-click operation? Now, Midscene supports a new `agent.aiRightClick()` method!

#### Function

Perform a right-click operation on the specified element, suitable for scenarios where right-click events are customized on web pages. Please note that Midscene cannot interact with the browser's native context menu after right-click.

#### Parameter description

- `locate`: Describe the element you want to operate in natural language
- `options`: Optional, supports `deepThink` (AI fine-grained positioning) and `cacheable` (result caching)

#### Example

```typescript
// Right-click on a contact in the contacts application, triggering a custom context menu
await agent.aiRightClick("Alice Johnson");

// Then you can click on the options in the menu
await agent.aiTap("Copy Info"); // Copy contact information to the clipboard
```

### A complete example

In this report file, we show a complete example of using the new `aiRightClick` API and new query parameters to extract contact data including hidden attributes.

Report file: [puppeteer-2025-06-04_20-34-48-zyh4ry4e.html](https://lf3-static.bytednsdoc.com/obj/eden-cn/nupipfups/Midscene/puppeteer-2025-06-04_20-34-48-zyh4ry4e.html)

The corresponding code can be found in our example repository: [puppeteer-demo/extract-data.ts](https://github.com/web-infra-dev/midscene-example/blob/main/puppeteer-demo/extract-data.ts)


### Refactor cache

Use xpath cache instead of coordinates, improve cache hit rate.

Refactor cache file format from json to yaml, improve readability.

## v0.16 - Support MCP

### Midscene MCP

🤖 Use Cursor / Trae to help write test cases.
🕹️ Quickly implement browser operations akin to the Manus platform.
🔧 Integrate Midscene capabilities swiftly into your platforms and tools.

<video src="https://lf3-static.bytednsdoc.com/obj/eden-cn/ozpmyhn_lm_hymuPild/ljhwZthlaukjlkulzlp/midscene/en-midscene-mcp-Sauce-Demo.mp4" controls/>

Read more: [MCP](./mcp#choosing-your-environment)

### Support structured API for agent

APIs: `aiBoolean`, `aiNumber`, `aiString`, `aiLocate`

Read more: [Use JavaScript to Optimize the AI Automation Code](./use-javascript-to-optimize-ai-automation-code.md)

## v0.15 - Android automation unlocked!

### Android automation unlocked!

🤖 AI Playground: natural‑language debugging
📱 Supports native, Lynx & WebView apps
🔁 Replayable runs
🛠️ YAML or JS SDK
⚡ Auto‑planning & Instant Actions APIs

Read more: [Android automation](./android-introduction)

### More features

* Allow custom midscene_run dir
* Enhance report filename generation with unique identifiers and support split mode
* Enhance timeout configurations and logging for network idle and navigation
* Adapt for gemini-2.5-pro

## v0.14 - Instant actions

"Instant Actions" introduces new atomic APIs, enhancing the accuracy of AI operations.

Read more: [Instant Actions](./blog-introducing-instant-actions-and-deep-think.md)

## v0.13 - DeepThink mode

### Atomic AI interaction methods

* Supports aiTap, aiInput, aiHover, aiScroll, and aiKeyboardPress for precise AI actions.

### DeepThink mode

* Enhances click accuracy with deeper contextual understanding.

![](/blog/0.13.jpeg)

## v0.12 - Integrate Qwen 2.5 VL

### Integrate Qwen 2.5 VL's native capabilities

* Keeps output accuracy.
* Supports more element interactions.
* Cuts operating cost by over 80%.

## v0.11.0 - UI-TARS model caching

### UI-TARS model support caching

* Enable caching by document 👉 : [Enable Caching](./caching.mdx)

* Enable effect

<video src="https://lf3-static.bytednsdoc.com/obj/eden-cn/nupipfups/Midscene/antd-form-cache.mp4" controls/>

![](/blog/0.11.0.png)

### Optimize DOM tree extraction strategy

* Optimize the information ability of the dom tree, accelerate the inference process of models like GPT 4o

![](/blog/0.11.0-2.png)


## v0.10.0 - UI-TARS model released

UI-TARS is a Native GUI agent model released by the **Seed** team. It is named after the [TARS robot](https://interstellarfilm.fandom.com/wiki/TARS) in the movie [Star Trek](https://en.wikipedia.org/wiki/Star_Trek), which has high intelligence and autonomous thinking capabilities. UI-TARS **takes images and human instructions as input information**, can correctly perceive the next action, and gradually approach the goal of human instructions, leading to the best performance in various benchmark tests of GUI automation tasks compared to open-source and closed-source commercial models.

![](/blog/0.10.0.png)

UI-TARS: Pioneering Automated GUI Interaction with Native Agents - Figure 1

![](/blog/0.10.0-2.png)

UI-TARS: Pioneering Automated GUI Interaction with Native - Figure 4

### Model advantage

UI-TARS has the following advantages in GUI tasks:

* **Target-driven**

* **Fast inference speed**

* **Native GUI agent model**

* **Private deployment without data security issues**


## v0.9.0 - Bridge mode released

With the Midscene browser extension, you can now use scripts to link with the desktop browser for automated operations!

We call it "Bridge Mode".

Compared to previous CI environment debugging, the advantages are:

1. You can reuse the desktop browser, especially Cookie, login state, and front-end interface state, and start automation without worrying about environment setup.

2. Support manual and script cooperation to improve the flexibility of automation tools.

3. Simple business regression, just run it locally with Bridge Mode.

![](/blog/0.9.0.png)

Documentation: [Use Chrome Extension to Experience Midscene](./bridge-mode.mdx)


## v0.8.0 - Chrome extension

### New Chrome extension, run Midscene anywhere

Through the Midscene browser extension, you can run Midscene on any page, without writing any code.

Experience it now 👉: [Use Chrome Extension to Experience Midscene](./quick-experience.mdx)

<video src="https://lf3-static.bytednsdoc.com/obj/eden-cn/nupipfups/Midscene/Midscene_extension.mov" controls/>



## v0.7.0 - Playground ability

### Playground ability, debug anytime

Now you don't have to keep re-running scripts to debug prompts!

On the new test report page, you can debug the AI execution results at any time, including page operations, page information extraction, and page assertions.

<video src="https://lf3-static.bytednsdoc.com/obj/eden-cn/nupipfups/Midscene/midscene-playground.mov" controls/>


## v0.6.0 - Doubao model support

### Doubao model support

* Support for calling Doubao models, reference the environment variables below to experience.

```bash
MIDSCENE_OPENAI_INIT_CONFIG_JSON='{"baseURL":"https://xxx.net/api/v3","apiKey":"xxx"}'
MIDSCENE_MODEL_NAME='ep-20240925111815-mpfz8'
MIDSCENE_MODEL_TEXT_ONLY='true'
```

Summarize the availability of Doubao models:

* Currently, Doubao only has pure text models, which means "seeing" is not available. In scenarios where pure text is used for reasoning, it performs well.

* If the use case requires combining UI analysis, it is completely unusable


Example:

✅ The price of a multi-meat grape (can be guessed from the order of the text on the interface)

✅ The language switch text button (can be guessed from the text content on the interface: Chinese, English text)

❌ The left-bottom play button (requires image understanding, failed)

### Support for GPT-4o structured output, cost reduction

By using the gpt-4o-2024-08-06 model, Midscene now supports structured output (structured-output) features, ensuring enhanced stability and reduced costs by 40%+.

Midscene now supports hitting GPT-4o prompt caching features, and the cost of AI calls will continue to decrease as the company's GPT platform is deployed.

### Test report: support animation playback

Now you can view the animation playback of each step in the test report, quickly debug your running script

<video src="https://lf3-static.bytednsdoc.com/obj/eden-cn/nupipfups/Midscene/midscene-play-all.mp4" controls/>

### Speed up: merge plan and locate operations, response speed increased by 30%

In the new version, we have merged the Plan and Locate operations in the prompt execution to a certain extent, which increases the response speed of AI by 30%.

> Before

![](/blog/0.6.0.png)

> after

![](/blog/0.6.0-2.png)

### Test report: the accuracy of different models

* GPT 4o series models, 100% correct rate

* doubao-pro-4k pure text model, approaching usable state

![](/blog/0.6.0-3.png)

![](/blog/0.6.0-4.png)

### Problem fix

* Optimize the page information extraction to avoid collecting obscured elements, improving success rate, speed, and AI call cost 🚀

> before

![](/blog/0.6.0-5.png)

> after

![](/blog/0.6.0-6.png)


## v0.5.0 - Support GPT-4o structured output

### New features

* Support for gpt-4o-2024-08-06 model to provide 100% JSON format limit, reducing Midscene task planning hallucination behavior

![](/blog/0.5.0.png)

* Support for Playwright AI behavior real-time visualization, improve the efficiency of troubleshooting

![](/blog/0.5.0-2.png)

* Cache generalization, cache capabilities are no longer limited to playwright, pagepass, puppeteer can also use cache

```diff
- playwright test --config=playwright.config.ts
# Enable cache
+ MIDSCENE_CACHE=true playwright test --config=playwright.config.ts
```

* Support for azure openAI

* Support for AI to add, delete, and modify the existing input

### Problem fix

* Optimize the page information extraction to avoid collecting obscured elements, improving success rate, speed, and AI call cost 🚀

* During the AI interaction process, unnecessary attribute fields were trimmed, reducing token consumption.

* Optimize the AI interaction process to reduce the likelihood of hallucination in KeyboardPress and Input events

* For pagepass, provide an optimization solution for the flickering behavior that occurs during the execution of Midscene

```javascript
// Currently, pagepass relies on a too low version of puppeteer, which may cause the interface to flicker and the cursor to be lost. The following solution can be used to solve this problem
const originScreenshot = puppeteerPage.screenshot;
puppeteerPage.screenshot = async (options) => {
  return await originScreenshot.call(puppeteerPage, {
    ...options,
    captureBeyondViewport: false
  });
};
```

## v0.4.0 - Support CLI usage

### New features

* Support for Cli usage, reducing the usage threshold of Midscene

```bash
# Headed mode (visible browser) access baidu.com and search "weather"
npx @midscene/cli --headed --url https://www.baidu.com --action "input 'weather', press enter" --sleep 3000

# Visit GitHub status page and save the status to ./status.json
npx @midscene/cli --url https://www.githubstatus.com/ \
  --query-output status.json \
  --query '{serviceName: string, status: string}[], github page status, return service name'
```

* Support for AI to wait for a certain time to continue the subsequent task execution

* Playwright AI task report shows the overall time and aggregates AI tasks by test group

### Problem fix

* Optimize the AI interaction process to reduce the likelihood of hallucination in KeyboardPress and Input events


## v0.3.0 - Support AI report HTML

### New features

* Generate html format AI report, aggregate AI tasks by test group, facilitate test report distribution

### Problem fix

* Fix the problem of AI report scrolling preview

## v0.2.0 - Control Puppeteer by natural language

### New features

* Support for using natural language to control puppeteer to implement page automation 🗣️💻

* Provide AI cache capabilities for playwright framework, improve stability and execution efficiency

* AI report visualization, aggregate AI tasks by test group, facilitate test report distribution

* Support for AI to assert the page, let AI judge whether the page meets certain conditions

## v0.1.0 - Control Playwright by natural language

### New features

* Support for using natural language to control puppeteer to implement page automation 🗣️💻

* Support for using natural language to extract page information 🔍🗂️

* AI report visualization, AI behavior, AI thinking visualization 🛠️👀

* Direct use of GPT-4o model, no training required 🤖🔧



---
url: /common/get-cdp-url.md
---

#### Getting a CDP WebSocket URL

You can get a CDP WebSocket URL from various sources, for example:

- **BrowserBase**: Sign up at https://browserbase.com and get your CDP URL
- **Browserless**: Use https://browserless.io or run your own instance
- **Local Chrome**: Run Chrome with `--remote-debugging-port=9222` and use `ws://localhost:9222/devtools/browser/...`
- **Docker**: Run Chrome in a Docker container with debugging port exposed



---
url: /common/prepare-android.md
---

## Preparation

### Install Node.js

Install [Node.js 18 or higher](https://nodejs.org/en/download/).

### Prepare API Key

Prepare an API Key for a Vision Language (VL) model.

You can check the models and configurations supported by Midscene.js in the [Model Strategy](../model-strategy) documentation.

### Install adb

`adb` is a command-line tool that allows you to communicate with Android devices. There are two ways to install `adb`:

- Method 1: Install using [Android Studio](https://developer.android.com/studio)
- Method 2: Install using [Android Command Line Tools](https://developer.android.com/studio#command-line-tools-only)

Verify that `adb` is installed successfully:

```bash
adb --version
```

When you see the following output, it means `adb` is installed successfully:

```log
Android Debug Bridge version 1.0.41
Version 34.0.4-10411341
Installed as /usr/local/bin//adb
Running on Darwin 24.3.0 (arm64) 
```

### Set the `ANDROID_HOME` environment variable

Refer to [Android Environment Variables](https://developer.android.com/tools/variables) to set the `ANDROID_HOME` environment variable.

Verify that the `ANDROID_HOME` variable is set successfully:

```bash
echo $ANDROID_HOME
```

When the above command has output, it means the `ANDROID_HOME` variable is set successfully:

```log
/Users/your_username/Library/Android/sdk
```

### Connect Android Device

In the developer options of your Android device, enable 'USB debugging'. If 'USB debugging (Security settings)' exists, enable it as well. Then connect your Android device using a USB cable.

<p align="center">
  <img src="/android-usb-debug-en.png" alt="android usb debug" width="400"/>
</p>

Verify the connection:

```bash
adb devices -l
```

When you see the following output, it means the connection is successful:

```log
List of devices attached
s4ey59	device usb:34603008X product:cezanne model:M2006J device:cezan transport_id:3
```



---
url: /common/prepare-ios.md
---

## Preparation

### Install Node.js

Install [Node.js 18 or higher](https://nodejs.org/en/download/).

### Prepare API Key

Prepare an API Key for a visual language (VL) model.

You can find supported models and configurations for Midscene.js in the [Model strategy](../model-strategy) documentation.

### Prepare WebDriver Server

Before getting started, you need to set up the iOS development environment:

- macOS (required for iOS development)
- Xcode and Xcode command line tools
- iOS Simulator or real device

#### Environment Configuration

Before using Midscene iOS, you need to prepare the WebDriverAgent service.

:::note Version Requirement

WebDriverAgent version must be **>= 7.0.0**

:::

Please refer to the official documentation for setup:

- **Simulator Configuration**: [Run Prebuilt WDA](https://appium.github.io/appium-xcuitest-driver/5.12/run-prebuilt-wda/)
- **Real Device Configuration**: [Real Device Configuration](https://appium.github.io/appium-xcuitest-driver/5.12/real-device-config/)

#### Verify Environment Configuration

After completing the configuration, you can verify whether the service is working properly by accessing WebDriverAgent's status endpoint:

**Access URL**: `http://localhost:8100/status`

**Correct Response Example**:
```json
{
  "value": {
    "build": {
      "version": "10.1.1",
      "time": "Sep 24 2025 18:56:41",
      "productBundleIdentifier": "com.facebook.WebDriverAgentRunner"
    },
    "os": {
      "testmanagerdVersion": 65535,
      "name": "iOS",
      "sdkVersion": "26.0",
      "version": "26.0"
    },
    "device": "iphone",
    "ios": {
      "ip": "10.91.115.63"
    },
    "message": "WebDriverAgent is ready to accept commands",
    "state": "success",
    "ready": true
  },
  "sessionId": "BCAD9603-F714-447C-A9E6-07D58267966B"
}
```

If you can successfully access this endpoint and receive a similar JSON response as shown above, it indicates that WebDriverAgent is properly configured and running.



---
url: /common/setup-env.md
---

## Set up API keys for model

Set your model configs into the environment variables. You may refer to [Model strategy](../model-strategy) for more details.

```bash
export MIDSCENE_MODEL_BASE_URL="https://replace-with-your-model-service-url/v1"
export MIDSCENE_MODEL_API_KEY="replace-with-your-api-key"
export MIDSCENE_MODEL_NAME="replace-with-your-model-name"
export MIDSCENE_MODEL_FAMILY="replace-with-your-model-family"
```

For more configuration details, please refer to [Model strategy](../model-strategy) and [Model configuration](../model-config).



---
url: /common/start-experience.md
---

After configuration, you can start using Midscene right away. It provides several key operation tabs:

- **Act**: interact with the page. This is Auto Planning, corresponding to `aiAct`. For example:
```
Type “Midscene” in the search box, run the search, and open the first result
```

```
Fill out the registration form and make sure every field passes validation
```

- **Query**: extract JSON data from the interface, corresponding to `aiQuery`.

Similar methods include `aiBoolean()`, `aiNumber()`, and `aiString()` for directly extracting booleans, numbers, and strings.

```
Extract the user ID from the page and return JSON data in the { id: string } structure
```

- **Assert**: understand the page and assert; if the condition is not met, throw an error, corresponding to `aiAssert`.

```
There is a login button on the page, with a user agreement link below it
```

- **Tap**: click on an element. This is Instant Action, corresponding to `aiTap`.
```
Click the login button
```

> For the difference between Auto Planning and Instant Action, see the [API](../api.mdx) document.



---
url: /common/troubleshooting-llm-connectivity.md
---

## Troubleshooting model service connectivity issues

If you want to troubleshoot connectivity issues, you can use the 'connectivity-test' folder in our example project: [https://github.com/web-infra-dev/midscene-example/tree/main/connectivity-test](https://github.com/web-infra-dev/midscene-example/tree/main/connectivity-test)

Put your `.env` file in the `connectivity-test` folder, and run the test with `npm i && npm run test`.



---
url: /computer-api-reference.md
---

# API Reference (PC Desktop)

This page documents the PC desktop-specific APIs provided by `@midscene/computer`.

For common APIs that work across all platforms, see [Common API Reference](./api).

## Agent Creation

`agentFromComputer(opts?): Promise<ComputerAgent>`

Create an agent for desktop automation.

**Parameters:**

```typescript
interface ComputerAgentOpt {
  // Agent options (inherited from AgentOpt)
  aiActionContext?: string;
  cache?: boolean;
  // ... other AgentOpt properties

  // Device options
  displayId?: string;
  customActions?: DeviceAction<any>[];
  headless?: boolean;
  xvfbResolution?: string;
}
```

- `displayId` (optional): Specify which display to control. Get available displays with `ComputerDevice.listDisplays()`.
- `customActions` (optional): Add custom actions to the device.
- `headless` (optional, Linux only): Set to `true` to start a virtual display via [Xvfb](https://www.x.org/releases/X11R7.6/doc/man/man1/Xvfb.1.xhtml), enabling desktop automation on headless Linux servers and CI environments without a physical display. Can also be set via the `MIDSCENE_COMPUTER_HEADLESS_LINUX=true` environment variable.
- `xvfbResolution` (optional): Resolution for the Xvfb virtual display. Defaults to `'1920x1080x24'`.

:::tip Example: Testing Electron Apps on Headless Linux CI
A complete demo of testing [Obsidian](https://obsidian.md/) (an Electron app) on headless Linux CI with `@midscene/computer`: [https://github.com/web-infra-dev/midscene-example/tree/main/computer/electron-demo](https://github.com/web-infra-dev/midscene-example/tree/main/computer/electron-demo)
:::

**Example:**

```typescript
import { agentFromComputer } from '@midscene/computer';

// Connect to primary display
const agent = await agentFromComputer({
  aiActionContext: 'You are automating a desktop application.',
});

// Connect to specific display
const displays = await ComputerDevice.listDisplays();
const agent2 = await agentFromComputer({
  displayId: displays[1].id,
});
```

## Device Management

`ComputerDevice.listDisplays(): Promise<DisplayInfo[]>`

List all available displays.

**Returns:**

```typescript
interface DisplayInfo {
  id: string;
  name: string;
  primary?: boolean;
}
```

**Example:**

```typescript
import { ComputerDevice } from '@midscene/computer';

const displays = await ComputerDevice.listDisplays();
console.log('Available displays:', displays);
// [
//   { id: '0', name: 'Built-in Display', primary: true },
//   { id: '1', name: 'External Display', primary: false }
// ]
```

`checkComputerEnvironment(): Promise<EnvironmentCheck>`

Check if the computer environment is properly configured.

**Returns:**

```typescript
interface EnvironmentCheck {
  available: boolean;
  error?: string;
  platform: string;
  displays: number;
}
```

**Example:**

```typescript
import { checkComputerEnvironment } from '@midscene/computer';

const env = await checkComputerEnvironment();
console.log('Environment check:', env);

if (!env.available) {
  console.error('Environment error:', env.error);
}
```

## ComputerAgent

The `ComputerAgent` class extends `PageAgent<ComputerDevice>` and inherits all common agent methods:

- `aiAct(action: string)`: Perform an action with AI
- `aiQuery(query: string)`: Extract information with AI
- `aiAssert(assertion: string)`: Assert a condition with AI
- `aiWaitFor(condition: string)`: Wait for a condition
- `aiLocate(description: string)`: Locate an element
- And more...

See [Common API Reference](./api) for details.

## Available Actions

The `ComputerDevice` supports the following actions:

### Mouse Actions

#### Tap (Click)

Single click at the target location.

```typescript
await agent.aiAct('click on the File menu');
await agent.aiAct('click at center of screen');
```

#### DoubleClick

Double-click at the target location.

```typescript
await agent.aiAct('double-click on the desktop icon');
```

#### RightClick

Right-click to open context menu.

```typescript
await agent.aiAct('right-click on the desktop');
await agent.aiAct('right-click on the file');
```

#### MouseMove

Move mouse to an element.

```typescript
await agent.aiAct('move mouse to the menu item');
```

#### DragAndDrop

Drag from one location and drop at another.

```typescript
await agent.aiAct('drag the file to the folder');
```

### Keyboard Actions

#### KeyboardPress

Press keyboard keys with optional modifiers.

**Supported keys:**
- Regular keys: `a-z`, `0-9`, `Enter`, `Escape`, `Space`, `Tab`, etc.
- Arrow keys: `ArrowUp`, `ArrowDown`, `ArrowLeft`, `ArrowRight`
- Function keys: `F1`-`F12`
- Modifiers: `Command`/`Cmd` (macOS), `Control`/`Ctrl`, `Alt`, `Shift`, `Win` (Windows)
- Media keys: `VolumeUp`, `VolumeDown`, `Mute`, etc.

**Examples:**

```typescript
// Simple key press
await agent.aiAct('press Enter');
await agent.aiAct('press Escape');

// Key combinations (platform-specific)
if (process.platform === 'darwin') {
  // macOS
  await agent.aiAct('press Cmd+Space');  // Open Spotlight
  await agent.aiAct('press Cmd+Tab');    // App switcher
  await agent.aiAct('press Cmd+C');      // Copy
  await agent.aiAct('press Cmd+V');      // Paste
} else {
  // Windows/Linux
  await agent.aiAct('press Windows key'); // Start menu
  await agent.aiAct('press Alt+Tab');     // App switcher
  await agent.aiAct('press Ctrl+C');      // Copy
  await agent.aiAct('press Ctrl+V');      // Paste
}

// Arrow keys
await agent.aiAct('press ArrowDown');
await agent.aiAct('press ArrowUp');

// Function keys
await agent.aiAct('press F5');  // Refresh
```

#### Input

Type text into an input field.

```typescript
await agent.aiAct('type "Hello World" in the search box');
await agent.aiAct('type "my-document.txt"');
```

#### ClearInput

Clear the content of an input field.

```typescript
await agent.aiAct('clear the text field');
```

### Scroll Actions

Scroll the screen or a specific area.

```typescript
// Scroll directions
await agent.aiAct('scroll down');
await agent.aiAct('scroll up');
await agent.aiAct('scroll left');
await agent.aiAct('scroll right');

// Scroll to positions
await agent.aiAct('scroll to top');
await agent.aiAct('scroll to bottom');
```

### Display Actions

#### ListDisplays

Get information about all connected displays.

```typescript
const displays = await ComputerDevice.listDisplays();
```

## Examples

### Open Application and Navigate

```typescript
import { agentFromComputer } from '@midscene/computer';

const agent = await agentFromComputer();

// Open application
if (process.platform === 'darwin') {
  await agent.aiAct('press Cmd+Space');
  await agent.aiAct('type "TextEdit" and press Enter');
} else {
  await agent.aiAct('press Windows key');
  await agent.aiAct('type "Notepad" and press Enter');
}

await agent.aiWaitFor('text editor window is visible');

// Type content
await agent.aiAct('type "Hello, Midscene!"');

// Save file
if (process.platform === 'darwin') {
  await agent.aiAct('press Cmd+S');
} else {
  await agent.aiAct('press Ctrl+S');
}
```

### Multi-Display Workflow

```typescript
import { ComputerDevice, agentFromComputer } from '@midscene/computer';

// List displays
const displays = await ComputerDevice.listDisplays();
console.log(`Found ${displays.length} displays`);

// Control primary display
const agent1 = await agentFromComputer({
  displayId: displays[0].id,
});
await agent1.aiAct('move mouse to center of screen');

// Control secondary display
if (displays.length > 1) {
  const agent2 = await agentFromComputer({
    displayId: displays[1].id,
  });
  await agent2.aiAct('move mouse to center of screen');
}
```

### Web Browser Automation

```typescript
import { agentFromComputer } from '@midscene/computer';

const agent = await agentFromComputer();

// Open browser
if (process.platform === 'darwin') {
  await agent.aiAct('press Cmd+Space');
  await agent.aiAct('type "Safari" and press Enter');
} else {
  await agent.aiAct('press Windows key');
  await agent.aiAct('type "Chrome" and press Enter');
}

await agent.aiWaitFor('browser window is open');

// Navigate
await agent.aiAct('click on address bar');
await agent.aiAct('type "example.com" and press Enter');
await agent.aiWaitFor('page has loaded');

// Extract information
const title = await agent.aiQuery('string, get the page title');
console.log('Page title:', title);
```

## TypeScript Types

```typescript
import type {
  ComputerAgent,
  ComputerAgentOpt,
  ComputerDevice,
  ComputerDeviceOpt,
  DisplayInfo,
  EnvironmentCheck,
} from '@midscene/computer';
```

## See Also

- [Common API Reference](./api) - APIs that work across all platforms
- [Model Configuration](./model-config) - Configure AI models
- [Caching](./caching) - Improve performance with caching



---
url: /computer-getting-started.md
---




import { PackageManagerTabs } from '@theme';

# PC Desktop Getting Started

This guide walks you through everything required to automate PC desktop applications with Midscene: install dependencies, configure model credentials, and run your first JavaScript script.

:::info Demo Projects

Control PC desktop with JavaScript: [https://github.com/web-infra-dev/midscene-example/tree/main/computer/javascript-sdk-demo](https://github.com/web-infra-dev/midscene-example/tree/main/computer/javascript-sdk-demo)

Integrate Vitest for testing: [https://github.com/web-infra-dev/midscene-example/tree/main/computer/vitest-demo](https://github.com/web-infra-dev/midscene-example/tree/main/computer/vitest-demo)

:::

## Set up API keys for model

Set your model configs into the environment variables. You may refer to [Model strategy](../model-strategy) for more details.

```bash
export MIDSCENE_MODEL_BASE_URL="https://replace-with-your-model-service-url/v1"
export MIDSCENE_MODEL_API_KEY="replace-with-your-api-key"
export MIDSCENE_MODEL_NAME="replace-with-your-model-name"
export MIDSCENE_MODEL_FAMILY="replace-with-your-model-family"
```

For more configuration details, please refer to [Model strategy](../model-strategy) and [Model configuration](../model-config).


## System Requirements

### Node.js

Node.js 18.19.0 or higher is required.

### Platform-Specific Dependencies

**macOS**: Accessibility permissions are required for keyboard and mouse control. When you run the script for the first time, macOS will prompt you to grant access. Go to **System Settings > Privacy & Security > Accessibility** and enable permissions for the application running your script (e.g., Terminal, iTerm2, VS Code, WebStorm, or other IDEs). For more details, see [nut.js macOS setup](https://github.com/nut-tree/nut.js#macos).

**Linux**: [ImageMagick](https://imagemagick.org/script/download.php) is required for screenshot functionality.

**Headless Linux (CI)**: To run desktop automation on a headless Linux server (e.g. GitHub Actions), install Xvfb and its dependencies, then enable headless mode:

```bash
# Install dependencies
sudo apt-get install -y xvfb x11-xserver-utils imagemagick
```

```typescript
// Option 1: Pass headless option
const agent = await agentFromComputer({ headless: true });

// Option 2: Set environment variable
// MIDSCENE_COMPUTER_HEADLESS_LINUX=true npx tsx example.ts
```

Xvfb creates a virtual display so that mouse, keyboard, and screenshot operations work without a physical monitor. See [API Reference](./computer-api-reference) for details.

:::tip Example: Testing Electron Apps on Headless Linux CI
A complete demo of testing [Obsidian](https://obsidian.md/) (an Electron app) on headless Linux CI with `@midscene/computer`: [https://github.com/web-infra-dev/midscene-example/tree/main/computer/electron-demo](https://github.com/web-infra-dev/midscene-example/tree/main/computer/electron-demo)
:::

## Try Playground (no code)

Playground is the fastest way to validate the connection and observe AI-driven steps without writing code. It shares the same core as `@midscene/computer`, so anything that works here will behave the same once scripted.

<video src="https://lf3-static.bytednsdoc.com/obj/eden-cn/nupipfups/Midscene/pc-twitter2.mp4" controls/>

1. Launch the Playground CLI:

```bash
npx --yes @midscene/computer-playground
```

2. Click the gear icon in the Playground window, then paste your API key configuration. Refer back to [Model configuration](./model-config) if you still need credentials.

## Start experiencing

After configuration, you can start using Midscene right away. It provides several key operation tabs:

- **Act**: interact with the page. This is Auto Planning, corresponding to `aiAct`. For example:
```
Type “Midscene” in the search box, run the search, and open the first result
```

```
Fill out the registration form and make sure every field passes validation
```

- **Query**: extract JSON data from the interface, corresponding to `aiQuery`.

Similar methods include `aiBoolean()`, `aiNumber()`, and `aiString()` for directly extracting booleans, numbers, and strings.

```
Extract the user ID from the page and return JSON data in the { id: string } structure
```

- **Assert**: understand the page and assert; if the condition is not met, throw an error, corresponding to `aiAssert`.

```
There is a login button on the page, with a user agreement link below it
```

- **Tap**: click on an element. This is Instant Action, corresponding to `aiTap`.
```
Click the login button
```

> For the difference between Auto Planning and Instant Action, see the [API](../api.mdx) document.


## Integration with Midscene Agent

Once Playground works, move to a repeatable script with the JavaScript SDK.

### Step 1. Install dependencies

<PackageManagerTabs command="install @midscene/computer" />

### Step 2. Write your first script

Create `example.ts`:

```typescript
import { agentFromComputer } from '@midscene/computer';

(async () => {
  // Create an agent
  const agent = await agentFromComputer({
    aiActionContext: 'You are controlling a desktop computer.',
  });

  // Take a screenshot and query information
  const screenInfo = await agent.aiQuery(
    '{width: number, height: number}, get screen resolution'
  );
  console.log('Screen resolution:', screenInfo);

  // Move mouse to center
  await agent.aiAct('move mouse to center of screen');

  // Assert screen has content
  await agent.aiAssert('The screen has visible content');

  console.log('Desktop automation completed!');
})();
```

### Step 3. Run the script

```bash
npx tsx example.ts
```

## Multi-Display Support

If you have multiple displays, you can control a specific one:

```typescript
import { ComputerDevice, agentFromComputer } from '@midscene/computer';

// List all displays
const displays = await ComputerDevice.listDisplays();
console.log('Available displays:', displays);

// Connect to a specific display
const agent = await agentFromComputer({
  displayId: displays[0].id,
});
```

## Example Usage

### Basic Mouse Operations

```typescript
// Click at center of screen
await agent.aiAct('click mouse at center of screen');

// Move mouse to a specific location
await agent.aiAct('move mouse to top-left corner');

// Double-click
await agent.aiAct('double-click on the desktop icon');

// Right-click
await agent.aiAct('right-click to open context menu');
```

### Keyboard Operations

```typescript
// Type text
await agent.aiAct('type "Hello World"');

// Press keyboard shortcuts
if (process.platform === 'darwin') {
  await agent.aiAct('press Cmd+Space to open Spotlight');
  await agent.aiAct('type "Calculator" and press Enter');
} else {
  await agent.aiAct('press Windows key');
  await agent.aiAct('type "Calculator" and press Enter');
}

// Press function keys
await agent.aiAct('press Escape');
await agent.aiAct('press Enter');
```

### Query Information

```typescript
// Extract screen information
const info = await agent.aiQuery(
  '{hasDesktop: boolean, visibleApps: string[]}, check if desktop is visible and list visible apps'
);

// Locate elements
const position = await agent.aiLocate('the File menu');
console.log('File menu position:', position);
```

### Complex Workflows

```typescript
// Open an application and interact with it
await agent.aiAct('open Finder');
await agent.aiWaitFor('Finder window is visible');

await agent.aiAct('click on Documents folder');
await agent.aiAct('press Cmd+N to create new folder');
await agent.aiAct('type "My Project"');
await agent.aiAct('press Enter');

await agent.aiAssert('A folder named "My Project" exists');
```

## Environment Check

You can check if your system is properly configured:

```typescript
import { checkComputerEnvironment } from '@midscene/computer';

const env = await checkComputerEnvironment();
console.log('Platform:', env.platform);
console.log('Available:', env.available);
console.log('Displays:', env.displays);

if (!env.available) {
  console.error('Environment not available:', env.error);
}
```

## Next Steps

- [API Reference](./computer-api-reference)
- [Use YAML format automation scripts](./automate-with-scripts-in-yaml)
- [YAML script runner](./yaml-script-runner)
- [Caching for efficiency](./caching)



---
url: /computer-introduction.md
---



# PC Desktop Automation Support

Midscene can drive native keyboard and mouse controls to support PC desktop automation on Windows, macOS, and Linux.

By leveraging a visual model solution, the automation process works with any desktop application—whether built with Electron, Qt, WPF, or native technologies. Developers only need to focus on the final user experience when debugging UI automation scripts.

The PC desktop automation solution comes with all the features of Midscene:

- Supports zero-code trial using Playground
- Supports JavaScript SDK for scripting
- Supports automation scripts in YAML format and command-line tools
- Supports HTML reports to replay all operation paths
- Works across Windows, macOS, and Linux platforms
- Headless mode for Linux CI via Xvfb (no physical display required)
- Multi-display support for complex setups

## Showcases

**Prompt** (**macOS**): Help me post a tweet promoting Midscene's support for AutoGLM through safari, with the following requirements:
1. Text content: Midscene now supports AutoGLM!
2. Media content: Use the AutoGLM video from the download folder!

<video src="https://lf3-static.bytednsdoc.com/obj/eden-cn/nupipfups/Midscene/pc-twitter2.mp4" height="300" controls />

View the full report for this task: [report.html](https://lf3-static.bytednsdoc.com/obj/eden-cn/nupipfups/Midscene/1.0-showcases/pc-twitter2-midscene_report.html)

**Prompt** (**Windows**): Open Sauce Demo e-commerce site, login and add items to cart

<video src="https://lf3-static.bytednsdoc.com/obj/eden-cn/nupipfups/Midscene/windows.mov" height="300" controls />

View the full report for this task: [report.html](https://lf3-static.bytednsdoc.com/obj/eden-cn/nupipfups/Midscene/shop-computer-2026-01-14_11-57-36-f8411b8f.html)

**Prompt** (**macOS**): Open Google and query San Jose tomorrow weather temperature

<video src="https://lf3-static.bytednsdoc.com/obj/eden-cn/nupipfups/Midscene/mac.mov" height="300" controls />

View the full report for this task: [report.html](https://lf3-static.bytednsdoc.com/obj/eden-cn/nupipfups/Midscene/weather-computer-2026-01-14_11-26-38-9592ecf5.html)

**Prompt** (**Linux**): Open TodoMVC, add multiple tasks and filter them

<video src="https://lf3-static.bytednsdoc.com/obj/eden-cn/nupipfups/Midscene/linux.mov" height="300" controls />

View the full report for this task: [report.html](https://lf3-static.bytednsdoc.com/obj/eden-cn/nupipfups/Midscene/todo-computer-2026-01-13_15-40-37-6f45fb0f.html)


See more showcases: [showcases](./showcases.mdx)

## Try with Playground

With Midscene.js playground, you can experience PC desktop automation capabilities without writing any code.

<video src="https://lf3-static.bytednsdoc.com/obj/eden-cn/nupipfups/Midscene/pc-twitter2.mp4" controls/>

See [Getting Started](./computer-getting-started#try-playground-no-code) to learn how to launch the Playground.

## Key Features

### Cross-platform Desktop Control

- **Mouse Operations**: Click, double-click, right-click, mouse move, drag-and-drop
- **Keyboard Input**: Type text, press keys with modifiers (Cmd/Ctrl/Alt/Shift)
- **Screen Capture**: Take screenshots of any display
- **Multi-display**: Work with multiple monitors simultaneously

### AI-Powered Automation

Using Midscene's AI capabilities, you can automate desktop applications with natural language:

```typescript
await agent.aiAct('open the File menu');
await agent.aiAct('click on Save As');
await agent.aiAct('type "my-document" in the filename field');
await agent.aiAct('press Enter');
```

## Use Cases

- **Desktop Application Testing**: Automate testing for Electron, Qt, or native apps
- **Workflow Automation**: Automate repetitive tasks across desktop applications
- **Cross-app Integration**: Control multiple applications in sequence
- **UI Testing**: Test desktop applications with natural language descriptions

## Next Steps

* [Getting Started](./computer-getting-started)
* [API Reference](./computer-api-reference)
* [Use YAML format automation scripts](./automate-with-scripts-in-yaml) and [YAML script runner](./yaml-script-runner)



---
url: /data-privacy.md
---

# Data privacy

⁠Midscene.js is an open-source project (GitHub: [Midscene](https://github.com/web-infra-dev/midscene/)) under the MIT license. You can see all the codes in the public repository.

When using Midscene.js, your page data (including the screenshot) is sent directly to the AI model provider you choose. No third-party platform will have access to this data. All you need to be concerned about is the data privacy policy of the model provider.

If you prefer building Midscene.js and its Chrome Extension in your own environment instead of using the published versions, you can refer to the [Contributing Guide](https://github.com/web-infra-dev/midscene/blob/main/CONTRIBUTING.md) to find building instructions.



---
url: /faq.md
---

# FAQ

## What data is sent to AI model?

The screenshot will be sent to the AI model. In some cases, like setting the `domIncluded` option to `true` when calling `aiAsk` or `aiQuery`, the DOM information will also be sent.

⁠If you are worried about data privacy issues, please refer to [Data Privacy](./data-privacy)

## How to improve the running time?

There are several ways to improve the running time:
1. Use instant action interface like `agent.aiTap('Login Button')` instead of `agent.ai('Click Login Button')`.
2. Use a lower resolution if possible, this will reduce the input token cost.
3. Change to a faster model service
4. Use caching to accelerate the debug process. Read more about it in [Caching](./caching).

## The webpage continues to flash when running in headed mode

In the local visualization interface, continuous flashing is usually caused by a mismatch between the viewport's `deviceScaleFactor` and the system/browser's pixel ratio (common on high-resolution or Retina screens).

This flashing does not affect Midscene's screenshots or automation execution, but it does affect the local preview experience. To resolve this, set `deviceScaleFactor` to match your browser's `window.devicePixelRatio`, or use Puppeteer's auto-adaptation feature.

```typescript
// Puppeteer: Set deviceScaleFactor to 0 to automatically use the device pixel ratio
await page.setViewport({
  deviceScaleFactor: 0,
});

// Playwright: Playwright does not support using 0 for auto-adaptation like Puppeteer
const page = await browser.newPage({
  deviceScaleFactor: 2, // Replace the number 2 with your window.devicePixelRatio
})
```

If you are unsure of your browser's pixel ratio, you can press F12 on any page to open the console and type `window.devicePixelRatio` to check; or paste the following into the Chrome address bar and press Enter to see the value in a popup:

```plain
data:text/html,<script>alert(`deviceScaleFactor of your browser: ${devicePixelRatio}`)</script>
```

## How do I configure the midscene_run directory?

Midscene saves runtime artifacts (reports, logs, cache, etc.) in the `midscene_run` directory. By default, this directory is created in the current working directory.

You can customize the directory location using the `MIDSCENE_RUN_DIR` environment variable, which accepts both relative and absolute paths:

```bash
# Using a relative path
export MIDSCENE_RUN_DIR="./my_custom_dir"

# Using an absolute path
export MIDSCENE_RUN_DIR="/tmp/midscene_output"
```

The directory contains the following subdirectories:

- `report/` - Test report files (HTML format)
- `log/` - Debug log files
- `cache/` - Cache files (see [Caching](./caching))

For more configuration options, see [Model configuration](./model-config).

## How do I control the report player's default replay style via a link?

You can override the default values of the **Focus on cursor** and **Show element markers** toggles by adding query parameters to the report URL, which determines whether the report highlights the cursor position and element markers. Use `focusOnCursor` and `showElementMarkers` with values such as `true`, `false`, `1`, or `0`. For example: `...?focusOnCursor=false&showElementMarkers=true`.
 
## Customize the network timeout

When doing interaction or navigation on web page, Midscene automatically waits for the network to be idle. It's a strategy to ensure the stability of the automation. Nothing would happen if the waiting process is timeout. 

The default timeout is configured as follows:

1. If it's a page navigation, the default wait timeout is 5000ms (the `waitForNavigationTimeout`)
2. If it's a click, input, etc., the default wait timeout is 2000ms (the `waitForNetworkIdleTimeout`)

You can also customize or disable the timeout by options:

- Use `waitForNetworkIdleTimeout` and `waitForNavigationTimeout` parameters in [Agent](/api#constructors).
- Use `waitForNetworkIdle` parameter in [Yaml](/automate-with-scripts-in-yaml#the-web-part) or [PlaywrightAiFixture](/integrate-with-playwright#step-2-extend-the-test-instance).

## Get an error 403 when using Ollama model in Chrome extension

`OLLAMA_ORIGINS="*"` is required to allow the Chrome extension to access the Ollama model.

## Inaccurate Element Positioning

If you encounter inaccurate element positioning when using Midscene, follow these steps to troubleshoot and resolve the issue:

### 1. Upgrade to the Latest Version

Make sure you are using the latest version of Midscene, as new versions typically include optimizations and improvements for positioning accuracy.

```bash
# Web automation
npm install @midscene/web@latest
# iOS automation
npm install @midscene/ios@latest
# CLI tool
npm install @midscene/cli@latest
# Or other packages corresponding to your platform
```

### 2. Use Better Vision Models

Midscene's element positioning capability relies on the AI model's visual understanding ability, so be sure to choose models that support visual capabilities.

Generally, newer versions and models with larger parameters perform better than older versions and smaller models. For example, Qwen3-VL performs better than Qwen2.5-VL, and its plus version performs better than the flash version.

For more model selection suggestions, please refer to [Model Strategy](./model-strategy).

### 3. Check Model Family Configuration

Verify that the `MIDSCENE_MODEL_FAMILY` parameter is set correctly in your model configuration. Incorrect `MIDSCENE_MODEL_FAMILY` configuration will affect Midscene's adaptation logic for the model. See [Model Configuration](./model-config) for details.

### 4. Analyze the Cause of Positioning Offset

Positioning offset typically occurs in two scenarios:

**Scenario 1: The model cannot understand semantics**
- Symptoms: Positioning results randomly fall on unrelated elements, with significant variation in results each time.
- Cause: The model may not understand the semantics behind icon buttons. For example, `aiTap('profile center')` is a functional description, and the model may not know the specific style of a profile center icon; whereas `aiTap('person avatar icon')` is a visual description, and the model can locate the element based on its visual characteristics.
- Solution: Optimize prompts by combining visual features and position information to describe the element.
  ```typescript
  // ❌ Using only functional description
  await agent.aiTap('profile center');

  // ✅ Using visual description
  await agent.aiTap('person avatar icon');

  // ✅ Combining visual features and position information
  await agent.aiTap('person avatar icon in the top right corner of the page');
  ```

**Scenario 2: The model recognizes accurately but the positioning has deviation**
- Symptoms: Positioning results fall near the target element but with a few pixels offset.
- Solution: Enabling `deepThink` will significantly improve positioning effectiveness.
  ```typescript
  await agent.aiTap('Login button', {
    deepThink: true
  });
  ```

For more information about `deepThink`, please refer to the [API documentation](/api).

## Does the Doubao phone use Midscene under the hood?

No.



---
url: /harmony-api-reference.md
---

# API Reference (HarmonyOS)

When you need to customize device behavior, integrate Midscene into a framework, or troubleshoot HDC issues, refer to this section. For common constructor parameters (reports, hooks, caching, etc.), see the platform-agnostic [API Reference](./api).

## Action Space

`HarmonyDevice` uses the following action space. The Midscene Agent can use these operations when planning tasks:

- `Tap` — Tap on an element.
- `DoubleClick` — Double-tap on an element.
- `Input` — Input text, supporting `replace`/`typeOnly`/`clear` modes.
- `Scroll` — Scroll from an element or screen center in any direction, supporting scroll-to-top/bottom/left/right.
- `DragAndDrop` — Drag from one element to another.
- `KeyboardPress` — Press a specific key.
- `LongPress` — Long-press a target element with optional custom duration.
- `ClearInput` — Clear input field contents.
- `Launch` — Open a HarmonyOS app (bundle name).
- `RunHdcShell` — Execute a raw `hdc shell` command.
- `HarmonyBackButton` — Trigger system back.
- `HarmonyHomeButton` — Return to home screen.
- `HarmonyRecentAppsButton` — Open recent apps / multitasking.

## HarmonyDevice {#harmonydevice}

Creates an HDC device instance that can be driven by HarmonyAgent.

### Import

```ts
import { HarmonyDevice, getConnectedDevices } from '@midscene/harmony';
```

### Constructor

```ts
const device = new HarmonyDevice(deviceId, {
  // device options...
});
```

### Device Options

- `deviceId: string` — Value from `hdc list targets` or `getConnectedDevices()`.
- `hdcPath?: string` — Custom path to the HDC executable. If not set, it searches `HDC_HOME` environment variable and common installation paths.
- `autoDismissKeyboard?: boolean` — Automatically hide keyboard after input, default `true`.
- `screenshotResizeScale?: number` — Scale screenshots before sending to the model. Not recommended to modify manually.
- `customActions?: DeviceAction[]` — Extend the planner's available actions via `defineAction`.

### Usage Notes

- Use `getConnectedDevices()` to discover devices. The returned `deviceId` matches `hdc list targets` output.
- If HDC is not in your system PATH, specify it via the `HDC_HOME` environment variable or the `hdcPath` option.

### Examples

#### Quick Start

```ts
import { HarmonyAgent, HarmonyDevice, getConnectedDevices } from '@midscene/harmony';

const [first] = await getConnectedDevices();
const device = new HarmonyDevice(first.deviceId, {});
await device.connect();

const agent = new HarmonyAgent(device, {
  aiActionContext: 'This is a HarmonyOS device. If any popup appears, agree to it.',
});

await agent.launch('com.huawei.hmos.settings');
await agent.aiAct('scroll down one screen');
const items = await agent.aiQuery(
  'string[], list all visible setting item names',
);
console.log(items);
```

#### Launch Apps

```ts
await agent.launch('com.huawei.hmos.settings'); // Open Settings
await agent.launch('com.huawei.hmos.camera');    // Open Camera
await agent.back();
await agent.home();
```

## HarmonyAgent {#harmonyagent}

Binds Midscene's AI planning capabilities to a HarmonyDevice for UI automation.

### Import

```ts
import { HarmonyAgent } from '@midscene/harmony';
```

### Constructor

```ts
const agent = new HarmonyAgent(device, {
  // common Agent options...
});
```

### HarmonyOS-Specific Options

- `customActions?: DeviceAction[]` — Extend the planner's available actions via `defineAction`.
- `appNameMapping?: Record<string, string>` — Map friendly app names to bundle names. When you pass an app name to `launch(target)`, the Agent looks up the corresponding bundle name in this mapping; if no mapping is found, it tries to launch `target` as-is.
- Other fields are the same as [API constructors](./api#common-parameters): `generateReport`, `reportFileName`, `aiActionContext`, `modelConfig`, `cacheId`, `createOpenAIClient`, `onTaskStartTip`, etc.

### Usage Notes

:::info

- One device connection corresponds to one Agent.
- HarmonyOS-specific helpers like `launch` and `runHdcShell` can also be used in YAML scripts. See [HarmonyOS platform-specific actions](./automate-with-scripts-in-yaml#the-harmony-part).
- For common interaction methods, see [API Reference (Common)](./api#interaction-methods).

:::

### HarmonyOS-Specific Methods

#### `agent.launch()`

Launch a HarmonyOS app.

```ts
function launch(uri: string): Promise<void>;
```

- `uri: string` — Can be an app bundle name (e.g., `com.huawei.hmos.settings`), or an app name registered in `appNameMapping`. If a URL starting with `http://` or `https://` is passed, it opens via the browser.

```ts
await agent.launch('com.huawei.hmos.settings'); // Open Settings
await agent.launch('com.huawei.hmos.camera');    // Open Camera
```

#### `agent.runHdcShell()`

Run a raw `hdc shell` command on the connected device.

```ts
function runHdcShell(command: string): Promise<string>;
```

- `command: string` — The command passed directly to `hdc shell`.

```ts
const result = await agent.runHdcShell('hidumper -s RenderService -a screen');
console.log(result);
```

#### Navigation Helpers

- `agent.back(): Promise<void>` — Trigger HarmonyOS system back.
- `agent.home(): Promise<void>` — Return to home screen.
- `agent.recentApps(): Promise<void>` — Open recent apps / multitasking.

### Utilities

#### `agentFromHdcDevice()`

Create a `HarmonyAgent` from any connected HDC device.

```ts
function agentFromHdcDevice(
  deviceId?: string,
  opts?: HarmonyAgentOpt & HarmonyDeviceOpt,
): Promise<HarmonyAgent>;
```

- `deviceId?: string` — Connect to a specific device; leave empty for "first available device".
- `opts?: HarmonyAgentOpt & HarmonyDeviceOpt` — Merge Agent options and [`HarmonyDevice`](#harmonydevice) settings in a single object.

```ts
import { agentFromHdcDevice } from '@midscene/harmony';

const agent = await agentFromHdcDevice('0123456789ABCDEF'); // specific device
const agent = await agentFromHdcDevice(); // first available device
```

#### `getConnectedDevices()`

List HDC devices that Midscene can drive.

```ts
function getConnectedDevices(
  hdcPath?: string,
): Promise<Array<{ deviceId: string }>>;
```

```ts
import { getConnectedDevices } from '@midscene/harmony';

const devices = await getConnectedDevices();
console.log(devices); // [{ deviceId: '0123456789ABCDEF' }]
```

### Related Reading

- [HarmonyOS Getting Started](./harmony-getting-started) for setup and script examples.



---
url: /harmony-getting-started.md
---




import { PackageManagerTabs } from '@theme';

# HarmonyOS Getting Started

This guide walks you through everything needed to automate HarmonyOS devices with Midscene: connecting a real device via HDC, configuring model API keys, trying the zero-code Playground, and running your first JavaScript script.

:::info Demo Projects

Control HarmonyOS devices with JavaScript: [https://github.com/web-infra-dev/midscene-example/blob/main/harmony/javascript-sdk-demo](https://github.com/web-infra-dev/midscene-example/blob/main/harmony/javascript-sdk-demo)

Integrate Vitest for testing: [https://github.com/web-infra-dev/midscene-example/tree/main/harmony/vitest-demo](https://github.com/web-infra-dev/midscene-example/tree/main/harmony/vitest-demo)

:::

## Set up API keys for model

Set your model configs into the environment variables. You may refer to [Model strategy](../model-strategy) for more details.

```bash
export MIDSCENE_MODEL_BASE_URL="https://replace-with-your-model-service-url/v1"
export MIDSCENE_MODEL_API_KEY="replace-with-your-api-key"
export MIDSCENE_MODEL_NAME="replace-with-your-model-name"
export MIDSCENE_MODEL_FAMILY="replace-with-your-model-family"
```

For more configuration details, please refer to [Model strategy](../model-strategy) and [Model configuration](../model-config).


## Prerequisites

Before writing scripts, verify that HDC can connect to your device and the device trusts the current computer.

### Install HDC

HDC (HarmonyOS Device Connector) is a command-line tool provided by HarmonyOS for communicating with devices. Installation options:

- Via [DevEco Studio](https://developer.huawei.com/consumer/en/deveco-studio/) (recommended)
- Via [HarmonyOS command-line tools](https://developer.huawei.com/consumer/en/download/) standalone installation

Verify HDC is installed:

```bash
hdc version
```

A version number in the output confirms successful installation.

:::tip Configuring HDC Path

If `hdc` is not in your system PATH, set the `HDC_HOME` environment variable to the directory containing HDC:

```bash
export HDC_HOME=/path/to/hdc/directory
```

:::

### Enable Developer Mode and Verify Device

In your HarmonyOS device settings, go to **Developer Options** and enable **USB Debugging**, then connect via USB cable.

Verify the connection:

```bash
hdc list targets
```

A device ID in the output confirms a successful connection:

```log
0123456789ABCDEF
```

## Try Playground (Zero Code)

Playground is the fastest way to verify your connection and observe the AI Agent, without writing any code. It shares the same code implementation as `@midscene/harmony`, so flows validated in Playground will work identically when run via scripts.

<video src="https://lf3-static.bytednsdoc.com/obj/eden-cn/nupipfups/Midscene/1.0-showcases/harmony.mp4" controls/>

1. Launch the Playground CLI:

```bash
npx --yes @midscene/harmony-playground
```

2. Click the gear button in the Playground window and paste your API Key configuration. If you don't have an API Key yet, go back to [Model Configuration](./model-config) to get one.

## Start Your Experience

After configuration, you can start using Midscene right away. It provides several key operation tabs:

- **Act**: interact with the page. This is Auto Planning, corresponding to `aiAct`. For example:
```
Type “Midscene” in the search box, run the search, and open the first result
```

```
Fill out the registration form and make sure every field passes validation
```

- **Query**: extract JSON data from the interface, corresponding to `aiQuery`.

Similar methods include `aiBoolean()`, `aiNumber()`, and `aiString()` for directly extracting booleans, numbers, and strings.

```
Extract the user ID from the page and return JSON data in the { id: string } structure
```

- **Assert**: understand the page and assert; if the condition is not met, throw an error, corresponding to `aiAssert`.

```
There is a login button on the page, with a user agreement link below it
```

- **Tap**: click on an element. This is Instant Action, corresponding to `aiTap`.
```
Click the login button
```

> For the difference between Auto Planning and Instant Action, see the [API](../api.mdx) document.


## Integrate Midscene Agent

Once Playground runs successfully, you can switch to reusable JavaScript scripts.

### Step 1: Install Dependencies

<PackageManagerTabs command="install @midscene/harmony --save-dev" />

### Step 2: Write a Script

The following example opens the Settings app on the device and performs scrolling operations.

```typescript title="./demo.ts"
import {
  HarmonyAgent,
  HarmonyDevice,
  getConnectedDevices,
} from '@midscene/harmony';

const sleep = (ms) => new Promise((r) => setTimeout(r, ms));
Promise.resolve(
  (async () => {
    const devices = await getConnectedDevices();
    const device = new HarmonyDevice(devices[0].deviceId, {});

    const agent = new HarmonyAgent(device, {
      aiActionContext:
        'This is a HarmonyOS device. The system language is Chinese. If any popup appears, dismiss or agree to it.',
    });
    await device.connect();

    // Open Settings app
    await agent.launch('com.huawei.hmos.settings');
    await sleep(2000);

    // Scroll down
    await agent.aiAct('scroll down one screen');

    // Query page content
    const items = await agent.aiQuery(
      'string[], list all visible setting item names',
    );
    console.log('Settings items', items);

    // Assert
    await agent.aiAssert('There is a settings item list on the page');
  })(),
);
```

### Step 3: Run the Example

```bash
npx tsx demo.ts
```

### Step 4: View the Report

After a successful run, the console outputs `Midscene - report file updated: /path/to/report/some_id.html`. Open this HTML file in a browser to replay each interaction, query, and assertion.

## Advanced

When you need to customize device behavior, integrate Midscene into a standalone framework, or troubleshoot HDC issues, refer to this section. See [API Reference (HarmonyOS)](./harmony-api-reference) for more constructor parameters.

### Extending Midscene on HarmonyOS

Use `defineAction()` to define custom gestures and pass them via `customActions`. Midscene appends custom actions to the planner, allowing AI to invoke your domain-specific action names.

```typescript
import { getMidsceneLocationSchema, z } from '@midscene/core';
import { defineAction } from '@midscene/core/device';
import { HarmonyAgent, HarmonyDevice, getConnectedDevices } from '@midscene/harmony';

const ContinuousClick = defineAction({
  name: 'continuousClick',
  description: 'Click the same target repeatedly',
  paramSchema: z.object({
    locate: getMidsceneLocationSchema(),
    count: z.number().int().positive().describe('How many times to click'),
  }),
  async call(param) {
    const { locate, count } = param;
    console.log('click target center', locate.center);
    console.log('click count', count);
  },
});

const devices = await getConnectedDevices();
const device = new HarmonyDevice(devices[0].deviceId, {});
await device.connect();

const agent = new HarmonyAgent(device, {
  customActions: [ContinuousClick],
});

await agent.aiAct('click the red button five times');
```

For more details on custom actions and action schemas, see [Integrate with Any Interface](./integrate-with-any-interface#define-a-custom-action).

## More

- View all Agent methods: [API Reference (Common)](./api#interaction-methods)
- HarmonyOS-specific parameters and interfaces: [HarmonyOS Agent API](./harmony-api-reference)
- Demo projects
  - HarmonyOS JavaScript SDK demo: [https://github.com/web-infra-dev/midscene-example/blob/main/harmony/javascript-sdk-demo](https://github.com/web-infra-dev/midscene-example/blob/main/harmony/javascript-sdk-demo)
  - HarmonyOS + Vitest demo: [https://github.com/web-infra-dev/midscene-example/tree/main/harmony/vitest-demo](https://github.com/web-infra-dev/midscene-example/tree/main/harmony/vitest-demo)

### Complete example (Vitest + HarmonyAgent)

```typescript
import type { TestStatus } from '@midscene/core';
import { ReportMergingTool } from '@midscene/core/report';
import { sleep } from '@midscene/core/utils';
import {
  HarmonyAgent,
  HarmonyDevice,
  getConnectedDevices,
} from '@midscene/harmony';
import {
  afterAll,
  afterEach,
  beforeAll,
  beforeEach,
  describe,
  it,
} from 'vitest';

describe('HarmonyOS Settings Test', () => {
  let device: HarmonyDevice;
  let agent: HarmonyAgent;
  let itTestStatus: TestStatus = 'passed';
  const reportMergingTool = new ReportMergingTool();

  beforeAll(async () => {
    const devices = await getConnectedDevices();
    device = new HarmonyDevice(devices[0].deviceId);
    await device.connect();
  });

  beforeEach((ctx) => {
    agent = new HarmonyAgent(device, {
      groupName: ctx.task.name,
    });
  });

  afterEach((ctx) => {
    if (ctx.task.result?.state === 'pass') {
      itTestStatus = 'passed';
    } else if (ctx.task.result?.state === 'skip') {
      itTestStatus = 'skipped';
    } else if (ctx.task.result?.errors?.[0].message.includes('timed out')) {
      itTestStatus = 'timedOut';
    } else {
      itTestStatus = 'failed';
    }
    reportMergingTool.append({
      reportFilePath: agent.reportFile as string,
      reportAttributes: {
        testId: `${ctx.task.name}`,
        testTitle: `${ctx.task.name}`,
        testDescription: 'description',
        testDuration: (Date.now() - ctx.task.result?.startTime!) | 0,
        testStatus: itTestStatus,
      },
    });
  });

  afterAll(() => {
    reportMergingTool.mergeReports('my-harmony-setting-test-report');
  });

  it('toggle WLAN', async () => {
    await device.home();
    await sleep(1000);
    await device.launch('com.huawei.hmos.settings');
    await sleep(1000);
    await agent.aiAct('find and enter WLAN setting');
    await agent.aiAct(
      'toggle WLAN status *once*, if WLAN is off please turn it on, otherwise turn it off.',
    );
  });

  it('toggle Bluetooth', async () => {
    await device.home();
    await sleep(1000);
    await device.launch('com.huawei.hmos.settings');
    await sleep(1000);
    await agent.aiAct('find and enter Bluetooth setting');
    await agent.aiAct(
      'toggle Bluetooth status *once*, if Bluetooth is off please turn it on, otherwise turn it off.',
    );
  });
});
```

:::tip

Merged reports are stored inside `midscene_run/report` by default. Override the directory with `MIDSCENE_RUN_DIR` when running in CI.

:::



---
url: /harmony-introduction.md
---



# HarmonyOS Automation Support

Midscene can drive the HDC (HarmonyOS Device Connector) tool to automate HarmonyOS NEXT devices.

Thanks to its visual model approach, the entire automation process works with any HarmonyOS app technology stack — whether ArkTS native or other frameworks. Developers only need to debug UI automation scripts against the final rendered interface.

The HarmonyOS UI automation solution includes all Midscene features:

- Zero-code trial via Playground.
- JavaScript SDK support.
- YAML-based automation scripts and CLI tools.
- HTML report generation for replaying all action paths.

## Showcases

**Prompt** : Open Settings, scroll to find "About phone", view device information.

<video src="https://lf3-static.bytednsdoc.com/obj/eden-cn/nupipfups/Midscene/1.0-showcases/harmony.mp4" poster="https://lf3-static.bytednsdoc.com/obj/eden-cn/nupipfups/Midscene/1.0-showcases/harmony.png" height="300" controls />

View the full report of this task: [report.html](https://lf3-static.bytednsdoc.com/obj/eden-cn/nupipfups/Midscene/1.0-showcases/harmony.html)


See more showcases: [showcases](./showcases.mdx)

## Try Midscene Playground on HarmonyOS

With Midscene.js Playground, you can experience HarmonyOS automation without writing any code.

<video src="https://lf3-static.bytednsdoc.com/obj/eden-cn/nupipfups/Midscene/1.0-showcases/harmony.mp4" controls/>

See [Getting Started](./harmony-getting-started#try-playground-zero-code) to learn how to launch Playground.

## Next Steps

* [Getting Started](./harmony-getting-started)
* [Using JavaScript SDK](./harmony-getting-started)
* [Using YAML automation scripts](./automate-with-scripts-in-yaml) and [CLI tools](./automate-with-scripts-in-yaml)



---
url: /index.md
---




---
url: /integrate-with-android.md
---




# Integrate with Android (adb)

After connecting the Android device with adb, you can use Midscene javascript SDK to control Android devices.

import { PackageManagerTabs } from '@theme';

:::info Demo Projects
Control Android devices with javascript: [https://github.com/web-infra-dev/midscene-example/blob/main/android/javascript-sdk-demo](https://github.com/web-infra-dev/midscene-example/blob/main/android/javascript-sdk-demo)

Integrate Vitest for testing: [https://github.com/web-infra-dev/midscene-example/tree/main/android/vitest-demo](https://github.com/web-infra-dev/midscene-example/tree/main/android/vitest-demo)
:::

:::info Showcases

[More showcases](./android-introduction.mdx)

<p align="center">
  <img src="/android.png" alt="android" width="400" />
</p>

:::


## Preparation

### Install Node.js

Install [Node.js 18 or higher](https://nodejs.org/en/download/).

### Prepare API Key

Prepare an API Key for a Vision Language (VL) model.

You can check the models and configurations supported by Midscene.js in the [Model Strategy](../model-strategy) documentation.

### Install adb

`adb` is a command-line tool that allows you to communicate with Android devices. There are two ways to install `adb`:

- Method 1: Install using [Android Studio](https://developer.android.com/studio)
- Method 2: Install using [Android Command Line Tools](https://developer.android.com/studio#command-line-tools-only)

Verify that `adb` is installed successfully:

```bash
adb --version
```

When you see the following output, it means `adb` is installed successfully:

```log
Android Debug Bridge version 1.0.41
Version 34.0.4-10411341
Installed as /usr/local/bin//adb
Running on Darwin 24.3.0 (arm64) 
```

### Set the `ANDROID_HOME` environment variable

Refer to [Android Environment Variables](https://developer.android.com/tools/variables) to set the `ANDROID_HOME` environment variable.

Verify that the `ANDROID_HOME` variable is set successfully:

```bash
echo $ANDROID_HOME
```

When the above command has output, it means the `ANDROID_HOME` variable is set successfully:

```log
/Users/your_username/Library/Android/sdk
```

### Connect Android Device

In the developer options of your Android device, enable 'USB debugging'. If 'USB debugging (Security settings)' exists, enable it as well. Then connect your Android device using a USB cable.

<p align="center">
  <img src="/android-usb-debug-en.png" alt="android usb debug" width="400"/>
</p>

Verify the connection:

```bash
adb devices -l
```

When you see the following output, it means the connection is successful:

```log
List of devices attached
s4ey59	device usb:34603008X product:cezanne model:M2006J device:cezan transport_id:3
```


## Set up API keys for model

Set your model configs into the environment variables. You may refer to [Model strategy](../model-strategy) for more details.

```bash
export MIDSCENE_MODEL_BASE_URL="https://replace-with-your-model-service-url/v1"
export MIDSCENE_MODEL_API_KEY="replace-with-your-api-key"
export MIDSCENE_MODEL_NAME="replace-with-your-model-name"
export MIDSCENE_MODEL_FAMILY="replace-with-your-model-family"
```

For more configuration details, please refer to [Model strategy](../model-strategy) and [Model configuration](../model-config).


## Integrate Midscene

### Step 1: Install dependencies

<PackageManagerTabs command="install @midscene/android --save-dev" />

### Step 2: Write scripts

Let's take a simple example: search for headphones on eBay using the browser in the Android device. （Of course, you can also use any other apps on the Android device.）

Write the following code, and save it as `./demo.ts`

```typescript title="./demo.ts"
import {
  AndroidAgent,
  AndroidDevice,
  getConnectedDevices,
} from '@midscene/android';

const sleep = (ms) => new Promise((r) => setTimeout(r, ms));
Promise.resolve(
  (async () => {
    const devices = await getConnectedDevices();
    const device = new AndroidDevice(devices[0].udid);

    // 👀 init Midscene agent
    const agent = new AndroidAgent(device, {
      aiActionContext:
        'If any location, permission, user agreement, etc. popup, click agree. If login page pops up, close it.',
    });
    await device.connect();

    // 👀 open browser and navigate to ebay.com (Please ensure that the current page has a browser app)
    await agent.aiAction('open browser and navigate to ebay.com');

    await sleep(5000);

    // 👀 type keywords, perform a search
    await agent.aiAction('type "Headphones" in search box, hit Enter');

    // 👀 wait for loading completed
    await agent.aiWaitFor('There is at least one headphone product');
    // or you can use a normal sleep:
    // await sleep(5000);

    // 👀 understand the page content, extract data
    const items = await agent.aiQuery(
      '{itemTitle: string, price: Number}[], find item in list and corresponding price',
    );
    console.log('headphones in stock', items);

    // 👀 assert by AI
    await agent.aiAssert('There is a category filter on the left');
  })(),
);
```

### Step 3: Run

Using `tsx` to run

```bash
# run
npx tsx demo.ts
```

After a while, you will see the following output:

```log
[
{
  itemTitle: 'Beats by Dr. Dre Studio Buds Totally Wireless Noise Cancelling In Ear + OPEN BOX',
  price: 505.15
},
{
  itemTitle: 'Skullcandy Indy Truly Wireless Earbuds-Headphones Green Mint',
  price: 186.69
}
]
```

### Step 4: View the report

After the above command executes successfully, the console will output: `Midscene - report file updated: /path/to/report/some_id.html`. You can open this file in a browser to view the report.

## Constructor and Interface

### `AndroidDevice` Constructor

The AndroidDevice constructor supports the following parameters:

- `deviceId: string` - The device id
- `opts?: AndroidDeviceOpt` - Optional, the options for the AndroidDevice
  - `autoDismissKeyboard?: boolean` - Optional, whether to dismiss the keyboard after inputting. (Default: true)
  - `keyboardDismissStrategy?: 'esc-first' | 'back-first'` - Optional, the strategy to dismiss the keyboard. 'esc-first' tries ESC key first, then back key if needed. 'back-first' tries back key first, then ESC key if needed. (Default: 'esc-first')
  - `androidAdbPath?: string` - Optional, the path to the adb executable.
  - `remoteAdbHost?: string` - Optional, the remote adb host.
  - `remoteAdbPort?: number` - Optional, the remote adb port.
  - `imeStrategy?: 'always-yadb' | 'yadb-for-non-ascii'` - Optional, controls when Midscene invokes [yadb](https://github.com/ysbing/YADB) for text input. `'yadb-for-non-ascii'` (default) automatically uses yadb for Unicode characters (including Latin Unicode like ö, é, ñ), Chinese, Japanese, and format specifiers (like %s, %d), while pure ASCII text uses the faster native `adb input text`. `'always-yadb'` forces yadb for all text input, providing maximum compatibility but slightly slower for pure ASCII. (Default: 'yadb-for-non-ascii')
  - `displayId?: number` - Optional, the display id to use. (Default: undefined, means use the current display)
  - `screenshotResizeScale?: number` - Optional, controls the size of the screenshot Midscene sends to the AI model. Default is `1 / devicePixelRatio`, so a 1200×800 display with a device pixel ratio of 3 sends an image of roughly 400×267 to the model. Adjusting this value manually is not recommended.
  - `alwaysRefreshScreenInfo?: boolean` - Optional, whether to re-fetch screen size and orientation information every time. Default is false (uses cache for better performance). Set to true if the device may rotate or you need real-time screen information.

### Additional Android Agent Interfaces

Except the common agent interfaces in [API Reference](./api.mdx), AndroidAgent also provides some other interfaces:

#### `agent.launch()`

Launch a webpage or native page.

- Type

```typescript
function launch(uri: string): Promise<void>;
```

- Parameters:

  - `uri: string` - The uri to open, can be a webpage url or a native app's package name or activity name, if the activity name exists, it should be separated by / (e.g. com.android.settings/.Settings).

- Return Value:

  - Returns a Promise that resolves to void when the page is opened.

- Examples:

```typescript
import { AndroidAgent, AndroidDevice } from '@midscene/android';

const device = new AndroidDevice('s4ey59');
const agent = new AndroidAgent(device);

await agent.launch('https://www.ebay.com'); // open a webpage
await agent.launch('com.android.settings'); // open a native page
await agent.launch('com.android.settings/.Settings'); // open a native page
```

#### `agent.runAdbShell()`

Execute a command through `adb shell` on the connected device.

> Note: This method wraps `adb shell` and forwards the command directly to the device.

- Type

```typescript
function runAdbShell(command: string): Promise<string>;
```

- Parameters:

  - `command: string` - The adb shell command to execute.

- Return Value:

  - `Promise<string>` - Returns a Promise that resolves to the command output.

- Examples:

```typescript
import { AndroidAgent, AndroidDevice } from '@midscene/android';

const device = new AndroidDevice('s4ey59');
const agent = new AndroidAgent(device);
await device.connect();

const result = await agent.runAdbShell('dumpsys battery');
// Equivalent to running `adb shell dumpsys battery`
console.log(result);
```

#### `agentFromAdbDevice()`

Create a AndroidAgent from a connected adb device.

- Type

```typescript
function agentFromAdbDevice(
  deviceId?: string,
  opts?: PageAgentOpt,
): Promise<AndroidAgent>;
```

- Parameters:

  - `deviceId?: string` - Optional, the adb device id to connect. If not provided, the first connected device will be used.
  - `opts?: PageAgentOpt & AndroidDeviceOpt` - Optional, the options for the AndroidAgent, PageAgentOpt refer to [constructor](./api.mdx), AndroidDeviceOpt refer to [AndroidDevice constructor](./integrate-with-android#androiddevice-constructor).

- Return Value:

  - `Promise<AndroidAgent>` Returns a Promise that resolves to an AndroidAgent.

- Examples:

```typescript
import { agentFromAdbDevice } from '@midscene/android';

const agent = await agentFromAdbDevice('s4ey59'); // create a AndroidAgent from a specific adb device
const agent = await agentFromAdbDevice(); // no deviceId, use the first connected device
```

#### `getConnectedDevices()`

Get all connected Android devices.

- Type

```typescript
function getConnectedDevices(): Promise<Device[]>;
interface Device {
  /**
   * The device udid.
   */
  udid: string;
  /**
   * Current device state, as it is visible in
   * _adb devices -l_ output.
   */
  state: string;
  port?: number;
}
```

- Return Value:

  - `Promise<Device[]>` Returns a Promise that resolves to an array of Device.

- Examples:

```typescript
import { agentFromAdbDevice, getConnectedDevices } from '@midscene/android';

const devices = await getConnectedDevices();
console.log(devices);
const agent = await agentFromAdbDevice(devices[0].udid);
```

## Extending Custom Interaction Actions

Use the `customActions` option to extend the agent's action space with your own actions defined via `defineAction`. When provided, these actions will be appended to the built-in ones so the agent can call them during planning.

```typescript
import { getMidsceneLocationSchema, z } from '@midscene/core';
import { defineAction } from '@midscene/core/device';
import { AndroidAgent, AndroidDevice } from '@midscene/android';

const ContinuousClick = defineAction({
  name: 'continuousClick',
  description: 'Click the same target repeatedly',
  paramSchema: z.object({
    locate: getMidsceneLocationSchema(),
    count: z
      .number()
      .int()
      .positive()
      .describe('How many times to click'),
  }),
  async call(param) {
    const { locate, count } = param;
    console.log('click target center', locate.center);
    console.log('click count', count);
    // carry out your clicking logic using locate + count
  },
});

const device = new AndroidDevice('your-device-id', {
  customActions: [ContinuousClick],
});
const agent = new AndroidAgent(device);

await agent.aiAction('click the red button five times');
```

Check [Integrate with any interface](./integrate-with-any-interface#define-a-custom-action) for more details about defining custom actions.

## More

- For all the APIs on the Agent, please refer to [API Reference](./api.mdx).

## FAQ

### Why can't I control the device even though I've connected it?
A typical error message is:
```
Error:
Exception occurred while executing 'tap':
java.lang.SecurityException: Injecting input events requires the caller (or the source of the instrumentation, if any) to have the INJECT_EVENTS permission.
```

Please check if the device is unlocked in the developer options of the system settings.

<p align="center">
  <img src="/android-usb-debug-en.png" alt="android usb debug" width="400" />
</p>

### How to use a custom adb path, remote adb host and port?

You can use the `MIDSCENE_ADB_PATH` environment variable to specify the path to the adb executable, `MIDSCENE_ADB_REMOTE_HOST` environment variable to specify the remote adb host, `MIDSCENE_ADB_REMOTE_PORT` environment variable to specify the remote adb port.

```bash
export MIDSCENE_ADB_PATH=/path/to/adb
export MIDSCENE_ADB_REMOTE_HOST=192.168.1.100
export MIDSCENE_ADB_REMOTE_PORT=5037
```

Additionally, you can also specify the adb path, remote adb host and port through the AndroidDevice constructor.

```typescript
const device = new AndroidDevice('s4ey59', {
  androidAdbPath: '/path/to/adb',
  remoteAdbHost: '192.168.1.100',
  remoteAdbPort: 5037,
});
```



---
url: /integrate-with-any-interface.md
---



# Integrate with any interface

You can use Midscene Agent to control any interface—such as IoT devices, in-house apps, and in-vehicle displays—by implementing a UI operation class that conforms to `AbstractInterface`.

After implementing the UI operation class, you get the full capabilities of Midscene Agent:

- the TypeScript GUI Automation Agent SDK, supporting integration with any interface
- the playground for debugging
- controlling the interface with YAML scripts
- an MCP service that exposes UI actions


## Demo and community project

We have prepared a demo project for you to learn how to define your own interface class. It's highly recommended to check it out.

* [Demo Project](https://github.com/web-infra-dev/midscene-example/tree/main/custom-interface) - A simple demo project that shows how to define your own interface class

* [Android (adb) Agent](https://github.com/web-infra-dev/midscene/blob/main/packages/android/src/device.ts) - This is the Android (adb) Agent for Midscene that implements this feature

* [iOS (WebDriverAgent) Agent](https://github.com/web-infra-dev/midscene/blob/main/packages/ios/src/device.ts) - This is the iOS (WebDriverAgent) Agent for Midscene that implements this feature

There are also some community projects that use this feature:

* [midscene-ios](https://github.com/lhuanyu/midscene-ios) - A project driving the OSX "iPhone Mirroring" app with Midscene

## Set up API keys for model

Set your model configs into the environment variables. You may refer to [Model strategy](../model-strategy) for more details.

```bash
export MIDSCENE_MODEL_BASE_URL="https://replace-with-your-model-service-url/v1"
export MIDSCENE_MODEL_API_KEY="replace-with-your-api-key"
export MIDSCENE_MODEL_NAME="replace-with-your-model-name"
export MIDSCENE_MODEL_FAMILY="replace-with-your-model-family"
```

For more configuration details, please refer to [Model strategy](../model-strategy) and [Model configuration](../model-config).


## Implement your own interface class

### Key concepts

* The `AbstractInterface` class: a predefined abstract class that can connect to the Midscene Agent
* The **action space**: a set of actions that describe the actions that can be performed on the interface. This will affect how the AI model plans the actions and executes them

### Step 1. Clone and setup from the demo project

We provide a demo project that runs all the features of this document below. It's the fastest way to get started.

```bash
# prepare the environment
git clone https://github.com/web-infra-dev/midscene-example.git
cd midscene-example/custom-interface
npm install
npm run build

# run the demo
npm run demo
```

### Step 2. Implement your interface class

Define a class that extends the `AbstractInterface` class, and implement the required methods.

You can get the sample implementation from the [`./src/sample-device.ts`](https://github.com/web-infra-dev/midscene-example/blob/main/custom-interface/src/sample-device.ts) file. Let's take a glance at it.

```typescript
import type { DeviceAction, Size } from '@midscene/core';
import { getMidsceneLocationSchema, z } from '@midscene/core';
import {
  type AbstractInterface,
  defineAction,
  defineActionTap,
  defineActionInput,
  // ... other action imports
} from '@midscene/core/device';

export interface SampleDeviceConfig {
  deviceName?: string;
  width?: number;
  height?: number;
}

/**
 * SampleDevice - A template implementation of AbstractInterface
 */
export class SampleDevice implements AbstractInterface {
  interfaceType = 'sample-device';
  private config: Required<SampleDeviceConfig>;

  constructor(config: SampleDeviceConfig = {}) {
    this.config = {
      deviceName: config.deviceName || 'Sample Device',
      width: config.width || 1920,
      height: config.height || 1080,
    };
  }

  /**
   * Required: Take a screenshot and return base64 string
   */
  async screenshotBase64(): Promise<string> {
    // TODO: Implement actual screenshot capture
    console.log('📸 Taking screenshot...');
    return 'data:image/png;base64,...'; // Your screenshot implementation
  }

  /**
   * Required: Get interface dimensions
   * The width and height here refer to the logical size of the interface, not considering the device pixel ratio (dpr). The coordinates obtained from actions like defineActionTap are also based on this logical coordinate system. You can convert logical coordinates to physical coordinates in your action implementations if needed.
   */
  async size(): Promise<Size> {
    return {
      width: this.config.width,
      height: this.config.height,
    };
  }

  /**
   * Required: Define available actions for AI model
   */
  actionSpace(): DeviceAction[] {
    return [
      // Basic tap action
      defineActionTap(async (param) => {
        // TODO: Implement tap at param.locate.center coordinates
        await this.performTap(param.locate.center[0], param.locate.center[1]);
      }),

      // Text input action  
      defineActionInput(async (param) => {
        // TODO: Implement text input
        await this.performInput(param.locate.center[0], param.locate.center[1], param.value);
      }),

      // Custom action example
      defineAction({
        name: 'CustomAction',
        description: 'Your custom device-specific action',
        paramSchema: z.object({
          locate: getMidsceneLocationSchema(),
          // ... custom parameters
        }),
        call: async (param) => {
          // TODO: Implement custom action
        },
      }),
    ];
  }

  async destroy(): Promise<void> {
    // TODO: Cleanup resources
  }

  // Private implementation methods
  private async performTap(x: number, y: number): Promise<void> {
    // TODO: Your actual tap implementation
  }

  private async performInput(x: number, y: number, text: string): Promise<void> {
    // TODO: Your actual input implementation  
  }
}
```

The key methods that you need to implement are:
- `screenshotBase64()`, `size()`: help the AI model to get the context of the interface
- `actionSpace()`: an array of `DeviceAction` objects defining the actions that can be performed on the interface. AI model will use these actions to perform the actions. Midscene has provided a set of predefined action spaces for the most common interfaces and devices. And there is also a method to define any custom action.

Use these commands to run the agent:

- `npm run build` to rebuild the agent
- `npm run demo` to run the agent with javascript
- `npm run demo:yaml` to run the agent with yaml script


### Step 3. Test the agent with the playground

Attach a playground server to the agent, and you can test the agent in the web browser.

```ts 
import 'dotenv/config'; // read Midscene environment variables from .env file
import { playgroundForAgent } from '@midscene/playground';

const sleep = (ms) => new Promise((r) => setTimeout(r, ms));

// instantiate device and agent
const device = new SampleDevice();
await device.launch();
const agent = new Agent(device);

// launch playground
const server = await playgroundForAgent(agent).launch();

// close playground
await sleep(10 * 60 * 1000);
await server.close();
console.log('Playground closed!');
```

### Step 4. Test the MCP server

(still in progress)

### Step 5. Release the npm package, and let your users use it

The agent and interface class have been exported in `./index.ts` file. Now you can publish it to npm.

Fill the `name` and `version` in the `package.json` file, and then run the following command:

```bash
npm publish
```

A typical usage of your npm package is like this:

```typescript
import 'dotenv/config'; // read Midscene environment variables from .env file
import { playgroundForAgent } from '@midscene/playground';

const sleep = (ms) => new Promise((r) => setTimeout(r, ms));

// instantiate device and agent
const device = new SampleDevice();
await device.launch();
const agent = new Agent(device);

await agent.aiAct('click the button');
```

### Step 6. Invoke your class in Midscene CLI and YAML script

Write a yaml script with the `interface` section to invoke your class.

```yaml
interface:
  module: 'my-pkg-name'
  # export: 'MyDeviceClass' # use this if this is a named export

config: 
  output: './data.json'
```

This config works same as this:
```typescript
import MyDeviceClass from 'my-pkg-name';
const device = new MyDeviceClass();
const agent = new Agent(device, {
  output: './data.json',
});
```

Other fields in the yaml script are the same as the [yaml script](./automate-with-scripts-in-yaml.html).

## API reference

### Agent constructor

```typescript
import { Agent } from '@midscene/core/agent';
import type { AbstractInterface } from '@midscene/core';
```

Create an agent by pairing your custom `AbstractInterface` implementation with the standard constructor:

```typescript
const device = new SampleDevice({
  deviceName: 'Demo Panel',
  width: 1920,
  height: 1080,
});

const agent = new Agent(device, {
  actionContext: 'Dismiss pop-ups before every task.',
  generateReport: true,
  customActions: [], // optional extra DeviceAction list
});
```

- `device: AbstractInterface` (required): Any class that fulfills `screenshotBase64`, `size`, and `actionSpace`. This is where you translate Midscene actions into real I/O calls for your hardware or desktop app.
- `options?: PageAgentOpt`: Shares the same option bag as the browser and mobile agents described in the [API constructors](./api#common-parameters). Commonly used fields include `generateReport`, `reportFileName`, `actionContext`/`aiActionContext`, `cacheId`, `modelConfig`, `createOpenAIClient`, `customActions`, and `onTaskStartTip`.
- The resulting agent instantly unlocks the regular automation surfaces: `aiAct`/`aiTap` APIs, YAML runner (`interface` block), [playground](#playgroundforagent-function), MCP server, and reporting pipeline.

### `AbstractInterface` class

```typescript
import { AbstractInterface } from '@midscene/core';
```

`AbstractInterface` is the key class for the agent to control the interface. 

These are the required methods that you need to implement:

- `interfaceType: string`: define a name for the interface, this will not be provided to the AI model
- `screenshotBase64(): Promise<string>`: take a screenshot of the interface and return the base64 string with the `'data:image/` prefix
- `size(): Promise<Size>`: the size of the interface, which is an object with the `width` and `height` properties
- `actionSpace(): DeviceAction[] | Promise<DeviceAction[]>`: the action space of the interface, which is an array of `DeviceAction` objects. Use predefined actions or define any custom action.

Type signatures:

```ts
import type { DeviceAction, Size, UIContext } from '@midscene/core';
import type { ElementNode } from '@midscene/shared/extractor';

abstract class AbstractInterface {
  // Required
  abstract interfaceType: string;
  abstract screenshotBase64(): Promise<string>;
  abstract size(): Promise<Size>;
  abstract actionSpace(): DeviceAction[] | Promise<DeviceAction[]>;

  // Optional lifecycle/hooks
  abstract destroy?(): Promise<void>;
  abstract describe?(): string;
  abstract beforeInvokeAction?(actionName: string, param: any): Promise<void>;
  abstract afterInvokeAction?(actionName: string, param: any): Promise<void>;
}
```

These are the optional methods that you can implement:

- `destroy?(): Promise<void>`: destroy the interface
- `describe?(): string`: describe the interface, this may be used for the report and the playground. But it will not be provided to the AI model.
- `beforeInvokeAction?(actionName: string, param: any): Promise<void>`: a hook function before invoking an action in action space
- `afterInvokeAction?(actionName: string, param: any): Promise<void>`: a hook function after invoking an action

### The action space

Action space is the set of actions that can be performed on the interface. AI model will use these actions to perform the actions. All the descriptions and parameter schemas of the actions will be provided to the AI model.

To help you easily define the action space, Midscene has provided a set of predefined action spaces for the most common interfaces and devices. And there is also a method to define any custom action.

This is how you can import the utils to define the action space:

```typescript
import {
	type ActionTapParam,
	defineAction,
	defineActionTap,
} from "@midscene/core/device";
```

#### The predefined actions

These are the predefined action spaces for the most common interfaces and devices. You can expose them to the customized interface by implementing the call method of the action.

You can find the parameters of the actions in the type definition of these functions.

* `defineActionTap()`: define the tap action. This is also the function to invoke for the `aiTap` method.
* `defineActionDoubleClick()`: define the double click action
* `defineActionInput()`: define the input action. This is also the function to invoke for the `aiInput` method. This is also the function to invoke for the `aiInput` method.
* `defineActionKeyboardPress()`: define the keyboard press action. This is also the function to invoke for the `aiKeyboardPress` method.
* `defineActionScroll()`: define the scroll action. This is also the function to invoke for the `aiScroll` method.
* `defineActionDragAndDrop()`: define the drag and drop action
* `defineActionLongPress()`: define the long press action
* `defineActionSwipe()`: define the swipe action

#### Define a custom action

You can define your own action by using the `defineAction()` function. You can also use this method to define more actions for the [PuppeteerAgent](./integrate-with-puppeteer), [AgentOverChromeBridge](./bridge-mode#constructor), and [AndroidAgent](./android-getting-started).

API Signature:

```typescript
import { defineAction } from "@midscene/core/device";

defineAction(
  {
    name: string,
    description: string,
    paramSchema: z.ZodType<T>;
    call: (param: z.infer<z.ZodType<T>>) => Promise<void>;
  }
)
```

* `name`: the name of the action, AI model will use this name to invoke the action
* `description`: the description of the action, AI model will use this description to understand what the action is doing. For complex actions, you can provide a more detailed example here.
* `paramSchema`: the [Zod](https://www.npmjs.com/package/zod) schema of the parameters of the action, AI model will help to fill the parameters according to this schema
* `call`: the function to invoke the action, you can get the parameters from the `param` parameter which conforms to the `paramSchema`


Example:

```typescript
defineAction({
  name: 'MyAction',
  description: 'My action',
  paramSchema: z.object({
    name: z.string(),
  }),
  call: async (param) => {
    console.log(param.name);
  },
});
```

If you want to get a param about the location of some element, you can use the `getMidsceneLocationSchema()` function to get the specific zod schema.

A more complex example about defining a custom action:

```typescript
import { getMidsceneLocationSchema } from "@midscene/core/device";

defineAction({
  name: 'LaunchApp',
  description: 'A an app on screen',
  paramSchema: z.object({
    name: z.string().describe('The name of the app to launch'),
    locate: getMidsceneLocationSchema().describe('The app icon to be launched'),
  }),
  call: async (param) => {
    console.log(`launching app: ${param.name}, ui located at: ${JSON.stringify(param.locate.center)}`);
  },
});
```

### `playgroundForAgent` function

```typescript
import { playgroundForAgent } from '@midscene/playground';
```

The `playgroundForAgent` function creates a playground launcher for a specific Agent, allowing you to test and debug your custom interface Agent in a web browser.

#### Function signature

```typescript
function playgroundForAgent(agent: Agent): {
  launch(options?: LaunchPlaygroundOptions): Promise<LaunchPlaygroundResult>
}
```

#### Parameters

- `agent: Agent`: The Agent instance to launch the playground for

#### Return value

Returns an object containing a `launch` method.

#### `launch` method options

```typescript
interface LaunchPlaygroundOptions {
  /**
   * Port to start the playground server on
   * @default 5800
   */
  port?: number;

  /**
   * Whether to automatically open the playground in browser
   * @default true
   */
  openBrowser?: boolean;

  /**
   * Custom browser command to open playground
   * @default 'open' on macOS, 'start' on Windows, 'xdg-open' on Linux
   */
  browserCommand?: string;

  /**
   * Whether to show server logs
   * @default true
   */
  verbose?: boolean;

  /**
   * Unique identifier for the playground server instance
   * Same ID shares playground chat history
   * @default undefined (generates random UUID)
   */
  id?: string;
}
```

#### `launch` method return value

```typescript
interface LaunchPlaygroundResult {
  /**
   * The playground server instance
   */
  server: PlaygroundServer;

  /**
   * The server port
   */
  port: number;

  /**
   * The server host
   */
  host: string;

  /**
   * Function to close the playground
   */
  close: () => Promise<void>;
}
```

#### Usage example

```typescript
import 'dotenv/config';
import { playgroundForAgent } from '@midscene/playground';
import { SampleDevice } from './sample-device';
import { Agent } from '@midscene/core/agent';

const sleep = (ms) => new Promise((r) => setTimeout(r, ms));

// Create device and agent instances
const device = new SampleDevice();
const agent = new Agent(device);

// Launch playground
const result = await playgroundForAgent(agent).launch({
  port: 5800,
  openBrowser: true,
  verbose: true
});

console.log(`Playground started: http://${result.host}:${result.port}`);

// Close playground when needed
await sleep(10 * 60 * 1000); // Wait 10 minutes
await result.close();
console.log('Playground closed!');
```

## FAQ

**My interface-controller is general-purpose; can it be included in this document?**

Yes, we are happy to gather creative projects and list them in this document.

[Raise an issue](https://github.com/web-infra-dev/midscene/issues) to us when it's ready.



---
url: /integrate-with-playwright.md
---



import { PackageManagerTabs } from '@theme';

# Integrate with Playwright

[Playwright.js](https://playwright.com/) is an open-source automation library developed by Microsoft, mainly used for end-to-end testing and web scraping of web applications.

There are two ways to integrate with Playwright:

- Directly integrate and call the Midscene Agent via script, suitable for quick prototyping, data scraping, and automation scripts.
- Integrate Midscene into Playwright test cases, suitable for UI testing scenarios.

## Set up API keys for model

Set your model configs into the environment variables. You may refer to [Model strategy](../model-strategy) for more details.

```bash
export MIDSCENE_MODEL_BASE_URL="https://replace-with-your-model-service-url/v1"
export MIDSCENE_MODEL_API_KEY="replace-with-your-api-key"
export MIDSCENE_MODEL_NAME="replace-with-your-model-name"
export MIDSCENE_MODEL_FAMILY="replace-with-your-model-family"
```

For more configuration details, please refer to [Model strategy](../model-strategy) and [Model configuration](../model-config).


## Direct integration with Midscene agent

:::info Example Project

You can find an example project of direct Playwright integration here: [https://github.com/web-infra-dev/midscene-example/blob/main/playwright-demo](https://github.com/web-infra-dev/midscene-example/blob/main/playwright-demo)

:::

### Step 1: Install dependencies

<PackageManagerTabs command="install @midscene/web playwright @playwright/test tsx --save-dev" />

### Step 2: Write the script

Save the following code as `./demo.ts`:

```typescript
import { chromium } from 'playwright';
import { PlaywrightAgent } from '@midscene/web/playwright';
import 'dotenv/config'; // read environment variables from .env file

const sleep = (ms) => new Promise((r) => setTimeout(r, ms));

Promise.resolve(
  (async () => {
    const browser = await chromium.launch({
      headless: true, // 'true' means we can't see the browser window
      args: ['--no-sandbox', '--disable-setuid-sandbox'],
    });

    const page = await browser.newPage();
    await page.setViewportSize({
      width: 1280,
      height: 768,
    });
    await page.goto('https://www.ebay.com');
    await sleep(5000); // 👀 init Midscene agent
    const agent = new PlaywrightAgent(page);

    // 👀 type keywords, perform a search
    await agent.aiAct('type "Headphones" in search box, hit Enter');

    // 👀 wait for the loading
    await agent.aiWaitFor('there is at least one headphone item on page');
    // or you may use a plain sleep:
    // await sleep(5000);

    // 👀 understand the page content, find the items
    const items = await agent.aiQuery(
      '{itemTitle: string, price: Number}[], find item in list and corresponding price',
    );
    console.log('headphones in stock', items);

    const isMoreThan1000 = await agent.aiBoolean(
      'Is the price of the headphones more than 1000?',
    );
    console.log('isMoreThan1000', isMoreThan1000);

    const price = await agent.aiNumber(
      'What is the price of the first headphone?',
    );
    console.log('price', price);

    const name = await agent.aiString(
      'What is the name of the first headphone?',
    );
    console.log('name', name);

    const location = await agent.aiLocate(
      'What is the location of the first headphone?',
    );
    console.log('location', location);

    // 👀 assert by AI
    await agent.aiAssert('There is a category filter on the left');

    // 👀 click on the first item
    await agent.aiTap('the first item in the list');

    await browser.close();
  })(),
);
```

For more Agent API details, please refer to [API Reference](./api#interaction-methods).

### Step 3: Run the script

Use `tsx` to run, and you will see the product information printed in the terminal:

```bash
# run
npx tsx demo.ts

# The terminal should output something like:
#  [
#   {
#     itemTitle: 'JBL Tour Pro 2 - True wireless Noise Cancelling earbuds with Smart Charging Case',
#     price: 551.21
#   },
#   {
#     itemTitle: 'Soundcore Space One Wireless Headphones 40H ANC Playtime 2XStronger Voice',
#     price: 543.94
#   }
# ]
```

### Step 4: View the run report

After the above command executes successfully, it will output: `Midscene - report file updated: /path/to/report/some_id.html`. Open this file in your browser to view the report.

## Integration in Playwright test cases

Here we assume you already have a repository with Playwright integration.

:::info Example Project

You can find an example project of Playwright test integration here: [https://github.com/web-infra-dev/midscene-example/blob/main/playwright-testing-demo](https://github.com/web-infra-dev/midscene-example/blob/main/playwright-testing-demo)

:::

### Step 1: Add dependencies and update configuration

Add dependencies

<PackageManagerTabs command="install @midscene/web --save-dev" />

Update playwright.config.ts

```diff
export default defineConfig({
  testDir: './e2e',
+ timeout: 90 * 1000,
+ reporter: [["list"], ["@midscene/web/playwright-reporter", { type: "merged" }]], // type optional, default is "merged", means multiple test cases generate one report, optional value is "separate", means one report for each test case
});
```

Reporter configuration options:
- `type`: Report mode, can be `merged` (default) or `separate`. `merged` means multiple test cases generate one merged report, `separate` means each test case generates its own report.
- `outputFormat`: Controls how the report is generated. `'single-html'` (default) embeds all screenshots as base64 in a single HTML file. `'html-and-external-assets'` saves screenshots as separate PNG files in a subdirectory, useful when report files are too large. **Note**: When using `'html-and-external-assets'`, reports must be served via HTTP server and cannot be opened directly using `file://` protocol (because browser CORS restrictions block loading local images via relative paths from the file protocol). Navigate to the report directory and run one of the following commands:
  - Using Node.js: `npx serve`
  - Using Python: `python -m http.server` or `python3 -m http.server`

  Then access the report via `http://localhost:3000` (or the port shown in the terminal).

### Step 2: Extend the `test` instance

Save the following code as `./e2e/fixture.ts`:

```typescript
import { test as base } from '@playwright/test';
import type { PlayWrightAiFixtureType } from '@midscene/web/playwright';
import { PlaywrightAiFixture } from '@midscene/web/playwright';

export const test = base.extend<PlayWrightAiFixtureType>(
  PlaywrightAiFixture({
    waitForNetworkIdleTimeout: 2000, // optional, the timeout for waiting for network idle between each action, default is 2000ms
  }),
);
```

### Step 3: Write test cases

Review the full catalog of action, query, and utility methods in the [Agent API reference](./api#interaction-methods). When you need lower-level control, you can use `agentForPage` to obtain the underlying `PageAgent` instance and call any API directly:

```typescript
test('case demo', async ({ agentForPage, page }) => {
  const agent = await agentForPage(page);

  await agent.recordToReport();
  const logContent = agent._unstableLogContent();
  console.log(logContent);
});
```

#### Example code

```typescript title="./e2e/ebay-search.spec.ts"
import { expect } from '@playwright/test';
import { test } from './fixture';

test.beforeEach(async ({ page }) => {
  page.setViewportSize({ width: 400, height: 905 });
  await page.goto('https://www.ebay.com');
  await page.waitForLoadState('networkidle');
});

test('search headphone on ebay', async ({
  ai,
  aiQuery,
  aiAssert,
  aiInput,
  aiTap,
  aiScroll,
  aiWaitFor,
  aiRightClick,
  recordToReport,
}) => {
  // Use aiInput to enter search keyword
  await aiInput('Headphones', 'search box');

  // Use aiTap to click search button
  await aiTap('search button');

  // Wait for search results to load
  await aiWaitFor('search results list loaded', { timeoutMs: 5000 });

  // Use aiScroll to scroll to bottom
  await aiScroll(
    {
      scrollType: 'untilBottom',
    },
    'search results list',
  );

  // Use aiQuery to get product information
  const items = await aiQuery<Array<{ title: string; price: number }>>(
    'get product titles and prices from search results',
  );

  console.log('headphones in stock', items);
  expect(items?.length).toBeGreaterThan(0);

  // Use aiAssert to verify filter functionality
  await aiAssert('category filter exists on the left side');

  // Use recordToReport to capture the current state
  await recordToReport('Search Results', {
    content: 'Final search results for headphones',
  });
});
```

For more Agent API details, please refer to [API Reference](./api#interaction-methods).

### Step 4. Run test cases

```bash
npx playwright test ./e2e/ebay-search.spec.ts
```

### Step 5. View test report

After the command executes successfully, it will output: `Midscene - report file updated: ./current_cwd/midscene_run/report/some_id.html`. Open this file in your browser to view the report.

## Advanced

### About opening in a new tab

Each Agent instance is bound to a single page. To make debugging easier, Midscene intercepts new tabs by default (for example, links with `target="_blank"`) and opens them in the current page.

If you want to restore opening in a new tab, set `forceSameTabNavigation` to `false`—but you’ll need to create a new Agent instance for each new tab.

```typescript
const mid = new PlaywrightAgent(page, {
  forceSameTabNavigation: false,
});
```

### Connect Midscene Agent to a Remote Playwright Browser

:::info Example Project

You can find an example project of remote Playwright integration here: [https://github.com/web-infra-dev/midscene-example/tree/main/remote-playwright-demo](https://github.com/web-infra-dev/midscene-example/tree/main/remote-playwright-demo)

:::

Connect to a remote Playwright browser when you already run browsers in your own infra or vendor grid. This keeps the browser close to the target environment, avoids repeated launches, and still lets Midscene drive it with the same AI APIs.

#### Prerequisites

<PackageManagerTabs command="install playwright @playwright/test @midscene/web --save-dev" />

#### Getting a CDP WebSocket URL

You can get a CDP WebSocket URL from various sources, for example:

- **BrowserBase**: Sign up at https://browserbase.com and get your CDP URL
- **Browserless**: Use https://browserless.io or run your own instance
- **Local Chrome**: Run Chrome with `--remote-debugging-port=9222` and use `ws://localhost:9222/devtools/browser/...`
- **Docker**: Run Chrome in a Docker container with debugging port exposed


#### Code example

```typescript
import { chromium } from 'playwright';
import { PlaywrightAgent } from '@midscene/web/playwright';

// CDP WebSocket URL from your remote browser service
const cdpWsUrl = 'ws://your-remote-browser.com/devtools/browser/your-session-id';

// Connect and pick a page
const browser = await chromium.connectOverCDP(cdpWsUrl);
const context = browser.contexts()[0];
const page = context.pages()[0] || await context.newPage();

// Create Midscene agent (usage matches any Playwright agent)
const agent = new PlaywrightAgent(page);

// Use AI methods as usual
await agent.aiAct('navigate to https://example.com');
await agent.aiAct('click the login button');
const result = await agent.aiQuery('get page title: {title: string}');

// Cleanup
await agent.destroy();
await browser.close();
```

Once connected, keep using `PlaywrightAgent` the same way you would with a locally launched browser.

### Provide custom actions

Use the `customActions` option to extend the agent's action space with your own actions defined via `defineAction`. When provided, these actions will be appended to the built-in ones so the agent can call them during planning.

```typescript
import { getMidsceneLocationSchema, z } from '@midscene/core';
import { defineAction } from '@midscene/core/device';

const ContinuousClick = defineAction({
  name: 'continuousClick',
  description: 'Click the same target repeatedly',
  paramSchema: z.object({
    locate: getMidsceneLocationSchema(),
    count: z
      .number()
      .int()
      .positive()
      .describe('How many times to click'),
  }),
  async call(param) {
    const { locate, count } = param;
    console.log('click target center', locate.center);
    console.log('click count', count);
    // carry out your clicking logic using locate + count
  },
});

const agent = new PlaywrightAgent(page, {
  customActions: [ContinuousClick],
});

await agent.aiAct('click the red button five times');
```

Check [Integrate with any interface](./integrate-with-any-interface#define-a-custom-action) for more details about defining custom actions.

## More

- For all the methods on the Agent, please refer to [API Reference](./api#interaction-methods).
- For the Playwright API reference, see [Playwright Agent API](./web-api-reference#playwright-agent).
- Demo projects
  - Direct integration: [https://github.com/web-infra-dev/midscene-example/blob/main/playwright-demo](https://github.com/web-infra-dev/midscene-example/blob/main/playwright-demo)
  - Playwright test integration: [https://github.com/web-infra-dev/midscene-example/blob/main/playwright-testing-demo](https://github.com/web-infra-dev/midscene-example/blob/main/playwright-testing-demo)
  - Remote Playwright integration: [https://github.com/web-infra-dev/midscene-example/tree/main/remote-playwright-demo](https://github.com/web-infra-dev/midscene-example/tree/main/remote-playwright-demo)



---
url: /integrate-with-puppeteer.md
---




# Integrate with Puppeteer

import { PackageManagerTabs } from '@theme';

[Puppeteer](https://pptr.dev/) is a Node.js library which provides a high-level API to control Chrome or Firefox over the DevTools Protocol or WebDriver BiDi. Puppeteer runs in the headless (no visible UI) by default but can be configured to run in a visible ("headful") browser.

:::info Demo Projects

you can check the demo project of Puppeteer here: [https://github.com/web-infra-dev/midscene-example/blob/main/puppeteer-demo](https://github.com/web-infra-dev/midscene-example/blob/main/puppeteer-demo)

There is also a demo of Puppeteer with Vitest: [https://github.com/web-infra-dev/midscene-example/tree/main/puppeteer-with-vitest-demo](https://github.com/web-infra-dev/midscene-example/tree/main/puppeteer-with-vitest-demo)

:::

## Set up API keys for model

Set your model configs into the environment variables. You may refer to [Model strategy](../model-strategy) for more details.

```bash
export MIDSCENE_MODEL_BASE_URL="https://replace-with-your-model-service-url/v1"
export MIDSCENE_MODEL_API_KEY="replace-with-your-api-key"
export MIDSCENE_MODEL_NAME="replace-with-your-model-name"
export MIDSCENE_MODEL_FAMILY="replace-with-your-model-family"
```

For more configuration details, please refer to [Model strategy](../model-strategy) and [Model configuration](../model-config).


## Integration with Midscene Agent

### Step 1. Install dependencies

<PackageManagerTabs command="install @midscene/web puppeteer tsx --save-dev" />

### Step 2. Write scripts

Write and save the following code as `./demo.ts`.

```typescript title="./demo.ts"
import puppeteer from "puppeteer";
import { PuppeteerAgent } from "@midscene/web/puppeteer";

const sleep = (ms: number) => new Promise((r) => setTimeout(r, ms));
Promise.resolve(
  (async () => {
    const browser = await puppeteer.launch({
      headless: false, // here we use headed mode to help debug
    });

    const page = await browser.newPage();
    await page.setViewport({
      width: 1280,
      height: 800,
      deviceScaleFactor: 1,
    });

    await page.goto("https://www.ebay.com");
    await sleep(5000);

    // 👀 init Midscene agent
    const agent = new PuppeteerAgent(page);

    // 👀 type keywords, perform a search
    await agent.aiAct('type "Headphones" in search box, hit Enter');
    await sleep(5000);

    // 👀 understand the page content, find the items
    const items = await agent.aiQuery(
      "{itemTitle: string, price: Number}[], find item in list and corresponding price"
    );
    console.log("headphones in stock", items);

    // 👀 assert by AI
    await agent.aiAssert("There is a category filter on the left");

    await browser.close();
  })()
);
```


### Step 3. Run

Using `tsx` to run, you will get the data of Headphones on eBay:

```bash
# run
npx tsx demo.ts

# it should print 
#  [
#   {
#     itemTitle: 'Beats by Dr. Dre Studio Buds Totally Wireless Noise Cancelling In Ear + OPEN BOX',
#     price: 505.15
#   },
#   {
#     itemTitle: 'Skullcandy Indy Truly Wireless Earbuds-Headphones Green Mint',
#     price: 186.69
#   }
# ]
```

For the complete catalog of agent methods, see the [API reference](./api#interaction-methods).

### Step 4: View the report

After the above command executes successfully, the console will output: `Midscene - report file updated: /path/to/report/some_id.html`. You can open this file in a browser to view the report.

<a id="puppeteeragent"></a>

## Advanced

### About opening in a new tab

Each Agent instance is bound to a single page. For easier debugging, Midscene intercepts new tabs by default (for example, links with `target="_blank"`) and opens them in the current page.

If you want to allow new tabs again, set `forceSameTabNavigation` to `false`—but you must create a new Agent instance for each new tab.

```typescript
const mid = new PuppeteerAgent(page, {
  forceSameTabNavigation: false,
});
```

### Connect Midscene Agent to a Remote Puppeteer Browser

:::info Example Project

You can find an example project of remote Puppeteer integration here: [https://github.com/web-infra-dev/midscene-example/tree/main/remote-puppeteer-demo](https://github.com/web-infra-dev/midscene-example/tree/main/remote-puppeteer-demo)

:::

Use this approach when you want to reuse a browser that already runs inside your own infrastructure—such as a persistent cloud worker, a third-party browser grid, or an on-prem desktop. Wiring Midscene into that remote Puppeteer instance keeps the browser close to the target environment, cuts repeated startup costs, and lets you centralize management while keeping the same AI automation APIs.

In practice you manually:
1. Obtain a CDP WebSocket URL from the remote browser service
2. Use Puppeteer to connect to the remote browser
3. Create a Midscene agent for AI-driven automation

#### Prerequisites

<PackageManagerTabs command="install puppeteer @midscene/web --save-dev" />

#### Getting a CDP WebSocket URL

You can get a CDP WebSocket URL from various sources, for example:

- **BrowserBase**: Sign up at https://browserbase.com and get your CDP URL
- **Browserless**: Use https://browserless.io or run your own instance
- **Local Chrome**: Run Chrome with `--remote-debugging-port=9222` and use `ws://localhost:9222/devtools/browser/...`
- **Docker**: Run Chrome in a Docker container with debugging port exposed


#### Basic Example

```typescript
import puppeteer from 'puppeteer';
import { PuppeteerAgent } from '@midscene/web/puppeteer';

// Assuming you already have a CDP WebSocket URL
const cdpWsUrl = 'ws://your-remote-browser.com/devtools/browser/your-session-id';

// Connect to remote browser
const browser = await puppeteer.connect({
  browserWSEndpoint: cdpWsUrl
});

// Get or create page
const pages = await browser.pages();
const page = pages[0] || await browser.newPage();

// Create Midscene agent
const agent = new PuppeteerAgent(page);

// Use AI methods
await agent.aiAct('navigate to https://example.com');
await agent.aiAct('click the login button');
const result = await agent.aiQuery('get page title: {title: string}');

// Cleanup
await agent.destroy();
await browser.disconnect();
```

### Provide custom actions

Use the `customActions` option to extend the agent's action space with your own actions defined via `defineAction`. When provided, these actions will be appended to the built-in ones so the agent can call them during planning.

```typescript
import { getMidsceneLocationSchema, z } from '@midscene/core';
import { defineAction } from '@midscene/core/device';

const ContinuousClick = defineAction({
  name: 'continuousClick',
  description: 'Click the same target repeatedly',
  paramSchema: z.object({
    locate: getMidsceneLocationSchema(),
    count: z
      .number()
      .int()
      .positive()
      .describe('How many times to click'),
  }),
  async call(param) {
    const { locate, count } = param;
    console.log('click target center', locate.center);
    console.log('click count', count);
    // carry out your clicking logic using locate + count
  },
});

const agent = new PuppeteerAgent(page, {
  customActions: [ContinuousClick],
});

await agent.aiAct('click the red button five times');
```

Check [Integrate with any interface](./integrate-with-any-interface#define-a-custom-action) for more details about defining custom actions.

## More

- For every Agent method, check the [API Reference](./api#interaction-methods).
- For the Puppeteer API reference, see [Puppeteer Agent API](./web-api-reference#puppeteer-agent).
- Demo projects
  - Puppeteer demo: [https://github.com/web-infra-dev/midscene-example/blob/main/puppeteer-demo](https://github.com/web-infra-dev/midscene-example/blob/main/puppeteer-demo)
  - Puppeteer + Vitest demo: [https://github.com/web-infra-dev/midscene-example/tree/main/puppeteer-with-vitest-demo](https://github.com/web-infra-dev/midscene-example/tree/main/puppeteer-with-vitest-demo)



---
url: /introduction.md
---

# Midscene.js - Vision-Driven Automation by AI

Driving all platforms UI automation with vision-based model

## 📣 v1.0 Release Announcement

> **We have released v1.0.** It is currently published on npm.  
> For the latest documentation and code, please visit [https://midscenejs.com/](https://midscenejs.com/) and the `main` branch.  
> For historical documentation, please visit [https://v0.midscenejs.com/](https://v0.midscenejs.com/).  
> v1.0 changelog: [https://midscenejs.com/changelog](https://midscenejs.com/changelog)

## Features

### Write Automation with Natural Language
- Describe your goals and steps, and Midscene will plan and operate the user interface for you.
- Use Javascript SDK or YAML to write your automation script.

### Web & Mobile App & Any Interface
- **Web Automation**: Either [integrate with Puppeteer](./integrate-with-puppeteer), [with Playwright](./integrate-with-playwright) or use [Bridge Mode](./bridge-mode) to control your desktop browser.
- **Android Automation**: Use [Javascript SDK](./android-getting-started) with adb to control your local Android device.
- **iOS Automation**: Use [Javascript SDK](./ios-getting-started) with WebDriverAgent to control your local iOS devices and simulators.
- **Any Interface Automation**: Use [Javascript SDK](./integrate-with-any-interface) to control your own interface.

### For Developers
- **Three kinds of APIs**:
  - [Interaction API](./api#interaction-methods): interact with the user interface.
  - [Data Extraction API](./api#data-extraction): extract data from the user interface and dom.
  - [Utility API](./api#more-apis): utility functions like `aiAssert()`, `aiLocate()`, `aiWaitFor()`.
- **MCP**: Midscene provides MCP services that expose atomic Midscene Agent actions as MCP tools so upper-layer agents can inspect and operate UIs with natural language. [Docs](./mcp)
- [**Caching for Efficiency**](./caching): Replay your script with cache and get the result faster.
- **Debugging Experience**: Midscene.js offers a visualized replay back report file, a built-in playground, and a Chrome Extension to simplify the debugging process. These are the tools most developers truly need.


## Showcases

Register the GitHub form autonomously in a web browser and pass all field validations:

<video src="https://lf3-static.bytednsdoc.com/obj/eden-cn/nupipfups/Midscene/1.0-showcases/github2.mp4" height="300" controls />

Plus these real-world showcases:
* [iOS Automation - Meituan coffee order](./showcases#ios)
* [iOS Automation - Auto-like the first @midscene_ai tweet](./showcases#ios)
* [Android Automation - DCar: Xiaomi SU7 specs](./showcases#android)
* [Android Automation - Booking a Tokyo hotel for Christmas](./showcases#android)
* [MCP Integration - Midscene MCP UI prepatch release](./showcases#mcp)


## Zero-code quick experience

- **[Chrome Extension](./quick-experience)**: Start in-browser experience immediately through [the Chrome Extension](./quick-experience), without writing any code.
- **[Android Playground](./android-getting-started#try-playground-no-code)**: There is also a built-in Android playground to control your local Android device.
- **[iOS Playground](./ios-getting-started#try-playground)**: There is also a built-in iOS playground to control your local iOS device.

## Driven by Visual Language Model

Midscene.js is all-in on the pure-vision route for UI actions: element localization and interactions are based on screenshots only. It supports visual-language models like `Qwen3-VL`, `Doubao-1.6-vision`, `gemini-3-flash`, and `UI-TARS`. For data extraction and page understanding, you can still opt in to include DOM when needed.

* Pure-vision localization for UI actions; the DOM extraction mode is removed.
* Works across web, mobile, desktop, and even `<canvas>` surfaces.
* Far fewer tokens by skipping DOM for actions, which cuts cost and speeds up runs.
* DOM can still be included for data extraction and page understanding when needed.
* Strong open-source options for self-hosting.

Read more about [Model Strategy](./model-strategy)

## Two styles of automation

### Auto planning

AI autonomously plans and executes the flow to complete the task.

```javascript
await aiAct('click all the records one by one. If one record contains the text "completed", skip it');
```

### Workflow style

Split complex logic into multiple steps to improve the stability of the automation code.

```javascript
const recordList = await agent.aiQuery('string[], the record list')
for (const record of recordList) {
  const hasCompleted = await agent.aiBoolean(`check if the record ${record}" contains the text "completed"`)
  if (!hasCompleted) {
    await agent.aiTap(record)
  }
}
```

> For more details about the workflow style, please refer to [Use JavaScript to Optimize the AI Automation Code](./use-javascript-to-optimize-ai-automation-code)

## Resources

* Home Page and Documentation: [https://midscenejs.com](https://midscenejs.com/)
* Sample Projects: [https://github.com/web-infra-dev/midscene-example](https://github.com/web-infra-dev/midscene-example)
* API Reference: [https://midscenejs.com/api.html](./api)
* GitHub: [https://github.com/web-infra-dev/midscene](https://github.com/web-infra-dev/midscene)

## Community

* [Discord](https://discord.gg/2JyBHxszE4)
* [Follow us on X](https://x.com/midscene_ai)
* [Lark Group(飞书交流群)](https://applink.larkoffice.com/client/chat/chatter/add_by_link?link_token=693v0991-a6bb-4b44-b2e1-365ca0d199ba)


## Credits

We would like to thank the following projects:

- [Rsbuild](https://github.com/web-infra-dev/rsbuild) and [Rslib](https://github.com/web-infra-dev/rslib) for the build tool.
- [UI-TARS](https://github.com/bytedance/ui-tars) for the open-source agent model UI-TARS.
- [Qwen2.5-VL](https://github.com/QwenLM/Qwen2.5-VL) for the open-source VL model Qwen2.5-VL.
- [scrcpy](https://github.com/Genymobile/scrcpy) and [yume-chan](https://github.com/yume-chan) allow us to control Android devices with browser.
- [appium-adb](https://github.com/appium/appium-adb) for the javascript bridge of adb.
- [appium-webdriveragent](https://github.com/appium/WebDriverAgent) for operating XCTest with JavaScript.
- [YADB](https://github.com/ysbing/YADB) for the yadb tool which improves the performance of text input.
- [libnut-core](https://github.com/nut-tree/libnut-core) for the cross-platform native keyboard and mouse control.
- [Puppeteer](https://github.com/puppeteer/puppeteer) for browser automation and control.
- [Playwright](https://github.com/microsoft/playwright) for browser automation and control and testing.

## License

Midscene.js is [MIT licensed](https://github.com/web-infra-dev/midscene/blob/main/LICENSE).



---
url: /ios-api-reference.md
---

# API reference (iOS)

Use this doc when you need to customize iOS device behavior, wire Midscene into WebDriverAgent-driven workflows, or troubleshoot WDA requests. For shared constructor options (reporting, hooks, caching, etc.), see the platform-agnostic [API reference (Common)](./api).

## Action Space

`IOSDevice` uses the following action space; the Midscene Agent can use these actions while planning tasks:

- `Tap` &mdash; Tap an element.
- `DoubleClick` &mdash; Double-tap an element.
- `Input` &mdash; Enter text with `replace`/`append`/`clear` modes and optional `autoDismissKeyboard`.
- `Scroll` &mdash; Scroll from an element or screen center in any direction, including scroll-to-top/bottom/left/right helpers.
- `DragAndDrop` &mdash; Drag from one element to another.
- `KeyboardPress` &mdash; Press a specified key.
- `LongPress` &mdash; Long-press a target element with optional duration.
- `ClearInput` &mdash; Clear the contents of an input field.
- `Launch` &mdash; Open a URL, bundle identifier, or URL scheme.
- `RunWdaRequest` &mdash; Call WebDriverAgent REST endpoints directly.
- `IOSHomeButton` &mdash; Trigger the iOS system Home action.
- `IOSAppSwitcher` &mdash; Open the iOS multitasking view.

## IOSDevice {#iosdevice}

Create a WebDriverAgent-backed instance that an IOSAgent can drive.

### Import

```ts
import { IOSDevice } from '@midscene/ios';
```

### Constructor

```ts
const device = new IOSDevice({
  // device options...
});
```

### Device options

- `wdaPort?: number` &mdash; WebDriverAgent port. Default `8100`.
- `wdaHost?: string` &mdash; WebDriverAgent host. Default `'localhost'`.
- `autoDismissKeyboard?: boolean` &mdash; Hide the keyboard after text input. Default `true`.
- `customActions?: DeviceAction<any>[]` &mdash; Additional device actions exposed to the agent.

### Usage notes

- Ensure Developer Mode is enabled and WDA can reach the device; use `iproxy` when forwarding ports from a real device.
- Use `wdaHost`/`wdaPort` to target remote devices or custom WDA deployments.
- For shared interaction methods, see [API reference (Common)](./api#interaction-methods).

### Examples

#### Quick start

```ts
import { IOSAgent, IOSDevice } from '@midscene/ios';

const device = new IOSDevice({ wdaHost: 'localhost', wdaPort: 8100 });
await device.connect();

const agent = new IOSAgent(device, {
  aiActionContext: 'If any permission dialog appears, accept it.',
});

await agent.launch('https://ebay.com');
await agent.aiAct('Search for "Headphones"');
const items = await agent.aiQuery(
  '{itemTitle: string, price: Number}[], list headphone products',
);
console.log(items);
```

#### Custom host and port

```ts
const device = new IOSDevice({
  wdaHost: '192.168.1.100',
  wdaPort: 8300,
});
await device.connect();
```

## IOSAgent {#iosagent}

Wire Midscene's AI planner to an IOSDevice for UI automation over WebDriverAgent.

### Import

```ts
import { IOSAgent } from '@midscene/ios';
```

### Constructor

```ts
const agent = new IOSAgent(device, {
  // common agent options...
});
```

### iOS-specific options

- `customActions?: DeviceAction<any>[]` &mdash; Extend planning with actions defined via `defineAction`.
- `appNameMapping?: Record<string, string>` &mdash; Map friendly app names to bundle identifiers. When you pass an app name to `launch(target)`, the agent will look up the bundle ID in this mapping. If no mapping is found, it will attempt to launch `target` as-is. User-provided mappings take precedence over default mappings.
- All other fields match [API constructors](./api#common-parameters): `generateReport`, `reportFileName`, `aiActionContext`, `modelConfig`, `cacheId`, `createOpenAIClient`, `onTaskStartTip`, and more.

### Usage notes

:::info

- Use one agent per device connection.
- iOS-only helpers such as `launch` and `runWdaRequest` are also exposed in YAML scripts. See [iOS platform-specific actions](./automate-with-scripts-in-yaml#the-ios-part).
- For shared interaction methods, see [API reference (Common)](./api#interaction-methods).

:::

### iOS-specific methods

#### `agent.launch()`

Launch a web URL, native application bundle, or custom scheme.

```ts
function launch(target: string): Promise<void>;
```

- `target: string` &mdash; Target address (web URL, Bundle Identifier, URL scheme, tel/mailto, etc.) or app name. If you pass an app name and it exists in `appNameMapping`, it will be automatically resolved to the mapped Bundle ID; otherwise, `target` will be launched as-is.

```ts
await agent.launch('https://www.apple.com');
await agent.launch('com.apple.Preferences');
await agent.launch('myapp://profile/user/123');
await agent.launch('tel:+1234567890');
```

#### `agent.runWdaRequest()`

Execute raw WebDriverAgent REST calls when you need low-level control.

```ts
function runWdaRequest(
  method: string,
  endpoint: string,
  data?: Record<string, any>,
): Promise<any>;
```

- `method: string` &mdash; HTTP verb (`GET`, `POST`, `DELETE`, etc.).
- `endpoint: string` &mdash; WebDriverAgent endpoint path.
- `data?: Record<string, any>` &mdash; Optional JSON body.

```ts
const screen = await agent.runWdaRequest('GET', '/wda/screen');
await agent.runWdaRequest('POST', '/session/test/wda/pressButton', { name: 'home' });
```

#### Navigation helpers

- `agent.home(): Promise<void>` &mdash; Return to the Home screen.
- `agent.appSwitcher(): Promise<void>` &mdash; Reveal the multitasking view.

### Helper utilities

#### `agentFromWebDriverAgent()` {#agentfromwebdriveragent}

Connect to WebDriverAgent and return a ready-to-use IOSAgent.

```ts
function agentFromWebDriverAgent(
  opts?: PageAgentOpt & IOSDeviceOpt,
): Promise<IOSAgent>;
```

- `opts?: PageAgentOpt & IOSDeviceOpt` &mdash; Combine common agent options with [`IOSDevice`](#iosdevice) settings.

```ts
import { agentFromWebDriverAgent } from '@midscene/ios';

const agent = await agentFromWebDriverAgent({
  wdaHost: 'localhost',
  wdaPort: 8100,
  aiActionContext: 'Accept permission dialogs automatically.',
});
```

### Extending custom interaction actions

Extend the Agent's action space by supplying `customActions` with handlers created via `defineAction`. These actions appear after the built-in ones and can be called during planning.

```ts
import { getMidsceneLocationSchema, z } from '@midscene/core';
import { defineAction } from '@midscene/core/device';
import { agentFromWebDriverAgent } from '@midscene/ios';

const ContinuousClick = defineAction({
  name: 'continuousClick',
  description: 'Click the same target repeatedly',
  paramSchema: z.object({
    locate: getMidsceneLocationSchema(),
    count: z.number().int().positive().describe('How many times to click'),
  }),
  async call({ locate, count }) {
    console.log('click target center', locate.center);
    console.log('click count', count);
  },
});

const agent = await agentFromWebDriverAgent({
  customActions: [ContinuousClick],
});

await agent.aiAct('Click the red button five times');
```

### See also

- [iOS getting started](./ios-getting-started) for setup and scripting steps.
- [Integrate with any interface](./integrate-with-any-interface#define-a-custom-action) for custom actions and schemas.



---
url: /ios-getting-started.md
---





import { PackageManagerTabs } from '@theme';

# iOS getting started

This guide walks you through everything required to automate an iOS device with Midscene: connect a real phone through WebDriverAgent, configure model credentials, try the no-code Playground, and run your first JavaScript script.

:::info Demo Projects

Control iOS devices with JavaScript: [https://github.com/web-infra-dev/midscene-example/blob/main/ios/javascript-sdk-demo](https://github.com/web-infra-dev/midscene-example/blob/main/ios/javascript-sdk-demo)

Integrate Vitest for testing: [https://github.com/web-infra-dev/midscene-example/tree/main/ios/vitest-demo](https://github.com/web-infra-dev/midscene-example/tree/main/ios/vitest-demo)

:::

## Set up API keys for model

Set your model configs into the environment variables. You may refer to [Model strategy](../model-strategy) for more details.

```bash
export MIDSCENE_MODEL_BASE_URL="https://replace-with-your-model-service-url/v1"
export MIDSCENE_MODEL_API_KEY="replace-with-your-api-key"
export MIDSCENE_MODEL_NAME="replace-with-your-model-name"
export MIDSCENE_MODEL_FAMILY="replace-with-your-model-family"
```

For more configuration details, please refer to [Model strategy](../model-strategy) and [Model configuration](../model-config).


## Preparation

### Install Node.js

Install [Node.js 18 or higher](https://nodejs.org/en/download/).

### Prepare API Key

Prepare an API Key for a visual language (VL) model.

You can find supported models and configurations for Midscene.js in the [Model strategy](../model-strategy) documentation.

### Prepare WebDriver Server

Before getting started, you need to set up the iOS development environment:

- macOS (required for iOS development)
- Xcode and Xcode command line tools
- iOS Simulator or real device

#### Environment Configuration

Before using Midscene iOS, you need to prepare the WebDriverAgent service.

:::note Version Requirement

WebDriverAgent version must be **>= 7.0.0**

:::

Please refer to the official documentation for setup:

- **Simulator Configuration**: [Run Prebuilt WDA](https://appium.github.io/appium-xcuitest-driver/5.12/run-prebuilt-wda/)
- **Real Device Configuration**: [Real Device Configuration](https://appium.github.io/appium-xcuitest-driver/5.12/real-device-config/)

#### Verify Environment Configuration

After completing the configuration, you can verify whether the service is working properly by accessing WebDriverAgent's status endpoint:

**Access URL**: `http://localhost:8100/status`

**Correct Response Example**:
```json
{
  "value": {
    "build": {
      "version": "10.1.1",
      "time": "Sep 24 2025 18:56:41",
      "productBundleIdentifier": "com.facebook.WebDriverAgentRunner"
    },
    "os": {
      "testmanagerdVersion": 65535,
      "name": "iOS",
      "sdkVersion": "26.0",
      "version": "26.0"
    },
    "device": "iphone",
    "ios": {
      "ip": "10.91.115.63"
    },
    "message": "WebDriverAgent is ready to accept commands",
    "state": "success",
    "ready": true
  },
  "sessionId": "BCAD9603-F714-447C-A9E6-07D58267966B"
}
```

If you can successfully access this endpoint and receive a similar JSON response as shown above, it indicates that WebDriverAgent is properly configured and running.


## Try Playground (no code)

Playground is the fastest way to validate the connection and observe AI-driven steps without writing code. It shares the same core as `@midscene/ios`, so anything that works here will behave the same once scripted.

<video src="https://lf3-static.bytednsdoc.com/obj/eden-cn/nupipfups/Midscene/1.0-showcases/x2.mp4" controls/>

1. Launch the Playground CLI:

```bash
npx --yes @midscene/ios-playground
```

2. Click the gear button to enter the configuration page and paste your API key config. Refer back to [Model configuration](./model-config) if you still need credentials.

![](/ios-set-env.png)

## Start experiencing

After configuration, you can start using Midscene right away. It provides several key operation tabs:

- **Act**: interact with the page. This is Auto Planning, corresponding to `aiAct`. For example:
```
Type “Midscene” in the search box, run the search, and open the first result
```

```
Fill out the registration form and make sure every field passes validation
```

- **Query**: extract JSON data from the interface, corresponding to `aiQuery`.

Similar methods include `aiBoolean()`, `aiNumber()`, and `aiString()` for directly extracting booleans, numbers, and strings.

```
Extract the user ID from the page and return JSON data in the { id: string } structure
```

- **Assert**: understand the page and assert; if the condition is not met, throw an error, corresponding to `aiAssert`.

```
There is a login button on the page, with a user agreement link below it
```

- **Tap**: click on an element. This is Instant Action, corresponding to `aiTap`.
```
Click the login button
```

> For the difference between Auto Planning and Instant Action, see the [API](../api.mdx) document.


## Integration with Midscene Agent

Once Playground works, move to a repeatable script with the JavaScript SDK.

### Step 1. Install dependencies

<PackageManagerTabs command="install @midscene/ios --save-dev" />

### Step 2. Write scripts

Save the following code as `./demo.ts`. It opens Safari on the device, searches eBay, and asserts the result list.

```typescript title="./demo.ts"
import {
  IOSAgent,
  IOSDevice,
  agentFromWebDriverAgent,
} from '@midscene/ios';

const sleep = (ms) => new Promise((r) => setTimeout(r, ms));
Promise.resolve(
  (async () => {
    // Method 1: Create device and agent directly
    const page = new IOSDevice({
      wdaPort: 8100,
      wdaHost: 'localhost',
    });

    // 👀 Initialize Midscene agent
    const agent = new IOSAgent(page, {
      aiActionContext:
        'If any location, permission, user agreement, etc. popup appears, click agree. If login page appears, close it.',
    });
    await page.connect();

    // Method 2: Or use convenience function (recommended)
    // const agent = await agentFromWebDriverAgent({
    //   wdaPort: 8100,
    //   wdaHost: 'localhost',
    //   aiActionContext: 'If any location, permission, user agreement, etc. popup appears, click agree. If login page appears, close it.',
    // });

    // 👀 Directly open ebay.com webpage (recommended approach)
    await page.launch('https://ebay.com');
    await sleep(3000);

    // 👀 Enter keywords and perform search
    await agent.aiAct('Search for "Headphones"');

    // 👀 Wait for loading to complete
    await agent.aiWaitFor('At least one headphone product is displayed on the page');
    // Or you can use a simple sleep:
    // await sleep(5000);

    // 👀 Understand page content and extract data
    const items = await agent.aiQuery(
      '{itemTitle: string, price: Number}[], find product titles and prices in the list',
    );
    console.log('Headphone product information', items);

    // 👀 Use AI assertion
    await agent.aiAssert('Multiple headphone products are displayed on the interface');

    await page.destroy();
  })(),
);
```

### Step 3. Run

```bash
npx tsx demo.ts
```

### Step 4: View the report

Successful runs print `Midscene - report file updated: /path/to/report/some_id.html`. Open the generated HTML file in a browser to replay every interaction, query, and assertion.

## API reference and more resources

Looking for constructors, helper methods, and platform-only device APIs? See the dedicated [iOS API reference](./ios-api-reference) for detailed parameter lists plus advanced topics like custom actions. For API surfaces shared across platforms, head to the [common API reference](./api).

## FAQ

### Why can't I control my device through WebDriverAgent even though it's connected?

Please check the following:

1. **Developer Mode**: Ensure it's enabled in Settings > Privacy & Security > Developer Mode
2. **UI Automation**: Ensure it's enabled in Settings > Developer > UI Automation
3. **Device Trust**: Ensure the device trusts the current Mac

### What are the differences between simulators and real devices?

| Feature | Real Device | Simulator |
|---------|-------------|-----------|
| Port Forwarding | Requires iproxy | Not required |
| Developer Mode | Must enable | Auto-enabled |
| UI Automation Settings | Must enable manually | Auto-enabled |
| Performance | Real device performance | Depends on Mac performance |
| Sensors | Real hardware | Simulated data |

### How to use custom WebDriverAgent port and host?

You can specify WebDriverAgent port and host through the `IOSDevice` constructor or `agentFromWebDriverAgent`:

```typescript
// Method 1: Using IOSDevice
const device = new IOSDevice({
  wdaPort: 8100,        // Custom port
  wdaHost: '192.168.1.100', // Custom host
});

// Method 2: Using convenience function (recommended)
const agent = await agentFromWebDriverAgent({
  wdaPort: 8100,        // Custom port
  wdaHost: '192.168.1.100', // Custom host
});
```

For remote devices, you also need to set up port forwarding accordingly:

```bash
iproxy 8100 8100 YOUR_DEVICE_ID
```

### How to get smoother live screen preview in Playground?

Playground's screen preview supports two modes:

- **Polling mode** (default): Captures screenshots one by one via the WDA screenshot API, achieving ~5-10fps.
- **Native MJPEG stream** (recommended): Proxies WDA's built-in MJPEG Server directly for higher frame rate and lower latency.

To enable the native MJPEG stream, forward the WDA MJPEG Server port (default 9100) to localhost:

```bash
# Required for real devices only (simulators don't need this)
iproxy 9100 9100 YOUR_DEVICE_ID
```

Playground automatically probes port 9100 on startup. If available, the log will show `MJPEG: streaming via native WDA MJPEG server`; otherwise it falls back to polling mode automatically.

## More

- For every Agent method, check the [API reference (Common)](./api#interaction-methods).
- For the iOS API reference, see [iOS Agent API](./ios-api-reference).
- Demo projects
  - iOS JavaScript SDK demo: [https://github.com/web-infra-dev/midscene-example/blob/main/ios/javascript-sdk-demo](https://github.com/web-infra-dev/midscene-example/blob/main/ios/javascript-sdk-demo)
  - iOS + Vitest demo: [https://github.com/web-infra-dev/midscene-example/tree/main/ios/vitest-demo](https://github.com/web-infra-dev/midscene-example/tree/main/ios/vitest-demo)



---
url: /ios-introduction.md
---



# iOS Automation Support

Midscene can drive WebDriver tools to support iOS automation.

By adapting a visual model solution, the automation process works with any app tech stack—whether built with Native, Flutter, React Native, or Lynx. Developers only need to focus on the final experience when debugging UI automation scripts.

The iOS UI automation solution comes with all the features of Midscene:

- Supports zero-code trial using Playground.
- Supports JavaScript SDK.
- Supports automation scripts in YAML format and command-line tools.
- Supports HTML reports to replay all operation paths.

## Showcases

**Prompt** : Open Twitter and auto-like the first tweet by @midscene_ai

<video src="https://lf3-static.bytednsdoc.com/obj/eden-cn/nupipfups/Midscene/1.0-showcases/x2.mp4" poster="https://lf3-static.bytednsdoc.com/obj/eden-cn/nupipfups/Midscene/1.0-showcases/x.png" height="300" controls />

View the full report of this task: [report.html](https://lf3-static.bytednsdoc.com/obj/eden-cn/nupipfups/Midscene/1.0-showcases/x.html)


See more showcases: [showcases](./showcases.mdx)

## Try with Playground

With Playground, you can experience Midscene's capabilities without writing any code.

<video src="https://lf3-static.bytednsdoc.com/obj/eden-cn/nupipfups/Midscene/1.0-showcases/x2.mp4" controls/>

See [Getting Started](./ios-getting-started#try-playground-no-code) to learn how to launch the Playground.


## Understand WebDriverAgent

WebDriver is a standard protocol established by W3C for browser automation, providing a unified API to control different browsers and applications. The WebDriver protocol defines the communication method between client and server, enabling automation tools to control various user interfaces across platforms.

Through the efforts of the Appium team and other open source communities, the industry now has many excellent libraries that convert desktop and mobile device automation operations into WebDriver protocol. These tools include:
- **Appium** - Cross-platform mobile automation framework
- **WebDriverAgent** - Service dedicated to iOS device automation
- **Selenium** - Web browser automation tool
- **WinAppDriver** - Windows application automation tool

**Midscene adapts to the WebDriver protocol**, which means developers can use AI models to perform intelligent automated operations on any device that supports WebDriver. Through this design, Midscene can not only control traditional operations like clicking and typing, but also:
- Understand interface content and context
- Execute complex multi-step operations
- Perform intelligent assertions and validations
- Extract and analyze interface data

On iOS platform, Midscene connects to iOS devices through WebDriverAgent, allowing you to control iOS apps and system using natural language descriptions.


## Next Steps

* [Use JavaScript SDK](/en/ios-getting-started)
* [Use YAML format automation scripts](/en/automate-with-scripts-in-yaml) and [command-line tools](/en/automate-with-scripts-in-yaml)



---
url: /llm-txt.md
---

# LLMs.txt documentation

How to get tools like Cursor, Windstatic, GitHub Copilot, ChatGPT, and Claude to understand Midscene.js.

We support LLMs.txt files for making the Midscene.js documentation available to large language models.

## Directory overview

The following files are available.

- [llms.txt](https://midscenejs.com/llms.txt): The main LLMs.txt file
- [llms-full.txt](https://midscenejs.com/llms-full.txt): The complete documentation for Midscene.js


## Usage

### Cursor

Use `@Docs` feature in Cursor to include the LLMs.txt files in your project.

[Read more](https://docs.cursor.com/context/@-symbols/@-docs)

### Windstatic

Reference the LLMs.txt files using `@` or in your `.windsurfrules` files.

[Read more](https://docs.windsurf.com/windsurf/getting-started#memories-and-rules)




---
url: /mcp-android.md
---



# MCP server

Midscene provides a MCP server that allows AI assistants to control Android devices, automate mobile app testing tasks.

:::info MCP Introduction
MCP ([Model Context Protocol](https://modelcontextprotocol.io/introduction)) is a standardized way for AI models to interact with external tools and capabilities. MCP servers expose a set of tools that AI models can invoke to perform various tasks. For Midscene, these tools allow AI models to connect to Android devices, launch apps, interact with UI elements, and more.
:::

## Use cases

- Execute automated testing on Android devices
- Control Android apps for UI interaction

## Setting up Midscene MCP

### Prerequisites

1. An OpenAI API key or another supported AI model provider. For more information, see [Choosing an AI Model](./model-config.mdx).
2. [Android adb](https://developer.android.com/tools/adb?hl=zh-cn) tool installed and configured
3. Android device with USB debugging enabled and connected to your computer

### Configuration

Add the Midscene MCP server to your MCP configuration, note that the `MIDSCENE_MCP_ANDROID_MODE` environment variable is required:

```json
{
  "mcpServers": {
    "mcp-midscene": {
      "command": "npx",
      "args": ["-y", "@midscene/mcp"],
      "env": {
        "MIDSCENE_MODEL_NAME": "REPLACE_WITH_YOUR_MODEL_NAME",
        "OPENAI_API_KEY": "REPLACE_WITH_YOUR_OPENAI_API_KEY",
        "MIDSCENE_MCP_ANDROID_MODE": "true",
        "MCP_SERVER_REQUEST_TIMEOUT": "800000"
      }
    }
  }
}
```

For more information about configuring AI models, see [Choosing an AI Model](./model-config.mdx).

## Available tools

Midscene MCP provides the following Android device automation tools:

| Category                         | Tool Name                     | Description                                       |
| -------------------------------- | ----------------------------- | ------------------------------------------------- |
| **Device Management**            | midscene_android_list_devices | List all connected Android devices                |
|                                  | midscene_android_connect      | Connect to a specific Android device              |
| **App Control**                  | midscene_android_launch       | Launch an app or open a webpage on Android device |
| **System Operations**            | midscene_android_back         | Press the back button on Android device           |
|                                  | midscene_android_home         | Press the home button on Android device           |
| **Page Interaction**             | midscene_aiTap                | Click on an element described in natural language |
|                                  | midscene_aiInput              | Input text into a form field or element           |
|                                  | midscene_aiKeyboardPress      | Press a specific keyboard key                     |
|                                  | midscene_aiScroll             | Scroll the page or a specific element             |
| **Verification and Observation** | midscene_aiWaitFor            | Wait for a condition to be true on the page       |
|                                  | midscene_aiAssert             | Assert that a condition is true on the page       |
|                                  | midscene_screenshot           | Take a screenshot of the current page             |

### Device management

- **midscene_android_list_devices**: List all connected Android devices available for automation

  ```
  Parameters: None
  ```

- **midscene_android_connect**: Connect to an Android device via ADB
  ```
  Parameters:
  - deviceId: (Optional) Device ID to connect to. If not provided, uses the first available device.
  - displayId: (Optional) Display ID for multi-display Android devices (e.g., 0, 1, 2). When specified, all ADB input operations will target this specific display.
  - alwaysRefreshScreenInfo: (Optional) Whether to always fetch screen size and orientation from the device on each call. Defaults to false (uses cache for better performance). Set to true if the device may rotate or you need real-time screen information.
  ```

### App control

- **midscene_android_launch**: Launch an app or navigate to a URL on Android device
  ```
  Parameters:
  - uri: Package name, activity name, or URL to launch
  ```

### System operations

- **midscene_android_back**: Press the back button on Android device

  ```
  Parameters: None
  ```

- **midscene_android_home**: Press the home button on Android device
  ```
  Parameters: None
  ```

### Page interaction

- **midscene_aiTap**: Click on an element described in natural language

  ```
  Parameters:
  - locate: Natural language description of the element to click
  ```

- **midscene_aiInput**: Input text into a form field or element

  ```
  Parameters:
  - value: The text to input
  - locate: Natural language description of the element to input text into
  ```

- **midscene_aiKeyboardPress**: Press a specific keyboard key

  ```
  Parameters:
  - key: The key to press (e.g., 'Enter', 'Tab', 'Escape')
  - locate: (Optional) Description of element to focus before pressing the key
  - deepThink: (Optional) If true, uses more precise element location
  ```

- **midscene_aiScroll**: Scroll the page or a specific element
  ```
  Parameters:
  - direction: 'up', 'down', 'left', or 'right'
  - scrollType: 'once', 'untilBottom', 'untilTop', 'untilLeft', or 'untilRight'
  - distance: (Optional) Distance to scroll in pixels
  - locate: (Optional) Description of the element to scroll
  - deepThink: (Optional) If true, uses more precise element location
  ```

### Verification and observation

- **midscene_aiWaitFor**: Wait for a condition to be true on the page

  ```
  Parameters:
  - assertion: Natural language description of the condition to wait for
  - timeoutMs: (Optional) Maximum time to wait in milliseconds
  - checkIntervalMs: (Optional) How often to check the condition
  ```

- **midscene_aiAssert**: Assert that a condition is true on the page

  ```
  Parameters:
  - assertion: Natural language description of the condition to check
  ```

- **midscene_screenshot**: Take a screenshot of the current page
  ```
  Parameters:
  - name: Name for the screenshot
  ```

## Common issues

### How to connect an Android device?

1. Ensure Android SDK is installed and ADB is configured
2. Enable Developer Options and USB debugging on your Android device
3. Connect the device to your computer via USB cable
4. Run `adb devices` to confirm the device is connected
5. Use `midscene_android_list_devices` in MCP to view available devices

### How to launch an Android app?

Use the `midscene_android_launch` tool with parameters that can be:

- App package name: e.g., `com.android.chrome`
- Activity name: e.g., `com.android.chrome/.MainActivity`
- Web URL: e.g., `https://www.example.com`

### Local port conflicts when multiple clients are used

> Problem description

When users simultaneously use Midscene MCP in multiple local clients (Claude Desktop, Cursor MCP, etc.), port conflicts may occur causing server errors

> Solution

- Temporarily close the MCP server in the extra clients
- Execute the command:

```bash
# For macOS/Linux:
lsof -i:3766 | awk 'NR>1 {print $2}' | xargs -r kill -9

# For Windows:
FOR /F "tokens=5" %i IN ('netstat -ano ^| findstr :3766') DO taskkill /F /PID %i
```

### How to access Midscene execution reports

After each task execution, a Midscene task report is generated. You can open this HTML report directly from the command line:

```bash
# Replace the opened address with your report filename
open report_file_name.html
```

![image](https://lf3-static.bytednsdoc.com/obj/eden-cn/ozpmyhn_lm_hymuPild/ljhwZthlaukjlkulzlp/midscene/image.png)



---
url: /mcp.md
---


import McpSharedTools from './common/mcp-shared-tools.mdx';
import McpReport from './common/mcp-report.mdx';

# Expose devices as an MCP service

[MCP](https://modelcontextprotocol.io/introduction) (Model Context Protocol) is a protocol standard that lets AI models interact with external tools and capabilities.

Midscene provides MCP services that expose atomic operations in Midscene Agent (each Action in the Action Space) as MCP tools. Upper-layer Agents can use natural language to inspect the UI, precisely operate UI elements, and run automation tasks without needing to understand the underlying implementation.

Because Midscene Agent relies on a vision model, configure the environment variables required by Midscene inside the MCP service instead of reusing the upstream Agent's model configuration.

## MCP tool list

| Tool name | Description |
|---------|---------|
| Device connections such as `web_connect`, `ios_connect`, `android_connect`, `computer_connect` | Connect to target devices such as browsers, iOS devices, Android devices, or computer desktops |
| `take_screenshot` | Get the latest screenshot |
| Device actions | Each Action in the Action Space, such as `Tap`, `Scroll`, etc. |

## View execution reports

After each interaction finishes, Midscene generates a task report. You can open it directly in the command line:

```bash
open report_file_name.html
```

The report includes detailed interaction information such as screenshots, operation logs, and error details to help with debugging and troubleshooting.

## Configure MCP

### Browser Bridge Mode

`@midscene/web-bridge-mcp` exposes the [Chrome extension Bridge Mode](./bridge-mode) as an MCP service.

**Environment preparation**

Refer to [Chrome Bridge Mode](./bridge-mode) to ensure the browser extension starts correctly. We recommend enabling **Background Bridge Mode**, which allows the connection to run persistently in the background without manual intervention and won't disconnect when closing the extension popup.

:::tip Background Bridge Mode
With background bridge mode enabled, the MCP service can connect at any time without user intervention. See [Background Bridge Mode](./bridge-mode#background-bridge-mode) for details.
:::

**Configuration**

Add the Midscene Web Bridge MCP server (`@midscene/web-bridge-mcp`) in your MCP client. For model configuration parameters, see [Model strategy](./model-strategy).

```json
{
  "mcpServers": {
    "midscene-web": {
      "command": "npx",
      "args": ["-y", "@midscene/web-bridge-mcp"],
      "env": {
        "MIDSCENE_MODEL_BASE_URL": "replace with your model service URL",
        "MIDSCENE_MODEL_API_KEY": "replace with your API Key",
        "MIDSCENE_MODEL_NAME": "replace with your model name",
        "MIDSCENE_MODEL_FAMILY": "replace with your model family",
        "MCP_SERVER_REQUEST_TIMEOUT": "600000"
      }
    }
  }
}
```

### iOS MCP service

**Environment preparation**

- **AI model service**: Prepare an OpenAI API Key or another supported AI model service. See [Model strategy](./model-strategy) for more details.
- **Device setup**: Follow [iOS Getting Started](./ios-getting-started) to configure WebDriverAgent, certificates, and device connections, and make sure WebDriverAgent is running. You can verify screenshots and basic operations in [iOS Playground](./ios-getting-started#try-playground).

**Configuration**

Add the Midscene iOS MCP server (`@midscene/ios-mcp`) in your MCP client. For model configuration parameters, see [Model strategy](./model-strategy).

```json
{
  "mcpServers": {
    "midscene-ios": {
      "command": "npx",
      "args": ["-y", "@midscene/ios-mcp"],
      "env": {
        "MIDSCENE_MODEL_BASE_URL": "replace with your model service URL",
        "MIDSCENE_MODEL_API_KEY": "replace with your API Key",
        "MIDSCENE_MODEL_NAME": "replace with your model name",
        "MIDSCENE_MODEL_FAMILY": "replace with your model family",
        "MCP_SERVER_REQUEST_TIMEOUT": "800000"
      }
    }
  }
}
```


### Android MCP service

**Environment preparation**

- **AI model service**: Prepare an OpenAI API Key or another supported AI model service. See [Model strategy](./model-strategy) for more details.
- **Device setup**: Follow [Android Getting Started](./android-getting-started) to configure adb and connect your device. Ensure `adb devices` can recognize the target device. Use [Android Playground](./android-getting-started#try-playground-no-code) to verify screenshots and basic operations.

**Configuration**

Add the Midscene Android MCP server (`@midscene/android-mcp`) in your MCP client. For model configuration parameters, see [Model strategy](./model-strategy).

```json
{
  "mcpServers": {
    "midscene-android": {
      "command": "npx",
      "args": ["-y", "@midscene/android-mcp"],
      "env": {
        "MIDSCENE_MODEL_BASE_URL": "replace with your model service URL",
        "MIDSCENE_MODEL_API_KEY": "replace with your API Key",
        "MIDSCENE_MODEL_NAME": "replace with your model name",
        "MIDSCENE_MODEL_FAMILY": "replace with your model family",
        "MCP_SERVER_REQUEST_TIMEOUT": "800000"
      }
    }
  }
}
```

### Computer Desktop MCP service

`@midscene/computer-mcp` exposes the computer desktop automation capabilities as an MCP service, allowing AI to control your computer through mouse, keyboard, and screenshot operations.

**Environment preparation**

- **AI model service**: Prepare an OpenAI API Key or another supported AI model service. See [Model strategy](./model-strategy) for more details.
- **System permissions**: On macOS, you need to grant accessibility and screen recording permissions to the terminal or application running the MCP service.

**Configuration**

Add the Midscene Computer MCP server (`@midscene/computer-mcp`) in your MCP client. For model configuration parameters, see [Model strategy](./model-strategy).

```json
{
  "mcpServers": {
    "midscene-computer": {
      "command": "npx",
      "args": ["-y", "@midscene/computer-mcp"],
      "env": {
        "MIDSCENE_MODEL_BASE_URL": "replace with your model service URL",
        "MIDSCENE_MODEL_API_KEY": "replace with your API Key",
        "MIDSCENE_MODEL_NAME": "replace with your model name",
        "MIDSCENE_MODEL_FAMILY": "replace with your model family",
        "MCP_SERVER_REQUEST_TIMEOUT": "800000"
      }
    }
  }
}
```

## Implement your own MCP

If you want to integrate Midscene tools into your own MCP service, you can use the `mcpKitForAgent` function to get tool definitions and expose your own MCP service as needed.

The tools provided by `mcpKitForAgent` include screenshots and every Action in the Action Space.

### Using mcpKitForAgent

The `mcpKitForAgent` function takes an Agent instance and returns an object containing description and tools list:

```typescript
import { mcpKitForAgent } from '@midscene/web/mcp-server';
import { Agent } from '@midscene/core/agent';

const agent = new Agent();
const { description, tools } = await mcpKitForAgent(agent);

// description - "Control the browser / device using natural language commands"
// tools - Tool[] - array of tool definitions
```

### Platform support

Each platform provides its corresponding `mcpKitForAgent` function:

**Web platform**
```typescript
import { mcpKitForAgent } from '@midscene/web/mcp-server';
```

**iOS platform**
```typescript
import { mcpKitForAgent } from '@midscene/ios/mcp-server';
```

**Android platform**
```typescript
import { mcpKitForAgent } from '@midscene/android/mcp-server';
```

**Computer platform**
```typescript
import { mcpKitForAgent } from '@midscene/computer/mcp-server';
```

### Integrate into custom MCP service

You can integrate the obtained tools into your own MCP service:

```typescript
import { McpServer } from '@modelcontextprotocol/sdk/server/mcp.js';
import { mcpKitForAgent } from '@midscene/web/mcp-server';

const agent = new Agent();
const { description, tools } = await mcpKitForAgent(agent);
const server = new McpServer({
  name: 'my-custom-mcp',
  version: '1.0.0',
  description
});

// Register Midscene tools to your MCP service
for (const tool of tools) {
  server.tool(tool.name, tool.description, tool.schema, tool.handler);
}
```



---
url: /model-common-config.md
---

import TroubleshootingLLMConnectivity from './common/troubleshooting-llm-connectivity.mdx';

# Common Model Configuration

## Ways to set environment variables

Midscene reads all model configuration from environment variables. Below are common approaches, but feel free to adopt any method used in your project.

### Method 1: Set variables in the system

> The Midscene Chrome extension also accepts this `export KEY="value"` format.

```bash
# Replace with your own API key
export MIDSCENE_MODEL_BASE_URL="https://.../compatible-mode/v1"
export MIDSCENE_MODEL_API_KEY="sk-abcde..."
export MIDSCENE_MODEL_NAME="qwen3-vl-plus"
export MIDSCENE_MODEL_FAMILY="qwen3-vl"
```

### Method 2: Create a `.env` file (for CLI tools)

Create a `.env` file in the directory where you run the project. Midscene CLI tools load this file automatically.

```bash
MIDSCENE_MODEL_BASE_URL="https://.../compatible-mode/v1"
MIDSCENE_MODEL_API_KEY="sk-abcdefghijklmnopqrstuvwxyz"
MIDSCENE_MODEL_NAME="qwen3-vl-plus"
MIDSCENE_MODEL_FAMILY="qwen3-vl"
```

Keep in mind:
1. You do **not** need to prefix each line with `export`.
2. Only the Midscene CLI automatically reads this file. For the JavaScript SDK, load it manually as shown below.

### Method 3: Load variables via dotenv

[Dotenv](https://www.npmjs.com/package/dotenv) is a zero-dependency npm package that loads variables from `.env` into Node.js `process.env`.

Our [demo project](https://github.com/web-infra-dev/midscene-example) uses this method.

```bash
# install dotenv
npm install dotenv --save
```

Create a `.env` file in the project root and add (no `export` prefix):

```bash
MIDSCENE_MODEL_API_KEY="sk-abcdefghijklmnopqrstuvwxyz"
```

Import dotenv in your script; it will read `.env` automatically:

```typescript
import 'dotenv/config';
```

## Common model configurations

This section lists common model configurations. For details on model differences and selection strategies, see [Recommended Vision Models](./model-strategy#recommended-vision-models).

### Doubao Seed Model {#doubao-seed-model}

Doubao-Seed-2.0-Lite is the recommended model. Doubao-Seed-1.6-Vision and Doubao-Seed-1.8 are also supported.

Obtain an API key from [Volcano Engine](https://volcengine.com) and set:

```bash
MIDSCENE_MODEL_BASE_URL="https://ark.cn-beijing.volces.com/api/v3"
MIDSCENE_MODEL_API_KEY="...."
MIDSCENE_MODEL_NAME="ep-..." # Inference endpoint ID or model name from Volcano Engine
MIDSCENE_MODEL_FAMILY="doubao-seed" # "doubao-vision" is also supported
# Optional: control reasoning effort (low, medium, high)
# MIDSCENE_MODEL_REASONING_EFFORT="medium"
```
### Qwen3.5 {#qwen35}

Using Alibaba Cloud's `qwen3.5-plus` as an example. It is recommended to disable the platform's default thinking mode to improve execution speed, the environment variable configuration is as follows:

```bash
MIDSCENE_MODEL_BASE_URL="https://dashscope.aliyuncs.com/compatible-mode/v1"
MIDSCENE_MODEL_API_KEY="......"
MIDSCENE_MODEL_NAME="qwen3.5-plus"
MIDSCENE_MODEL_FAMILY="qwen3.5"
MIDSCENE_MODEL_REASONING_ENABLED="false"
```

To enable thinking mode, remove the `MIDSCENE_MODEL_REASONING_ENABLED="false"` line and add `MIDSCENE_MODEL_REASONING_BUDGET="500"` to control thinking cost.

You can also use Qwen3.5 from [OpenRouter](https://openrouter.ai/qwen).

### Qwen3-VL {#qwen3-vl}

Using Alibaba Cloud's `qwen3-vl-plus` as an example:

```bash
MIDSCENE_MODEL_BASE_URL="https://dashscope.aliyuncs.com/compatible-mode/v1"
MIDSCENE_MODEL_API_KEY="......"
MIDSCENE_MODEL_NAME="qwen3-vl-plus"
MIDSCENE_MODEL_FAMILY="qwen3-vl"
```

You can also use Qwen3-VL from [OpenRouter](https://openrouter.ai/qwen).

### Qwen2.5-VL {#qwen25-vl}

Using Alibaba Cloud's `qwen-vl-max-latest` as an example:

```bash
MIDSCENE_MODEL_BASE_URL="https://dashscope.aliyuncs.com/compatible-mode/v1"
MIDSCENE_MODEL_API_KEY="......"
MIDSCENE_MODEL_NAME="qwen-vl-max-latest"
MIDSCENE_MODEL_FAMILY="qwen2.5-vl"
```

### Zhipu GLM-V {#glm-v}

Zhipu GLM-V is an open-source vision model from Zhipu AI. Using `GLM-4.6V` as an example:

Obtain an API key from [Z.AI (Global)](https://z.ai/manage-apikey/apikey-list) or [BigModel (CN)](https://bigmodel.cn/usercenter/proj-mgmt/apikeys), and set:

```bash
MIDSCENE_MODEL_BASE_URL="https://api.z.ai/api/paas/v4" # Or https://open.bigmodel.cn/api/paas/v4
MIDSCENE_MODEL_API_KEY="......"
MIDSCENE_MODEL_NAME="glm-4.6v"
MIDSCENE_MODEL_FAMILY="glm-v"
```

**Learn more about Zhipu GLM-V**

- Github: [https://github.com/zai-org/GLM-V](https://github.com/zai-org/GLM-V)
- Hugging Face: [https://huggingface.co/zai-org/GLM-4.6V](https://huggingface.co/zai-org/GLM-4.6V)

### Zhipu AutoGLM {#auto-glm}

Zhipu AutoGLM is an open-source mobile UI automation model (9B parameters) from Zhipu AI.

After obtaining an API key from [Z.AI (Global)](https://z.ai/manage-apikey/apikey-list) or [BigModel (CN)](https://bigmodel.cn/usercenter/proj-mgmt/apikeys), configure:

```bash
MIDSCENE_MODEL_BASE_URL="https://api.z.ai/api/paas/v4" # Or https://open.bigmodel.cn/api/paas/v4
MIDSCENE_MODEL_API_KEY="......"
MIDSCENE_MODEL_NAME="autoglm-phone"
MIDSCENE_MODEL_FAMILY="auto-glm" # Or "auto-glm-multilingual"
```

**About `MIDSCENE_MODEL_FAMILY` Configuration**

AutoGLM provides two model versions, distinguished by `MIDSCENE_MODEL_FAMILY`:

- `auto-glm` - Corresponds to AutoGLM-Phone-9B, optimized for **Chinese mobile applications**
- `auto-glm-multilingual` - Corresponds to AutoGLM-Phone-9B-Multilingual, supports **English and other languages**

Choose the appropriate version based on your application language.

:::note
AutoGLM is best suited for mobile interactions and operations. If you need to use `aiAssert`, `aiQuery`, or other APIs requiring page understanding/assertions, you should configure separate `MIDSCENE_INSIGHT_MODEL_...` environment variables to let an independent Insight model handle page comprehension. See [Model Strategy](./model-strategy) for more information about multi-model configuration.
:::

**Learn more about Zhipu AutoGLM**

- Github: [https://github.com/zai-org/Open-AutoGLM](https://github.com/zai-org/Open-AutoGLM)
- Hugging Face: [https://huggingface.co/zai-org/AutoGLM-Phone-9B](https://huggingface.co/zai-org/AutoGLM-Phone-9B)

### Gemini-3-Pro and Gemini-3-Flash {#gemini-3-pro}

After requesting an API key from [Google Gemini](https://gemini.google.com/), configure. Use your specific Gemini-3-Pro or Gemini-3-Flash release name for `MIDSCENE_MODEL_NAME`:

```bash
MIDSCENE_MODEL_BASE_URL="https://generativelanguage.googleapis.com/v1beta/openai/"
MIDSCENE_MODEL_API_KEY="......"
MIDSCENE_MODEL_NAME="gemini-3.0-pro-preview" # Or a gemini-3-flash release name
MIDSCENE_MODEL_FAMILY="gemini"
```

### UI-TARS {#ui-tars}

Use the deployed `doubao-1.5-ui-tars` on [Volcano Engine](https://volcengine.com):

```bash
MIDSCENE_MODEL_BASE_URL="https://ark.cn-beijing.volces.com/api/v3"
MIDSCENE_MODEL_API_KEY="...."
MIDSCENE_MODEL_NAME="ep-2025..." # Inference endpoint ID or model name from Volcano Engine
MIDSCENE_MODEL_FAMILY="vlm-ui-tars-doubao-1.5"
```

**About `MIDSCENE_MODEL_FAMILY`**

This variable selects the UI-TARS version. Supported values:
- `vlm-ui-tars` – for the 1.0 release
- `vlm-ui-tars-doubao` – for the 1.5 release deployed on Volcano Engine (equivalent to `vlm-ui-tars-doubao-1.5`)
- `vlm-ui-tars-doubao-1.5` – for the 1.5 release deployed on Volcano Engine

:::tip

The legacy configurations `MIDSCENE_USE_VLM_UI_TARS=DOUBAO` or `MIDSCENE_USE_VLM_UI_TARS=1.5` are still supported but deprecated. Please migrate to `MIDSCENE_MODEL_FAMILY`.

Migration mapping:
- `MIDSCENE_USE_VLM_UI_TARS=1.0` → `MIDSCENE_MODEL_FAMILY="vlm-ui-tars"`
- `MIDSCENE_USE_VLM_UI_TARS=1.5` → `MIDSCENE_MODEL_FAMILY="vlm-ui-tars-doubao-1.5"`
- `MIDSCENE_USE_VLM_UI_TARS=DOUBAO` → `MIDSCENE_MODEL_FAMILY="vlm-ui-tars-doubao"`

:::

### ~~GPT-4o~~

Starting with version 1.0, Midscene no longer supports gpt series models as the default model. See [Model strategy](./model-strategy) for details.

## Multi-model example: GPT-5.1 planning/insight + Qwen3-VL for vision {#gpt51-planning-insight-qwen3}

For more information on combining multiple models, see [Advanced: Combining Multiple Models](./model-strategy#advanced-combining-multiple-models).

Below is an example using GPT-5.1 for Planning/Insight and Qwen3-VL for vision. Use GPT-5.1 for Planning and/or Insight to handle heavy reasoning, while Qwen3-VL focuses on visual grounding. You can enable either role or both—toggle them based on your workload.

```bash
# Default vision model: Qwen3-VL
export MIDSCENE_MODEL_BASE_URL="https://..."       # Qwen3-VL endpoint
export MIDSCENE_MODEL_API_KEY="..."                # Your Qwen3-VL API key
export MIDSCENE_MODEL_NAME="qwen3-vl-plus"
export MIDSCENE_MODEL_FAMILY="qwen3-vl"

# Planning model: GPT-5.1
export MIDSCENE_PLANNING_MODEL_API_KEY="sk-..."    # Your GPT-5.1 API key
export MIDSCENE_PLANNING_MODEL_BASE_URL="https://..." 
export MIDSCENE_PLANNING_MODEL_NAME="gpt-5.1"

# Insight model: GPT-5.1
export MIDSCENE_INSIGHT_MODEL_API_KEY="sk-..."     # Your GPT-5.1 API key
export MIDSCENE_INSIGHT_MODEL_BASE_URL="https://..."
export MIDSCENE_INSIGHT_MODEL_NAME="gpt-5.1"
```

## More

For additional, advanced model settings, see [Model configuration](./model-config).


<TroubleshootingLLMConnectivity />



---
url: /model-config.md
---

# Model configuration

Midscene reads all model configuration from operating-system environment variables.

Midscene integrates the OpenAI SDK by default for AI calls. The SDK defines the parameter shape, and most model providers (or deployment tools) offer compatible endpoints.

This doc focuses on Midscene model configuration. For how we choose models, see [Model strategy](./model-strategy). For quick recipes for popular models, see [Common model configuration](./model-common-config).


## Required settings

You need to set a default model for Midscene; see [Model strategy](./model-strategy) for details.


| Name | Description |
|------|-------------|
| `MIDSCENE_MODEL_API_KEY` | Model API key, e.g., `"sk-abcd..."` |
| `MIDSCENE_MODEL_BASE_URL` | API endpoint URL, usually ending with a version (e.g., `/v1`); do not append `/chat/completion` here since the underlying sdk will add it automatically |
| `MIDSCENE_MODEL_NAME` | Model name |
| `MIDSCENE_MODEL_FAMILY` | Model family, determine the way of dealing with the coordinates |

## Advanced settings (optional)

| Name | Description |
|------|-------------|
| `MIDSCENE_MODEL_TIMEOUT` | Timeout for AI API calls in milliseconds (default intent). Defaults to OpenAI SDK default (10 minutes) |
| `MIDSCENE_MODEL_TEMPERATURE` | Sampling temperature for model responses |
| `MIDSCENE_MODEL_MAX_TOKENS` | `max_tokens` for responses, default 2048 |
| `MIDSCENE_MODEL_RETRY_COUNT` | Number of retries when AI call fails, default 1 (i.e., retry once after failure) |
| `MIDSCENE_MODEL_RETRY_INTERVAL` | Interval between retries in milliseconds, default 2000 |
| `MIDSCENE_MODEL_REASONING_ENABLED` | Enable or disable model reasoning/thinking. Set to `"true"` or `"false"`. Passed through as `enable_thinking` for qwen, `thinking.type` for doubao/glm-v. Unsupported families ignore this setting |
| `MIDSCENE_MODEL_REASONING_EFFORT` | Reasoning effort level string. Passed through as `reasoning_effort` for doubao, `reasoning.effort` for gpt-5. Common values: `low`, `medium`, `high` |
| `MIDSCENE_MODEL_REASONING_BUDGET` | Thinking token budget (number). Passed through as `thinking_budget` for qwen. Unsupported families ignore this setting |
| `MIDSCENE_MODEL_HTTP_PROXY` | HTTP/HTTPS proxy, e.g., `http://127.0.0.1:8080` or `https://proxy.example.com:8080`. Takes precedence over `MIDSCENE_MODEL_SOCKS_PROXY` |
| `MIDSCENE_MODEL_SOCKS_PROXY` | SOCKS proxy, e.g., `socks5://127.0.0.1:1080` |
| `MIDSCENE_MODEL_INIT_CONFIG_JSON` | JSON blob that overrides the OpenAI SDK initialization config |
| `MIDSCENE_RUN_DIR` | Run artifact directory, like report and logs. Defaults to `midscene_run` in the current working directory; accepts absolute or relative paths |
| `MIDSCENE_PREFERRED_LANGUAGE` | Optional. Preferred response language. Defaults to `Chinese` if timezone is GMT+8, otherwise `English` |

> Note: Control replanning behavior with the agent option `replanningCycleLimit` (defaults to 20, or 40 for `vlm-ui-tars`), not with environment variables.

### Configure a dedicated Insight model

Set the following if the Insight intent needs a different model:

| Name | Description |
|------|-------------|
| `MIDSCENE_INSIGHT_MODEL_API_KEY` | API key |
| `MIDSCENE_INSIGHT_MODEL_BASE_URL` | API endpoint URL (omit the trailing `/chat/completion`) |
| `MIDSCENE_INSIGHT_MODEL_NAME` | Model name |
| `MIDSCENE_INSIGHT_MODEL_TIMEOUT` | Optional; timeout for Insight intent AI API calls in milliseconds |
| `MIDSCENE_INSIGHT_MODEL_TEMPERATURE` | Optional; sampling temperature for Insight intent responses |
| `MIDSCENE_INSIGHT_MODEL_RETRY_COUNT` | Optional; same effect as `MIDSCENE_MODEL_RETRY_COUNT` |
| `MIDSCENE_INSIGHT_MODEL_RETRY_INTERVAL` | Optional; same effect as `MIDSCENE_MODEL_RETRY_INTERVAL` |
| `MIDSCENE_INSIGHT_MODEL_HTTP_PROXY` | Optional; same effect as `MIDSCENE_MODEL_HTTP_PROXY` |
| `MIDSCENE_INSIGHT_MODEL_SOCKS_PROXY` | Optional; same effect as `MIDSCENE_MODEL_SOCKS_PROXY` |
| `MIDSCENE_INSIGHT_MODEL_INIT_CONFIG_JSON` | Optional; same effect as `MIDSCENE_MODEL_INIT_CONFIG_JSON` |

### Configure a dedicated Planning model

Set the following if the Planning intent needs a different model:

| Name | Description |
|------|-------------|
| `MIDSCENE_PLANNING_MODEL_API_KEY` | API key |
| `MIDSCENE_PLANNING_MODEL_BASE_URL` | API endpoint URL (omit the trailing `/chat/completion`) |
| `MIDSCENE_PLANNING_MODEL_NAME` | Model name |
| `MIDSCENE_PLANNING_MODEL_TIMEOUT` | Optional; timeout for Planning intent AI API calls in milliseconds |
| `MIDSCENE_PLANNING_MODEL_TEMPERATURE` | Optional; sampling temperature for Planning intent responses |
| `MIDSCENE_PLANNING_MODEL_RETRY_COUNT` | Optional; same effect as `MIDSCENE_MODEL_RETRY_COUNT` |
| `MIDSCENE_PLANNING_MODEL_RETRY_INTERVAL` | Optional; same effect as `MIDSCENE_MODEL_RETRY_INTERVAL` |
| `MIDSCENE_PLANNING_MODEL_HTTP_PROXY` | Optional; same effect as `MIDSCENE_MODEL_HTTP_PROXY` |
| `MIDSCENE_PLANNING_MODEL_SOCKS_PROXY` | Optional; same effect as `MIDSCENE_MODEL_SOCKS_PROXY` |
| `MIDSCENE_PLANNING_MODEL_INIT_CONFIG_JSON` | Optional; same effect as `MIDSCENE_MODEL_INIT_CONFIG_JSON` |

### Debug logging switches

Enable the following variables to print richer debug logs. Regardless of the switches, logs are also saved under `./midscene_run/log`.

| Name | Description |
|------|-------------|
| `DEBUG=midscene:ai:profile:stats` | Prints model latency, token usage, etc., comma-separated for easier analysis |
| `DEBUG=midscene:ai:profile:detail` | Prints detailed token-usage logs |
| `DEBUG=midscene:ai:call` | Prints AI response details |
| `DEBUG=midscene:android:adb` | Prints Android adb command details |
| `DEBUG=midscene:*` | Prints every debug log |

## Still-compatible configs (not recommended)

The following environment variables are deprecated but still compatible. We recommend migrating to the new configuration approach.

### Planning model configuration

| Name | Description | New approach |
|------|-------------|--------------|
| `MIDSCENE_USE_DOUBAO_VISION` | Deprecated. Enables Doubao vision model | Use `MIDSCENE_MODEL_FAMILY="doubao-vision"` |
| `MIDSCENE_USE_QWEN3_VL` | Deprecated. Enables Qwen3-VL model | Use `MIDSCENE_MODEL_FAMILY="qwen3-vl"` |
| `MIDSCENE_USE_QWEN_VL` | Deprecated. Enables Qwen2.5-VL model | Use `MIDSCENE_MODEL_FAMILY="qwen2.5-vl"` |
| `MIDSCENE_USE_GEMINI` | Deprecated. Enables Gemini model | Use `MIDSCENE_MODEL_FAMILY="gemini"` |
| `MIDSCENE_USE_VLM_UI_TARS` | Deprecated. Enables UI-TARS model | Use `MIDSCENE_MODEL_FAMILY="vlm-ui-tars"` |

### General configuration

| Name | Description | New approach |
|------|-------------|--------------|
| `OPENAI_API_KEY` | Deprecated but supported | Prefer `MIDSCENE_MODEL_API_KEY` |
| `OPENAI_BASE_URL` | Deprecated but supported | Prefer `MIDSCENE_MODEL_BASE_URL` |
| `MIDSCENE_OPENAI_INIT_CONFIG_JSON` | Deprecated but supported | Prefer `MIDSCENE_MODEL_INIT_CONFIG_JSON` |
| `MIDSCENE_OPENAI_HTTP_PROXY` | Deprecated but supported | Prefer `MIDSCENE_MODEL_HTTP_PROXY` |
| `MIDSCENE_OPENAI_SOCKS_PROXY` | Deprecated but supported | Prefer `MIDSCENE_MODEL_SOCKS_PROXY` |
| `OPENAI_MAX_TOKENS` | Deprecated but supported | Prefer `MIDSCENE_MODEL_MAX_TOKENS` |

## Configure settings via JavaScript

You can configure models for each agent in JavaScript. See the [API reference](./api) for details.

```typescript
const agent = new Agent(page, {
  // Configure via modelConfig
  modelConfig: {
    MIDSCENE_MODEL_TIMEOUT: '60000', // 60 seconds
    MIDSCENE_MODEL_NAME: 'qwen3-vl-plus',
    // ... other configurations
  }
});
```

## FAQ

### How can I monitor token usage?

Set `DEBUG=midscene:ai:profile:stats` to print usage and latency.

You can also find usage statistics inside the generated report files.


### Using LangSmith

LangSmith is a platform for debugging large language models. Midscene provides auto-integration support - just install the dependency and set environment variables.

**Step 1: Install dependency**

```bash
npm install langsmith
```

**Step 2: Set environment variables**

```bash
# Enable Midscene's LangSmith auto-integration
export MIDSCENE_LANGSMITH_DEBUG=1

# LangSmith configuration
export LANGCHAIN_API_KEY="your-langchain-api-key-here"
export LANGCHAIN_TRACING=true
export LANGCHAIN_ENDPOINT="https://api.smith.langchain.com"
# export LANGCHAIN_ENDPOINT="https://eu.api.smith.langchain.com" # If signed up in the EU region
```

After starting Midscene, you should see logs similar to:

```log
DEBUGGING MODE: langsmith wrapper enabled
```

Notes:
- LangSmith and Langfuse can be enabled simultaneously.
- Node.js only; browser environments will throw errors.
- If you use [`createOpenAIClient`](./api#custom-openai-client), it overrides the env-based auto-integration.

For finer-grained control (e.g., enabling LangSmith only for specific tasks), use [`createOpenAIClient`](./api#custom-openai-client) to wrap the client manually.

### Using Langfuse

[Langfuse](https://langfuse.com) is another popular LLM observability platform. Midscene has integrated Langfuse's `observeOpenAI` wrapper to automatically trace all OpenAI API calls.

Since Langfuse tracing is built on top of OpenTelemetry, you need to initialize the OpenTelemetry SDK at application startup.

**Step 1: Install dependencies**

```bash
npm install @langfuse/openai @langfuse/otel @opentelemetry/sdk-node
```

**Step 2: Initialize OpenTelemetry (Required)**

Add the following code at the **very top** of your application entry file:

```typescript
import { NodeSDK } from "@opentelemetry/sdk-node";
import { LangfuseSpanProcessor } from "@langfuse/otel";

const sdk = new NodeSDK({
  spanProcessors: [new LangfuseSpanProcessor()],
});
sdk.start();
```

**Step 3: Set environment variables**

```bash
# Enable Midscene's Langfuse auto-integration
export MIDSCENE_LANGFUSE_DEBUG=1

# Langfuse configuration
export LANGFUSE_PUBLIC_KEY="your-langfuse-public-key-here"
export LANGFUSE_SECRET_KEY="your-langfuse-secret-key-here"
export LANGFUSE_BASE_URL="https://cloud.langfuse.com" # 🇪🇺 EU region
# export LANGFUSE_BASE_URL="https://us.cloud.langfuse.com" # 🇺🇸 US region
```

After starting Midscene, you should see logs similar to:

```log
OpenTelemetry SDK initialized for Langfuse tracing
DEBUGGING MODE: langfuse wrapper enabled
```

**Learn More**: Check out the [Langfuse OpenAI Integration documentation](https://langfuse.com/integrations/model-providers/openai-js) for additional configuration options and best practices.

Notes:
- LangSmith and Langfuse can be enabled simultaneously.
- Node.js only; browser environments will throw errors.
- If you use [`createOpenAIClient`](./api#custom-openai-client), it overrides the env-based auto-integration.



---
url: /model-strategy.md
---



# Model Strategy

:::info Quick start

If you want to try Midscene right away, pick a model and follow its configuration guide:
* [Doubao Seed Model](./model-common-config#doubao-seed-model)
* Qwen Models: [Qwen3.5](./model-common-config#qwen35), [Qwen3-VL](./model-common-config#qwen3-vl), [Qwen2.5-VL](./model-common-config#qwen25-vl)
* [Zhipu GLM-V](./model-common-config#glm-v)
* [Zhipu AutoGLM](./model-common-config#auto-glm)
* [Gemini-3-Pro / Gemini-3-Flash](./model-common-config#gemini-3-pro)
* [UI-TARS](./model-common-config#ui-tars)

:::

This guide focuses on how we choose models in Midscene. If you need configuration instructions, head to [Model configuration](./model-config).

## Background - UI automation approaches

Driving UI automation with AI hinges on two challenges: planning a reasonable set of actions and precisely locating the elements that require interaction. The strength of element localization directly determines how successful an automation task will be.

To solve element localization, UI automation frameworks traditionally follow one of two approaches:

* **DOM + annotated screenshots**: Extract the DOM tree beforehand, annotate screenshots with DOM metadata, and ask the model to “pick” the right nodes.
* **Pure vision**: Perform all analysis on screenshots alone by using the visual grounding capabilities of the model. The model only receives the image—no DOM, no annotations.

## Pure vision for element localization

Earlier Midscene releases supported both **DOM localization** and **pure vision** so developers could compare them. After dozens of versions and hundreds of project tests, we have some new findings.

The DOM approach has been less stable than expected. It often misfires with canvas elements, controls rendered via CSS `background-image`, content inside cross-origin iframes, or elements without strong accessibility annotations. These intermittent failures can send teams into long debugging sessions or even a frustrating prompt-tuning loop.

Meanwhile, the pure-vision route has started to show clear advantages:

- **Stable results**: These models pair strong UI planning with element targeting and interface understanding, helping developers get productive quickly.
- **Works everywhere**: Automation no longer depends on how the UI is rendered. Android, iOS, desktop apps, or a browser `<canvas>`—if you can capture a screenshot, Midscene can interact with it.
- **Developer-friendly**: Removing selectors and DOM fuss makes the model “handshake” easier. Even teammates unfamiliar with rendering tech can become productive quickly.
- **Far fewer tokens**: Compared to the DOM approach, pure vision can cut token usage by up to ~80%, reducing cost and speeding up local runs.
- **Open-source options**: Open-source vision models keep improving. Teams can self-host options such as Qwen3-VL 8B/30B with solid results.

Given these advantages, **Midscene 1.0 and later only support the pure-vision approach**—the DOM-extraction compatibility mode has been removed. This applies to UI actions and element localization; for data extraction or page understanding you can still opt in to include DOM when needed.

## Recommended vision models

Based on extensive real-world usage, we recommend these defaults for Midscene: Doubao Seed, Qwen VL, Gemini-3 (Pro/Flash), and UI-TARS.

They offer strong element-localization skills and solid performance in planning and screen understanding.

If you are unsure where to start, pick whichever model is easiest to access today, then run side-by-side comparisons later.

| Model family | Deployment | Midscene notes |
| --- | --- | --- |
| Doubao Seed Model<br />[Quick setup](./model-common-config#doubao-seed-model) | Volcano Engine:<br />[Doubao-Seed-1.6-Vision](https://www.volcengine.com/docs/82379/1799865)<br />[Doubao-Seed-2.0-Lite](https://www.volcengine.com/docs/82379/1799865) | ⭐⭐⭐⭐<br />Strong at UI planning and targeting<br />Slightly slower |
| Qwen3.5<br />[Quick setup](./model-common-config#qwen35) | [Alibaba Cloud](https://help.aliyun.com/zh/model-studio/vision)<br/>[OpenRouter](https://openrouter.ai/qwen) | ⭐⭐⭐⭐<br />Stronger than Qwen3-VL and Qwen2.5-VL |
| Zhipu GLM-4.6V<br />[Quick setup](./model-common-config#glm-v) | [Z.AI (Global)](https://docs.z.ai/guides/vlm/glm-4.6v)<br/>[BigModel (CN)](https://docs.bigmodel.cn/cn/guide/models/vlm/glm-4.6v) | Newly integrated, welcome to try it out<br />Weights open-sourced on [HuggingFace](https://huggingface.co/zai-org/GLM-4.6V) |
| Gemini-3-Pro / Gemini-3-Flash<br />[Quick setup](./model-common-config#gemini-3-pro) | [Google Cloud](https://ai.google.dev/gemini-api/docs/models/gemini) | ⭐⭐⭐<br />Gemini-3-Flash is supported<br />Price is higher than Doubao and Qwen |
| UI-TARS<br />[Quick setup](./model-common-config#ui-tars) | [Volcano Engine](https://www.volcengine.com/docs/82379/1536429) | ⭐⭐<br />Strong exploratory ability but results vary by scenario<br />Open-source versions available ([HuggingFace](https://huggingface.co/bytedance-research/UI-TARS-72B-SFT) / [GitHub](https://github.com/bytedance/ui-tars)) |

:::info Why not use multimodal models like gpt-5 as the default?

Midscene requires strong UI localization (visual grounding). Models like gpt-5 perform poorly here, so they cannot serve as the default. You can still use them as dedicated “planning models,” which we cover below.

:::

## Advanced: combining multiple models

The default model strategy covers most projects at kick-off. As prompts grow more complex and you need stronger generalization, the default models’ planning ability may fall short. Consider this GitHub sign-up prompt:

```text
Fill out the GitHub sign-up form. Ensure no fields are missing and pick “United States” as the region.

Make sure every field passes validation.

Only fill the form; do not submit a real registration request.

Return the actual field values you filled.
```

This sounds simple, but the model must understand per-field rules, identify each control, operate a multi-step region selector, paginate and trigger validation, and still locate every element. Default models often cannot satisfy all of these at once, so success rates drop.

In these situations, configure separate models for *Planning* and *Insight*, while **keeping the default model** as your baseline. This is not “only Planning and Insight”—it is **default model + (optional) Planning/Insight**. Multi-model setups are the most practical and effective way to lift UI automation success, at a small cost in latency and tokens.

### Model responsibility overview

By default, the system follows these rules (unset intents fall back to the default model):

- **Default model**: handles **element localization** (Locate), plus anything not explicitly assigned to Planning/Insight.
- **Planning model (optional)**: handles **task planning** (Planning in `aiAct` / `ai`).
- **Insight model (optional)**: handles **data extraction, assertions, and UI understanding** (`aiQuery` / `aiAsk` / `aiAssert`).

In short: **Planning goes to the Planning model, localization stays on the default model, and extraction/assertion goes to Insight** when configured.

For a ready-to-use environment snippet, including an Insight model, see the [multi-model example](./model-common-config#gpt51-planning-insight-qwen3).

### *Planning* intent

For `aiAct` or `ai` planning tasks, add settings with the `MIDSCENE_PLANNING_MODEL_` prefix to use a dedicated model for the Planning intent.

We recommend `gpt-5.1` or other multimodal models that understand UI workflows.

### *Insight* intent

Midscene provides page-understanding APIs such as AI assertions (`aiAssert`) and data extraction (`aiQuery`, `aiAsk`). We group these workloads under the *Insight* intent, which depends on visual question answering (VQA) skills.

Add settings with the `MIDSCENE_INSIGHT_MODEL_` prefix to use a dedicated model for Insight workloads.

We recommend `gpt-5.1` or other models with strong VQA capability.

## How do I tune execution results?

If you see success rates below your target, try these steps.

0. Review Midscene's replay report to confirm the task timeline is correct and the flow didn't veer into the wrong page or branch.
1. Use the best, newest, and larger model versions to materially improve success rates. For example, Qwen3-VL outperforms Qwen2.5-VL, and a 72B build is more accurate than a 30B build.
2. Ensure the `MIDSCENE_MODEL_FAMILY` environment variable is set correctly; otherwise element localization can drift significantly.
3. Tweak the `planningStrategy` option in `aiAct` to increase planning depth.
4. Try different models, or combine multiple models to compensate for weaker understanding.

## More

### Limitations of the pure-vision approach

Vision models are a highly general approach: they do not depend on a specific UI rendering stack and can analyze screenshots directly, letting developers ramp up and tune quickly across any app surface.

The flip side is higher demands on the model itself.

For example, on mobile UI automation, if you have a component tree plus full a11y annotations, you can lean on a small text-only model and reason over structure, potentially pushing performance further. Pure vision skips those annotations and is more universal, saving UI labeling effort but consuming more model resources at runtime.

### Model configuration doc

See [Model configuration](./model-config).

### About the `deepThink` option in `aiAct`

The `deepThink` option controls whether models use deep reasoning during planning. Currently supported for `qwen3-vl`, `qwen3.5`, `doubao-seed-1.6`, `doubao-seed-2.0` and `glm-v` model families.

When enabled, you'll see a `Reasoning` section in the report. Planning takes longer but produces more accurate results.

**Default behavior varies by provider:**
- `qwen3.5` on Alibaba Cloud: enabled by default (slower, more accurate)
- `qwen3-vl` on Alibaba Cloud: disabled by default (faster, less accurate)
- `doubao-seed-1.6` on Volcano Engine: enabled by default (slower, more accurate)
- `doubao-seed-2.0` on Volcano Engine: enabled by default (slower, more accurate)
- `glm-v` on Z.AI: enabled by default (slower, more accurate)

You can explicitly set `deepThink: true` or `deepThink: false` to override the provider's default.

`deepThink` accepts `'unset' | true | false`. The default is `'unset'` (same as omitting the option, following the model provider's default strategy).

Note: The implementation details behind `deepThink` may evolve in the future as model providers change.

### API style for model service provider

Midscene expects providers to expose an OpenAI-compatible API (this does **not** mean you must use OpenAI models).

Most major vendors and deployment tools already support this pattern.

### How do I inspect token usage?

Set `DEBUG=midscene:ai:profile:stats` to print cost and latency information. You can also review token usage in the generated report files.

### "MIDSCENE_MODEL_FAMILY is required" error

If you see "No visual language model (VL model) detected" or "MIDSCENE_MODEL_FAMILY is required," make sure the `MIDSCENE_MODEL_FAMILY` environment variable for your VL model is set correctly.

Starting with version 1.0, Midscene recommends using `MIDSCENE_MODEL_FAMILY` to specify the vision model type. Legacy `MIDSCENE_USE_...` configs remain compatible but are deprecated.

See [Model configuration](./model-config) for setup details.

### Can each Agent instance use its own model?

Yes. You can configure per-agent models via the `modelConfig` parameter. See the [API reference](./api) for details.

### Want to send browser DOM info to the model?

By default, Midscene does not send browser DOM data to models. If you need to include it for UI understanding (for example, to pass details not visible in a screenshot), set `domIncluded` to `true` in the options for interfaces like `aiAsk` or `aiQuery`. See the [API reference](./api) for details.

### Backward compatibility

From version 1.0 onward, Midscene.js recommends these environment variable names, for example:

- `MIDSCENE_MODEL_API_KEY`
- `MIDSCENE_MODEL_BASE_URL`

For compatibility, we still support these OpenAI-style names, though they are no longer preferred:

- `OPENAI_API_KEY`
- `OPENAI_BASE_URL`

When both are present, Midscene prefers the new `MIDSCENE_MODEL_*` variables.

## Troubleshooting model service connectivity issues

If you want to troubleshoot connectivity issues, you can use the 'connectivity-test' folder in our example project: [https://github.com/web-infra-dev/midscene-example/tree/main/connectivity-test](https://github.com/web-infra-dev/midscene-example/tree/main/connectivity-test)

Put your `.env` file in the `connectivity-test` folder, and run the test with `npm i && npm run test`.




---
url: /quick-experience.md
---





# Quick experience by Chrome extension

Midscene.js provides a Chrome extension. By using it, you can quickly experience the main features of Midscene on any webpage, without needing to set up a code project.

⁠The extension shares the same code as the npm `@midscene/web` packages, so you can think of it as a playground or a way to debug with Midscene.

**Prompt** : Sign up for Github, you need to pass the form validation, but don't actually click. 

<video src="https://lf3-static.bytednsdoc.com/obj/eden-cn/nupipfups/Midscene/1.0-showcases/github2.mp4" poster="https://lf3-static.bytednsdoc.com/obj/eden-cn/nupipfups/Midscene/1.0-showcases/github.png" height="300" controls />

View the full report of this task: [report.html](https://lf3-static.bytednsdoc.com/obj/eden-cn/nupipfups/Midscene/1.0-showcases/github.html)


## Install Chrome extension

<a href="https://chromewebstore.google.com/detail/midscene/gbldofcpkknbggpkmbdaefngejllnief" target="_blank"><img src="https://lf3-static.bytednsdoc.com/obj/eden-cn/vhaeh7vhabf/chrome_extension_store_btn.png" width="200" /></a>

Install Midscene extension from chrome web store: [Midscene](https://chromewebstore.google.com/detail/midscene/gbldofcpkknbggpkmbdaefngejllnief)

Start the extension (may be folded by Chrome extension icon), you should see the sidebar named "Midscene" in the right side of the browser.


## Set up API keys for model

Set your model configs into the environment variables. You may refer to [Model strategy](../model-strategy) for more details.

```bash
export MIDSCENE_MODEL_BASE_URL="https://replace-with-your-model-service-url/v1"
export MIDSCENE_MODEL_API_KEY="replace-with-your-api-key"
export MIDSCENE_MODEL_NAME="replace-with-your-model-name"
export MIDSCENE_MODEL_FAMILY="replace-with-your-model-family"
```

For more configuration details, please refer to [Model strategy](../model-strategy) and [Model configuration](../model-config).


See more showcases: [showcases](./showcases.mdx)

## Start experiencing

After configuration, you can start using Midscene right away. It provides several key operation tabs:

- **Act**: interact with the page. This is Auto Planning, corresponding to `aiAct`. For example:
```
Type “Midscene” in the search box, run the search, and open the first result
```

```
Fill out the registration form and make sure every field passes validation
```

- **Query**: extract JSON data from the interface, corresponding to `aiQuery`.

Similar methods include `aiBoolean()`, `aiNumber()`, and `aiString()` for directly extracting booleans, numbers, and strings.

```
Extract the user ID from the page and return JSON data in the { id: string } structure
```

- **Assert**: understand the page and assert; if the condition is not met, throw an error, corresponding to `aiAssert`.

```
There is a login button on the page, with a user agreement link below it
```

- **Tap**: click on an element. This is Instant Action, corresponding to `aiTap`.
```
Click the login button
```

> For the difference between Auto Planning and Instant Action, see the [API](../api.mdx) document.


## FAQ

### Can I install the extension manually?

If you cannot access the Chrome Web Store, you can download the installation package from the [Releases page](https://github.com/web-infra-dev/midscene/releases) on GitHub and install it manually. However, this is not recommended as automatic updates will not be available.

### Extension fails to run and shows 'Cannot access a chrome-extension:// URL of different extension'

It's mainly due to conflicts with other extensions injecting `<iframe />` or `<script />` into the page. Try disabling the suspicious plugins and refresh.

To find the suspicious plugins:

1. Open the Devtools of the page, find the `<script>` or `<iframe>` with a url like `chrome-extension://{ID-of-the-suspicious-plugin}/...`.
2. Copy the ID from the url, open `chrome://extensions/` , use cmd+f to find the plugin with the same ID, disable it.
3. Refresh the page, try again.



---
url: /showcases-android.md
---

**Prompt** : Open the Booking App, search for a hotel in Tokyo for four adults on Christmas, with a score of 8 or above.

<video src="https://lf3-static.bytednsdoc.com/obj/eden-cn/nupipfups/Midscene/1.0-showcases/booking2.mp4" poster="https://lf3-static.bytednsdoc.com/obj/eden-cn/nupipfups/Midscene/1.0-showcases/booking.png" height="300" controls />

View the full report of this task: [report.html](https://lf3-static.bytednsdoc.com/obj/eden-cn/nupipfups/Midscene/1.0-showcases/booking.html)



---
url: /showcases-computer.md
---

**Prompt** (**macOS**): Help me post a tweet promoting Midscene's support for AutoGLM through safari, with the following requirements:
1. Text content: Midscene now supports AutoGLM!
2. Media content: Use the AutoGLM video from the download folder!

<video src="https://lf3-static.bytednsdoc.com/obj/eden-cn/nupipfups/Midscene/pc-twitter2.mp4" height="300" controls />

View the full report for this task: [report.html](https://lf3-static.bytednsdoc.com/obj/eden-cn/nupipfups/Midscene/1.0-showcases/pc-twitter2-midscene_report.html)

**Prompt** (**Windows**): Open Sauce Demo e-commerce site, login and add items to cart

<video src="https://lf3-static.bytednsdoc.com/obj/eden-cn/nupipfups/Midscene/windows.mov" height="300" controls />

View the full report for this task: [report.html](https://lf3-static.bytednsdoc.com/obj/eden-cn/nupipfups/Midscene/shop-computer-2026-01-14_11-57-36-f8411b8f.html)

**Prompt** (**macOS**): Open Google and query San Jose tomorrow weather temperature

<video src="https://lf3-static.bytednsdoc.com/obj/eden-cn/nupipfups/Midscene/mac.mov" height="300" controls />

View the full report for this task: [report.html](https://lf3-static.bytednsdoc.com/obj/eden-cn/nupipfups/Midscene/weather-computer-2026-01-14_11-26-38-9592ecf5.html)

**Prompt** (**Linux**): Open TodoMVC, add multiple tasks and filter them

<video src="https://lf3-static.bytednsdoc.com/obj/eden-cn/nupipfups/Midscene/linux.mov" height="300" controls />

View the full report for this task: [report.html](https://lf3-static.bytednsdoc.com/obj/eden-cn/nupipfups/Midscene/todo-computer-2026-01-13_15-40-37-6f45fb0f.html)



---
url: /showcases-harmony.md
---

**Prompt** : Open Settings, scroll to find "About phone", view device information.

<video src="https://lf3-static.bytednsdoc.com/obj/eden-cn/nupipfups/Midscene/1.0-showcases/harmony.mp4" poster="https://lf3-static.bytednsdoc.com/obj/eden-cn/nupipfups/Midscene/1.0-showcases/harmony.png" height="300" controls />

View the full report of this task: [report.html](https://lf3-static.bytednsdoc.com/obj/eden-cn/nupipfups/Midscene/1.0-showcases/harmony.html)



---
url: /showcases-ios.md
---

**Prompt** : Open Twitter and auto-like the first tweet by @midscene_ai

<video src="https://lf3-static.bytednsdoc.com/obj/eden-cn/nupipfups/Midscene/1.0-showcases/x2.mp4" poster="https://lf3-static.bytednsdoc.com/obj/eden-cn/nupipfups/Midscene/1.0-showcases/x.png" height="300" controls />

View the full report of this task: [report.html](https://lf3-static.bytednsdoc.com/obj/eden-cn/nupipfups/Midscene/1.0-showcases/x.html)



---
url: /showcases-mcp.md
---

**Prompt** : Use GitHub MCP to check if the latest commit of Midscene has released a version. If not, invoke the Midscene MCP to navigate to the action page and run the release action to publish a prepatch version.

<video src="https://lf3-static.bytednsdoc.com/obj/eden-cn/nupipfups/Midscene/1.0-showcases/mcp2.mp4" poster="https://lf3-static.bytednsdoc.com/obj/eden-cn/nupipfups/Midscene/1.0-showcases/mcp.png" height="300" controls />

View the full report of this task: [report.html](https://lf3-static.bytednsdoc.com/obj/eden-cn/nupipfups/Midscene/1.0-showcases/mcp.html)



---
url: /showcases-web.md
---

**Prompt** : Sign up for Github, you need to pass the form validation, but don't actually click. 

<video src="https://lf3-static.bytednsdoc.com/obj/eden-cn/nupipfups/Midscene/1.0-showcases/github2.mp4" poster="https://lf3-static.bytednsdoc.com/obj/eden-cn/nupipfups/Midscene/1.0-showcases/github.png" height="300" controls />

View the full report of this task: [report.html](https://lf3-static.bytednsdoc.com/obj/eden-cn/nupipfups/Midscene/1.0-showcases/github.html)



---
url: /showcases.md
---

# Showcases

This doc showcases some cases built with Midscene.









## Web

**Prompt** : Sign up for Github, you need to pass the form validation, but don't actually click. 

<video src="https://lf3-static.bytednsdoc.com/obj/eden-cn/nupipfups/Midscene/1.0-showcases/github2.mp4" poster="https://lf3-static.bytednsdoc.com/obj/eden-cn/nupipfups/Midscene/1.0-showcases/github.png" height="300" controls />

View the full report of this task: [report.html](https://lf3-static.bytednsdoc.com/obj/eden-cn/nupipfups/Midscene/1.0-showcases/github.html)


## iOS

**Prompt** : Open Twitter and auto-like the first tweet by @midscene_ai

<video src="https://lf3-static.bytednsdoc.com/obj/eden-cn/nupipfups/Midscene/1.0-showcases/x2.mp4" poster="https://lf3-static.bytednsdoc.com/obj/eden-cn/nupipfups/Midscene/1.0-showcases/x.png" height="300" controls />

View the full report of this task: [report.html](https://lf3-static.bytednsdoc.com/obj/eden-cn/nupipfups/Midscene/1.0-showcases/x.html)


## Android

**Prompt** : Open the Booking App, search for a hotel in Tokyo for four adults on Christmas, with a score of 8 or above.

<video src="https://lf3-static.bytednsdoc.com/obj/eden-cn/nupipfups/Midscene/1.0-showcases/booking2.mp4" poster="https://lf3-static.bytednsdoc.com/obj/eden-cn/nupipfups/Midscene/1.0-showcases/booking.png" height="300" controls />

View the full report of this task: [report.html](https://lf3-static.bytednsdoc.com/obj/eden-cn/nupipfups/Midscene/1.0-showcases/booking.html)


## HarmonyOS

**Prompt** : Open Settings, scroll to find "About phone", view device information.

<video src="https://lf3-static.bytednsdoc.com/obj/eden-cn/nupipfups/Midscene/1.0-showcases/harmony.mp4" poster="https://lf3-static.bytednsdoc.com/obj/eden-cn/nupipfups/Midscene/1.0-showcases/harmony.png" height="300" controls />

View the full report of this task: [report.html](https://lf3-static.bytednsdoc.com/obj/eden-cn/nupipfups/Midscene/1.0-showcases/harmony.html)


## Computer

**Prompt** (**macOS**): Help me post a tweet promoting Midscene's support for AutoGLM through safari, with the following requirements:
1. Text content: Midscene now supports AutoGLM!
2. Media content: Use the AutoGLM video from the download folder!

<video src="https://lf3-static.bytednsdoc.com/obj/eden-cn/nupipfups/Midscene/pc-twitter2.mp4" height="300" controls />

View the full report for this task: [report.html](https://lf3-static.bytednsdoc.com/obj/eden-cn/nupipfups/Midscene/1.0-showcases/pc-twitter2-midscene_report.html)

**Prompt** (**Windows**): Open Sauce Demo e-commerce site, login and add items to cart

<video src="https://lf3-static.bytednsdoc.com/obj/eden-cn/nupipfups/Midscene/windows.mov" height="300" controls />

View the full report for this task: [report.html](https://lf3-static.bytednsdoc.com/obj/eden-cn/nupipfups/Midscene/shop-computer-2026-01-14_11-57-36-f8411b8f.html)

**Prompt** (**macOS**): Open Google and query San Jose tomorrow weather temperature

<video src="https://lf3-static.bytednsdoc.com/obj/eden-cn/nupipfups/Midscene/mac.mov" height="300" controls />

View the full report for this task: [report.html](https://lf3-static.bytednsdoc.com/obj/eden-cn/nupipfups/Midscene/weather-computer-2026-01-14_11-26-38-9592ecf5.html)

**Prompt** (**Linux**): Open TodoMVC, add multiple tasks and filter them

<video src="https://lf3-static.bytednsdoc.com/obj/eden-cn/nupipfups/Midscene/linux.mov" height="300" controls />

View the full report for this task: [report.html](https://lf3-static.bytednsdoc.com/obj/eden-cn/nupipfups/Midscene/todo-computer-2026-01-13_15-40-37-6f45fb0f.html)


## MCP

**Prompt** : Use GitHub MCP to check if the latest commit of Midscene has released a version. If not, invoke the Midscene MCP to navigate to the action page and run the release action to publish a prepatch version.

<video src="https://lf3-static.bytednsdoc.com/obj/eden-cn/nupipfups/Midscene/1.0-showcases/mcp2.mp4" poster="https://lf3-static.bytednsdoc.com/obj/eden-cn/nupipfups/Midscene/1.0-showcases/mcp.png" height="300" controls />

View the full report of this task: [report.html](https://lf3-static.bytednsdoc.com/obj/eden-cn/nupipfups/Midscene/1.0-showcases/mcp.html)


## Community showcases

Some community developers have successfully built on Midscene's capability to [integrate with any interface](./integrate-with-any-interface), extending it with a robotic arm plus vision and voice models for in-vehicle large-screen testing scenarios.

<video src="https://lf3-static.bytednsdoc.com/obj/eden-cn/vhaeh7vhabf/AI_Vision_Powered_Robotic_Arm.mp4" height="300" controls />



---
url: /skills.md
---

# Control any platform with Skills

[Agent Skills](https://github.com/anthropics/agent-skills) are a format for extending AI coding agents with specialized capabilities. Midscene provides Agent Skills that let AI coding tools (like Claude Code, Cline, etc.) drive UI automation through CLI commands — no MCP server setup required.

Unlike [MCP integration](./mcp), Skills work by running CLI commands directly in the terminal. The AI agent acts as the brain: it takes screenshots, analyzes the UI, and decides which actions to perform next.

## Supported platforms

| Skill | Package | CLI command | Description |
|-------|---------|-------------|-------------|
| Browser Automation | `@midscene/web` | `npx @midscene/web` | Headless Chrome via Puppeteer, opens new browser tabs |
| Chrome Bridge Automation | `@midscene/web` | `npx @midscene/web --bridge` | User's own Chrome browser, preserves cookies and sessions |
| Desktop Computer Automation | `@midscene/computer` | `npx @midscene/computer` | macOS, Windows, Linux desktop control |
| Android Device Automation | `@midscene/android` | `npx @midscene/android` | Android device control via ADB |
| iOS Device Automation | `@midscene/ios` | `npx @midscene/ios` | iOS device control via WebDriverAgent |

## Installation

Make sure [Node.js](https://nodejs.org) is installed, then run:

```bash
# General installation
npx skills add web-infra-dev/midscene-skills

# Claude Code
npx skills add web-infra-dev/midscene-skills -a claude-code

# OpenClaw
npx skills add web-infra-dev/midscene-skills -a openclaw
```

Skills repository: [github.com/web-infra-dev/midscene-skills](https://github.com/web-infra-dev/midscene-skills)

## Model configuration

Midscene skills require a vision model with strong visual grounding capabilities. Configure the following environment variables — either as system environment variables or in a `.env` file in the current working directory (Midscene loads `.env` automatically):

```bash
MIDSCENE_MODEL_API_KEY="your-api-key"
MIDSCENE_MODEL_NAME="model-name"
MIDSCENE_MODEL_BASE_URL="https://..."
MIDSCENE_MODEL_FAMILY="family-identifier"
```

For supported models and configuration details, see [Model strategy](./model-strategy) and [Common model configuration](./model-common-config).

## Use skills

In your AI chat assistant, you can use the following command to use skills:

```markdown
Open photo app, see what is the first photo in the album.
```

## More

Please refer to the [Skills Repository](https://github.com/web-infra-dev/midscene-skills) for more details.





---
url: /use-javascript-to-optimize-ai-automation-code.md
---

# Use JavaScript to optimize the AI automation code

Many developers love using `aiAct` or `ai` to accomplish automation tasks, even packing long, complex logic into a single natural-language instruction. It feels “smart,” but in practice you may run into unstable reproducibility and slower performance.

This article shares an approach for writing automation scripts with JavaScript and structured APIs.

## Write automation scripts with JavaScript and structured APIs

Midscene provides structured API methods like `aiBoolean`, `aiString`, and `aiNumber` to extract state from the UI. Combined with instant action methods like `aiTap`, `aiInput`, `aiScroll`, and `aiHover`, you can break complex logic into steps to make automation more stable.

### A simple example

Take this prompt as an example:

```txt
click all the records one by one. If one record contains the text "completed", skip it
```

By composing the structured APIs, you can convert the prompt into more reliable, maintainable code:
```javascript
const recordList = await agent.aiQuery('string[], the record list')
for (const record of recordList) {
  const hasCompleted = await agent.aiBoolean(`check if the record ${record}" contains the text "completed"`)
  if (!hasCompleted) {
    await agent.aiTap(record)
  }
}
```

After changing the coding style, the process becomes more reliable and easier to maintain, and you can use traditional debugging to control the execution flow.

### A complex example

Here is another example, shown before rewriting:

```javascript
aiAct(`
1. click the first unfollowed user, enter the user's homepage
2. click the follow button
3. go back to the previous page
4. if all users are followed, scroll down one screen
5. repeat the above steps until all users are followed
`)
```

Using structured APIs, you can lock this flow into code:

```javascript
let user = await agent.aiQuery('string[], the unfollowed user names in the list')
let currentUserIndex = 0

while (user.length > 0) {
  console.log('current user is', user[currentUserIndex])
  await agent.aiTap(user[currentUserIndex])
  try {
    await agent.aiTap('follow button')
  } catch (e) {
    // ignore if error
  }
  // Go back to the previous page
  await agent.aiTap('back button')
  
  currentUserIndex++
  
  // Check if we've gone through all users in the current list
  if (currentUserIndex >= user.length) {
    // Scroll down to load more users
    await agent.aiScroll({
      direction: 'down',
      scrollType: 'once',
    })
    
    // Get the updated user list
    user = await agent.aiQuery('string[], the unfollowed user names in the list')
    currentUserIndex = 0
  }
}
```

## Commonly-used structured API methods

Here are some commonly-used structured API methods:

### `aiBoolean` - conditional decision

* Use Case: Condition Judgment, State Detection
* Advantage: Convert fuzzy descriptions into clear boolean values

Example:
```javascript
const hasAlreadyChat = await agent.aiBoolean('check if I have already sent a message to him/her')
if (hasAlreadyChat) {
  // ...
}
```

### `aiString` - text extraction

* Use Case: Text Content Retrieval
* Advantage: Avoid Ambiguity in Natural Language Descriptions

Example:
```javascript
const username = await agent.aiString('the nickname of the first user in the list')
console.log('username is', username)
```

### `aiNumber` - numerical extraction

* Use Case: Counting, Numerical Comparison, Loop Control
* Advantage: Ensure Return Standard Numeric Types

Example:
```javascript
const unreadCount = await agent.aiNumber('the number of unread messages on the message icon')
for (let i = 0; i < unreadCount; i++) {
  // ...
}
```

### `aiQuery` - general data extraction

* Use Case: Extract any data type
* Advantage: Flexible Data Type Handling

Example:
```javascript
const userList = await agent.aiQuery('string[], the user list')
```

### Instant action methods

Midscene provides some instant action methods, like `aiTap`, `aiInput`, `aiScroll`, `aiHover`, etc., They are also commonly used in the automation code. You can check them in the [API](./api.mdx) page.

## Which approach is best: `aiAct` or structured code?

There is no standard answer. It depends on the model's ability, the complexity of the actual business.

Generally, if you encounter the following situations, you should consider abandoning the `aiAct` method:

- The success rate of `aiAct` does not meet the requirements after multiple retries
- You have already felt tired and spent too much time repeatedly tuning the `aiAct` prompt
- You need to debug the script step by step

## Want to write structured code easily?

If you think the JavaScript code above is hard to write, now is the time to use an AI IDE.

Use your AI IDE to index our docs:

- https://midscenejs.com/use-javascript-to-optimize-ai-automation-code.md
- https://midscenejs.com/api.md

To learn how to add Midscene docs to your AI IDE, see [this article](./llm-txt.mdx#usage).



---
url: /web-api-reference.md
---

# API reference (Web)

Use this doc when you need to customize Midscene's browser automation agents or review browser-only constructor options. For shared parameters (reporting, hooks, caching, etc.), see the platform-agnostic [API reference (Common)](./api).

## Action Space

PuppeteerAgent, PlaywrightAgent, and Chrome Bridge share one action space; the Midscene Agent can use these actions while planning tasks:

- `Tap` &mdash; Left-click an element.
- `RightClick` &mdash; Right-click an element.
- `DoubleClick` &mdash; Double-click an element.
- `Hover` &mdash; Hover over an element.
- `Input` &mdash; Enter text with `replace`/`append`/`clear` modes.
- `KeyboardPress` &mdash; Press a specified key (optionally focusing a target element first).
- `Scroll` &mdash; Scroll from an element or screen center; supports scroll-to-top/bottom/left/right helpers.
- `DragAndDrop` &mdash; Drag from one element to another.
- `LongPress` &mdash; Long-press a target element with optional duration.
- `Swipe` &mdash; Touch-style swipe gesture (available when `enableTouchEventsInActionSpace` is `true`).
- `ClearInput` &mdash; Clear the contents of an input field.
- `Navigate` &mdash; Open a URL in the current tab.
- `Reload` &mdash; Reload the page.
- `GoBack` &mdash; Navigate back in history.

## PuppeteerAgent {#puppeteer-agent}

Use Midscene against a Puppeteer-controlled browser when you need AI actions in your own Puppeteer workflows.

### Import

```ts
import { PuppeteerAgent } from '@midscene/web/puppeteer';
```

### Constructor

```ts
const agent = new PuppeteerAgent(page, {
  // browser-specific options...
});
```

### Browser-specific options

In addition to the base agent options, Puppeteer exposes:

- `forceSameTabNavigation: boolean` &mdash; Restrict navigation to the current tab. Default `true`.
- `waitForNavigationTimeout: number` &mdash; Maximum wait when a step causes navigation. Default `5000` (set `0` to skip waiting).
- `waitForNetworkIdleTimeout: number` &mdash; Wait for network idle between actions to reduce flakiness. Default `2000` (set `0` to skip waiting).
- `enableTouchEventsInActionSpace: boolean` &mdash; Add touch gestures (like swipe) to the action space so the agent can handle touch-only interactions. Default `false`.
- `forceChromeSelectRendering: boolean` &mdash; Force `select` elements to render with Chrome's base-select styling so they're visible in screenshots/element extraction; requires Puppeteer > `24.6.0`.
- `customActions: DeviceAction[]` &mdash; Register bespoke actions defined via `defineAction` so planning can call domain-specific steps.

### Usage notes

:::info

- One agent per page: by default (`forceSameTabNavigation: true`), Midscene opens new links in the current tab for easier debugging. Set it to `false` if you want new tabs, and create a new agent per tab.
- For the full list of interaction methods, see [API reference (Common)](./api#interaction-methods).

:::

### Examples

#### Quick start

```ts
import puppeteer from 'puppeteer';
import { PuppeteerAgent } from '@midscene/web/puppeteer';

const browser = await puppeteer.launch({ headless: false });
const page = await browser.newPage();
await page.goto('https://www.ebay.com');

const agent = new PuppeteerAgent(page, {
  actionContext: 'When a cookie dialog appears, accept it.',
});

await agent.aiAct('search "Noise cancelling headphones" and open first result');
const items = await agent.aiQuery(
  '{itemTitle: string, price: number}[], list two products with price',
);
console.log(items);

await agent.aiAssert('there is a category filter on the left sidebar');
await browser.close();
```

#### Connect to a remote Puppeteer browser

```ts
import puppeteer from 'puppeteer';
import { PuppeteerAgent } from '@midscene/web/puppeteer';

const browser = await puppeteer.connect({
  browserWSEndpoint: process.env.REMOTE_CDP_URL!,
});

const [page = await browser.newPage()] = await browser.pages();
const agent = new PuppeteerAgent(page, {
  waitForNetworkIdleTimeout: 0,
});

await agent.aiAct('open https://example.com and click the login button');
await agent.destroy();
await browser.disconnect();
```

### See also

- [Integrate with Puppeteer](./integrate-with-puppeteer) for installation, fixtures, and remote-CDP guidance.

## PlaywrightAgent {#playwright-agent}

Use Midscene inside a Playwright browser for AI-driven testing or automation alongside your Playwright flows.

### Import

```ts
import { PlaywrightAgent } from '@midscene/web/playwright';
```

### Constructor

```ts
const agent = new PlaywrightAgent(page, {
  // browser-specific options...
});
```

### Browser-specific options

- `forceSameTabNavigation: boolean` &mdash; Keep automation inside the active tab. Default `true`.
- `waitForNavigationTimeout: number` &mdash; Wait time for navigation completion. Default `5000` (set `0` to disable).
- `waitForNetworkIdleTimeout: number` &mdash; Wait between actions for network idle. Default `2000` (set `0` to disable).
- `enableTouchEventsInActionSpace: boolean` &mdash; Add touch gestures (like swipe) to the action space so the agent can handle touch-only interactions. Default `false`.
- `forceChromeSelectRendering: boolean` &mdash; Force `select` elements to render with Chrome's base-select styling so they're visible in screenshots/element extraction; requires Playwright ≥ `1.52.0`.
- `customActions: DeviceAction[]` &mdash; Extend planning with project-specific actions.

### Usage notes

:::info

- One agent per page: with `forceSameTabNavigation` (default `true`), Midscene intercepts new tabs for stability. Set it to `false` to allow new tabs and create a separate agent for each.
- For the full list of interaction methods, see [API reference (Common)](./api#interaction-methods).

:::

### Examples

#### Quick start

```ts
import { chromium } from 'playwright';
import { PlaywrightAgent } from '@midscene/web/playwright';

const browser = await chromium.launch({ headless: true });
const page = await browser.newPage();
await page.goto('https://www.ebay.com');

const agent = new PlaywrightAgent(page);
await agent.aiAct('search "Noise cancelling headphones" and wait for results');
await agent.aiWaitFor('the results grid becomes visible');

const price = await agent.aiNumber('price of the first headphone');
console.log('first price', price);

await agent.aiTap('click the first result card');
await browser.close();
```

#### Extend Playwright tests with Midscene fixtures

```ts
// playwright.config.ts
export default defineConfig({
  reporter: [['list'], ['@midscene/web/playwright-reporter']],
});

// e2e/fixture.ts
import { test as base } from '@playwright/test';
import { PlaywrightAiFixture } from '@midscene/web/playwright';

export const test = base.extend(
  PlaywrightAiFixture({ waitForNetworkIdleTimeout: 1000 }),
);

// e2e/examples.spec.ts
test('search flow', async ({ agentForPage, page }) => {
  await page.goto('https://www.ebay.com');
  const agent = await agentForPage(page);
  await agent.aiAct('search "keyboard" and open first listing');
  await agent.aiAssert('a product detail page is opened');
});
```

### See also

- [Integrate with Playwright](./integrate-with-playwright) for setup, fixtures, and advanced configuration.

## Chrome Bridge Agent {#chrome-bridge-agent}

Bridge Mode lets Midscene operate your currently active desktop Chrome tab through the extension instead of launching a dedicated automation browser.

### Import

```ts
import { AgentOverChromeBridge } from '@midscene/web/bridge-mode';
```

### Constructor

```ts
const agent = new AgentOverChromeBridge({
  allowRemoteAccess: false,
  // other bridge options...
});
```

### Bridge options

- `closeNewTabsAfterDisconnect?: boolean` &mdash; Close any bridge-created tabs when the agent is destroyed. Default `false`.
- `allowRemoteAccess?: boolean` &mdash; Allow remote machines to attach. Defaults to `false` (binds to `127.0.0.1`).
- `host?: string` &mdash; Override the interface for the bridge server. Takes precedence over `allowRemoteAccess`.
- `port?: number` &mdash; TCP port for the bridge server. Default `3766`.

See [Bridge Mode by Chrome extension](./bridge-mode#constructor) for full installation and capability details.

### Usage notes

:::info

Call `connectCurrentTab` or `connectNewTabWithUrl` before issuing other actions. Each `AgentOverChromeBridge` instance can only attach to one tab; create a new instance after `destroy`.

:::

### Bridge methods

#### `connectCurrentTab()`

```ts
function connectCurrentTab(options?: {
  forceSameTabNavigation?: boolean;
}): Promise<void>;
```

- `options.forceSameTabNavigation` (default `true`) intercepts new tabs and opens them in the current tab to simplify debugging; set to `false` if you want normal new-tab behavior (create a separate agent per tab).
- Resolves on a successful handshake with the active tab; rejects if the extension is not allowed to connect.

#### `connectNewTabWithUrl()`

```ts
function connectNewTabWithUrl(
  url: string,
  options?: { forceSameTabNavigation?: boolean },
): Promise<void>;
```

- `url` &mdash; Address to open in a new desktop tab before attaching.
- `options` &mdash; Same as `connectCurrentTab`.
- Resolves when the new tab is opened and the bridge is connected.

#### `destroy()`

```ts
function destroy(closeNewTabsAfterDisconnect?: boolean): Promise<void>;
```

- `closeNewTabsAfterDisconnect` &mdash; Optional runtime override for the constructor setting; `true` closes bridge-created tabs on teardown.
- Resolves after the bridge connection and local server are fully cleaned up.

### Examples

#### Open a new desktop tab

```ts
import { AgentOverChromeBridge } from '@midscene/web/bridge-mode';

const agent = new AgentOverChromeBridge();
await agent.connectNewTabWithUrl('https://www.bing.com');

await agent.ai('search "AI automation" and summarise first result');
await agent.aiAssert('some search results show up');
await agent.destroy();
```

#### Attach to current tab

```ts
import { AgentOverChromeBridge } from '@midscene/web/bridge-mode';

const agent = new AgentOverChromeBridge({
  allowRemoteAccess: false,
  closeNewTabsAfterDisconnect: true,
});

await agent.connectCurrentTab({ forceSameTabNavigation: true });
await agent.aiAct('open Gmail and report how many unread emails are visible');
await agent.destroy();
```

### See also

- [API reference (Common)](./api#interaction-methods) for shared agent methods.
- [Bridge mode](./bridge-mode) for extension setup, command sequence, and YAML usage.



---
url: /yaml-script-runner.md
---



# YAML script runner

Midscene defines a YAML-based scripting format so you can quickly author automation scripts, then run them from the command line without extra setup. For more details on YAML scripts, see [Automate with scripts in YAML](./automate-with-scripts-in-yaml).

For example, you can write a YAML script like this:

```yaml
web:
  url: https://www.bing.com

tasks:
  - name: Search for weather
    flow:
      - ai: Search for "today's weather"
      - sleep: 3000
      - aiAssert: The results show weather information
```

Run it with one command:

```bash
midscene ./bing-search.yaml
```

The CLI prints execution progress and generates a visual report when it finishes, while keeping setup simple.

## Configure environment variables with `.env`

The Midscene CLI uses [dotenv](https://www.npmjs.com/package/dotenv) to load a `.env` file from the directory where you run the tool. Create a `.env` file and add:

```ini filename=.env
MIDSCENE_MODEL_BASE_URL="replace with your model service URL/v1"
MIDSCENE_MODEL_API_KEY="replace with your API Key"
MIDSCENE_MODEL_NAME="replace with your model name"
MIDSCENE_MODEL_FAMILY="replace with your model family"
```

For more configuration details, see the [Model strategy](./model-strategy) guide.

Notes:

- The file is optional; you can also set global environment variables instead.
- Do not add an `export` prefix—this is how dotenv expects values.
- Place `.env` in the directory where you run the tool, not necessarily next to the YAML file.
- These values do **not** override existing global environment variables unless you enable `--dotenv-override` (see below).
- Use `--dotenv-debug` if you need to debug how environment variables load.

## Get started

### Install the CLI

Install `@midscene/cli` globally (recommended for first-time users):

```bash
npm i -g @midscene/cli
```

Or install it per project:

```bash
npm i @midscene/cli --save-dev
```

### Write your first script

Create `bing-search.yaml` to drive a web browser:

```yaml
web:
  url: https://www.bing.com

tasks:
  - name: Search for weather
    flow:
      - ai: Search for "today's weather"
      - sleep: 3000
      - aiAssert: The results show weather information
```

Drive an Android device connected over adb:

```yaml
android:
  deviceId: s4ey59 # find the device id with `adb devices`

tasks:
  - name: Maps Navigation
    flow:
      - ai: Open the Maps app
      - ai: Input 'West Lake, Hangzhou' in the search bar, and click the search button
      - ai: Click the first search result, enter the details page
      - ai: Click "Directions" button, enter the route planning page
      - ai: Click "Start" button to start navigation
```

Or drive an iOS device with WebDriverAgent configured:

```yaml
ios:
  wdaPort: 8100

tasks:
  - name: Change System Settings
    flow:
      - ai: Open the Settings app
      - ai: Tap "Display & Brightness"
      - ai: Turn on "Dark Mode"
      - aiAssert: Dark Mode is enabled
```

### Run the script

```bash
midscene ./bing-search.yaml
# If Midscene is installed in your project
npx midscene ./bing-search.yaml
```

The CLI prints execution progress and generates a visual report when it finishes.

## Advanced usage of the command-line tool

### Use environment variables in `.yaml`

Reference environment variables in your scripts with `${variable-name}`.

```ini filename=.env
topic=weather today
```

```yaml
# ...
- ai: type ${topic} in input box
# ...
```

### Run multiple scripts

`@midscene/cli` supports glob patterns to batch-execute scripts, which is a shorthand for the `--files` argument.

```bash
# Run a single script
midscene ./bing-search.yaml

# Use a glob pattern to run all matching scripts
midscene './scripts/**/*.yaml'
```

### Analyze command-line output

After execution, the output directory contains:

- A JSON summary specified by `--summary` (defaults to `index.json`) with execution status and statistics for all scripts.
- Individual execution results for each YAML file (JSON).
- Visual reports for each script (HTML).

### Run in headed mode

> `web` scenarios only

Headed mode opens the browser window. By default, scripts run headless.

```bash
# Run in headed mode
midscene /path/to/yaml --headed

# Run in headed mode and keep the window after finishing
midscene /path/to/yaml --keep-window
```

### Use bridge mode

> `web` scenarios only

Bridge mode lets YAML scripts drive your existing desktop browser so you can reuse cookies, extensions, or state. Install the Chrome extension, then add:

```diff
web:
  url: https://www.bing.com
+ bridgeMode: newTabWithUrl
```

See [Bridge Mode via Chrome Extension](./bridge-mode) for details.

### Run YAML scripts with JavaScript

Call the Agent's [`runYaml`](./api#runyaml) method to execute YAML from JavaScript. This runs only the `tasks` section of the script.

## Command-line options

The CLI provides parameters to control how scripts run:

- `--files <file1> <file2> ...`: List of script files. Executes in order, sequentially by default (`--concurrent` is `1`), or concurrently when `--concurrent` is set. Supports [glob](https://www.npmjs.com/package/glob) patterns.
- `--concurrent <number>`: Number of concurrent executions. Default `1`.
- `--continue-on-error`: Continue running remaining scripts even if one fails. Default off.
- `--share-browser-context`: Share browser context (cookies, `localStorage`, etc.) across scripts. Default off.
- `--summary <filename>`: Path for the JSON summary report.
- `--headed`: Run in a headed browser instead of headless.
- `--keep-window`: Keep the browser window after execution; enables `--headed` automatically.
- `--config <filename>`: Config file whose values become defaults for CLI arguments.
- `--web.userAgent <ua>`: Override `web.userAgent` for all scripts.
- `--web.viewportWidth <width>`: Override `web.viewportWidth` for all scripts.
- `--web.viewportHeight <height>`: Override `web.viewportHeight` for all scripts.
- `--android.deviceId <device-id>`: Override `android.deviceId` for all scripts.
- `--ios.wdaPort <port>`: Override `ios.wdaPort` for all scripts.
- `--ios.wdaHost <host>`: Override `ios.wdaHost` for all scripts.
- `--dotenv-debug`: Enable dotenv debug logs. Default off.
- `--dotenv-override`: Allow dotenv to override global environment variables. Default off.

Examples:

Use `--files` to specify execution order:

```bash
midscene --files ./login.yaml ./buy/*.yaml ./checkout.yaml
```

Run all scripts with a concurrency of 4 and continue when errors occur:

```bash
midscene --files './scripts/**/*.yaml' --concurrent 4 --continue-on-error
```

### Write command-line arguments in a file

You can place arguments in a YAML config file and reference it with `--config`. Command-line arguments take priority over the config file.

```yaml
files:
  - './scripts/login.yaml'
  - './scripts/search.yaml'
  - './scripts/**/*.yaml'

concurrent: 4
continueOnError: true
shareBrowserContext: true
```

Run with:

```bash
midscene --config ./config.yaml
```

## FAQ

**How can I export cookies from Chrome as JSON?**

Use this [Chrome extension](https://chromewebstore.google.com/detail/get-cookiestxt-locally/cclelndahbckbenkjhflpdbgdldlbecc) to export cookies.

**How can I view dotenv debug logs?**

Use the `--dotenv-debug` flag:

```bash
midscene /path/to/yaml --dotenv-debug=true
```