---
url: /API.md
---

# API Reference

> In the documentation below, you might see function calls prefixed with `agent.`. If you utilize destructuring in Playwright (e.g., `async ({ ai, aiQuery }) => { /* ... */ }`), you can call these functions without the `agent.` prefix. This is merely a syntactical difference.

## Constructors

Each Agent in Midscene has its own constructor.

* In Puppeteer, use [PuppeteerAgent](/en/integrate-with-puppeteer.md)
* In Bridge Mode, use [AgentOverChromeBridge](/en/bridge-mode-by-chrome-extension.md#constructor)

These Agents share some common constructor parameters:

* `generateReport: boolean`: If true, a report file will be generated. (Default: true)
* `autoPrintReportMsg: boolean`: If true, report messages will be printed. (Default: true)
* `cacheId: string | undefined`: If provided, this cacheId will be used to save or match the cache. (Default: undefined, means cache feature is disabled)
* `actionContext: string`: Some background knowledge that should be sent to the AI model when calling `agent.aiAction()`, like 'close the cookie consent dialog first if it exists' (Default: undefined)

In Puppeteer, there are 3 additional parameters:

* `forceSameTabNavigation: boolean`: If true, page navigation is restricted to the current tab. (Default: true)
* `waitForNetworkIdleTimeout: number`: The timeout for waiting for network idle between each action. (Default: 2000ms, set to 0 to disable the timeout)
* `waitForNavigationTimeout: number`: The timeout for waiting for navigation finished. (Default: 5000ms, set to 0 to disable the timeout)

## Interaction Methods

Below are the main APIs available for the various Agents in Midscene.

:::info Auto Planning v.s. Instant Action

In Midscene, you can choose to use either auto planning or instant action.

* `agent.ai()` is for Auto Planning: Midscene will automatically plan the steps and execute them. It's more smart and looks like more fashionable style for AI agents. But it may be slower and heavily rely on the quality of the AI model.
* `agent.aiTap()`, `agent.aiHover()`, `agent.aiInput()`, `agent.aiKeyboardPress()`, `agent.aiScroll()` are for Instant Action: Midscene will directly perform the specified action, while the AI model is responsible for basic tasks such as locating elements. It's faster and more reliable if you are certain about the action you want to perform.

:::

### `agent.aiAction()` or `.ai()`

This method allows you to perform a series of UI actions described in natural language. Midscene automatically plans the steps and executes them.

* Type

```typescript
function aiAction(prompt: string): Promise<void>;
function ai(prompt: string): Promise<void>; // shorthand form
```

* Parameters:
  * `prompt: string` - A natural language description of the UI steps.

* Return Value:
  * Returns a Promise that resolves to void when all steps are completed; if execution fails, an error is thrown.

* Examples:

```typescript
// Basic usage
await agent.aiAction('Type "JavaScript" into the search box, then click the search button');

// Using the shorthand .ai form
await agent.ai('Click the login button at the top of the page, then enter "test@example.com" in the username field');

// When using UI Agent models like ui-tars, you can try a more high-level prompt
await agent.aiAction('Post a Tweet "Hello World"');
```

:::tip

Under the hood, Midscene uses AI model to split the instruction into a series of steps (a.k.a. "Planning"). It then executes these steps sequentially. If Midscene determines that the actions cannot be performed, an error will be thrown.

For optimal results, please provide clear and detailed instructions for `agent.aiAction()`. For guides about writing prompts, you may read this doc: [Tips for Writing Prompts](/en/prompting-tips.md).

Related Documentation:

* [Choose a model](/en/choose-a-model.md)

:::

### `agent.aiTap()`

Tap something.

* Type

```typescript
function aiTap(locate: string, options?: Object): Promise<void>;
```

* Parameters:
  * `locate: string` - A natural language description of the element to tap.
  * `options?: Object` - Optional, a configuration object containing:
    * `deepThink?: boolean` - If true, Midscene will call AI model twice to precisely locate the element.

* Return Value:
  * Returns a `Promise<void>`

* Examples:

```typescript
await agent.aiTap('The login button at the top of the page');

// Use deepThink feature to precisely locate the element
await agent.aiTap('The login button at the top of the page', { deepThink: true });
```

### `agent.aiHover()`

Move mouse over something.

* Type

```typescript
function aiHover(locate: string, options?: Object): Promise<void>;
```

* Parameters:
  * `locate: string` - A natural language description of the element to hover over.
  * `options?: Object` - Optional, a configuration object containing:
    * `deepThink?: boolean` - If true, Midscene will call AI model twice to precisely locate the element.

* Return Value:
  * Returns a `Promise<void>`

* Examples:

```typescript
await agent.aiHover('The version number of the current page');
```

### `agent.aiInput()`

Input text into something.

* Type

```typescript
function aiInput(text: string, locate: string, options?: Object): Promise<void>;
```

* Parameters:
  * `text: string` - The final text content that should be placed in the input element. Use blank string to clear the input.
  * `locate: string` - A natural language description of the element to input text into.
  * `options?: Object` - Optional, a configuration object containing:
    * `deepThink?: boolean` - If true, Midscene will call AI model twice to precisely locate the element.

* Return Value:
  * Returns a `Promise<void>`

* Examples:

```typescript
await agent.aiInput('Hello World', 'The search input box');
```

### `agent.aiKeyboardPress()`

Press a keyboard key.

* Type

```typescript
function aiKeyboardPress(key: string, locate?: string, options?: Object): Promise<void>;
```

* Parameters:
  * `key: string` - The web key to press, e.g. 'Enter', 'Tab', 'Escape', etc. Key Combination is not supported.
  * `locate?: string` - Optional, a natural language description of the element to press the key on.
  * `options?: Object` - Optional, a configuration object containing:
    * `deepThink?: boolean` - If true, Midscene will call AI model twice to precisely locate the element.

* Return Value:
  * Returns a `Promise<void>`

* Examples:

```typescript
await agent.aiKeyboardPress('Enter', 'The search input box');
```

### `agent.aiScroll()`

Scroll a page or an element.

* Type

```typescript
function aiScroll(scrollParam: PlanningActionParamScroll, locate?: string, options?: Object): Promise<void>;
```

* Parameters:
  * `scrollParam: PlanningActionParamScroll` - The scroll parameter
    * `direction: 'up' | 'down' | 'left' | 'right'` - The direction to scroll.
    * `scrollType: 'once' | 'untilBottom' | 'untilTop' | 'untilRight' | 'untilLeft'` - Optional, the type of scroll to perform.
    * `distance: number` - Optional, the distance to scroll in px.
  * `locate?: string` - Optional, a natural language description of the element to scroll on. If not provided, Midscene will perform scroll on the current mouse position.
  * `options?: Object` - Optional, a configuration object containing:
    * `deepThink?: boolean` - If true, Midscene will call AI model twice to precisely locate the element.

* Return Value:
  * Returns a `Promise<void>`

* Examples:

```typescript
await agent.aiScroll({ direction: 'up', distance: 100, scrollType: 'once' }, 'The form panel');
```

:::tip About the `deepThink` feature

The `deepThink` feature is a powerful feature that allows Midscene to call AI model twice to precisely locate the element. It is useful when the AI model find it hard to distinguish the element from its surroundings.

:::

## Data Extraction

### `agent.aiQuery()`

This method allows you to extract data directly from the UI using multimodal AI reasoning capabilities. Simply define the expected format (e.g., string, number, JSON, or an array) in the `dataDemand`, and Midscene will return a result that matches the format.

* Type

```typescript
function aiQuery<T>(dataShape: string | Object): Promise<T>;
```

* Parameters:
  * `dataShape: T`: A description of the expected return format.

* Return Value:
  * Returns any valid basic type, such as string, number, JSON, array, etc.
  * Just describe the format in `dataDemand`, and Midscene will return a matching result.

* Examples:

```typescript
const dataA = await agent.aiQuery({
  time: 'The date and time displayed in the top-left corner as a string',
  userInfo: 'User information in the format {name: string}',
  tableFields: 'An array of table field names, string[]',
  tableDataRecord: 'Table records in the format {id: string, [fieldName]: string}[]',
});

// You can also describe the expected return format using a string:

// dataB will be an array of strings
const dataB = await agent.aiQuery('string[], list of task names');

// dataC will be an array of objects
const dataC = await agent.aiQuery('{name: string, age: string}[], table data records');
```

### `agent.aiBoolean()`

Extract a boolean value from the UI.

* Type

```typescript
function aiBoolean(prompt: string): Promise<boolean>;
```

* Parameters:
  * `prompt: string` - A natural language description of the expected value.

* Return Value:
  * Returns a `Promise<boolean>` when AI returns a boolean value.

* Examples:

```typescript
const bool = await agent.aiBoolean('Whether there is a login dialog');
```

### `agent.aiNumber()`

Extract a number value from the UI.

* Type

```typescript
function aiNumber(prompt: string): Promise<number>;
```

* Parameters:
  * `prompt: string` - A natural language description of the expected value.

* Return Value:
  * Returns a `Promise<number>` when AI returns a number value.

* Examples:

```typescript
const number = await agent.aiNumber('The remaining points of the account');
```

### `agent.aiString()`

Extract a string value from the UI.

* Type

```typescript
function aiString(prompt: string): Promise<string>;
```

* Parameters:
  * `prompt: string` - A natural language description of the expected value.

* Return Value:
  * Returns a `Promise<string>` when AI returns a string value.

* Examples:

```typescript
const string = await agent.aiString('The first item in the list');
```

## More APIs

### `agent.aiAssert()`

Specify an assertion in natural language, and the AI determines whether the condition is true. If the assertion fails, the SDK throws an error that includes both the optional `errorMsg` and a detailed reason generated by the AI.

* Type

```typescript
function aiAssert(assertion: string, errorMsg?: string): Promise<void>;
```

* Parameters:
  * `assertion: string` - The assertion described in natural language.
  * `errorMsg?: string` - An optional error message to append if the assertion fails.

* Return Value:
  * Returns a Promise that resolves to void if the assertion passes; if it fails, an error is thrown with `errorMsg` and additional AI-provided information.

* Example:

```typescript
await agent.aiAssert('The price of "Sauce Labs Onesie" is 7.99');
```

:::tip
Assertions are critical in test scripts. To reduce the risk of errors due to AI hallucination (e.g., missing an error), you can also combine `.aiQuery` with standard JavaScript assertions instead of using `.aiAssert`.

For example, you might replace the above code with:

```typescript
const items = await agent.aiQuery(
  '"{name: string, price: number}[], return product names and prices'
);
const onesieItem = items.find(item => item.name === 'Sauce Labs Onesie');
expect(onesieItem).toBeTruthy();
expect(onesieItem.price).toBe(7.99);
```

:::

### `agent.aiLocate()`

Locate an element using natural language.

* Type

```typescript
function aiLocate(locate: string, options?: Object): Promise<{
  rect: {
    left: number;
    top: number;
    width: number;
    height: number;
  };
  center: [number, number];
}>;
```

* Parameters:
  * `locate: string` - A natural language description of the element to locate.
  * `options?: Object` - Optional, a configuration object containing:
    * `deepThink?: boolean` - If true, Midscene will call AI model twice to precisely locate the element.

* Return Value:
  * Returns a `Promise` when the element is located parsed as an locate info object.

* Examples:

```typescript
const locateInfo = await agent.aiLocate('The login button at the top of the page');
console.log(locateInfo);
```

### `agent.aiWaitFor()`

Wait until a specified condition, described in natural language, becomes true. Considering the cost of AI calls, the check interval will not exceed the specified `checkIntervalMs`.

* Type

```typescript
function aiWaitFor(
  assertion: string, 
  options?: { 
    timeoutMs?: number;
    checkIntervalMs?: number;
  }
): Promise<void>;
```

* Parameters:
  * `assertion: string` - The condition described in natural language.
  * `options?: object` - An optional configuration object containing:
    * `timeoutMs?: number` - Timeout in milliseconds (default: 15000).
    * `checkIntervalMs?: number` - Interval for checking in milliseconds (default: 3000).

* Return Value:
  * Returns a Promise that resolves to void if the condition is met; if not, an error is thrown when the timeout is reached.

* Examples:

```typescript
// Basic usage
await agent.aiWaitFor("There is at least one headphone information displayed on the interface");

// Using custom options
await agent.aiWaitFor("The shopping cart icon shows a quantity of 2", {
  timeoutMs: 30000,    // Wait for 30 seconds
  checkIntervalMs: 5000  // Check every 5 seconds
});
```

:::tip
Given the time consumption of AI services, `.aiWaitFor` might not be the most efficient method. Sometimes, using a simple sleep function may be a better alternative.
:::

### `agent.runYaml()`

Execute an automation script written in YAML. Only the `tasks` part of the script is executed, and it returns the results of all `.aiQuery` calls within the script.

* Type

```typescript
function runYaml(yamlScriptContent: string): Promise<{ result: any }>;
```

* Parameters:
  * `yamlScriptContent: string` - The YAML-formatted script content.

* Return Value:
  * Returns an object with a `result` property that includes the results of all `.aiQuery` calls.

* Example:

```typescript
const { result } = await agent.runYaml(`
tasks:
  - name: search weather
    flow:
      - ai: input 'weather today' in input box, click search button
      - sleep: 3000

  - name: query weather
    flow:
      - aiQuery: "the result shows the weather info, {description: string}"
`);
console.log(result);
```

:::tip
For more information about YAML scripts, please refer to [Automate with Scripts in YAML](/en/automate-with-scripts-in-yaml.md).
:::

### `agent.setAIActionContext()`

Set the background knowledge that should be sent to the AI model when calling `agent.aiAction()`.

* Type

```typescript
function setAIActionContext(actionContext: string): void;
```

* Parameters:
  * `actionContext: string` - The background knowledge that should be sent to the AI model.

* Example:

```typescript
await agent.setAIActionContext('Close the cookie consent dialog first if it exists');
```

### `agent.evaluateJavaScript()`

Evaluate a JavaScript expression in the web page context.

* Type

```typescript
function evaluateJavaScript(script: string): Promise<any>;
```

* Parameters:
  * `script: string` - The JavaScript expression to evaluate.

* Return Value:
  * Returns the result of the JavaScript expression.

* Example:

```typescript
const result = await agent.evaluateJavaScript('document.title');
console.log(result);
```

## Properties

### `.reportFile`

The path to the report file.

## Additional Configurations

### Setting Environment Variables at Runtime

You can override environment variables at runtime by calling the `overrideAIConfig` method.

```typescript
import { overrideAIConfig } from '@midscene/web/puppeteer'; // or another Agent

overrideAIConfig({
  OPENAI_BASE_URL: "...",
  OPENAI_API_KEY: "...",
  MIDSCENE_MODEL_NAME: "..."
});
```

### Print usage information for each AI call

Set the `DEBUG=midscene:ai:profile:stats` to view the execution time and usage for each AI call.

```bash
export DEBUG=midscene:ai:profile:stats
```

### Customize the run artifact directory

Set the `MIDSCENE_RUN_DIR` variable to customize the run artifact directory.

```bash
export MIDSCENE_RUN_DIR=midscene_run # The default value is the midscene_run in the current working directory, you can set it to an absolute path or a relative path
```

### Using LangSmith

LangSmith is a platform for debugging large language models. To integrate LangSmith, follow these steps:

```bash
# Set environment variables

# Enable debug mode
export MIDSCENE_LANGSMITH_DEBUG=1 

# LangSmith configuration
export LANGSMITH_TRACING_V2=true
export LANGSMITH_ENDPOINT="https://api.smith.langchain.com"
export LANGSMITH_API_KEY="your_key_here"
export LANGSMITH_PROJECT="your_project_name_here"
```

After starting Midscene, you should see logs similar to:

```log
DEBUGGING MODE: langsmith wrapper enabled
```



---
url: /automate-with-scripts-in-yaml.md
---

# Automate with Scripts in YAML

In most cases, developers write automation just to perform some smoke tests, like checking the appearance of some content, or verifying that the key user path is accessible. Maintaining a large test project is unnecessary in this situation.

⁠Midscene offers a way to do this kind of automation with `.yaml` files, which helps you to focus on the script itself instead of the test infrastructure. Any team member can write an automation script without learning any API.

Here is an example of `.yaml` script, you may have already understood how it works by reading its content.

```yaml
web:
  url: https://www.bing.com

tasks:
  - name: search weather
    flow:
      - ai: search for 'weather today'
      - sleep: 3000

  - name: check result
    flow:
      - aiAssert: the result shows the weather info
```

:::info Demo Project

You can find the demo project with YAML scripts
[https://github.com/web-infra-dev/midscene-example/tree/main/yaml-scripts-demo](https://github.com/web-infra-dev/midscene-example/tree/main/yaml-scripts-demo)

* [Web](https://github.com/web-infra-dev/midscene-example/tree/main/yaml-scripts-demo)
* [Android](https://github.com/web-infra-dev/midscene-example/tree/main/android/yaml-scripts-demo)

:::

## Setup AI model service

Set your model configs into the environment variables. You may refer to [choose a model](/en/choose-a-model.md) for more details.

```bash
# replace with your own
export OPENAI_API_KEY="sk-abcdefghijklmnopqrstuvwxyz"
```

or you can use a `.env` file locate at the same directory as you run the command to store the configuration, Midscene command line tool will automatically load it.

```env filename=.env
OPENAI_API_KEY="sk-abcdefghijklmnopqrstuvwxyz"
```

## Install Command Line Tool

Install `@midscene/cli` globally

```bash
npm i -g @midscene/cli
# or if you prefer a project-wide installation
npm i @midscene/cli --save-dev
```

Write a yaml file to `bing-search.yaml` to automate in web browser :

```yaml
web:
  url: https://www.bing.com

tasks:
  - name: search weather
    flow:
      - ai: search for 'weather today'
      - sleep: 3000
      - aiAssert: the result shows the weather info
```

or to automate in Android device connected by adb :

```yaml
android:
  # launch: https://www.bing.com
  deviceId: s4ey59

tasks:
  - name: search weather
    flow:
      - ai: open browser and navigate to bing.com
      - ai: search for 'weather today'
      - sleep: 3000
      - aiAssert: the result shows the weather info
```

Run this script

```bash
midscene ./bing-search.yaml
# or if you installed midscene inside the project
npx midscene ./bing-search.yaml
```

You should see that the output shows the progress of the running process and the report file.

## Command line usage

### Run single `.yaml` file

```bash
midscene /path/to/yaml
```

### Run all `.yaml` files under a folder

```bash
midscene /dir/of/yaml/

# glob is also supported
midscene /dir/**/yaml/
```

## YAML file schema

There are two parts in a `.yaml` file, the `web/android` and the `tasks`.

The `web/android` part defines the basic of a task. Use `web` parameter (also previously named as `target`) for web browser automation, and use `android` parameter for Android device automation. They are mutually exclusive.

### The `web` part

```yaml
web:
  # The URL to visit, required. If `serve` is provided, provide the path to the file to visit
  url: <url>

  # Serve the local path as a static server, optional
  serve: <root-directory>

  # The user agent to use, optional
  userAgent: <ua>

  # number, the viewport width, default is 1280, optional
  viewportWidth: <width>

  # number, the viewport height, default is 960, optional
  viewportHeight: <height>

  # number, the device scale factor (dpr), default is 1, optional
  deviceScaleFactor: <scale>

  # string, the path to the json format cookie file, optional
  cookie: <path-to-cookie-file>
  
  # object, the strategy to wait for network idle, optional
  waitForNetworkIdle:
    # number, the timeout in milliseconds, 2000ms for default, optional
    timeout: <ms>
    # boolean, continue on network idle error, true for default
    continueOnNetworkIdleError: <boolean>
  
  # string, the path to save the aiQuery result, optional
  output: <path-to-output-file>

  # boolean, if limit the popup to the current page, true for default in yaml script
  forceSameTabNavigation: <boolean>

  # string, the bridge mode to use, optional, default is false, can be 'newTabWithUrl' or 'currentTab'. More details see the following section
  bridgeMode: false | 'newTabWithUrl' | 'currentTab'

  # boolean, if close the new tabs after the bridge is disconnected, optional, default is false
  closeNewTabsAfterDisconnect: <boolean>

  # boolean, if allow insecure https certs, optional, default is false
  acceptInsecureCerts: <boolean>

  # string, the background knowledge to send to the AI model when calling aiAction, optional
  aiActionContext: <string>
```

### The `android` part

```yaml
android:
  # The device id to use, optional, default is the first connected device
  deviceId: <device-id>

  # The url to launch, optional, default is the current page
  launch: <url>
```

### The `tasks` part

The `tasks` part is an array indicates the tasks to do. Remember to write a `-` before each item which means an array item.

The interfaces of the `flow` part are almost the same as the [API](/en/API.md), except for some parameter levels.

```yaml
tasks:
  - name: <name>
    continueOnError: <boolean> # optional, default is false
    flow:
      # Auto Planning (.ai)
      # ----------------

      # perform an action, this is the shortcut for aiAction
      - ai: <prompt>

      # this is the same as ai
      - aiAction: <prompt>

      # Instant Action(.aiTap, .aiHover, .aiInput, .aiKeyboardPress, .aiScroll)
      # ----------------

      # tap an element located by prompt
      - aiTap: <prompt>
        deepThink: <boolean> # optional, whether to use deepThink to precisely locate the element

      # hover an element located by prompt
      - aiHover: <prompt>
        deepThink: <boolean> # optional, whether to use deepThink to precisely locate the element

      # input text into an element located by prompt
      - aiInput: <final text content of the input>
        locate: <prompt>
        deepThink: <boolean> # optional, whether to use deepThink to precisely locate the element

      # press a key (like Enter, Tab, Escape, etc.) on an element located by prompt
      - aiKeyboardPress: <key>
        locate: <prompt>
        deepThink: <boolean> # optional, whether to use deepThink to precisely locate the element


      # scroll globally or on an element located by prompt
      - aiScroll:
        direction: 'up' # or 'down' | 'left' | 'right'
        scrollType: 'once' # or 'untilTop' | 'untilBottom' | 'untilLeft' | 'untilRight'
        distance: <number> # optional, distance to scroll in px
        locate: <prompt> # optional, the element to scroll on
        deepThink: <boolean> # optional, whether to use deepThink to precisely locate the element

      # Data Extraction
      # ----------------

      # perform a query, return a json object
      - aiQuery: <prompt> # remember to describe the format of the result in the prompt
        name: <name> # the name of the result, will be used as the key in the output json

      # More APIs
      # ----------------

      # wait for a condition to be met with a timeout (ms, optional, default 30000)
      - aiWaitFor: <prompt>
        timeout: <ms>

      # perform an assertion
      - aiAssert: <prompt>

      # sleep for a number of milliseconds
      - sleep: <ms>

      # evaluate a javascript expression in web page context
      - javascript: <javascript>
        name: <name> # assign a name to the return value, will be used as the key in the output json, optional

  - name: <name>
    flow:
      # ...
```

## More features

### Use environment variables in `.yaml` file

You can use environment variables in `.yaml` file by `${variable-name}`.

For example, if you have a `.env` file with the following content:

```env filename=.env
topic=weather today
```

You can use the environment variable in the `.yaml` file like this:

```yaml
#...
 - ai: type ${topic} in input box
#...
```

### Debug in headed mode

> `web` scenario only

'headed mode' means the browser will be visible. The default behavior is to run in headless mode.

To turn on headed mode, you can use `--headed` option. Besides, if you want to keep the browser window open after the script finishes, you can use `--keep-window` option. `--keep-window` implies `--headed`.

When running in headed mode, it will consume more resources, so we recommend you to use it locally only when needed.

```bash
# run in headed mode
midscene /path/to/yaml --headed

# run in headed mode and keep the browser window open after the script finishes
midscene /path/to/yaml --keep-window
```

### Use bridge mode

> `web` scenario only
> By using bridge mode, you can utilize YAML scripts to automate the web browser on your desktop. This is particularly useful if you want to reuse cookies, plugins, and page states, or if you want to manually interact with automation scripts.

To use bridge mode, you should install the Chrome extension first, and use this configuration in the `target` section:

```diff
web:
  url: https://www.bing.com
+ bridgeMode: newTabWithUrl
```

See [Bridge Mode by Chrome Extension](/en/bridge-mode-by-chrome-extension.md) for more details.

### Run yaml script with javascript

You can also run a yaml script with javascript by using the [`runYaml`](/en/api.md#runyaml) method of the Midscene agent. Only the `tasks` part of the yaml script will be executed.

## FAQ

**How to get cookies in JSON format from Chrome?**

You can use this [chrome extension](https://chromewebstore.google.com/detail/get-cookiestxt-locally/cclelndahbckbenkjhflpdbgdldlbecc) to export cookies in JSON format.

## More

You may also be interested in [Prompting Tips](/en/prompting-tips.md)



---
url: /blog-introducing-instant-actions-and-deep-think.md
---

# Introducing Instant Actions and Deep Think

From Midscene v0.14.0, we have introduced two new features: Instant Actions and Deep Think.

## Instant Actions - A More Predictable Way to Perform Actions

You may have already been familiar with our `.ai` interface. It's an auto-planning interface to interact with web pages. For example, when performing a search, you can do this:

```typescript
await agent.ai('type "Headphones" in search box, hit Enter');
```

Behind the scene, Midscene will call the LLM to plan the steps and execute them. You can see the report file to see the process. It's a very common way for AI agents to these kinds of tasks.

![](/blog/report-planning.png)

In the meantime, there are many testing engineers who want a faster way to perform actions. When using AI models with complex prompts, some of the LLM models may find it hard to plan the proper steps, or the coordinates of the elements may not be accurate. It could be frustrating for debugging the unpredictable process.

To solve this problem, we have introduced the `aiTap()`, `aiHover()`, `aiInput()`, `aiKeyboardPress()`, `aiScroll()` interfaces. They are call the **"instant actions"**. These interfaces will directly perform the specified action as the interface name suggests, while the AI model is responsible for the easier tasks such as locating elements. The whole process can be obviously faster and more reliable after using them.

For example, the search action above can be rewritten as:

```typescript
await agent.aiInput('Headphones', 'search-box');
await agent.aiKeyboardPress('Enter');
```

The typical workflow in the report file is like this, as you can see there is no planning process in the report file:

![](/blog/report-instant-action.png)

The scripts with instant actions seems a little bit redundant (or not 'ai-style'), but we believe these structured interfaces are a good way to save time debugging when the action is already clear.

## Deep Think - A More Accurate Way to Locate Elements

When using Midscene with some complex widgets, the LLM may find it hard to locate the target element. We have introduced a new option named `deepThink` to the instant actions.

The signature of the instant actions with `deepThink` is like this:

```typescript
await agent.aiTap('target', { deepThink: true });
```

`deepThink` is a strategy of locating elements. It will first find an area that contains the target element, then "focus" on this area to search the element again. By this way, the coordinates of the target element will be more accurate. 

Let's take the workflow editor page of Coze.com as an example. There are many customized icons on the sidebar. This is usually hard for LLMs to distinguish the target element from its surroundings.

![](/blog/coze-sidebar.png)

After using `deepThink` in instant actions, the yaml scripts will be like this (of course, you can also use the javascript interface):

```yaml
tasks:
  - name: edit input panel
    flow:
      - aiTap: the triangle icon on the left side of the text "Input"
        deepThink: true
      - aiTap: the first checkbox in the Input form
        deepThink: true
      - aiTap: the expand button on the second row of the Input form (on the right of the checkbox)
        deepThink: true
      - aiTap: the delete button on the second last row of the Input form
        deepThink: true
      - aiTap: the add button on the last row of the Input form （second button from the right）
        deepThink: true
```

By viewing the report file, you can see Midscene has found every target element in the area.

![](/blog/report-coze-deep-think.png)

Just like the example above, the highly-detailed prompt for `deepThink` adheres to [the prompting tips](./prompting-tips). This is always the key to make result stable.

`deepThink` is only available with the models that support visual grounding like qwen2.5-vl. If you are using LLM models like gpt-4o, it won't work.



---
url: /blog-programming-practice-using-structured-api.md
---

# Use Structured API to Optimize Automation Code

Many developers using `aiAction` will fall into a misconception: trying to describe all complex logic in a single natural language instruction. Although it looks "smart", in practice, it will bring a series of problems.

The most common mistake is writing a large logic storm with long descriptions, such as:

```javascript
aiAction(`
1. click the first user
2. click the chat bubble on the right side of the user page
3. if I have already sent a message to him/her, go back to the previous page
4. if I have not sent a message to him/her, input a greeting text and click send
`)
```

```javascript
await agent.aiAction('get all prices on the product list page, then go to the shopping cart page, and select the product with the lowest price')
```

This kind of writing will gather all operation instructions into a single Prompt, which actually has very high requirements for the understanding and stability of the AI model. Any step failure may cause the whole process to fail. Developers may also be lost in the circle of Prompt tuning.

Another misconception is to split the code into multiple `aiAction` methods, while the multiple `aiAction` methods still have context relationships, but `aiAction` cannot pass context to the next `aiAction`, so the same problem will occur.

```javascript
aiAction('click the first user')
aiAction('click the chat bubble on the right side of the user page')
aiAction('if I have already sent a message to him/her, go back to the previous page')
aiAction('if I have not sent a message to him/her, input a greeting text and click send')
```

## Use Structured API to Optimize Code

Midscene provides `aiBoolean` `aiString` `aiNumber` and other data extraction methods, which can be used to split complex logic into multiple steps to improve the stability of the automation code.

For the above examples, you can convert the natural language into this code form:

```javascript
aiAction('click the first user')
aiAction('click the chat bubble on the right side of the user page')

const hasAlreadyChat = await agent.aiBoolean('check if I have already sent a message to him/her')

if (hasAlreadyChat) {
  aiAction('go back to the previous page')
} else {
  aiAction('input a greeting text and click send')
}
```

Use this way to write automation code, can reduce the dependency on AI model, and improve the stability of the code.

Another example:

```javascript
aiAction(`
1. click the first user, enter the user's homepage
2. click the follow button
3. go back to the previous page
4. if all users are followed, scroll down one screen
5. repeat the above steps until all users are followed
`)


can be converted to:

```javascript
let user = await agent.aiQuery('string[], the user names in the list')
let currentUserIndex = 0

while (true) {
  await agent.aiAction(`click the first user, enter the user's homepage`)
  await agent.aiTap('follow button')
  await agent.aiAction('go back to the previous page')

  if (currentUserIndex === user.length - 1) {
    await agent.aiAction('scroll down one screen')
    user = await agent.aiQuery('string[], the user names in the list')
    currentUserIndex = 0
  } else {
    currentUserIndex++
  }
}
```

## Common Structured API Methods

### `aiBoolean` - Conditional Decision

* Use Case: Condition Judgment, State Detection
* Advantage: Convert fuzzy descriptions into clear boolean values

Example:
```javascript
const hasAlreadyChat = await agent.aiBoolean('check if I have already sent a message to him/her')
if (hasAlreadyChat) {
  // ...
}
```

### `aiString` - Text Extraction

* Use Case: Text Content Retrieval
* Advantage: Avoid Ambiguity in Natural Language Descriptions

Example:
```javascript
const username = await agent.aiString('the nickname of the first user in the list')
console.log('username is', username)
```

### `aiNumber` - Numerical Extraction

* Use Case: Counting, Numerical Comparison, Loop Control
* Advantage: Ensure Return Standard Numeric Types

Example:
```javascript
const unreadCount = await agent.aiNumber('the number of unread messages on the message icon')
for (let i = 0; i < unreadCount; i++) {
  // ...
}
```



---
url: /blog-support-android-automation.md
---

# Support Android Automation

From Midscene v0.15, we are happy to announce the support for Android automation. The era for AI-driven Android automation is here!

## Showcases

### Navigation to attraction

Open Maps, search for a destination, and navigate to it.

### Auto-like tweets

Open Twitter, auto-like the first tweet by [@midscene\_ai](https://x.com/midscene_ai).

## Suitable for ALL apps

For our developers, all you need is the adb connection and a visual-language model (vl-model) service. Everything is ready!

Behind the scenes, we utilize the visual grounding capabilities of vl-model to locate target elements on the screen. So, regardless of whether it's a native app, a [Lynx](https://github.com/lynx-family/lynx) page, or a hybrid app with a webview, it makes no difference. Developers can write automation scripts without the burden of worrying about the technology stack of the app.

## With ALL the power of Midscene

When using Midscene to do web automation, our users loves the tools like playgrounds and reports. Now, we bring the same power to Android automation!

### Use the playground to run automation without any code

### Use the report to replay the whole process

### Write the automation scripts by yaml file

Connect to the device, open ebay.com, and get some items info.

```yaml
# search headphone on ebay, extract the items info into a json file, and assert the shopping cart icon

android:
  deviceId: s4ey59

tasks:
  - name: search headphones
    flow:
      - aiAction: open browser and navigate to ebay.com
      - aiAction: type 'Headphones' in ebay search box, hit Enter
      - sleep: 5000
      - aiAction: scroll down the page for 800px

  - name: extract headphones info
    flow:
      - aiQuery: >
          {name: string, price: number, subTitle: string}[], return item name, price and the subTitle on the lower right corner of each item
        name: headphones

  - name: assert Filter button
    flow:
      - aiAssert: There is a Filter button on the page
```

### Use the javascript SDK

Use the javascript SDK to do the automation by code.

```ts
import { AndroidAgent, AndroidDevice, getConnectedDevices } from '@midscene/android';
import "dotenv/config"; // read environment variables from .env file

const sleep = (ms) => new Promise((r) => setTimeout(r, ms));
Promise.resolve(
  (async () => {
    const devices = await getConnectedDevices();
    const page = new AndroidDevice(devices[0].udid);

    // 👀 init Midscene agent
    const agent = new AndroidAgent(page,{
      aiActionContext:
        'If any location, permission, user agreement, etc. popup, click agree. If login page pops up, close it.',
    });
    await page.connect();
    await page.launch('https://www.ebay.com');

    await sleep(5000);

    // 👀 type keywords, perform a search
    await agent.aiAction('type "Headphones" in search box, hit Enter');

    // 👀 wait for the loading
    await agent.aiWaitFor("there is at least one headphone item on page");
    // or you may use a plain sleep:
    // await sleep(5000);

    // 👀 understand the page content, find the items
    const items = await agent.aiQuery(
      "{itemTitle: string, price: Number}[], find item in list and corresponding price"
    );
    console.log("headphones in stock", items);

    // 👀 assert by AI
    await agent.aiAssert("There is a category filter on the left");
  })()
);

```

### Two style APIs to do interaction

The auto-planning style:

```javascript
await agent.ai('input "Headphones" in search box, hit Enter');
```

The instant action style:

```javascript
await agent.aiInput('Headphones', 'search box');
await agent.aiKeyboardPress('Enter');
```

## Quick start

You can use the playground to experience the Android automation without any code. Please refer to [Quick experience with Android](/en/quick-experience-with-android.md) for more details.

After the experience, you can integrate with the Android device by javascript code. Please refer to [Integrate with Android(adb)](/en/integrate-with-android.md) for more details.

If you prefer the yaml file for automation scripts, please refer to [Automate with scripts in yaml](/en/automate-with-scripts-in-yaml.md).

### Demo projects

We have prepared a demo project for javascript SDK:

[JavaScript demo project](https://github.com/web-infra-dev/midscene-example/blob/main/android/javascript-sdk-demo)

If you want to use the automation for testing purpose, you can use the javascript with vitest. We have setup a demo project for you to see how it works:

[Vitest demo project](https://github.com/web-infra-dev/midscene-example/blob/main/android/vitest-demo)

You can also write the automation scripts by yaml file:

[YAML demo project](https://github.com/web-infra-dev/midscene-example/blob/main/android/yaml-scripts-demo)

## Limitations

1. Caching feature for element locator is not supported. Since no view-hierarchy is collected, we cannot cache the element identifier and reuse it.
2. LLMs like gpt-4o or deepseek are not supported. Only some known vl models with visual grounding ability are supported for now. If you want to introduce other vl models, please let us know.
3. The performance is not good enough for now. We are still working on it.
4. The vl model may not perform well on `.aiQuery` and `.aiAssert`. We will give a way to switch model for different kinds of tasks.
5. Due to some security restrictions, you may got a blank screenshot for the password input and Midscene will not be able to work for that.

## Credits

We would like to thank the following projects:

* [scrcpy](https://github.com/Genymobile/scrcpy) and [yume-chan](https://github.com/yume-chan) allow us to control Android devices with browser.
* [appium-adb](https://github.com/appium/appium-adb) for the javascript bridge of adb.
* [YADB](https://github.com/ysbing/YADB) for the yadb tool which improves the performance of text input.



---
url: /bridge-mode-by-chrome-extension.md
---

# Bridge Mode by Chrome Extension



The bridge mode in the Midscene Chrome extension is a tool that allows you to use local scripts to control the desktop version of Chrome. Your scripts can connect to either a new tab or the currently active tab.

Using the desktop version of Chrome allows you to reuse all cookies, plugins, page status, and everything else you want. You can work with automation scripts to complete your tasks. This mode is commonly referred to as 'man-in-the-loop' in the context of automation.

![bridge mode](/midscene-bridge-mode.jpg)

:::info Demo Project
check the demo project of bridge mode: [https://github.com/web-infra-dev/midscene-example/blob/main/bridge-mode-demo](https://github.com/web-infra-dev/midscene-example/blob/main/bridge-mode-demo)
:::

## Setup AI model service

Set your model configs into the environment variables. You may refer to [choose a model](/en/choose-a-model.md) for more details.

```bash
# replace with your own
export OPENAI_API_KEY="sk-abcdefghijklmnopqrstuvwxyz"
```

> In bridge mode, the AI models configs should be set in the Node.js side instead of the browser side.

## Step 1. Install Midscene extension from Chrome web store

Install [Midscene extension from Chrome web store](https://chromewebstore.google.com/detail/midscene/gbldofcpkknbggpkmbdaefngejllnief)

## Step 2. install dependencies

## Step 3. write scripts

Write and save the following code as `./demo-new-tab.ts`.

```typescript
import { AgentOverChromeBridge } from "@midscene/web/bridge-mode";

const sleep = (ms) => new Promise((r) => setTimeout(r, ms));
Promise.resolve(
  (async () => {
    const agent = new AgentOverChromeBridge();

    // This will connect to a new tab on your desktop Chrome
    // remember to start your chrome extension, click 'allow connection' button. Otherwise you will get an timeout error
    await agent.connectNewTabWithUrl("https://www.bing.com");

    // these are the same as normal Midscene agent
    await agent.ai('type "AI 101" and hit Enter');
    await sleep(3000);

    await agent.aiAssert("there are some search results");
    await agent.destroy();
  })()
);
```

## Step 4. start the chrome extension

Start the chrome extension and switch to 'Bridge Mode' tab. Click "Allow connection".

## Step 5. run the script

Run your scripts

```bash
tsx demo-new-tab.ts
```

After executing the script, you should see the status of the Chrome extension switched to 'connected', and a new tab has been opened. Now this tab is controlled by your scripts.

:::tip
⁠Whether the scripts are run before or after clicking 'Allow connection' in the browser is not significant.
:::

## Constructor

```typescript
import { AgentOverChromeBridge } from "@midscene/web/bridge-mode";

const agent = new AgentOverChromeBridge();
```

Except [the normal parameters in the agent constructor](/en/api.md), `AgentOverChromeBridge` accepts one more parameter:

* `closeNewTabsAfterDisconnect?: boolean`: If true, the newly created tab will be closed when the bridge is destroyed. Default is false.

## Methods

Except [the normal agent interface](/en/api.md), `AgentOverChromeBridge` provides some other interfaces to control the desktop Chrome.

:::info
You should always call `connectCurrentTab` or `connectNewTabWithUrl` before doing further actions.

Each of the agent instance can only connect to one tab instance, and it cannot be reconnected after destroy.
:::

### `connectCurrentTab()`

Connect to the currently active tab.

* Type

```typescript
function connectCurrentTab(options?: { 
  forceSameTabNavigation?: boolean 
}): Promise<void>;
```

* Parameters:
  * `options?: object` - Optional configuration object
    * `forceSameTabNavigation?: boolean` - If true (default), restricts pages from opening new tabs, forcing new pages to open in the current tab to prevent AI operation failures due to manual tab switching. This configuration usually doesn't need to be changed

* Returns:
  * Returns a Promise that resolves to void when connected successfully; throws an error if connection fails

* Example:

```typescript
try {
  await agent.connectCurrentTab();
  console.log('Successfully connected to current tab');
} catch (err) {
  console.error('Connection failed:', err.message);
}
```

### `connectNewTabWithUrl()`

Create a new tab and connect to it immediately.

* Type

```typescript
function connectNewTabWithUrl(
  url: string,
  options?: { 
    forceSameTabNavigation?: boolean 
  }
): Promise<void>;
```

* Parameters:
  * `url: string` - URL to open in the new tab
  * `options?: object` - Optional configuration object (same parameters as connectCurrentTab)

* Returns:
  * Returns a Promise that resolves to void when connected successfully; throws an error if connection fails

* Example:

```typescript
// Open Bing and wait for connection
await agent.connectNewTabWithUrl(
  "https://www.bing.com",
  { forceSameTabNavigation: false }
);
```

### `destroy()`

Destroy the connection and release resources.

* Type

```typescript
function destroy(closeNewTabsAfterDisconnect?: boolean): Promise<void>;
```

* Parameters:
  * `closeNewTabsAfterDisconnect?: boolean` - If true, the newly created tab will be closed when the bridge is destroyed. Default is false. The will override the `closeNewTabsAfterDisconnect` parameter in the constructor.

* Returns:
  * Returns a Promise that resolves to void when destruction completes

* Example:

```typescript
// Destroy connection after completing operations
await agent.ai('Perform final operation...');
await agent.destroy();
```

## Use bridge mode in yaml-script

[Yaml scripts](/en/automate-with-scripts-in-yaml.md) is a way for developers to write automation scripts in yaml format, which is easy to read and write comparing to javascript.

To use bridge mode in yaml script, set the `bridgeMode` property in the `target` section. If you want to use the current tab, set it to `currentTab`, otherwise set it to `newTabWithUrl`.

Set `closeNewTabsAfterDisconnect` to true if you want to close the newly created tabs when the bridge is destroyed. This is optional and the default value is false.

For example, the following script will open a new tab by Chrome extension bridge:

```diff
target:
  url: https://www.bing.com
+ bridgeMode: newTabWithUrl
+ closeNewTabsAfterDisconnect: true
tasks:
```

Run the script:

```bash
midscene ./bing.yaml
```

Remember to start the chrome extension and click 'Allow connection' button after the script is running.

### Unsupported options

In bridge mode, these options will be ignored (they will follow your desktop browser's settings):

* userAgent
* viewportWidth
* viewportHeight
* viewportScale
* waitForNetworkIdle
* cookie

## FAQ

* Where should I config the `OPENAI_API_KEY`, in the browser or in the terminal?

When using bridge mode, you should config the `OPENAI_API_KEY` in the terminal.



---
url: /caching.md
---

# Caching

Midscene supports caching the planning steps and DOM XPaths to reduce calls to AI models and greatly improve execution efficiency.

Caching is not supported in Android automation.

**Effect**

After enabling the cache, the execution time of AI service related steps can be significantly reduced.

* **before using cache, 39s**

![](/cache/no-cache-time.png)

* **after using cache, 13s**

![](/cache/use-cache-time.png)

## Instructions

There are two key points to use caching:

1. Set `MIDSCENE_CACHE=1` in the environment variable to enable matching cache.
2. Set `cacheId` to specify the cache file name. It's automatically set in Playwright and Yaml mode. If you are using javascript SDK, you should set it manually.

### Playwright

In playwright mode, you can use the `MIDSCENE_CACHE=1` environment variable to enable caching.

The `cacheId` will be automatically set to the test file name.

```diff
- playwright test --config=playwright.config.ts
+ MIDSCENE_CACHE=1 playwright test --config=playwright.config.ts
```

### Javascript agent, like PuppeteerAgent, AgentOverChromeBridge

Enable caching by setting the `MIDSCENE_CACHE=1` environment variable.
And also, you should set the `cacheId` to specify the cache identifier.

```diff
- tsx demo.ts 
+ MIDSCENE_CACHE=1 tsx demo.ts
```

```javascript
const mid = new PuppeteerAgent(originPage, {
  cacheId: 'puppeteer-swag-sab', // specify cache id
});
```

### Yaml

Enable caching by setting the `MIDSCENE_CACHE=1` environment variable.
The `cacheId` will be automatically set to the yaml filename.

```diff
- npx midscene ./bing-search.yaml
+ # Add cache identifier, cacheId is the yaml filename
+ MIDSCENE_CACHE=1 npx midscene ./bing-search.yaml
```

## Cache strategy

Cache contents will be saved in the `./midscene_run/cache` directory with the `.cache.yaml` as the extension name.

These two types of content will be cached:

1. the result of planning, like calls to `.ai` `.aiAction`
2. The XPaths for elements located by AI, such as `.aiLocate`, `.aiTap`, etc.

The query results like `aiBoolean`, `aiQuery`, `aiAssert` will never be cached.

If the cache is not hit, Midscene will call AI model again and the result in cache file will be updated.

## Common issues

### How to check if the cache is hit?

You can view the report file. If the cache is hit, you will see the `cache` tip and the time cost is obviously reduced.

### Why the cache is missed on CI?

You should commit the cache file to the repository (which is in the `./midscene_run/cache` directory). And also, check whether the prompt is the same as the one in the cache file.

### Does it mean that AI services are no longer needed after using cache?

No. Caching is the way to accelerate the execution, but it's not a tool for ensuring long-term script stability. We have noticed many scenarios where the cache may miss when the DOM structure changes. AI services are still needed to reevaluate the task when the cache miss occurs.

### How to manually remove the cache?

You can remove the cache file in the `cache` directory, or edit the contents in the cache file.



---
url: /choose-a-model.md
---

# Choose a Model

In this article, we will talk about what kind of models are supported by Midscene.js and the features of each model.

## Quick Config for using Midscene.js

Choose one of the following models, obtain the API key, complete the configuration, and you are ready to go. Choose the model that is easiest to obtain if you are a beginner.

If you want to see the detailed configuration of model services, see [Config Model and Provider](/en/model-provider.md).

### GPT-4o (can't be used in Android automation)

```bash
OPENAI_API_KEY="......"
OPENAI_BASE_URL="https://custom-endpoint.com/compatible-mode/v1" # optional, if you want an endpoint other than the default one from OpenAI.
MIDSCENE_MODEL_NAME="gpt-4o-2024-11-20" # optional. The default is "gpt-4o".
```

### Qwen-2.5-VL on Openrouter or Aliyun

After applying for the API key on [Openrouter](https://openrouter.ai) or [Aliyun](https://aliyun.com), you can use the following config:

```bash
# openrouter.ai
export OPENAI_BASE_URL="https://openrouter.ai/api/v1"
export OPENAI_API_KEY="......"
export MIDSCENE_MODEL_NAME="qwen/qwen2.5-vl-72b-instruct"
export MIDSCENE_USE_QWEN_VL=1

# or from Aliyun.com
OPENAI_BASE_URL="https://dashscope.aliyuncs.com/compatible-mode/v1"
export OPENAI_API_KEY="......"
MIDSCENE_MODEL_NAME="qwen-vl-max-latest"
MIDSCENE_USE_QWEN_VL=1
```

### Gemini-2.5-Pro on Google Gemini

After applying for the API key on [Google Gemini](https://gemini.google.com/), you can use the following config:

```bash
OPENAI_BASE_URL="https://generativelanguage.googleapis.com/v1beta/openai/"
OPENAI_API_KEY="......"
MIDSCENE_MODEL_NAME="gemini-2.5-pro-preview-03-25"
MIDSCENE_USE_GEMINI=1
```

### Doubao-1.5-thinking-vision-pro on Volcano Engine

You can use the `Doubao-1.5-thinking-vision-pro` model on [Volcano Engine](https://volcengine.com). After obtaining an API key from Volcano Engine, you can use the following configuration:

```bash
OPENAI_BASE_URL="https://ark.cn-beijing.volces.com/api/v3" 
OPENAI_API_KEY="...."
MIDSCENE_MODEL_NAME="ep-..." # Inference endpoint ID from Volcano Engine
MIDSCENE_USE_DOUBAO_VISION=1
```

### UI-TARS on volcengine.com

You can use `doubao-1.5-ui-tars` on [Volcengine](https://www.volcengine.com):

```bash
OPENAI_BASE_URL="https://ark.cn-beijing.volces.com/api/v3" 
OPENAI_API_KEY="...."
MIDSCENE_MODEL_NAME="ep-2025..."
MIDSCENE_USE_VLM_UI_TARS=DOUBAO
```

## Models in Depth

Midscene supports two types of models, which are:

1. **general-purpose multimodal LLMs**: Models that can understand text and image input. *GPT-4o* is this kind of model.
2. **models with visual grounding capabilities (VL models)**: Besides the ability to understand text and image input, these models can also return the coordinates of target elements on the page. We have adapted *Qwen-2.5-VL-72B*, *Gemini-2.5-Pro* and *UI-TARS* as VL models.

And we are primarily concerned with two features of the model:

1. The ability to understand the screenshot and *plan* the steps to achieve the goal.
2. The ability to *locate* the target elements on the page.

The main difference between different models is the way they handle the *locating* capability.

When using LLMs like GPT-4o, locating is accomplished through the model's understanding of the UI hierarchy tree and the markup on the screenshot, which consumes more tokens and does not always yield accurate results. In contrast, when using VL models, locating relies on the model's visual grounding capabilities, providing a more native and reliable solution in complex situations.

In the Android automation scenario, we decided to use the VL models since the infrastructure of the App in the real world is so complex that we don't want to do any adaptive work on the App UI stack any more. The VL models can provide us with more reliable results, and it should be a better approach to this type of work.

## The Recommended Models

### GPT-4o

GPT-4o is a multimodal LLM by OpenAI, which supports image input. This is the default model for Midscene.js. When using GPT-4o, a step-by-step prompting is preferred.

**Features**

* **Easy to achieve**: you can get the stable API service from many providers and just pay for the token.
* **Performing steadily**: it performs well on interaction, assertion, and query.

**Limitations when used in Midscene.js**

* **High token cost**: dom tree and screenshot will be sent together to the model. For example, it will use 6k input tokens for ebay homepage under 1280x800 resolution, and 9k for search result page. As a result, the cost will be higher than other models. And it will also take longer time to generate the response.
* **Content limitation**: it will not work if the target element is inside a cross-origin `<iframe />` or `<canvas />`.
* **Low resolution support**: the upper limit of the resolution is 2000 x 768. For images larger than this, the output quality will be lower.
* **Not good at small icon recognition**: it may not work well if the target element is a small icon.
* **Not supported for Android automation**: it does not support Android automation.

**Config**

```bash
OPENAI_API_KEY="......"
OPENAI_BASE_URL="https://custom-endpoint.com/compatible-mode/v1" # optional, if you want an endpoint other than the default one from OpenAI.
MIDSCENE_MODEL_NAME="gpt-4o-2024-11-20" # optional. The default is "gpt-4o".
```

### Qwen-2.5-VL 72B Instruct

From 0.12.0 version, Midscene.js supports Qwen-2.5-VL-72B-Instruct model.

Qwen-2.5-VL is an open-source model published by Alibaba. It provides Visual Grounding ability, which can accurately return the coordinates of target elements on the page. When using it for interaction, assertion and query, it performs quite well. We recommend using the largest version (72B) for reliable output.

Qwen-2.5-VL indeed has an action planning feature to control the application, but we still recommend using detailed prompts to provide a more stable and reliable result.

**Features**

* **Low cost**: the model can accurately tell the exact coordinates of target elements on the page(Visual Grounding), so we don't have to send the DOM tree to the model. You will achieve a token saving of 30% to 50% compared to GPT-4o.
* **Higher resolution support**: Qwen-2.5-VL supports higher resolution input than GPT-4o. It's enough for most of the cases.
* **Open-source**: this is an open-source model, so you can both use the API already deployed by cloud providers or deploy it on your own server.

**Limitations when used in Midscene.js**

* **Not good at small icon recognition**: to recognize small icons, you may need to [enable the `deepThink` parameter](/en/blog-introducing-instant-actions-and-deep-think.md) and optimize the description, otherwise the recognition result may not be accurate.
* **Perform not that good on assertion**: it may not work as well as GPT-4o on assertion.

**Config**

Except for the regular config, you need to include the `MIDSCENE_USE_QWEN_VL=1` config to turn on Qwen-2.5-VL mode. Otherwise, it will be the default GPT-4o mode (much more tokens used).

```bash
OPENAI_BASE_URL="https://openrouter.ai/api/v1"
OPENAI_API_KEY="......"
MIDSCENE_MODEL_NAME="qwen/qwen2.5-vl-72b-instruct"
MIDSCENE_USE_QWEN_VL=1
```

**Note about the model name on Aliyun.com**

⁠While the open-source version of Qwen-2.5-VL (72B) is named `qwen2.5-vl-72b-instruct`, there is also an enhanced and more stable version named `qwen-vl-max-latest` officially hosted on Aliyun.com. When using the `qwen-vl-max-latest` model on Aliyun, you will get larger context support and a much lower price (likely only 19% of the open-source version).

In short, if you want to use the Aliyun service, use `qwen-vl-max-latest`.

**Links**

* [Qwen 2.5 on 🤗 HuggingFace](https://huggingface.co/Qwen/Qwen2.5-VL-72B-Instruct)
* [Qwen 2.5 on Github](https://github.com/QwenLM/Qwen2.5-VL)
* [Qwen 2.5 on Aliyun](https://bailian.console.aliyun.com/#/model-market/detail/qwen-vl-max-latest)
* [Qwen 2.5 on openrouter.ai](https://openrouter.ai/qwen/qwen2.5-vl-72b-instruct)

### Gemini-2.5-Pro

Gemini-2.5-Pro is a model provided by Google Cloud. It works somehow similar to Qwen-2.5-VL, but it's not open-source.

From 0.15.1 version, Midscene.js supports Gemini-2.5-Pro model.

When using Gemini-2.5-Pro, you should use the `MIDSCENE_USE_GEMINI=1` config to turn on the Gemini-2.5-Pro mode.

```bash
OPENAI_BASE_URL="https://generativelanguage.googleapis.com/v1beta/openai/"
OPENAI_API_KEY="......"
MIDSCENE_MODEL_NAME="gemini-2.5-pro"
MIDSCENE_USE_GEMINI=1
```

**Links**

* [Gemini 2.5 on Google Cloud](https://cloud.google.com/gemini-api/docs/gemini-25-overview)

### Doubao-1.5-thinking-vision-pro

Doubao-1.5-thinking-vision-pro is a model provided by Volcano Engine. It works better on visual grounding and assertion.

**Links**

* [Doubao-1.5-thinking-vision-pro on Volcano Engine](https://www.volcengine.com/docs/82379/1536428)

### UI-TARS

UI-TARS is an end-to-end GUI agent model based on VLM architecture. It solely perceives screenshots as input and performs human-like interactions (e.g., keyboard and mouse operations), achieving state-of-the-art performance on 10+ GUI benchmarks. UI-TARS is an open-source model, and provides different versions of size.

When using UI-TARS, you can use target-driven style prompts, like "Login with user name foo and password bar", and it will plan the steps to achieve the goal.

**Features**

* **Exploratory**: it performs well on exploratory tasks, like "Help me send a tweet". It can try different paths to achieve the goal.
* **Speed**: Take the `doubao-1.5-ui-tars` deployed on volcengine as an example, its response time is obviously faster than other models.
* **Native image recognition**: Like Qwen-2.5-VL, UI-TARS can recognize the image directly from the screenshot, so Midscene.js does not need to extract the dom tree.
* **Open-source**: you can deploy it on your own server and your data will no longer be sent to the cloud.

**Limitations when used in Midscene.js**

* **Perform not good on assertion**: it may not work as well as GPT-4o and Qwen 2.5 on assertion and query.
* **Not stable on action path**: It may try different paths to achieve the goal, so the action path is not stable each time you call it.

**Config**

Except for the regular config, you need to include the `MIDSCENE_USE_VLM_UI_TARS` parameter to specify the UI-TARS version, supported values are `1.0` `1.5` `DOUBAO` (volcengine version). Otherwise, you will get some JSON parsing error.

```bash
OPENAI_BASE_URL="....."
OPENAI_API_KEY="......" 
MIDSCENE_MODEL_NAME="ui-tars-72b-sft"
MIDSCENE_USE_VLM_UI_TARS=1 # remember to include this for UI-TARS mode !
```

**Use the version provided by Volcengine**

On the Volcengine, there is a `doubao-1.5-ui-tars` model that has been deployed. Developers can access the model directly via API calls and pay based on usage. Docs link: https://www.volcengine.com/docs/82379/1536429

When using the Volcengine version of the model, you need to create an inference access point(like `ep-2025...`). After collecting the API Key and inference access point ID, configure should look like this:

```bash
OPENAI_BASE_URL="https://ark.cn-beijing.volces.com/api/v3" 
OPENAI_API_KEY="...."
MIDSCENE_MODEL_NAME="ep-2025..."
MIDSCENE_USE_VLM_UI_TARS=DOUBAO
```

Links:

* [UI-TARS on 🤗 HuggingFace](https://huggingface.co/bytedance-research/UI-TARS-72B-SFT)
* [UI-TARS on Github](https://github.com/bytedance/ui-tars)
* [UI-TARS - Model Deployment Guide](https://juniper-switch-f10.notion.site/UI-TARS-Model-Deployment-Guide-17b5350241e280058e98cea60317de71)

## Choose other multimodal LLMs

Other models are also supported by Midscene.js. Midscene will use the same prompt and strategy as GPT-4o for these models. If you want to use other models, please follow these steps:

1. A multimodal model is required, which means it must support image input.
2. The larger the model, the better it works. However, it needs more GPU or money.
3. Find out how to to call it with an OpenAI SDK compatible endpoint. Usually you should set the `OPENAI_BASE_URL`, `OPENAI_API_KEY` and `MIDSCENE_MODEL_NAME`. Config are described in [Config Model and Provider](/en/model-provider.md).
4. If you find it not working well after changing the model, you can try using some short and clear prompt, or roll back to the previous model. See more details in [Prompting Tips](/en/prompting-tips.md).
5. Remember to follow the terms of use of each model and provider.
6. Don't include the `MIDSCENE_USE_VLM_UI_TARS` and `MIDSCENE_USE_QWEN_VL` config unless you know what you are doing.

### Config

```bash
MIDSCENE_MODEL_NAME="....."
OPENAI_BASE_URL="......"
OPENAI_API_KEY="......"
```

For more details and sample config, see [Config Model and Provider](/en/model-provider.md).

## FAQ

### How can i check the model's token usage?

By setting `DEBUG=midscene:ai:profile:stats` in the environment variables, you can print the model's usage info and response time.

## More

* [Config Model and Provider](/en/model-provider.md)
* [Prompting Tips](/en/prompting-tips.md)



---
url: /common/prepare-android.md
---

## Preparation

### Install Node.js

Install [Node.js 18 or above](https://nodejs.org/en/download/) globally.

### Prepare an API Key

Prepare an API key from a visual-language (VL) model. You will use it later.

You can check the supported models in [Choose a model](/en/common/choose-a-model.md)

### Install adb

`adb` is a command-line tool that allows you to communicate with an Android device. There are two ways to install `adb`:

* way 1: use [Android Studio](https://developer.android.com/studio) to install
* way 2: use [Android command-line tools](https://developer.android.com/studio#command-line-tools-only) to install

Verify adb is installed successfully:

```bash
adb --version
```

When you see the following output, adb is installed successfully:

```log
Android Debug Bridge version 1.0.41
Version 34.0.4-10411341
Installed as /usr/local/bin//adb
Running on Darwin 24.3.0 (arm64)
```

### Set environment variable ANDROID\_HOME

Reference [Android environment variables](https://developer.android.com/tools/variables), set the environment variable `ANDROID_HOME`.

Verify the `ANDROID_HOME` variable is set successfully:

```bash
echo $ANDROID_HOME
```

When the command has any output, the `ANDROID_HOME` variable is set successfully:

```log
/Users/your_username/Library/Android/sdk
```

### Connect Android device with adb

In the developer options of the system settings, enable the 'USB debugging' of the Android device, if the 'USB debugging (secure settings)' exists, also enable it, then connect the Android device with a USB cable

Verify the connection:

```bash
adb devices -l
```

When you see the following output, the connection is successful:

```log
List of devices attached
s4ey59	device usb:34603008X product:cezanne model:M2006J device:cezan transport_id:3
```



---
url: /common/prepare-key-for-further-use.md
---

Prepare the config for the AI model you want to use. You can check the supported models in [Choose a model](/en/common/choose-a-model.md)



---
url: /common/setup-env.md
---

## Setup AI model service

Set your model configs into the environment variables. You may refer to [choose a model](/en/common/choose-a-model.md) for more details.

```bash
# replace with your own
export OPENAI_API_KEY="sk-abcdefghijklmnopqrstuvwxyz"
```



---
url: /common/start-experience.md
---

## Start experiencing

After the configuration, you can immediately experience Midscene. There are three main tabs in the extension:

* **Action**: interact with the web page. This is also known as "Auto Planning". For example:

```
type Midscene in the search box
click the login button
```

* **Query**: extract JSON data from the web page

```
extract the user id from the page, return in \{ id: string \}
```

* **Assert**: validate the page

```
the page title is "Midscene"
```

* **Tap**: perform a single tap on the element where you want to click. This is also known as "Instant Action".

```
the login button
```

Enjoy !

> For the different between "Auto Planning" and "Instant Action", please refer to the [API](/en/common/API.md) document.

## Want to write some code ?

After experiencing, you may want to write some code to integrate Midscene. There are multiple ways to do that. Please refer to the documents below:

* [Automate with Scripts in YAML](/en/common/automate-with-scripts-in-yaml.md)



---
url: /data-privacy.md
---

# Data Privacy

⁠Midscene.js is an open-source project (GitHub: [Midscene](https://github.com/web-infra-dev/midscene/)) under the MIT license. You can see all the codes in the public repository.

When using Midscene.js, your page data (including the screenshot) is sent directly to the AI model provider you choose. No third-party platform will have access to this data. All you need to be concerned about is the data privacy policy of the model provider.

If you prefer building Midscene.js and its Chrome Extension in your own environment instead of using the published versions, you can refer to the [Contributing Guide](https://github.com/web-infra-dev/midscene/blob/main/CONTRIBUTING.md) to find building instructions.




---
url: /faq.md
---

# FAQ

## Can Midscene smartly plan the actions according to my one-line goal? Like executing "Tweet 'hello world'"

It's only recommended to use this kind of goal-oriented prompt when you are using GUI agent models like *UI-TARS*.

## Why does Midscene require developers to provide detailed steps while other AI agents are demonstrating "autonomous planning"? Is this an outdated approach?

Midscene has a lot of tool developers, who are more concerned with the stability and performance of UI automation tools. To ensure that the Agent can run accurately in complex systems, clear prompts are still the optimal solution.

To further improve stability, we also provide features like Instant Action interface, Playback Report, and Playground. They may seem traditional and not AI-like, but after extensive practice, we believe these features are the real key to improving efficiency.

If you are interested in "smart GUI Agent", you can check out [UI-TARS](https://github.com/bytedance/ui-tars), which Midscene also supports.

Related Docs: 
* [Choose a model](./choose-a-model)
* [Prompting Tips](./prompting-tips)

## Limitations

There are some limitations with Midscene. We are still working on them.

1. The interaction types are limited to only tap, hover, drag (in UI-TARS model only), type, keyboard press, and scroll.
2. AI model is not 100% stable. Following the [Prompting Tips](./prompting-tips) will help improve stability.
3. You cannot interact with the elements inside the cross-origin iframe and canvas when using GPT-4o. This is not a problem when using Qwen and UI-TARS model.
4. We cannot access the native elements of Chrome, like the right-click context menu or file upload dialog.
5. Do not use Midscene to bypass CAPTCHA. Some LLM services are set to decline requests that involve CAPTCHA-solving (e.g., OpenAI), while the DOM of some CAPTCHA pages is not accessible by regular web scraping methods. Therefore, using Midscene to bypass CAPTCHA is not a reliable method.

## Which models are supported?

Please refer to [Choose a model](./choose-a-model).

## What data is sent to AI model?

The screenshot will be sent to the AI model. If you are using GPT-4o, some key information extracted from the DOM will also be sent.

⁠If you are worried about data privacy issues, please refer to [Data Privacy](./data-privacy)

## The automation process is running more slowly than the traditional one

When using multimodal LLM in Midscene.js, the running time may increase by a factor of 3 to 10 compared to traditional Playwright scripts, for instance from 5 seconds to 20 seconds. To make the result more stable, the token and time cost is inevitable.

There are several ways to improve the running time:
1. Use instant action interface like `agent.aiTap('Login Button')` instead of `agent.ai('Click Login Button')`. Read more about it in [API](./API).
2. Use a dedicated model and deploy it yourself, like UI-TARS. This is the recommended way. Read more about it in [Choose a model](./choose-a-model).
3. Use a lower resolution if possible.
4. Use caching to accelerate the debug process. Read more about it in [Caching](./caching).

## The webpage continues to flash when running in headed mode

It's common when the viewport `deviceScaleFactor` does not match your system settings. Setting it to 2 in OSX will solve the issue.

```typescript
await page.setViewport({
  deviceScaleFactor: 2,
});
```

## Where are the report files saved?

The report files are saved in `./midscene-run/report/` by default.

## How can I learn about Midscene's working process?

⁠By reviewing the report file after running the script, you can gain an overview of how Midscene works. 

## Customize the network timeout

When doing interaction or navigation on web page, Midscene automatically waits for the network to be idle. It's a strategy to ensure the stability of the automation. Nothing would happen if the waiting process is timeout. 

The default timeout is configured as follows:

1. If it's a page navigation, the default wait timeout is 5000ms (the `waitForNavigationTimeout`)
2. If it's a click, input, etc., the default wait timeout is 2000ms (the `waitForNetworkIdleTimeout`)

You can also customize or disable the timeout by options:

- Use `waitForNetworkIdleTimeout` and `waitForNavigationTimeout` parameters in [Agent](/api.html#constructors).
- Use `waitForNetworkIdle` parameter in [Yaml](/automate-with-scripts-in-yaml.html#the-web-part) or [PlaywrightAiFixture](/integrate-with-playwright.html#step-2-extend-the-test-instance).


---
url: /index.md
---

# Midscene.js - Joyful Automation by AI

Your AI Operator for Web, Android, Automation & Testing

## Interact, query and assert by natural language

There are three main capabilities: **action**, **query**, **assert**.

* Use **action (`.ai`, `.aiAction`)** to execute a series of actions by describing the steps
* Use **query (`.aiQuery`)** to extract customized data from the UI. Describe the JSON format you want, and AI will give the answer based on its "understanding" of the page
* Use **assert (`.aiAssert`)** to perform assertions on the page.

All these methods accept natural language prompt as param. Obviously, the cost of script maintenance will be greatly decreased.

## Start with Chrome extension

To quickly experience the main features of Midscene, you can use the Midscene Chrome extension. It allows you to use Midscene on any webpage without writing any code.

Click [here](https://chromewebstore.google.com/detail/midscene/gbldofcpkknbggpkmbdaefngejllnief) to install Midscene extension from Chrome Web Store.

For instructions, please refer to [Quick Experience](/en/quick-experience.md).

## Multiple ways to integrate

Maintaining automation scripts by Midscene could be a brand new experience. For example, to search for headphones on a website, you can do this:

```typescript
// 👀 type keywords, perform a search
await ai('type "Headphones" in search box, hit Enter');

// 👀 find the items, return in JSON
const items = await aiQuery(
  "{itemTitle: string, price: Number}[], find item in list and corresponding price"
);

console.log("headphones in stock", items);

// 👀 assert by natural language
await aiAssert("There is a category filter on the left");
```

There are several ways to integrate Midscene into your code project:

* [Automate with Scripts in YAML](/en/automate-with-scripts-in-yaml.md), use this if you prefer to write YAML file instead of code
* [Bridge Mode by Chrome Extension](/en/bridge-mode-by-chrome-extension.md), use this to control the desktop Chrome by scripts
* [Integrate with Puppeteer](/en/integrate-with-puppeteer.md)
* [Integrate with Playwright](/en/integrate-with-playwright.md)
* [Integrate with Android](/en/integrate-with-android.md)

## Visualized report

Midscene wants to provide a way to make automation more stable and easier to debug, so we provide a visual report after each run. With this report, you can review the animated replay and view the details of each step in the process.

What's more, there is a playground in the report file for you to adjust your prompt without re-running all your scripts.

## ✨ Model Choices

You can use multimodal LLMs like `gpt-4o`, or visual-language models like `Qwen2.5-VL`, `gemini-2.5-pro` and `UI-TARS`. In which `UI-TARS` is an open-source model dedicated for UI automation.

Read more about [Choose a model](https://midscenejs.com/choose-a-model)

## 👀 Comparing to ...

There are so many UI automation tools out there, and each one seems to be all-powerful. What's special about Midscene.js?

* Debugging Experience: You will soon realize that debugging and maintaining automation scripts is the real challenge. No matter how magical the demo looks, ensuring stability over time requires careful debugging. Midscene.js offers a visualized report file, a built-in playground, and a Chrome Extension to simplify the debugging process. These are the tools most developers truly need, and we’re continually working to improve the debugging experience.

* Open Source, Free, Deploy as you want: Midscene.js is an open-source project. It's decoupled from any cloud service and model provider, you can choose either public or private deployment. There is always a suitable plan for your business.

* Integrate with Javascript: You can always bet on Javascript 😎

## Just you and model provider, no third-party services

All data gathered from pages will be sent directly to OpenAI or the custom model provider according to your configuration. Therefore, no third-party platform will access the data.

For more details, please refer to [Data Privacy](/en/data-privacy.md).

## Follow us

* [GitHub - give us a star if you like it!](https://github.com/web-infra-dev/midscene)
* [Twitter](https://x.com/midscene_ai)
* [Discord](https://discord.gg/2JyBHxszE4)
* [Lark](https://applink.larkoffice.com/client/chat/chatter/add_by_link?link_token=291q2b25-e913-411a-8c51-191e59aab14d)



---
url: /integrate-with-android.md
---

# Integrate with Android (adb)

After connecting the Android device with adb, you can use Midscene javascript SDK to control Android devices.



:::info Demo Project
Control Android devices with javascript: [https://github.com/web-infra-dev/midscene-example/blob/main/android/javascript-sdk-demo](https://github.com/web-infra-dev/midscene-example/blob/main/android/javascript-sdk-demo)

Integrate Vitest for testing: [https://github.com/web-infra-dev/midscene-example/tree/main/android/vitest-demo](https://github.com/web-infra-dev/midscene-example/tree/main/android/vitest-demo)
:::

## Preparation

### Install Node.js

Install [Node.js 18 or above](https://nodejs.org/en/download/) globally.

### Prepare an API Key

Prepare an API key from a visual-language (VL) model. You will use it later.

You can check the supported models in [Choose a model](/en/choose-a-model.md)

### Install adb

`adb` is a command-line tool that allows you to communicate with an Android device. There are two ways to install `adb`:

* way 1: use [Android Studio](https://developer.android.com/studio) to install
* way 2: use [Android command-line tools](https://developer.android.com/studio#command-line-tools-only) to install

Verify adb is installed successfully:

```bash
adb --version
```

When you see the following output, adb is installed successfully:

```log
Android Debug Bridge version 1.0.41
Version 34.0.4-10411341
Installed as /usr/local/bin//adb
Running on Darwin 24.3.0 (arm64)
```

### Set environment variable ANDROID\_HOME

Reference [Android environment variables](https://developer.android.com/tools/variables), set the environment variable `ANDROID_HOME`.

Verify the `ANDROID_HOME` variable is set successfully:

```bash
echo $ANDROID_HOME
```

When the command has any output, the `ANDROID_HOME` variable is set successfully:

```log
/Users/your_username/Library/Android/sdk
```

### Connect Android device with adb

In the developer options of the system settings, enable the 'USB debugging' of the Android device, if the 'USB debugging (secure settings)' exists, also enable it, then connect the Android device with a USB cable

Verify the connection:

```bash
adb devices -l
```

When you see the following output, the connection is successful:

```log
List of devices attached
s4ey59	device usb:34603008X product:cezanne model:M2006J device:cezan transport_id:3
```

## Setup AI model service

Set your model configs into the environment variables. You may refer to [choose a model](/en/choose-a-model.md) for more details.

```bash
# replace with your own
export OPENAI_API_KEY="sk-abcdefghijklmnopqrstuvwxyz"
```

## Step 1. install dependencies

## Step 2. write scripts

Let's take a simple example: search for headphones on eBay using the browser in the Android device. （Of course, you can also use any other apps on the Android device.）

Write the following code, and save it as `./demo.ts`

```typescript title="./demo.ts"
import { AndroidAgent, AndroidDevice, getConnectedDevices } from '@midscene/android';

const sleep = (ms) => new Promise((r) => setTimeout(r, ms));
Promise.resolve(
  (async () => {
    const devices = await getConnectedDevices();
    const page = new AndroidDevice(devices[0].udid);

    // 👀 init Midscene agent
    const agent = new AndroidAgent(page,{
      aiActionContext:
        'If any location, permission, user agreement, etc. popup, click agree. If login page pops up, close it.',
    });
    await page.connect();

    // 👀 open browser and navigate to ebay.com (Please ensure that the current page has a browser app)
    await agent.aiAction('open browser and navigate to ebay.com');

    await sleep(5000);

    // 👀 type keywords, perform a search
    await agent.aiAction('type "Headphones" in search box, hit Enter');

    // 👀 wait for loading completed
    await agent.aiWaitFor("There is at least one headphone product");
    // or you can use a normal sleep:
    // await sleep(5000);

    // 👀 understand the page content, extract data
    const items = await agent.aiQuery(
      "{itemTitle: string, price: Number}[], find item in list and corresponding price"
    );
    console.log("headphones in stock", items);

    // 👀 assert by AI
    await agent.aiAssert("There is a category filter on the left");
  })()
);

```

## Step 3. run

Using `tsx` to run

```bash
# run
npx tsx demo.ts
```

After a while, you will see the following output:

```log
[
{
  itemTitle: 'Beats by Dr. Dre Studio Buds Totally Wireless Noise Cancelling In Ear + OPEN BOX',
  price: 505.15
},
{
  itemTitle: 'Skullcandy Indy Truly Wireless Earbuds-Headphones Green Mint',
  price: 186.69
}
]
```

## Step 4: view the report

After the above command executes successfully, the console will output: `Midscene - report file updated: /path/to/report/some_id.html`. You can open this file in a browser to view the report.

## More interfaces in AndroidAgent

Except the common agent interfaces in [API Reference](/en/API.md), AndroidAgent also provides some other interfaces:

### `agent.launch()`

Launch a webpage or native page.

* Type

```typescript
function launch(uri: string): Promise<void>;
```

* Parameters:
  * `uri: string` - The uri to open, can be a webpage url or a native app's package name or activity name, if the activity name exists, it should be separated by / (e.g. com.android.settings/.Settings).

* Return Value:
  * Returns a Promise that resolves to void when the page is opened.

* Examples:

```typescript
import { AndroidAgent, AndroidDevice } from '@midscene/android';

const page = new AndroidDevice('s4ey59');
const agent = new AndroidAgent(page);

await agent.launch('https://www.ebay.com'); // open a webpage
await agent.launch('com.android.settings'); // open a native page
await agent.launch('com.android.settings/.Settings'); // open a native page
```

### `agentFromAdbDevice()`

Create a AndroidAgent from a connected adb device.

* Type

```typescript
function agentFromAdbDevice(deviceId?: string, opts?: PageAgentOpt): Promise<AndroidAgent>;
```

* Parameters:
  * `deviceId?: string` - Optional, the adb device id to connect. If not provided, the first connected device will be used.
  * `opts?: PageAgentOpt` - Optional, the options for the AndroidAgent, refer to [constructor](/en/API.md).

* Return Value:
  * `Promise<AndroidAgent>` Returns a Promise that resolves to an AndroidAgent.

* Examples:

```typescript
import { agentFromAdbDevice } from '@midscene/android';

const agent = await agentFromAdbDevice('s4ey59'); // create a AndroidAgent from a specific adb device
const agent = await agentFromAdbDevice(); // no deviceId, use the first connected device
```

### `getConnectedDevices()`

Get all connected Android devices.

* Type

```typescript
function getConnectedDevices(): Promise<Device[]>;
interface Device {
  /**
   * The device udid.
   */
  udid: string;
  /**
   * Current device state, as it is visible in
   * _adb devices -l_ output.
   */
  state: string;
  port?: number;
}
```

* Return Value:
  * `Promise<Device[]>` Returns a Promise that resolves to an array of Device.

* Examples:

```typescript
import { agentFromAdbDevice, getConnectedDevices } from '@midscene/android';

const devices = await getConnectedDevices();
console.log(devices);
const agent = await agentFromAdbDevice(devices[0].udid);
```

## More

* For all the APIs on the Agent, please refer to [API Reference](/en/API.md).
* For more details about prompting, please refer to [Prompting Tips](/en/prompting-tips.md)

## FAQ

### Why can't I control the device even though I've connected it?

Please check if the device is unlocked in the developer options of the system settings.

### How to use a custom adb path?

You can use the `MIDSCENE_ADB_PATH` environment variable to specify the path to the adb executable.

```bash
export MIDSCENE_ADB_PATH=/path/to/adb
```



---
url: /integrate-with-playwright.md
---

# Integrate with Playwright



[Playwright.js](https://playwright.com/) is an open-source automation library developed by Microsoft, primarily used for end-to-end testing and web scraping of web applications.

Here we assume you already have a repository with Playwright integration.

:::info Example Project
You can find an example project of Playwright integration here: [https://github.com/web-infra-dev/midscene-example/blob/main/playwright-demo](https://github.com/web-infra-dev/midscene-example/blob/main/playwright-demo)
:::

## Setup AI model service

Set your model configs into the environment variables. You may refer to [choose a model](/en/choose-a-model.md) for more details.

```bash
# replace with your own
export OPENAI_API_KEY="sk-abcdefghijklmnopqrstuvwxyz"
```

## Step 1: add dependencies and update configuration

Add dependencies

Update playwright.config.ts

```diff
export default defineConfig({
  testDir: './e2e',
+ timeout: 90 * 1000,
+ reporter: [["list"], ["@midscene/web/playwright-report", { type: "merged" }]], // type optional, default is "merged", means multiple test cases generate one report, optional value is "separate", means one report for each test case
});
```

## Step 2: extend the `test` instance

Save the following code as `./e2e/fixture.ts`:

```typescript
import { test as base } from '@playwright/test';
import type { PlayWrightAiFixtureType } from '@midscene/web/playwright';
import { PlaywrightAiFixture } from '@midscene/web/playwright';

export const test = base.extend<PlayWrightAiFixtureType>(PlaywrightAiFixture({
  waitForNetworkIdleTimeout: 2000, // optional, the timeout for waiting for network idle between each action, default is 2000ms
}));
```

## Step 3: write test cases

### Basic AI Operation APIs

* `ai` or `aiAction` - General AI interaction
* `aiTap` - Click operation
* `aiHover` - Hover operation
* `aiInput` - Input operation
* `aiKeyboardPress` - Keyboard operation
* `aiScroll` - Scroll operation

### Query

* `aiQuery` - AI Query

### More APIs

* `aiAssert` - AI Assertion
* `aiWaitFor` - AI Wait

You can find more details in [API Reference](/en/API.md).

### Example Code

```typescript title="./e2e/ebay-search.spec.ts"
import { expect } from "@playwright/test";
import { test } from "./fixture";

test.beforeEach(async ({ page }) => {
  page.setViewportSize({ width: 400, height: 905 });
  await page.goto("https://www.ebay.com");
  await page.waitForLoadState("networkidle");
});

test("search headphone on ebay", async ({ 
  ai, 
  aiQuery, 
  aiAssert,
  aiInput,
  aiTap,
  aiScroll,
  aiWaitFor
}) => {
  // Use aiInput to enter search keyword
  await aiInput('Headphones', 'search box');
  
  // Use aiTap to click search button
  await aiTap('search button');
  
  // Wait for search results to load
  await aiWaitFor('search results list loaded', { timeoutMs: 5000 });
  
  // Use aiScroll to scroll to bottom
  await aiScroll(
    { 
      direction: 'down',
      scrollType: 'untilBottom'
    },
    'search results list'
  );
  
  // Use aiQuery to get product information
  const items = await aiQuery<Array<{title: string, price: number}>>(
    'get product titles and prices from search results'
  );
  
  console.log("headphones in stock", items);
  expect(items?.length).toBeGreaterThan(0);
  
  // Use aiAssert to verify filter functionality
  await aiAssert("category filter exists on the left side");
});
```

For more Agent API details, please refer to [API Reference](/en/API.md).

## Step 4. run test cases

```bash
npx playwright test ./e2e/ebay-search.spec.ts
```

## Step 5. view test report

After the command executes successfully, it will output: `Midscene - report file updated: ./current_cwd/midscene_run/report/some_id.html`. Open this file in your browser to view the report.

## More

* For all the methods on the Agent, please refer to [API Reference](/en/API.md).
* For more details about prompting, please refer to [Prompting Tips](/en/prompting-tips.md)



---
url: /integrate-with-puppeteer.md
---

# Integrate with Puppeteer



[Puppeteer](https://pptr.dev/) is a Node.js library which provides a high-level API to control Chrome or Firefox over the DevTools Protocol or WebDriver BiDi. Puppeteer runs in the headless (no visible UI) by default but can be configured to run in a visible ("headful") browser.

:::info Demo Project
you can check the demo project of Puppeteer here: [https://github.com/web-infra-dev/midscene-example/blob/main/puppeteer-demo](https://github.com/web-infra-dev/midscene-example/blob/main/puppeteer-demo)

There is also a demo of Puppeteer with Vitest: [https://github.com/web-infra-dev/midscene-example/tree/main/puppeteer-with-vitest-demo](https://github.com/web-infra-dev/midscene-example/tree/main/puppeteer-with-vitest-demo)
:::

## Setup AI model service

Set your model configs into the environment variables. You may refer to [choose a model](/en/choose-a-model.md) for more details.

```bash
# replace with your own
export OPENAI_API_KEY="sk-abcdefghijklmnopqrstuvwxyz"
```

## Step 1. install dependencies

## Step 2. write scripts

Write and save the following code as `./demo.ts`.

```typescript title="./demo.ts"
import puppeteer from "puppeteer";
import { PuppeteerAgent } from "@midscene/web/puppeteer";

const sleep = (ms: number) => new Promise((r) => setTimeout(r, ms));
Promise.resolve(
  (async () => {
    const browser = await puppeteer.launch({
      headless: false, // here we use headed mode to help debug
    });

    const page = await browser.newPage();
    await page.setViewport({
      width: 1280,
      height: 800,
      deviceScaleFactor: 1,
    });

    await page.goto("https://www.ebay.com");
    await sleep(5000);

    // 👀 init Midscene agent
    const agent = new PuppeteerAgent(page);

    // 👀 type keywords, perform a search
    await agent.aiAction('type "Headphones" in search box, hit Enter');
    await sleep(5000);

    // 👀 understand the page content, find the items
    const items = await agent.aiQuery(
      "{itemTitle: string, price: Number}[], find item in list and corresponding price"
    );
    console.log("headphones in stock", items);

    // 👀 assert by AI
    await agent.aiAssert("There is a category filter on the left");

    await browser.close();
  })()
);
```

## Step 3. run

Using `tsx` to run, you will get the data of Headphones on eBay:

```bash
# run
npx tsx demo.ts

# it should print 
#  [
#   {
#     itemTitle: 'Beats by Dr. Dre Studio Buds Totally Wireless Noise Cancelling In Ear + OPEN BOX',
#     price: 505.15
#   },
#   {
#     itemTitle: 'Skullcandy Indy Truly Wireless Earbuds-Headphones Green Mint',
#     price: 186.69
#   }
# ]
```

For the agent's more APIs, please refer to [API](/en/API.md).

## Step 4: view the report

After the above command executes successfully, the console will output: `Midscene - report file updated: /path/to/report/some_id.html`. You can open this file in a browser to view the report.

## More options in PuppeteerAgent constructor

### To limit the popup to the current page

If you want to limit the popup to the current page (like clicking a link with `target="_blank"`), you can set the `forceSameTabNavigation` option to `true`:

```typescript
const mid = new PuppeteerAgent(page, {
  forceSameTabNavigation: true,
});
```

## More

* For all the APIs on the Agent, please refer to [API Reference](/en/API.md).
* For more details about prompting, please refer to [Prompting Tips](/en/prompting-tips.md)



---
url: /llm-txt.md
---

# LLMs.txt Documentation

How to get tools like Cursor, Windstatic, GitHub Copilot, ChatGPT, and Claude to understand Midscene.js.

We support LLMs.txt files for making the Midscene.js documentation available to large language models.

## Directory Overview

The following files are available.

* [llms.txt](https://midscenejs.com/llms.txt): The main LLMs.txt file
* [llms-full.txt](https://midscenejs.com/llms-full.txt): The complete documentation for Midscene.js

## Usage

### Cursor

Use `@Docs` feature in Cursor to include the LLMs.txt files in your project.

[Read more](https://docs.cursor.com/context/@-symbols/@-docs)

### Windstatic

Reference the LLMs.txt files using `@` or in your `.windsurfrules` files.

[Read more](https://docs.windsurf.com/windsurf/getting-started#memories-and-rules)



---
url: /mcp.md
---

# MCP Server

Midscene provides a MCP server that allows AI assistants to control browsers, automate web tasks and write automation scripts for Midscene.

:::info MCP Introduction
MCP ([Model Context Protocol](https://modelcontextprotocol.io/introduction)) is a standardized way for AI models to interact with external tools and capabilities. MCP servers expose a set of tools that AI models can invoke to perform various tasks. In Midscene's case, these tools allow AI models to control browsers, navigate web pages, interact with UI elements, and more.
:::

## Use Cases

* Control browsers to execute automation tasks
* Automatically generate Midscene automation scripts

### Examples

> Generate Midscene test cases for the Sauce Demo site

## Setting Up Midscene MCP

### Prerequisites

1. An OpenAI API key or another supported AI model provider. For more information, see [Choosing an AI Model](/en/choose-a-model.md).
2. For Chrome browser integration (Bridge Mode):
   * Install the Midscene Chrome extension (download from [Chrome Web Extension](https://chromewebstore.google.com/detail/midscenejs/gbldofcpkknbggpkmbdaefngejllnief?hl=en\&utm_source=ext_sidebar))
   * Switch to "Bridge Mode" in the extension and click "Allow Connection"

### Configuration

Add the Midscene MCP server to your MCP configuration:

```json
{
  "mcpServers": {
    "mcp-midscene": {
      "command": "npx",
      "args": ["-y", "@midscene/mcp"],
      "env": {
        "MIDSCENE_MODEL_NAME": "REPLACE_WITH_YOUR_MODEL_NAME",
        "OPENAI_API_KEY": "REPLACE_WITH_YOUR_OPENAI_API_KEY",
        "MCP_SERVER_REQUEST_TIMEOUT": "800000"
      }
    }
  }
}
```

For more information about configuring AI models, see [Choosing an AI Model](/en/choose-a-model.md).

## Available Tools

Midscene MCP provides the following browser automation tools:

| Category | Tool Name | Description |
|---------|---------|---------|
| **Navigation** | midscene\_navigate | Navigate to a specified URL in the current tab |
| **Tab Management** | midscene\_get\_tabs | Get a list of all open browser tabs |
| | midscene\_set\_active\_tab | Switch to a specific tab by ID |
| **Page Interaction** | midscene\_aiTap | Click on an element described in natural language |
| | midscene\_aiInput | Input text into a form field or element |
| | midscene\_aiHover | Hover over an element |
| | midscene\_aiKeyboardPress | Press a specific keyboard key |
| | midscene\_aiScroll | Scroll the page or a specific element |
| **Verification and Observation** | midscene\_aiWaitFor | Wait for a condition to be true on the page |
| | midscene\_aiAssert | Assert that a condition is true on the page |
| | midscene\_screenshot | Take a screenshot of the current page |
| **Playwright Code Example** | midscene\_playwright\_example | Provides Playwright code examples for Midscene |

### Navigation

* **midscene\_navigate**: Navigate to a specified URL in the current tab
  ```
  Parameters:
  - url: The URL to navigate to
  ```

### Tab Management

* **midscene\_get\_tabs**: Get a list of all open browser tabs, including their IDs, titles, and URLs
  ```
  Parameters: None
  ```

* **midscene\_set\_active\_tab**: Switch to a specific tab by ID
  ```
  Parameters:
  - tabId: The ID of the tab to activate
  ```

### Page Interaction

* **midscene\_aiTap**: Click on an element described in natural language
  ```
  Parameters:
  - locate: Natural language description of the element to click
  ```

* **midscene\_aiInput**: Input text into a form field or element
  ```
  Parameters:
  - value: The text to input
  - locate: Natural language description of the element to input text into
  ```

* **midscene\_aiHover**: Hover over an element
  ```
  Parameters:
  - locate: Natural language description of the element to hover over
  ```

* **midscene\_aiKeyboardPress**: Press a specific keyboard key
  ```
  Parameters:
  - key: The key to press (e.g., 'Enter', 'Tab', 'Escape')
  - locate: (Optional) Description of element to focus before pressing the key
  - deepThink: (Optional) If true, uses more precise element location
  ```

* **midscene\_aiScroll**: Scroll the page or a specific element
  ```
  Parameters:
  - direction: 'up', 'down', 'left', or 'right'
  - scrollType: 'once', 'untilBottom', 'untilTop', 'untilLeft', or 'untilRight'
  - distance: (Optional) Distance to scroll in pixels
  - locate: (Optional) Description of the element to scroll
  - deepThink: (Optional) If true, uses more precise element location
  ```

### Verification and Observation

* **midscene\_aiWaitFor**: Wait for a condition to be true on the page
  ```
  Parameters:
  - assertion: Natural language description of the condition to wait for
  - timeoutMs: (Optional) Maximum time to wait in milliseconds
  - checkIntervalMs: (Optional) How often to check the condition
  ```

* **midscene\_aiAssert**: Assert that a condition is true on the page
  ```
  Parameters:
  - assertion: Natural language description of the condition to check
  ```

* **midscene\_screenshot**: Take a screenshot of the current page
  ```
  Parameters:
  - name: Name for the screenshot
  ```

## Common Issues

### What advantages does Midscene MCP have over other browser MCPs?

* Midscene MCP supports Bridge mode by default, allowing you to **directly control your current browser** without needing to **log in again or download a browser**
* Midscene MCP includes built-in optimal prompt templates and operation execution practices for browser page control, providing **more stable and reliable browser automation experiences** compared to other MCP implementations
* Midscene MCP automatically generates execution case reports after completing tasks, allowing you to **view the execution process at any time**

### Local port conflicts when multiple Clients are used

> Problem description

When users simultaneously use Midscene MCP in multiple local clients (Claude Desktop, Cursor MCP, etc.), port conflicts may occur causing server errors

> Solution

* Temporarily close the MCP server in the extra clients
* Execute the command:

```bash
# For macOS/Linux:
lsof -i:3766 | awk 'NR>1 {print $2}' | xargs -r kill -9

# For Windows:
FOR /F "tokens=5" %i IN ('netstat -ano ^| findstr :3766') DO taskkill /F /PID %i
```

### How to Access Midscene Execution Reports

After each task execution, a Midscene task report is generated. You can open this HTML report directly from the command line:

```bash
# Replace the opened address with your report filename
open report_file_name.html
```

![image](https://lf3-static.bytednsdoc.com/obj/eden-cn/ozpmyhn_lm_hymuPild/ljhwZthlaukjlkulzlp/midscene/image.png)



---
url: /model-provider.md
---

# Config Model and Provider

Midscene uses the OpenAI SDK to call AI services. Using this SDK limits the input and output schema of AI services, but it doesn't mean you can only use OpenAI's services. You can use any model service that supports the same interface (most platforms or tools support this).

In this article, we will show you how to config AI service provider and how to choose a different model. You may read [Choose a model](/en/choose-a-model.md) first to learn more about how to choose a model.

## Configs

### Common configs

These are the most common configs, in which `OPENAI_API_KEY` is required.

| Name | Description |
|------|-------------|
| `OPENAI_API_KEY` | Required. Your OpenAI API key (e.g. "sk-abcdefghijklmnopqrstuvwxyz") |
| `OPENAI_BASE_URL` | Optional. Custom endpoint URL for API endpoint. Use it to switch to a provider other than OpenAI (e.g. "https://some\_service\_name.com/v1") |
| `MIDSCENE_MODEL_NAME` | Optional. Specify a different model name other than `gpt-4o` |

Extra configs to use `Qwen 2.5 VL` model:

| Name | Description |
|------|-------------|
| `MIDSCENE_USE_QWEN_VL` | Set to "1" to use the adapter of Qwen 2.5 VL model |

Extra configs to use `UI-TARS` model:

| Name | Description |
|------|-------------|
| `MIDSCENE_USE_VLM_UI_TARS` | Version of UI-TARS model, supported values are `1.0` `1.5` `DOUBAO` (volcengine version) |

Extra configs to use `Gemini 2.5 Pro` model:

| Name | Description |
|------|-------------|
| `MIDSCENE_USE_GEMINI` | Set to "1" to use the adapter of Gemini 2.5 Pro model |

For more information about the models, see [Choose a model](/en/choose-a-model.md).

### Advanced configs

Some advanced configs are also supported. Usually you don't need to use them.

| Name | Description |
|------|-------------|
| `OPENAI_USE_AZURE` | Optional. Set to "true" to use Azure OpenAI Service. See more details in the following section. |
| `MIDSCENE_OPENAI_INIT_CONFIG_JSON` | Optional. Custom JSON config for OpenAI SDK initialization |
| `MIDSCENE_OPENAI_SOCKS_PROXY` | Optional. Proxy configuration (e.g. "socks5://127.0.0.1:1080") |
| `OPENAI_MAX_TOKENS` | Optional. Maximum tokens for model response |

### Debug configs

By setting the following configs, you can see more logs for debugging. And also, they will be printed into the `./midscene_run/log` folder.

| Name | Description |
|------|-------------|
| `DEBUG=midscene:ai:profile:stats` | Optional. Set this to print the AI service cost time, token usage, etc. in comma separated format, useful for analysis |
| `DEBUG=midscene:ai:profile:detail` | Optional. Set this to print the AI token usage details |
| `DEBUG=midscene:ai:call` | Optional. Set this to print the AI response details |
| `DEBUG=midscene:android:adb` | Optional. Set this to print the adb command calling details |

## Two ways to config environment variables

Pick one of the following ways to config environment variables.

### 1. Set environment variables in your system

```bash
# replace by your own
export OPENAI_API_KEY="sk-abcdefghijklmnopqrstuvwxyz"
```

### 2. Set environment variables using dotenv

This is what we used in our [demo project](https://github.com/web-infra-dev/midscene-example).

[Dotenv](https://www.npmjs.com/package/dotenv) is a zero-dependency module that loads environment variables from a `.env` file into `process.env`.

```bash
# install dotenv
npm install dotenv --save
```

Create a `.env` file in your project root directory, and add the following content. There is no need to add `export` before each line.

```
OPENAI_API_KEY=sk-abcdefghijklmnopqrstuvwxyz
```

Import the dotenv module in your script. It will automatically read the environment variables from the `.env` file.

```typescript
import 'dotenv/config';
```

## Using Azure OpenAI Service

There are some extra configs when using Azure OpenAI Service.

### Use ADT token provider

This mode cannot be used in Chrome extension.

```bash
# this is always true when using Azure OpenAI Service
export MIDSCENE_USE_AZURE_OPENAI=1

export MIDSCENE_AZURE_OPENAI_SCOPE="https://cognitiveservices.azure.com/.default"
export AZURE_OPENAI_ENDPOINT="..."
export AZURE_OPENAI_API_VERSION="2024-05-01-preview"
export AZURE_OPENAI_DEPLOYMENT="gpt-4o"
```

### Use keyless authentication

```bash
export MIDSCENE_USE_AZURE_OPENAI=1
export AZURE_OPENAI_ENDPOINT="..."
export AZURE_OPENAI_KEY="..."
export AZURE_OPENAI_API_VERSION="2024-05-01-preview"
export AZURE_OPENAI_DEPLOYMENT="gpt-4o"
```

## Set Config by Javascript

You can also override the config by javascript. Remember to call this before running Midscene codes.

```typescript
import { overrideAIConfig } from "@midscene/web/puppeteer";
// or import { overrideAIConfig } from "@midscene/web/playwright";
// or import { overrideAIConfig } from "@midscene/android";


overrideAIConfig({
  MIDSCENE_MODEL_NAME: "...",
  // ...
});
```

## Example: Using `gpt-4o` from OpenAI

Configure the environment variables:

```bash
export OPENAI_API_KEY="sk-..."
export OPENAI_BASE_URL="https://endpoint.some_other_provider.com/v1" # config this if you want to use a different endpoint
export MIDSCENE_MODEL_NAME="gpt-4o-2024-11-20" # optional, the default is "gpt-4o"
```

## Example: Using `qwen-vl-max-latest` from Aliyun

Configure the environment variables:

```bash
export OPENAI_API_KEY="sk-..."
export OPENAI_BASE_URL="https://dashscope.aliyuncs.com/compatible-mode/v1"
export MIDSCENE_MODEL_NAME="qwen-vl-max-latest"
export MIDSCENE_USE_QWEN_VL=1
```

## Example: Using `Doubao-1.5-thinking-vision-pro` from Volcano Engine

Configure the environment variables:

```bash
export OPENAI_BASE_URL="https://ark-cn-beijing.bytedance.net/api/v3"
export OPENAI_API_KEY="..."
export MIDSCENE_MODEL_NAME='ep-...'
export MIDSCENE_USE_DOUBAO_VISION=1
```

## Example: Using `ui-tars-72b-sft` hosted by yourself

Configure the environment variables:

```bash
export OPENAI_API_KEY="sk-..."
export OPENAI_BASE_URL="http://localhost:1234/v1"
export MIDSCENE_MODEL_NAME="ui-tars-72b-sft"
export MIDSCENE_USE_VLM_UI_TARS=1
```

## Example: Config `claude-3-opus-20240229` from Anthropic

When configuring `MIDSCENE_USE_ANTHROPIC_SDK=1`, Midscene will use Anthropic SDK (`@anthropic-ai/sdk`) to call the model.

Configure the environment variables:

```bash
export MIDSCENE_USE_ANTHROPIC_SDK=1
export ANTHROPIC_API_KEY="....."
export MIDSCENE_MODEL_NAME="claude-3-opus-20240229"
```

## Example: config request headers (like for openrouter)

```bash
export OPENAI_BASE_URL="https://openrouter.ai/api/v1"
export OPENAI_API_KEY="..."
export MIDSCENE_MODEL_NAME="..."
export MIDSCENE_OPENAI_INIT_CONFIG_JSON='{"defaultHeaders":{"HTTP-Referer":"...","X-Title":"..."}}'
```

## Troubleshooting LLM Service Connectivity Issues

If you want to troubleshoot connectivity issues, you can use the 'connectivity-test' folder in our example project: [https://github.com/web-infra-dev/midscene-example/tree/main/connectivity-test](https://github.com/web-infra-dev/midscene-example/tree/main/connectivity-test)

Put your `.env` file in the `connectivity-test` folder, and run the test with `npm i && npm run test`.



---
url: /prompting-tips.md
---

# Prompting Tips

The natural language parameter passed to Midscene will be part of the prompt sent to the AI model. There are certain techniques in prompt engineering that can help improve the understanding of user interfaces.

## The goal is to get a stable response from AI

Since AI has the nature of heuristic, the purpose of prompt tuning should be to obtain stable responses from the AI model across runs. In most cases, to expect a consistent response from AI model by using a good prompt is entirely feasible.

## Use detailed descriptions and samples

Detailed descriptions and examples are always welcome.

For example: 

❌ Don't
```log
Search 'headphone'
```

✅ Do
```log
Click the search box (it should be along with a region switch, such as 'domestic' or 'international'), type 'headphone', and hit Enter.
```

❌ Don't
```log
Assert: food delivery service is in normal state
```

✅ Do
```log
Assert: There is a 'food delivery service' on page, and is in normal state
```

### Use instant action interface if you are sure about what you want to do

For example:

`agent.ai('Click Login Button')` is the auto planning mode, Midscene will plan the steps and then execute them. It will cost more time and tokens.

By using `agent.aiTap('Login Button')`, you can directly using the locating result from the AI model and perform the click action. It's faster and more accurate compared to the auto planning mode.

For more details, please refer to [API](./API).

### Understand the reason why `.ai` is wrong

**Understanding the report**

By reviewing the report, you can see there are two main steps of each `.ai` call:

1. Planning
2. Locating

First, you should find out whether the AI is wrong in the planning step or the locating step.

When you see the steps are not as expected (more steps or less steps), it means the AI is wrong in the planning step. So you can try to give more details in the task flow.

For example:

❌ Don't
```log
Select "include" in the "range" dropdown menu
```

You can try:

✅ Do
```log
Click the "range" dropdown menu, and select "include"
```

When you see the locating result is not as expected (wrong element or biased coordinates), try to give more details in the locate parameter.

For example:

❌ Don't
```log
Click the "Add" button
```

You can try:

✅ Do
```log
Click the "Add" button on the top-right corner, it's on the right side of the "range" dropdown menu
```

**Other ways to improve**

* Use a larger and stronger AI model
* Use instant action interface like `agent.aiTap()` instead of `.ai` if you are sure about what you want to do

## One prompt should only do one thing

Use `.ai` each time to do one task. Although Midscene has an auto-replanning strategy, it's still preferable to keep the prompt concise. Otherwise the LLM output will likely be messy. The token cost between a long prompt and a short prompt is almost the same.

❌ Don't
```log
Click Login button, then click Sign up button, fill the form with 'test@test.com' in the email field, 'test' in the password field, and click Sign up button
```

✅ Split the task into the following steps into multiple `.ai` calls
```log
Click Login Button
Click Sign up button
Fill the form with 'test@test.com' in the email field
Fill the form with 'test' in the password field
Click Sign up button
```

### LLMs can NOT tell the exact number like coords or hex-style color, give it some choices

For example:

❌ Don't
```log
string, hex value of text color
```

❌ Don't
```log
[number, number], the [x, y] coords of the main button
```

✅ Do
```log
string, color of text, one of blue / red / yellow / green / white / black / others
```

## Use report file and playground tool to debug

Open the report file, you will see the detailed information about the steps. If you want to rerun a prompt together with UI context from the report file, just launch a Playground server and click "Send to Playground".

To launch the local Playground server:
```
npx --yes @midscene/web
```

![Playground](/midescene-playground-entry.jpg)

## Infer or assert from the interface, not the DOM properties or browser status

All the data sent to the LLM is in the form of screenshots and element coordinates. The DOM and the browser instance are almost invisible to the LLM. Therefore, ensure everything you expect is visible on the screen.

❌ Don't
```log
The title has a `test-id-size` property
```

❌ Don't
```log
The browser has two active tabs
```

❌ Don't
```log
The request has finished.
```

✅ Do
```log
The title is blue
```

## Cross-check the result using assertion

LLM could behave incorrectly. A better practice is to check its result after running.

For example, you can check the list content of the to-do app after inserting a record.

```typescript
await ai('Enter "Learning AI the day after tomorrow" in the task box, then press Enter to create');

// check the result
const taskList = await aiQuery<string[]>('string[], tasks in the list');
expect(taskList.length).toBe(1);
expect(taskList[0]).toBe('Learning AI the day after tomorrow');
```

## Non-English prompting is acceptable

Since most AI models can understand many languages, feel free to write the prompt in any language you prefer. It usually works even if the prompt is in a language different from the page's language.

✅ Good
```log
点击顶部左侧导航栏中的“首页”链接
```



---
url: /quick-experience-with-android.md
---

# Quick Experience with Android

By using Midscene.js playground, you can quickly experience the main features of Midscene on Android devices, without needing to write any code.

![](/android-playground.png)

## Preparation

### Install Node.js

Install [Node.js 18 or above](https://nodejs.org/en/download/) globally.

### Prepare an API Key

Prepare an API key from a visual-language (VL) model. You will use it later.

You can check the supported models in [Choose a model](/en/choose-a-model.md)

### Install adb

`adb` is a command-line tool that allows you to communicate with an Android device. There are two ways to install `adb`:

* way 1: use [Android Studio](https://developer.android.com/studio) to install
* way 2: use [Android command-line tools](https://developer.android.com/studio#command-line-tools-only) to install

Verify adb is installed successfully:

```bash
adb --version
```

When you see the following output, adb is installed successfully:

```log
Android Debug Bridge version 1.0.41
Version 34.0.4-10411341
Installed as /usr/local/bin//adb
Running on Darwin 24.3.0 (arm64)
```

### Set environment variable ANDROID\_HOME

Reference [Android environment variables](https://developer.android.com/tools/variables), set the environment variable `ANDROID_HOME`.

Verify the `ANDROID_HOME` variable is set successfully:

```bash
echo $ANDROID_HOME
```

When the command has any output, the `ANDROID_HOME` variable is set successfully:

```log
/Users/your_username/Library/Android/sdk
```

### Connect Android device with adb

In the developer options of the system settings, enable the 'USB debugging' of the Android device, if the 'USB debugging (secure settings)' exists, also enable it, then connect the Android device with a USB cable

Verify the connection:

```bash
adb devices -l
```

When you see the following output, the connection is successful:

```log
List of devices attached
s4ey59	device usb:34603008X product:cezanne model:M2006J device:cezan transport_id:3
```

## Run Playground

```bash
npx --yes @midscene/android-playground
```

## Config API Key

Click the gear button to enter the configuration page and paste your API key config.

![](/android-set-env.png)

Refer to [Config Model and Provider](/en/model-provider.md) document, config the API Key.

## Start experiencing

After the configuration, you can immediately experience Midscene. There are three main tabs in the extension:

* **Action**: interact with the web page. This is also known as "Auto Planning". For example:

```
type Midscene in the search box
click the login button
```

* **Query**: extract JSON data from the web page

```
extract the user id from the page, return in \{ id: string \}
```

* **Assert**: validate the page

```
the page title is "Midscene"
```

* **Tap**: perform a single tap on the element where you want to click. This is also known as "Instant Action".

```
the login button
```

Enjoy !

> For the different between "Auto Planning" and "Instant Action", please refer to the [API](/en/API.md) document.

## Want to write some code ?

After experiencing, you may want to write some code to integrate Midscene. There are multiple ways to do that. Please refer to the documents below:

* [Automate with Scripts in YAML](/en/automate-with-scripts-in-yaml.md)

* [Integrate javascript SDK with Android](/en/integrate-with-android.md)



---
url: /quick-experience.md
---

# Quick Experience by Chrome Extension

Midscene.js provides a Chrome extension. By using it, you can quickly experience the main features of Midscene on any webpage, without needing to set up a code project.

⁠The extension shares the same code as the npm `@midscene/web` packages, so you can think of it as a playground or a way to debug with Midscene.

![](/midscene-extension.jpg)

## Preparation

Prepare the config for the AI model you want to use. You can check the supported models in [Choose a model](/en/choose-a-model.md)

## Install and config

Install Midscene extension from chrome web store: [Midscene](https://chromewebstore.google.com/detail/midscene/gbldofcpkknbggpkmbdaefngejllnief)

Start the extension (may be folded by Chrome extension icon), setup the config by pasting the config in the K=V format:

```bash
OPENAI_API_KEY="sk-replace-by-your-own"
# ...all other configs here (if any)
```

## Start experiencing

After the configuration, you can immediately experience Midscene. There are three main tabs in the extension:

* **Action**: interact with the web page. This is also known as "Auto Planning". For example:

```
type Midscene in the search box
click the login button
```

* **Query**: extract JSON data from the web page

```
extract the user id from the page, return in \{ id: string \}
```

* **Assert**: validate the page

```
the page title is "Midscene"
```

* **Tap**: perform a single tap on the element where you want to click. This is also known as "Instant Action".

```
the login button
```

Enjoy !

> For the different between "Auto Planning" and "Instant Action", please refer to the [API](/en/API.md) document.

## Want to write some code ?

After experiencing, you may want to write some code to integrate Midscene. There are multiple ways to do that. Please refer to the documents below:

* [Automate with Scripts in YAML](/en/automate-with-scripts-in-yaml.md)

* [Bridge Mode by Chrome Extension](/en/bridge-mode-by-chrome-extension.md)

* [Integrate with Puppeteer](/en/integrate-with-puppeteer.md)

* [Integrate with Playwright](/en/integrate-with-playwright.md)

## FAQ

### Extension fails to run and shows 'Cannot access a chrome-extension:// URL of different extension'

It's mainly due to conflicts with other extensions injecting `<iframe />` or `<script />` into the page. Try disabling the suspicious plugins and refresh.

To find the suspicious plugins:

1. Open the Devtools of the page, find the `<script>` or `<iframe>` with a url like `chrome-extension://{ID-of-the-suspicious-plugin}/...`.
2. Copy the ID from the url, open `chrome://extensions/` , use cmd+f to find the plugin with the same ID, disable it.
3. Refresh the page, try again.