> For AI agents: the complete documentation index is available at /llms.txt, the full documentation bundle is available at /llms-full.txt.

# The Basics

This page explains Midscene's vision-based approach, usage options, and key APIs to help you understand and use the GUI Agent. To try it first, go to [Get started](/get-started.md).

## A vision-based approach to automation

Midscene interprets screenshots, locates targets, performs actions, and observes the result. UI localization and actions do not depend on DOM data, selectors, or extra annotations. This has four main advantages:

- **Serve the purpose of E2E testing**: Operate software as a user would and verify complete workflows from their perspective. Act on visible text, icons, and layout, then check the rendered result, including selected states, error messages, and obscured elements.
- **Work across UI stacks**: HTML, Canvas, mobile, and desktop apps all appear as screenshots. Once a platform provides screenshots and actions, the same Agent APIs apply, reducing dependence on application internals.
- **Align with mainstream model approaches**: [Gemini Computer Use](https://ai.google.dev/gemini-api/docs/computer-use) and [UI-TARS](https://github.com/bytedance/UI-TARS) both use screenshot-driven interaction. Midscene's model adapters let users choose models through a consistent API and benefit from improvements in model capabilities.
- **Affordable model costs**: In the [AppControlBench evaluation](/app-control-bench-report.md), Doubao Seed 2.1 Turbo ran 60 tasks and passed 58, with a total model API cost of about $0.59. Actual costs vary with the model, task, and cache usage.

This approach requires a model with UI localization capabilities; see [supported models](/model-selection.md). Data extraction and page understanding can still include DOM data when needed; see the [API reference](/reference.md#extraction-location-assertion).

## Choose how to use Midscene {#choose-your-path}

Midscene provides a cross-platform GUI Agent for web, mobile, and desktop apps. It follows natural-language instructions to interact with interfaces, check results, and extract data. For testing, choose an integration approach based on your existing project setup and how you prefer to write test cases.

If you prefer YAML-based testing, use **Midscene Test**, an E2E testing framework for web, mobile, and desktop apps. Describe test steps and expected results in YAML, extend business operations with TypeScript, and let the framework handle setup, cleanup, retries, and execution reports.

If you already have a JavaScript/TypeScript test project or prefer writing tests in code, use the SDK to integrate the GUI Agent into frameworks such as Playwright Test, Rstest, or Vitest to add visual actions and assertions to your test cases.

| Your goal                                      | Start here                                                                                                             |
| ---------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------- |
| Try natural-language actions and assertions    | [Get started](/get-started.md)                                                                                         |
| Organize web, mobile, or desktop tests in YAML | [Create a test project](/midscene-test/use.md)                                                                         |
| Write tests in a JavaScript/TypeScript project | [Integrate with Playwright Test](/integrate-with-playwright.md), [Rstest](/sdk/rstest.md), or [Vitest](/sdk/vitest.md) |
| Run standalone automation workflows            | [Standalone scripts](/sdk/standalone.md)                                                                               |
| Let an AI coding agent operate apps            | [Midscene Skills](/skills.md)                                                                                          |

## Key APIs {#javascript-orchestration}

### Plan and interact

**`aiAct`**

[`aiAct`](/reference.md#agentaiact) accepts a goal described in natural language. It observes the interface, plans the next steps, locates the target elements, and executes the actions until the goal is complete. The prompt can also include assertions. Midscene verifies these assertions during execution and throws an error promptly if one fails.

`aiAct` is flexible and autonomous, so it works well when a task has multiple steps, conditional branches, or an uncertain execution path. During execution, `aiAct` continuously uses AI to plan from the latest interface state, which means it usually takes more time and tokens than an instant interaction.

Typical usage:

```typescript
await agent.aiAct(
  'Search for headphones, add the first item to the cart, and confirm that the cart count changes to 1',
);
```

### Instant interactions

Instant interaction APIs perform one specified action. Their main job is to locate a UI element and execute a fixed operation on it.

These APIs do not plan a sequence of steps. A request such as “close the popup if it appears, then click the checkout button” should use `aiAct`; an instant interaction treats its prompt as a description of the target element, not as a workflow.

**`aiTap`**

[`aiTap`](/reference.md#agentaitap) locates and taps or clicks one element.

Typical usage:

```typescript
await agent.aiTap('The checkout button in the shopping cart');
```

**`aiInput`**

[`aiInput`](/reference.md#agentaiinput) locates an input field and enters a specified value. Its default `replace` mode clears the existing content before entering the new value.

Typical usage:

```typescript
await agent.aiInput('The email address input', {
  value: 'user@example.com',
});
```

Other input modes are `typeOnly`, which preserves the existing content, and `clear`, which only clears the field.

Other instant interaction APIs include `aiHover`, `aiClearInput`, `aiKeyboardPress`, `aiScroll`, `aiPinch`, `aiLongPress`, `aiDoubleClick`, and `aiRightClick`. Platform support varies; see [Planning and interaction](/reference.md#planning-interaction) in the API reference.

### Insight

Insight APIs observe the interface and return an analysis result without interacting with it. They use the current screenshot by default. On web pages, you can also pass `domIncluded` when the task requires DOM information that is not visible in the screenshot.

**`aiAssert`**

[`aiAssert`](/reference.md#agentaiassert) checks a condition described in natural language. It resolves when the condition is true. When the condition is false, it throws an error that includes the reason returned by the model.

Typical usage:

```typescript
await agent.aiAssert('The shopping cart contains one item and shows a subtotal');
```

**`aiQuery`**

[`aiQuery`](/reference.md#agentaiquery) extracts structured data from the interface. Describe both the required data and its expected type or shape in the prompt.

Typical usage:

```typescript
const items = await agent.aiQuery<
  Array<{ name: string; price: number }>
>('The products in the shopping cart, {name: string, price: number}[]');
// Example items value: [{ name: 'Wireless headphones', price: 99.9 }]
```

**`aiBoolean`**

[`aiBoolean`](/reference.md#agentaiboolean) answers a question about the interface and returns a boolean.

Typical usage:

```typescript
const loginDialogVisible = await agent.aiBoolean(
  'Is the login dialog visible?',
);
// Example loginDialogVisible value: true
```

Related convenience methods include [`aiNumber`](/reference.md#agentainumber) for numbers and [`aiString`](/reference.md#agentaistring) or [`aiAsk`](/reference.md#agentaiask) for strings.

### Improve planning and localization when needed

- **`deepThink`: enhance planning**. Available on `aiAct`, it focuses on task decomposition and separates planning and element localization into different model calls. Use it for complex, multi-step tasks. See [`deepThink` planning mode](/reference.md#aiact-deepthink).
- **`deepLocate`: enhance localization**. Available on APIs that locate elements, such as `aiAct` and `aiTap`, it uses an additional model call to improve accuracy for small or visually ambiguous targets. See [deep localization](/reference.md#deep-locate-deeplocate).

Both options are off by default. Enable them independently or combine them in `aiAct`. They add model-call overhead, so start with the defaults and enable them when you encounter planning or localization problems.

```typescript
await agent.aiAct('Complete the checkout form and stop before placing the order', {
  deepThink: true,
});
await agent.aiTap('The cart icon in the upper-right corner', { deepLocate: true });
```

## Next steps

- [Model Selection Guide](/model-selection.md): Learn when to use one Default model and when to combine models.
- [Get started](/get-started.md): Set up your platform and model, then complete a first run.
- [Create a test project](/midscene-test/use.md): Write and run YAML tests.
- [API reference](/reference.md): Look up full options and platform support.