#The Basics

This page provides an overview of Midscene's core concepts, including Agent usage, architecture, abstractions, and capability boundaries. You will learn how an Agent connects AI models to a target interface and how to perform interactions, assertions, and other operations.

#Plan and interact

#aiAct

aiAct accepts a goal described in natural language. It observes the interface, plans the next steps, locates the target elements, and executes the actions until the goal is complete. The prompt can also include assertions. Midscene verifies these assertions during execution and throws an error promptly if one fails.

aiAct is flexible and autonomous, so it works well when a task has multiple steps, conditional branches, or an uncertain execution path. During execution, aiAct continuously uses AI to plan from the latest interface state, which means it usually takes more time and tokens than an instant interaction.

Typical usage:

await agent.aiAct(
  'Search for headphones, add the first item to the cart, and confirm that the cart count changes to 1',
);

To give every subsequent aiAct call more business context, use agent.setAIActContext():

agent.setAIActContext(
  'Close the cookie consent dialog first if it appears. Prices are shown in USD.',
);

aiAct provides the following per-call options:

• deepThink: focuses more on task decomposition and uses separate model calls for planning and element localization. It can make complex tasks more stable, but increases model calls and latency.
• deepLocate: uses an additional model call to improve element localization accuracy. Enable it when a target is small or difficult to distinguish from nearby elements.
• context: provides business knowledge or other background for this call only. For aiAct, it overrides the Agent-level aiActContext, including when it is explicitly set to an empty string.

await agent.aiAct('Complete the checkout form and stop before placing the order', {
  deepThink: true,
  deepLocate: true,
  context: 'If an address confirmation dialog appears, select the default shipping address.',
});

#Instant interactions

Instant interaction APIs perform one specified action. Their main job is to locate a UI element and execute a fixed operation on it.

These APIs do not plan a sequence of steps. A request such as “close the popup if it appears, then click the checkout button” should use aiAct; an instant interaction treats its prompt as a description of the target element, not as a workflow.

#aiTap

aiTap locates and taps or clicks one element.

Typical usage:

await agent.aiTap('The checkout button in the shopping cart');

When the target is small or visually ambiguous, enable deepLocate (multi-pass deep localization):

await agent.aiTap('The cart icon in the upper-right corner', {
  deepLocate: true,
});

#aiInput

aiInput locates an input field and enters a specified value. Its default replace mode clears the existing content before entering the new value.

Typical usage:

await agent.aiInput('The email address input', {
  value: '[email protected]',
});

Other input modes are typeOnly, which preserves the existing content, and clear, which only clears the field.

Other instant interaction APIs include aiHover, aiClearInput, aiKeyboardPress, aiScroll, aiPinch, aiLongPress, aiDoubleClick, and aiRightClick. Platform support varies; see Planning and interaction in the API reference.

#Insight

Insight APIs observe the interface and return an analysis result without interacting with it. They use the current screenshot by default. On web pages, you can also pass domIncluded when the task requires DOM information that is not visible in the screenshot.

#aiAssert

aiAssert checks a condition described in natural language. It resolves when the condition is true. When the condition is false, it throws an error that includes the reason returned by the model.

Typical usage:

await agent.aiAssert('The shopping cart contains one item and shows a subtotal');

#aiQuery

aiQuery extracts structured data from the interface. Describe both the required data and its expected type or shape in the prompt.

Typical usage:

const items = await agent.aiQuery<
  Array<{ name: string; price: number }>
>('The products in the shopping cart, {name: string, price: number}[]');
// Example items value: [{ name: 'Wireless headphones', price: 99.9 }]

#aiBoolean

aiBoolean answers a question about the interface and returns a boolean.

Typical usage:

const loginDialogVisible = await agent.aiBoolean(
  'Is the login dialog visible?',
);
// Example loginDialogVisible value: true

Related convenience methods include aiNumber for numbers and aiString or aiAsk for strings.

#Orchestrate workflows with JavaScript

Midscene offers two basic ways to orchestrate automation: use aiAct or use JavaScript orchestration.

The following code shows both approaches completing the same task.

Using aiAct:

await agent.aiAct(
  'Check every record in the list and mark any incomplete record as completed',
);

Using JavaScript orchestration:

const recordNames = await agent.aiQuery<string[]>('All record names in the list');

for (const recordName of recordNames) {
  const completed = await agent.aiBoolean(
    `Is the record named "${recordName}" marked as completed?`,
  );

  if (!completed) {
    await agent.aiTap(`The record named "${recordName}"`);
  }
}

As the examples above show, in aiAct mode, the Agent plans the execution path. With JavaScript orchestration, developers must define conditions, loops, and step order in code.

JavaScript orchestration has an explicit execution path, allowing developers to control the behavior of each branch precisely. This makes the approach highly deterministic.

However, JavaScript orchestration can respond only to UI changes already handled in code. For example, a resolution change may require additional scrolling. If the code does not handle such changes, the workflow will fail.

Use the following guidelines when choosing an orchestration approach:

1. Use aiAct by default to execute operation goals. The Agent determines the specific steps based on the latest interface state, so it can better adapt to UI changes.
2. Use JavaScript orchestration only when the operation flow is well understood and stable.
3. If JavaScript orchestration code becomes increasingly difficult to maintain, or its success rate continues to decline, switch to aiAct.