API Reference#

In the documentation below, you might see function calls prefixed with agent.. If you utilize destructuring in Playwright (e.g., async ({ ai, aiQuery }) => { /* ... */ }), you can call these functions without the agent. prefix. This is merely a syntactical difference.

Constructors#

Each Agent in Midscene has its own constructor.

• In Puppeteer, use PuppeteerAgent
• In Bridge Mode, use AgentOverChromeBridge

These Agents share some common constructor parameters:

• generateReport: boolean: If true, a report file will be generated. (Default: true)
• autoPrintReportMsg: boolean: If true, report messages will be printed. (Default: true)
• cacheId: string | undefined: If provided, this cacheId will be used to save or match the cache. (Default: undefined, means cache feature is disabled)

In Puppeteer, there is an additional parameter:

• forceSameTabNavigation: boolean: If true, page navigation is restricted to the current tab. (Default: true)

Interaction Methods#

Below are the main APIs available for the various Agents in Midscene.

agent.aiAction() or .ai()#

This method allows you to perform a series of UI actions described in natural language. Midscene automatically plans the steps and executes them.

• Type

function aiAction(prompt: string): Promise<void>;
function ai(prompt: string): Promise<void>; // shorthand form

• Parameters:

  • prompt: string - A natural language description of the UI steps.

• Return Value:

  • Returns a Promise that resolves to void when all steps are completed; if execution fails, an error is thrown.

• Examples:

// Basic usage
await agent.aiAction('Type "JavaScript" into the search box, then click the search button');

// Using the shorthand .ai form
await agent.ai('Click the login button at the top of the page, then enter "test@example.com" in the username field');

// When using UI Agent models like ui-tars, you can try a more high-level prompt
await agent.aiAction('Post a Tweet "Hello World"');

agent.aiTap()#

Tap something.

• Type

function aiTap(locate: string): Promise<void>;

• Parameters:

  • locate: string - A natural language description of the element to tap.

• Return Value:

  • Returns a Promise<void>

• Examples:

await agent.aiTap('The login button at the top of the page');

agent.aiHover()#

Move mouse over something.

• Type

function aiHover(locate: string): Promise<void>;

• Parameters:

  • locate: string - A natural language description of the element to hover over.

• Return Value:

  • Returns a Promise<void>

• Examples:

await agent.aiHover('The version number of the current page');

agent.aiInput()#

Input text into something.

• Type

function aiInput(text: string, locate: string): Promise<void>;

• Parameters:

  • text: string - The final text content that should be placed in the input element. Use blank string to clear the input.
  • locate: string - A natural language description of the element to input text into.

• Return Value:

  • Returns a Promise<void>

• Examples:

await agent.aiInput('Hello World', 'The search input box');

agent.aiKeyboardPress()#

Press a keyboard key.

• Type

function aiKeyboardPress(key: string, locate?: string): Promise<void>;

• Parameters:

  • key: string - The web key to press, e.g. 'Enter', 'Tab', 'Escape', etc. Key Combination is not supported.
  • locate?: string - Optional, a natural language description of the element to press the key on.

• Return Value:

  • Returns a Promise<void>

• Examples:

await agent.aiKeyboardPress('Enter', 'The search input box');

agent.aiScroll()#

Scroll a page or an element.

• Type

function aiScroll(scrollParam: PlanningActionParamScroll, locate?: string): Promise<void>;

• Parameters:

  • scrollParam: PlanningActionParamScroll - The scroll parameter
    • direction: 'up' | 'down' | 'left' | 'right' - The direction to scroll.
    • scrollType: 'once' | 'untilBottom' | 'untilTop' | 'untilRight' | 'untilLeft' - Optional, the type of scroll to perform.
    • distance: number - Optional, the distance to scroll in px.
  • locate?: string - Optional, a natural language description of the element to scroll on. If not provided, Midscene will perform scroll on the current mouse position.

• Return Value:

  • Returns a Promise<void>

• Examples:

await agent.aiScroll({ direction: 'up', distance: 100, scrollType: 'once' }, 'The form panel');

Data Extraction#

agent.aiQuery()#

This method allows you to extract data directly from the UI using multimodal AI reasoning capabilities. Simply define the expected format (e.g., string, number, JSON, or an array) in the dataDemand, and Midscene will return a result that matches the format.

• Type

function aiQuery<T>(dataShape: string | Object): Promise<T>;

• Parameters:

  • dataShape: T: A description of the expected return format.

• Return Value:

  • Returns any valid basic type, such as string, number, JSON, array, etc.
  • Just describe the format in dataDemand, and Midscene will return a matching result.

• Examples:

const dataA = await agent.aiQuery({
  time: 'The date and time displayed in the top-left corner as a string',
  userInfo: 'User information in the format {name: string}',
  tableFields: 'An array of table field names, string[]',
  tableDataRecord: 'Table records in the format {id: string, [fieldName]: string}[]',
});

// You can also describe the expected return format using a string:

// dataB will be an array of strings
const dataB = await agent.aiQuery('string[], list of task names');

// dataC will be an array of objects
const dataC = await agent.aiQuery('{name: string, age: string}[], table data records');

More APIs#

agent.aiAssert()#

This method lets you specify an assertion in natural language, and the AI determines whether the condition is true. If the assertion fails, the SDK throws an error that includes both the optional errorMsg and a detailed reason generated by the AI.

• Type

function aiAssert(assertion: string, errorMsg?: string): Promise<void>;

• Parameters:

  • assertion: string - The assertion described in natural language.
  • errorMsg?: string - An optional error message to append if the assertion fails.

• Return Value:

  • Returns a Promise that resolves to void if the assertion passes; if it fails, an error is thrown with errorMsg and additional AI-provided information.

• Example:

await agent.aiAssert('The price of "Sauce Labs Onesie" is 7.99');

const items = await agent.aiQuery(
  '"{name: string, price: number}[], return product names and prices'
);
const onesieItem = items.find(item => item.name === 'Sauce Labs Onesie');
expect(onesieItem).toBeTruthy();
expect(onesieItem.price).toBe(7.99);

agent.aiWaitFor()#

This method allows you to wait until a specified condition, described in natural language, becomes true. Considering the cost of AI calls, the check interval will not exceed the specified checkIntervalMs.

• Type

function aiWaitFor(
  assertion: string, 
  options?: { 
    timeoutMs?: number;
    checkIntervalMs?: number;
  }
): Promise<void>;

• Parameters:

  • assertion: string - The condition described in natural language.
  • options?: object - An optional configuration object containing:
    • timeoutMs?: number - Timeout in milliseconds (default: 15000).
    • checkIntervalMs?: number - Interval for checking in milliseconds (default: 3000).

• Return Value:

  • Returns a Promise that resolves to void if the condition is met; if not, an error is thrown when the timeout is reached.

• Examples:

// Basic usage
await agent.aiWaitFor("There is at least one headphone information displayed on the interface");

// Using custom options
await agent.aiWaitFor("The shopping cart icon shows a quantity of 2", {
  timeoutMs: 30000,    // Wait for 30 seconds
  checkIntervalMs: 5000  // Check every 5 seconds
});

agent.runYaml()#

This method executes an automation script written in YAML. Only the tasks part of the script is executed, and it returns the results of all .aiQuery calls within the script.

• Type

function runYaml(yamlScriptContent: string): Promise<{ result: any }>;

• Parameters:

  • yamlScriptContent: string - The YAML-formatted script content.

• Return Value:

  • Returns an object with a result property that includes the results of all .aiQuery calls.

• Example:

const { result } = await agent.runYaml(`
tasks:
  - name: search weather
    flow:
      - ai: input 'weather today' in input box, click search button
      - sleep: 3000

  - name: query weather
    flow:
      - aiQuery: "the result shows the weather info, {description: string}"
`);
console.log(result);

Properties#

.reportFile#

The path to the report file.

Additional Configurations#

Setting Environment Variables at Runtime#

You can override environment variables at runtime by calling the overrideAIConfig method.

import { overrideAIConfig } from '@midscene/web/puppeteer'; // or another Agent

overrideAIConfig({
  OPENAI_BASE_URL: "...",
  OPENAI_API_KEY: "...",
  MIDSCENE_MODEL_NAME: "..."
});

Print usage information for each AI call#

Set the MIDSCENE_DEBUG_AI_PROFILE variable to view the execution time and usage for each AI call.

export MIDSCENE_DEBUG_AI_PROFILE=1

Using LangSmith#

LangSmith is a platform for debugging large language models. To integrate LangSmith, follow these steps:

# Set environment variables

# Enable debug mode
export MIDSCENE_LANGSMITH_DEBUG=1 

# LangSmith configuration
export LANGSMITH_TRACING_V2=true
export LANGSMITH_ENDPOINT="https://api.smith.langchain.com"
export LANGSMITH_API_KEY="your_key_here"
export LANGSMITH_PROJECT="your_project_name_here"

After starting Midscene, you should see logs similar to:

DEBUGGING MODE: langsmith wrapper enabled