iOS

Midscene connects to iOS devices through WebDriverAgent to automate apps and system interfaces.

This guide covers WebDriverAgent setup, model configuration, Playground, and JavaScript SDK integration with @midscene/ios.

See it in action

Prompt: Open Twitter and like the first post from @midscene_ai.

View the full report, or explore more Midscene showcases.

Get started

Prepare the iOS environment

WebDriver is a W3C standard protocol for browser automation. It provides a common API for controlling browsers and applications. The protocol defines how clients and servers communicate, which allows automation tools to control interfaces across platforms.

The Appium team and other open source communities maintain tools that expose desktop and mobile automation through WebDriver. These tools include:

Appium: a cross-platform mobile automation framework.
WebDriverAgent: a service for iOS device automation.
Selenium: a Web browser automation tool.
WinAppDriver: a Windows application automation tool.

Midscene supports the WebDriver protocol. You can use AI models to automate any compatible device. Midscene can understand interface context, perform multistep operations, validate results, and extract data in addition to clicking and typing.

On iOS, Midscene connects through WebDriverAgent. You can then use natural-language instructions to control iOS apps and system interfaces.

Before continuing, make sure WebDriverAgent can communicate with the device.

Install Node.js

Install Node.js 18 or higher.

Set up WebDriverAgent

Before getting started, you need to set up the iOS development environment:

macOS (required for iOS development)
Xcode and Xcode command line tools
iOS Simulator or real device

Configure WebDriverAgent

Before using Midscene iOS, you need to prepare the WebDriverAgent service.

Version Requirement

WebDriverAgent version must be >= 7.0.0

Please refer to the official documentation for setup:

Simulator Configuration: Run Prebuilt WDA
Real Device Configuration: Real Device Configuration

Verify WebDriverAgent

After completing the configuration, you can verify whether the service is working properly by accessing WebDriverAgent's status endpoint:

Access URL: http://localhost:8100/status

Correct Response Example:

{
  "value": {
    "build": {
      "version": "10.1.1",
      "time": "Sep 24 2025 18:56:41",
      "productBundleIdentifier": "com.facebook.WebDriverAgentRunner"
    },
    "os": {
      "testmanagerdVersion": 65535,
      "name": "iOS",
      "sdkVersion": "26.0",
      "version": "26.0"
    },
    "device": "iphone",
    "ios": {
      "ip": "10.91.115.63"
    },
    "message": "WebDriverAgent is ready to accept commands",
    "state": "success",
    "ready": true
  },
  "sessionId": "BCAD9603-F714-447C-A9E6-07D58267966B"
}

If you can successfully access this endpoint and receive a similar JSON response as shown above, it indicates that WebDriverAgent is properly configured and running.

Launch Playground

Playground is the fastest way to validate the connection and try core capabilities such as aiAct, aiQuery, and aiAssert without writing code. It shares the same core as @midscene/ios, so anything that works here will behave the same once scripted.

Launch the Playground CLI:

npx --yes @midscene/ios-playground

Click the gear button to enter the configuration page and paste your API key config. Refer back to Model configuration if you still need credentials.

Use the JavaScript SDK

Once Playground works, move to a repeatable script with the JavaScript SDK.

Configure the model

Set the model configuration through environment variables. See Model strategy for guidance on choosing a model.

export MIDSCENE_MODEL_BASE_URL="https://replace-with-your-model-service-url/v1"
export MIDSCENE_MODEL_API_KEY="replace-with-your-api-key"
export MIDSCENE_MODEL_NAME="replace-with-your-model-name"
export MIDSCENE_MODEL_FAMILY="replace-with-your-model-family"

For all configuration options, see Model configuration.

Install dependencies

npm

yarn

pnpm

bun

deno

npm install @midscene/ios dotenv --save-dev

Write a script

Save the following code as ./demo.ts. It opens Safari on the device, searches eBay, and asserts the result list.

./demo.ts

import 'dotenv/config'; // load Midscene environment variables from .env if present
import {
  IOSAgent,
  IOSDevice,
  agentFromWebDriverAgent,
} from '@midscene/ios';

const sleep = (ms) => new Promise((r) => setTimeout(r, ms));
Promise.resolve(
  (async () => {
    // Method 1: Create device and agent directly
    const page = new IOSDevice({
      wdaPort: 8100,
      wdaHost: 'localhost',
    });

    // 👀 Initialize Midscene agent
    const agent = new IOSAgent(page, {
      aiActionContext:
        'If any location, permission, user agreement, etc. popup appears, click agree. If login page appears, close it.',
    });
    await page.connect();

    // Method 2: Or use convenience function (recommended)
    // const agent = await agentFromWebDriverAgent({
    //   wdaPort: 8100,
    //   wdaHost: 'localhost',
    //   aiActionContext: 'If any location, permission, user agreement, etc. popup appears, click agree. If login page appears, close it.',
    // });

    // 👀 Directly open ebay.com webpage (recommended approach)
    await page.launch('https://ebay.com');
    await sleep(3000);

    // 👀 Enter keywords and perform search
    await agent.aiAct('Search for "Headphones"');

    // 👀 Wait for loading to complete
    await agent.aiWaitFor('At least one headphone product is displayed on the page');
    // Or you can use a simple sleep:
    // await sleep(5000);

    // 👀 Understand page content and extract data
    const items = await agent.aiQuery(
      '{itemTitle: string, price: Number}[], find product titles and prices in the list',
    );
    console.log('Headphone product information', items);

    // 👀 Use AI assertion
    await agent.aiAssert('Multiple headphone products are displayed on the interface');

    await page.destroy();
  })(),
);

Run the script

npx tsx demo.ts

View the report

Successful runs print Midscene - report file updated: /path/to/report/some_id.html. Open the generated HTML file in a browser to replay every interaction, query, and assertion.

API reference and more resources

For constructors, helper methods, and platform-specific device APIs, see the iOS section of the API reference. It includes detailed parameter lists and advanced topics such as custom actions. For APIs shared across platforms, see the Common section.

FAQ

Why can't I control my device through WebDriverAgent even though it's connected?

Please check the following:

Developer Mode: Ensure it's enabled in Settings > Privacy & Security > Developer Mode
UI Automation: Ensure it's enabled in Settings > Developer > UI Automation
Device Trust: Ensure the device trusts the current Mac

What are the differences between simulators and real devices?

Feature	Real Device	Simulator
Port Forwarding	Requires iproxy	Not required
Developer Mode	Must enable	Auto-enabled
UI Automation Settings	Must enable manually	Auto-enabled
Performance	Real device performance	Depends on Mac performance
Sensors	Real hardware	Simulated data

How to use custom WebDriverAgent port and host?

You can specify WebDriverAgent port and host through the IOSDevice constructor or agentFromWebDriverAgent:

// Method 1: Using IOSDevice
const device = new IOSDevice({
  wdaPort: 8100,        // Custom port
  wdaHost: '192.168.1.100', // Custom host
});

// Method 2: Using convenience function (recommended)
const agent = await agentFromWebDriverAgent({
  wdaPort: 8100,        // Custom port
  wdaHost: '192.168.1.100', // Custom host
});

For remote devices, you also need to set up port forwarding accordingly:

iproxy 8100 8100 YOUR_DEVICE_ID

How to get smoother live screen preview in Playground?

Playground's screen preview supports two modes:

Polling mode (default): Captures screenshots one by one via the WDA screenshot API, achieving ~5-10fps.
Native MJPEG stream (recommended): Proxies WDA's built-in MJPEG Server directly for higher frame rate and lower latency.

To enable the native MJPEG stream, forward the WDA MJPEG Server port (default 9100) to localhost:

# Required for real devices only (simulators don't need this)
iproxy 9100 9100 YOUR_DEVICE_ID

Playground automatically probes port 9100 on startup. If available, the log will show MJPEG: streaming via native WDA MJPEG server; otherwise it falls back to polling mode automatically.

For every Agent method, check the API reference (Common).
For iOS-specific APIs, see API reference (iOS).
Use YAML automation scripts and command-line tools.
Demo projects
- iOS JavaScript SDK demo: https://github.com/web-infra-dev/midscene-example/blob/main/ios/javascript-sdk-demo
- iOS + Vitest demo: https://github.com/web-infra-dev/midscene-example/tree/main/ios/vitest-demo

#iOS

#See it in action

#Get started

#Prepare the iOS environment

#Install Node.js

#Set up WebDriverAgent

#Launch Playground

#Use the JavaScript SDK

#Configure the model

#Install dependencies

#Write a script

#Run the script

#View the report

#API reference and more resources

#FAQ

#Why can't I control my device through WebDriverAgent even though it's connected?

#What are the differences between simulators and real devices?

#How to use custom WebDriverAgent port and host?

#How to get smoother live screen preview in Playground?

#More

iOS