• English

      • Android

      Midscene connects to Android devices through adb to automate apps and system interfaces.

      This guide covers device connection, model configuration, Playground, and JavaScript SDK integration with @midscene/android.

      See it in action

      Prompt: Open the Booking app. Search for a hotel in Tokyo for four adults on Christmas, with a score of 8 or above.

      View the full report, or explore more Midscene showcases.

      Get started

      Prepare your Android device

      Before scripting, confirm adb can talk to your device and the device trusts your machine.

      Install adb and set ANDROID_HOME

      adb --version

      Example output indicates success:

      Android Debug Bridge version 1.0.41
Version 34.0.4-10411341
Installed as /usr/local/bin//adb
Running on Darwin 24.3.0 (arm64)
      echo $ANDROID_HOME

      Any non-empty output means it is configured:

      /Users/your_username/Library/Android/sdk

      Enable USB debugging and verify the device

      In the system settings developer options, enable USB debugging (and USB debugging (Security settings) if present), then connect the device via USB.

      android usb debug

      Verify the connection:

      adb devices -l

      Example success output:

      List of devices attached
s4ey59	device usb:34603008X product:cezanne model:M2006J device:cezan transport_id:3

      Launch Playground

      Playground is the fastest way to validate the connection and try core capabilities such as aiAct, aiQuery, and aiAssert without writing code. It shares the same core as @midscene/android, so anything that works here will behave the same once scripted.

      1. Launch the Playground CLI:
      npx --yes @midscene/android-playground
      1. Click the gear icon in the Playground window, then paste your API key configuration. Refer back to Model configuration if you still need credentials.

      Use the JavaScript SDK

      Once Playground works, move to a repeatable script with the JavaScript SDK.

      Configure the model

      Set the model configuration through environment variables. See Model strategy for guidance on choosing a model.

      export MIDSCENE_MODEL_BASE_URL="https://replace-with-your-model-service-url/v1"
export MIDSCENE_MODEL_API_KEY="replace-with-your-api-key"
export MIDSCENE_MODEL_NAME="replace-with-your-model-name"
export MIDSCENE_MODEL_FAMILY="replace-with-your-model-family"

      For all configuration options, see Model configuration.

      Install dependencies

      npm
      yarn
      pnpm
      bun
      deno
      npm install @midscene/android dotenv --save-dev

      Write a script

      Save the following code as ./demo.ts. It opens the browser on the device, searches eBay, and asserts the result list.

      ./demo.ts
      import 'dotenv/config'; // load Midscene environment variables from .env if present
import {
  AndroidAgent,
  AndroidDevice,
  getConnectedDevices,
} from '@midscene/android';

const sleep = (ms) => new Promise((r) => setTimeout(r, ms));
Promise.resolve(
  (async () => {
    const devices = await getConnectedDevices();
    const device = new AndroidDevice(devices[0].udid);

    const agent = new AndroidAgent(device, {
      aiActionContext:
        'If any location, permission, user agreement, etc. popup, click agree. If login page pops up, close it.',
    });
    await device.connect();

    await agent.aiAct('open browser and navigate to ebay.com');
    await sleep(5000);
    await agent.aiAct('type "Headphones" in search box, hit Enter');
    await agent.aiWaitFor('There is at least one headphone product');

    const items = await agent.aiQuery(
      '{itemTitle: string, price: Number}[], find item in list and corresponding price',
    );
    console.log('headphones in stock', items);

    await agent.aiAssert('There is a category filter on the left');
  })(),
);

      Run the script

      npx tsx demo.ts

      View the report

      Successful runs print Midscene - report file updated: /path/to/report/some_id.html. Open the generated HTML file in a browser to replay every interaction, query, and assertion.

      Advanced

      Use this section to customize device behavior, integrate Midscene into your framework, or troubleshoot adb issues. For detailed constructor parameters, see the Android section of the API reference.

      Extend Midscene on Android

      Use defineAction() for custom gestures and pass them through customActions. Midscene will append them to the planner so AI can call your domain-specific action names.

      import { getMidsceneLocationSchema, z } from '@midscene/core';
import { defineAction } from '@midscene/core/device';
import { AndroidAgent, AndroidDevice, getConnectedDevices } from '@midscene/android';

const ContinuousClick = defineAction({
  name: 'continuousClick',
  description: 'Click the same target repeatedly',
  paramSchema: z.object({
    locate: getMidsceneLocationSchema(),
    count: z.number().int().positive().describe('How many times to click'),
  }),
  async call(param) {
    const { locate, count } = param;
    console.log('click target center', locate.center);
    console.log('click count', count);
  },
});

const devices = await getConnectedDevices();
const device = new AndroidDevice(devices[0].udid);
await device.connect();

const agent = new AndroidAgent(device, {
  customActions: [ContinuousClick],
});

await agent.aiAct('click the red button five times');

      See Integrate with any interface for a deeper explanation of custom actions and action schemas.

      More

      Complete example (Vitest + AndroidAgent)

      import {
  AndroidAgent,
  AndroidDevice,
  getConnectedDevices,
} from '@midscene/android';
import type { TestStatus } from '@midscene/core';
import { ReportMergingTool } from '@midscene/core/report';
import { sleep } from '@midscene/core/utils';
import type ADB from 'appium-adb';
import {
  afterAll,
  afterEach,
  beforeAll,
  beforeEach,
  describe,
  it,
} from 'vitest';

describe('Android Settings Test', () => {
  let page: AndroidDevice;
  let adb: ADB;
  let agent: AndroidAgent;
  let startTime: number;
  let itTestStatus: TestStatus = 'passed';
  const reportMergingTool = new ReportMergingTool();

  beforeAll(async () => {
    const devices = await getConnectedDevices();
    page = new AndroidDevice(devices[0].udid);
    adb = await page.getAdb();
  });

  beforeEach((ctx) => {
    startTime = performance.now();
    agent = new AndroidAgent(page, {
      groupName: ctx.task.name,
    });
  });

  afterEach((ctx) => {
    if (ctx.task.result?.state === 'pass') {
      itTestStatus = 'passed';
    } else if (ctx.task.result?.state === 'skip') {
      itTestStatus = 'skipped';
    } else if (ctx.task.result?.errors?.[0].message.includes('timed out')) {
      itTestStatus = 'timedOut';
    } else {
      itTestStatus = 'failed';
    }
    reportMergingTool.append({
      reportFilePath: agent.reportFile as string,
      reportAttributes: {
        testId: `${ctx.task.name}`,
        testTitle: `${ctx.task.name}`,
        testDescription: 'description',
        testDuration: (Date.now() - ctx.task.result?.startTime!) | 0,
        testStatus: itTestStatus,
      },
    });
  });

  afterAll(() => {
    reportMergingTool.mergeReports('my-android-setting-test-report');
  });

  it('toggle wlan', async () => {
    await adb.shell('input keyevent KEYCODE_HOME');
    await sleep(1000);
    await adb.shell('am start -n com.android.settings/.Settings');
    await sleep(1000);
    await agent.aiAct('find and enter WLAN setting');
    await agent.aiAct(
      'toggle WLAN status *once*, if WLAN is off pls turn it on, otherwise turn it off.',
    );
  });

  it('toggle bluetooth', async () => {
    await adb.shell('input keyevent KEYCODE_HOME');
    await sleep(1000);
    await adb.shell('am start -n com.android.settings/.Settings');
    await sleep(1000);
    await agent.aiAct('find and enter bluetooth setting');
    await agent.aiAct(
      'toggle bluetooth status *once*, if bluetooth is off pls turn it on, otherwise turn it off.',
    );
  });
});
      Tip

      Merged reports are stored inside midscene_run/report by default. Override the directory with MIDSCENE_RUN_DIR when running in CI.

      FAQ

      Why can't I control the device even though I've connected it?

      A common error is:

      Error:
Exception occurred while executing 'tap':
java.lang.SecurityException: Injecting input events requires the caller (or the source of the instrumentation, if any) to have the INJECT_EVENTS permission.

      Make sure USB debugging is enabled and the device is unlocked in developer options.

      android usb debug

      Text input is cleared or lost after typing

      After entering text, Midscene automatically dismisses the keyboard. The default behavior sends an ESC key event. However, some input fields (especially those inside WebView) listen for the ESC key event, which can cause side effects such as:

      • Clearing the text just entered
      • Closing the popup/modal containing the input field
      • Navigating away from the current page

      You can try the following solutions in order of priority:

      Option 1: Use the BACK key (Android back button) to dismiss the keyboard

      Set keyboardDismissStrategy to 'back-first' to use the Android BACK key instead of ESC to dismiss the keyboard:

      const device = new AndroidDevice('device-id', {
  keyboardDismissStrategy: 'back-first',
});

      Option 2: Disable auto keyboard dismiss

      If your input field also listens for the BACK key, you can disable auto keyboard dismiss entirely and let the AI Agent or subsequent actions manage the keyboard state:

      const device = new AndroidDevice('device-id', {
  autoDismissKeyboard: false,
});

      With auto dismiss disabled, the keyboard will remain visible and may cover a large portion of the screen. You can work around this by:

      • Using aiAct to manually dismiss the keyboard, e.g. await agent.aiAct('tap the collapse button on the keyboard')
      • Installing and switching to ADBKeyBoard — a minimal virtual keyboard that takes up very little screen space, so it barely affects screen interactions even when visible

      English text is rewritten by the Android keyboard

      If the report shows the correct input parameter, but the app receives different text, missing text, or Chinese/pinyin candidates, the active Android input method may be rewriting the text. This can happen when pure ASCII text goes through the native adb input text path while a Chinese keyboard or autocorrect keyboard is active.

      Use the existing imeStrategy option and force all text input through yadb:

      const device = new AndroidDevice('device-id', {
  imeStrategy: 'always-yadb',
});

      For YAML scripts:

      android:
  imeStrategy: always-yadb

      Or set the environment variable:

      export MIDSCENE_ANDROID_IME_STRATEGY=always-yadb

      This is different from text being cleared after typing. If the text is entered correctly and then disappears, check keyboardDismissStrategy or autoDismissKeyboard instead.

      How do I use a custom adb path or remote adb server?

      Set the environment variables first:

      export MIDSCENE_ADB_PATH=/path/to/adb
export MIDSCENE_ADB_REMOTE_HOST=192.168.1.100
export MIDSCENE_ADB_REMOTE_PORT=5037

      You can also provide the same information via the constructor:

      const device = new AndroidDevice('s4ey59', {
  androidAdbPath: '/path/to/adb',
  remoteAdbHost: '192.168.1.100',
  remoteAdbPort: 5037,
});