• English

      • HarmonyOS

      Midscene connects to HarmonyOS NEXT devices through HarmonyOS Device Connector (HDC) to automate apps and system interfaces.

      This guide covers device connection, model configuration, Playground, and JavaScript SDK integration with @midscene/harmony.

      See it in action

      Prompt: Open Settings, find About phone, and view the device information.

      View the full report, or explore more Midscene showcases.

      Get started

      Prepare your HarmonyOS device

      Before writing scripts, verify that HDC can connect to your device and the device trusts the current computer.

      Install HDC

      HDC (HarmonyOS Device Connector) is a command-line tool provided by HarmonyOS for communicating with devices. Installation options:

      Verify HDC is installed:

      hdc version

      A version number in the output confirms successful installation.

      Configuring HDC Path

      If hdc is not in your system PATH, set the HDC_HOME environment variable to the directory containing HDC:

      export HDC_HOME=/path/to/hdc/directory

      Enable Developer Mode and verify the device

      In your HarmonyOS device settings, go to Developer Options and enable USB Debugging, then connect via USB cable.

      Verify the connection:

      hdc list targets

      A device ID in the output confirms a successful connection:

      0123456789ABCDEF

      Launch Playground

      Playground is the fastest way to validate the connection and try core capabilities such as aiAct, aiQuery, and aiAssert without writing code. It shares the same core as @midscene/harmony, so anything that works here will behave the same once scripted.

      1. Launch the Playground CLI:
      npx --yes @midscene/harmony-playground
      1. Click the gear button in the Playground window and paste your API Key configuration. If you don't have an API Key yet, go back to Model Configuration to get one.

      Use the JavaScript SDK

      Once Playground runs successfully, you can switch to reusable JavaScript scripts.

      Configure the model

      Set the model configuration through environment variables. See Model strategy for guidance on choosing a model.

      export MIDSCENE_MODEL_BASE_URL="https://replace-with-your-model-service-url/v1"
export MIDSCENE_MODEL_API_KEY="replace-with-your-api-key"
export MIDSCENE_MODEL_NAME="replace-with-your-model-name"
export MIDSCENE_MODEL_FAMILY="replace-with-your-model-family"

      For all configuration options, see Model configuration.

      Install dependencies

      npm
      yarn
      pnpm
      bun
      deno
      npm install @midscene/harmony dotenv --save-dev

      Write a script

      The following example opens the Settings app on the device and performs scrolling operations.

      ./demo.ts
      import 'dotenv/config'; // load Midscene environment variables from .env if present
import {
  HarmonyAgent,
  HarmonyDevice,
  getConnectedDevices,
} from '@midscene/harmony';

const sleep = (ms) => new Promise((r) => setTimeout(r, ms));
Promise.resolve(
  (async () => {
    const devices = await getConnectedDevices();
    const device = new HarmonyDevice(devices[0].deviceId, {});

    const agent = new HarmonyAgent(device, {
      aiActionContext:
        'This is a HarmonyOS device. The system language is Chinese. If any popup appears, dismiss or agree to it.',
    });
    await device.connect();

    // Open Settings app
    await agent.launch('com.huawei.hmos.settings');
    await sleep(2000);

    // Scroll down
    await agent.aiAct('scroll down one screen');

    // Query page content
    const items = await agent.aiQuery(
      'string[], list all visible setting item names',
    );
    console.log('Settings items', items);

    // Assert
    await agent.aiAssert('There is a settings item list on the page');
  })(),
);

      Run the script

      npx tsx demo.ts

      View the report

      After a successful run, the console outputs Midscene - report file updated: /path/to/report/some_id.html. Open this HTML file in a browser to replay each interaction, query, and assertion.

      Advanced

      Use this section to customize device behavior, integrate Midscene into a standalone framework, or troubleshoot HDC issues. See the HarmonyOS section of the API reference for more constructor parameters.

      Extending Midscene on HarmonyOS

      Use defineAction() to define custom gestures and pass them via customActions. Midscene appends custom actions to the planner, allowing AI to invoke your domain-specific action names.

      import { getMidsceneLocationSchema, z } from '@midscene/core';
import { defineAction } from '@midscene/core/device';
import { HarmonyAgent, HarmonyDevice, getConnectedDevices } from '@midscene/harmony';

const ContinuousClick = defineAction({
  name: 'continuousClick',
  description: 'Click the same target repeatedly',
  paramSchema: z.object({
    locate: getMidsceneLocationSchema(),
    count: z.number().int().positive().describe('How many times to click'),
  }),
  async call(param) {
    const { locate, count } = param;
    console.log('click target center', locate.center);
    console.log('click count', count);
  },
});

const devices = await getConnectedDevices();
const device = new HarmonyDevice(devices[0].deviceId, {});
await device.connect();

const agent = new HarmonyAgent(device, {
  customActions: [ContinuousClick],
});

await agent.aiAct('click the red button five times');

      For more details on custom actions and action schemas, see Integrate with Any Interface.

      More

      Complete example (Vitest + HarmonyAgent)

      import type { TestStatus } from '@midscene/core';
import { ReportMergingTool } from '@midscene/core/report';
import { sleep } from '@midscene/core/utils';
import {
  HarmonyAgent,
  HarmonyDevice,
  getConnectedDevices,
} from '@midscene/harmony';
import {
  afterAll,
  afterEach,
  beforeAll,
  beforeEach,
  describe,
  it,
} from 'vitest';

describe('HarmonyOS Settings Test', () => {
  let device: HarmonyDevice;
  let agent: HarmonyAgent;
  let itTestStatus: TestStatus = 'passed';
  const reportMergingTool = new ReportMergingTool();

  beforeAll(async () => {
    const devices = await getConnectedDevices();
    device = new HarmonyDevice(devices[0].deviceId);
    await device.connect();
  });

  beforeEach((ctx) => {
    agent = new HarmonyAgent(device, {
      groupName: ctx.task.name,
    });
  });

  afterEach((ctx) => {
    if (ctx.task.result?.state === 'pass') {
      itTestStatus = 'passed';
    } else if (ctx.task.result?.state === 'skip') {
      itTestStatus = 'skipped';
    } else if (ctx.task.result?.errors?.[0].message.includes('timed out')) {
      itTestStatus = 'timedOut';
    } else {
      itTestStatus = 'failed';
    }
    reportMergingTool.append({
      reportFilePath: agent.reportFile as string,
      reportAttributes: {
        testId: `${ctx.task.name}`,
        testTitle: `${ctx.task.name}`,
        testDescription: 'description',
        testDuration: (Date.now() - ctx.task.result?.startTime!) | 0,
        testStatus: itTestStatus,
      },
    });
  });

  afterAll(() => {
    reportMergingTool.mergeReports('my-harmony-setting-test-report');
  });

  it('toggle WLAN', async () => {
    await device.home();
    await sleep(1000);
    await device.launch('com.huawei.hmos.settings');
    await sleep(1000);
    await agent.aiAct('find and enter WLAN setting');
    await agent.aiAct(
      'toggle WLAN status *once*, if WLAN is off please turn it on, otherwise turn it off.',
    );
  });

  it('toggle Bluetooth', async () => {
    await device.home();
    await sleep(1000);
    await device.launch('com.huawei.hmos.settings');
    await sleep(1000);
    await agent.aiAct('find and enter Bluetooth setting');
    await agent.aiAct(
      'toggle Bluetooth status *once*, if Bluetooth is off please turn it on, otherwise turn it off.',
    );
  });
});
      Tip

      Merged reports are stored inside midscene_run/report by default. Override the directory with MIDSCENE_RUN_DIR when running in CI.

      FAQ

      Keyboard is not dismissed or the page goes back after typing

      Midscene automatically dismisses the keyboard after entering text. By default, HarmonyOS uses the ESC key so the current page is less likely to navigate back. If ESC does not close the keyboard in your app, switch to Back first:

      const device = new HarmonyDevice('device-id', {
  keyboardDismissStrategy: 'back-first',
});

      If your input field listens for Back and clears or closes in response, disable auto keyboard dismiss:

      const device = new HarmonyDevice('device-id', {
  autoDismissKeyboard: false,
});

      With auto dismiss disabled, the keyboard will remain visible. You can use aiAct to manually dismiss it, e.g. await agent.aiAct('dismiss the keyboard').

      How to use a custom HDC path?

      Set the HDC_HOME environment variable to point to the HDC directory:

      export HDC_HOME=/path/to/hdc/directory

      Or pass it via the constructor:

      const device = new HarmonyDevice('0123456789ABCDEF', {
  hdcPath: '/path/to/hdc',
});