HarmonyOS Getting Started

This guide walks you through everything needed to automate HarmonyOS devices with Midscene: connecting a real device via HDC, configuring model API keys, trying the zero-code Playground, and running your first JavaScript script.

Demo Projects

Control HarmonyOS devices with JavaScript: https://github.com/web-infra-dev/midscene-example/blob/main/harmony/javascript-sdk-demo

Integrate Vitest for testing: https://github.com/web-infra-dev/midscene-example/tree/main/harmony/vitest-demo

Set up API keys for model

Set your model configs into the environment variables. You may refer to Model strategy for more details.

export MIDSCENE_MODEL_BASE_URL="https://replace-with-your-model-service-url/v1"
export MIDSCENE_MODEL_API_KEY="replace-with-your-api-key"
export MIDSCENE_MODEL_NAME="replace-with-your-model-name"
export MIDSCENE_MODEL_FAMILY="replace-with-your-model-family"

For more configuration details, please refer to Model strategy and Model configuration.

Prerequisites

Before writing scripts, verify that HDC can connect to your device and the device trusts the current computer.

Install HDC

HDC (HarmonyOS Device Connector) is a command-line tool provided by HarmonyOS for communicating with devices. Installation options:

Via DevEco Studio (recommended)
Via HarmonyOS command-line tools standalone installation

Verify HDC is installed:

hdc version

A version number in the output confirms successful installation.

Configuring HDC Path

If hdc is not in your system PATH, set the HDC_HOME environment variable to the directory containing HDC:

export HDC_HOME=/path/to/hdc/directory

Enable Developer Mode and Verify Device

In your HarmonyOS device settings, go to Developer Options and enable USB Debugging, then connect via USB cable.

Verify the connection:

hdc list targets

A device ID in the output confirms a successful connection:

0123456789ABCDEF

Try Playground (Zero Code)

Playground is the fastest way to verify your connection and observe the AI Agent, without writing any code. It shares the same code implementation as @midscene/harmony, so flows validated in Playground will work identically when run via scripts.

Launch the Playground CLI:

npx --yes @midscene/harmony-playground

Click the gear button in the Playground window and paste your API Key configuration. If you don't have an API Key yet, go back to Model Configuration to get one.

Start Your Experience

After configuration, you can start using Midscene right away. It provides several key operation tabs:

Act: interact with the page. This is Auto Planning, corresponding to aiAct. For example:

Type “Midscene” in the search box, run the search, and open the first result

Fill out the registration form and make sure every field passes validation

Query: extract JSON data from the interface, corresponding to aiQuery.

Similar methods include aiBoolean(), aiNumber(), and aiString() for directly extracting booleans, numbers, and strings.

Extract the user ID from the page and return JSON data in the { id: string } structure

Assert: understand the page and assert; if the condition is not met, throw an error, corresponding to aiAssert.

There is a login button on the page, with a user agreement link below it

Tap: click on an element. This is Instant Action, corresponding to aiTap.

Click the login button

For the difference between Auto Planning and Instant Action, see the API document.

Integrate Midscene Agent

Once Playground runs successfully, you can switch to reusable JavaScript scripts.

Step 1: Install Dependencies

npm

yarn

pnpm

bun

deno

npm install @midscene/harmony --save-dev

Step 2: Write a Script

The following example opens the Settings app on the device and performs scrolling operations.

./demo.ts

import {
  HarmonyAgent,
  HarmonyDevice,
  getConnectedDevices,
} from '@midscene/harmony';

const sleep = (ms) => new Promise((r) => setTimeout(r, ms));
Promise.resolve(
  (async () => {
    const devices = await getConnectedDevices();
    const device = new HarmonyDevice(devices[0].deviceId, {});

    const agent = new HarmonyAgent(device, {
      aiActionContext:
        'This is a HarmonyOS device. The system language is Chinese. If any popup appears, dismiss or agree to it.',
    });
    await device.connect();

    // Open Settings app
    await agent.launch('com.huawei.hmos.settings');
    await sleep(2000);

    // Scroll down
    await agent.aiAct('scroll down one screen');

    // Query page content
    const items = await agent.aiQuery(
      'string[], list all visible setting item names',
    );
    console.log('Settings items', items);

    // Assert
    await agent.aiAssert('There is a settings item list on the page');
  })(),
);

Step 3: Run the Example

npx tsx demo.ts

Step 4: View the Report

After a successful run, the console outputs Midscene - report file updated: /path/to/report/some_id.html. Open this HTML file in a browser to replay each interaction, query, and assertion.

Advanced

When you need to customize device behavior, integrate Midscene into a standalone framework, or troubleshoot HDC issues, refer to this section. See API Reference (HarmonyOS) for more constructor parameters.

Extending Midscene on HarmonyOS

Use defineAction() to define custom gestures and pass them via customActions. Midscene appends custom actions to the planner, allowing AI to invoke your domain-specific action names.

import { getMidsceneLocationSchema, z } from '@midscene/core';
import { defineAction } from '@midscene/core/device';
import { HarmonyAgent, HarmonyDevice, getConnectedDevices } from '@midscene/harmony';

const ContinuousClick = defineAction({
  name: 'continuousClick',
  description: 'Click the same target repeatedly',
  paramSchema: z.object({
    locate: getMidsceneLocationSchema(),
    count: z.number().int().positive().describe('How many times to click'),
  }),
  async call(param) {
    const { locate, count } = param;
    console.log('click target center', locate.center);
    console.log('click count', count);
  },
});

const devices = await getConnectedDevices();
const device = new HarmonyDevice(devices[0].deviceId, {});
await device.connect();

const agent = new HarmonyAgent(device, {
  customActions: [ContinuousClick],
});

await agent.aiAct('click the red button five times');

For more details on custom actions and action schemas, see Integrate with Any Interface.

View all Agent methods: API Reference (Common)
HarmonyOS-specific parameters and interfaces: HarmonyOS Agent API
Demo projects
- HarmonyOS JavaScript SDK demo: https://github.com/web-infra-dev/midscene-example/blob/main/harmony/javascript-sdk-demo
- HarmonyOS + Vitest demo: https://github.com/web-infra-dev/midscene-example/tree/main/harmony/vitest-demo

Complete example (Vitest + HarmonyAgent)

import type { TestStatus } from '@midscene/core';
import { ReportMergingTool } from '@midscene/core/report';
import { sleep } from '@midscene/core/utils';
import {
  HarmonyAgent,
  HarmonyDevice,
  getConnectedDevices,
} from '@midscene/harmony';
import {
  afterAll,
  afterEach,
  beforeAll,
  beforeEach,
  describe,
  it,
} from 'vitest';

describe('HarmonyOS Settings Test', () => {
  let device: HarmonyDevice;
  let agent: HarmonyAgent;
  let itTestStatus: TestStatus = 'passed';
  const reportMergingTool = new ReportMergingTool();

  beforeAll(async () => {
    const devices = await getConnectedDevices();
    device = new HarmonyDevice(devices[0].deviceId);
    await device.connect();
  });

  beforeEach((ctx) => {
    agent = new HarmonyAgent(device, {
      groupName: ctx.task.name,
    });
  });

  afterEach((ctx) => {
    if (ctx.task.result?.state === 'pass') {
      itTestStatus = 'passed';
    } else if (ctx.task.result?.state === 'skip') {
      itTestStatus = 'skipped';
    } else if (ctx.task.result?.errors?.[0].message.includes('timed out')) {
      itTestStatus = 'timedOut';
    } else {
      itTestStatus = 'failed';
    }
    reportMergingTool.append({
      reportFilePath: agent.reportFile as string,
      reportAttributes: {
        testId: `${ctx.task.name}`,
        testTitle: `${ctx.task.name}`,
        testDescription: 'description',
        testDuration: (Date.now() - ctx.task.result?.startTime!) | 0,
        testStatus: itTestStatus,
      },
    });
  });

  afterAll(() => {
    reportMergingTool.mergeReports('my-harmony-setting-test-report');
  });

  it('toggle WLAN', async () => {
    await device.home();
    await sleep(1000);
    await device.launch('com.huawei.hmos.settings');
    await sleep(1000);
    await agent.aiAct('find and enter WLAN setting');
    await agent.aiAct(
      'toggle WLAN status *once*, if WLAN is off please turn it on, otherwise turn it off.',
    );
  });

  it('toggle Bluetooth', async () => {
    await device.home();
    await sleep(1000);
    await device.launch('com.huawei.hmos.settings');
    await sleep(1000);
    await agent.aiAct('find and enter Bluetooth setting');
    await agent.aiAct(
      'toggle Bluetooth status *once*, if Bluetooth is off please turn it on, otherwise turn it off.',
    );
  });
});

Tip

Merged reports are stored inside midscene_run/report by default. Override the directory with MIDSCENE_RUN_DIR when running in CI.

#HarmonyOS Getting Started

#Set up API keys for model

#Prerequisites

#Install HDC

#Enable Developer Mode and Verify Device

#Try Playground (Zero Code)

#Start Your Experience

#Integrate Midscene Agent

#Step 1: Install Dependencies

#Step 2: Write a Script

#Step 3: Run the Example

#Step 4: View the Report

#Advanced

#Extending Midscene on HarmonyOS

#More

#Complete example (Vitest + HarmonyAgent)