#Android environment setup

Midscene connects to and controls Android devices over adb, supporting UI automation across native apps, hybrid apps, and system interfaces.

This guide prepares the Android environment: install adb and connect the device. Playground, Midscene Test, and the JavaScript SDK all use this connection.

#Connect the device through adb

Midscene uses adb on your computer to communicate with the target Android device over either USB or wireless adb. If adb is already installed and your device is connected, skip directly to Verify the device connection.

#Install adb

• Install adb via Android Studio or the standalone Android command-line tools.
• Run the following command to verify the installation:

adb --version

Output similar to the following confirms a working installation:

Android Debug Bridge version 1.0.41
Version 34.0.4-10411341
Installed as /usr/local/bin/adb
Running on Darwin 24.3.0 (arm64)

• Configure the ANDROID_HOME environment variable as described in Android environment variables, then check that it takes effect:

echo $ANDROID_HOME

The output should point to your local Android SDK directory, for example:

/Users/your_username/Library/Android/sdk

#Enable debugging and connect the device

The steps below use a USB connection as an example. If your device is already connected over wireless adb, you can keep using that connection.

1. In your phone's Developer options, enable USB debugging (and USB debugging (Security settings) if available, so Midscene can inject touch events).
2. Connect the phone to your computer using a USB cable that supports data transfer.
3. Keep the phone unlocked and accept the Allow USB debugging prompt on the screen.

#Verify the device connection

Run the following command to check the connection status:

adb devices -l

When the device status shows device, the connection is ready:

List of devices attached
s4ey59	device usb:34603008X product:cezanne model:M2006J device:cezan transport_id:3

If multiple devices are connected, note the device ID in the first column (such as s4ey59 above) so you can select the target device in Playground or your project configuration.

If the target device does not appear with the device status, check the following:

• Empty list: For USB, check the data cable (ensure it is not charge-only), USB connection mode, and USB debugging toggle; for wireless adb, check the network and adb connect status.
• unauthorized: Unlock the phone screen and accept the Allow USB debugging prompt on the device.
• offline: Reconnect the USB cable or wireless adb session, and restart adb if needed. Proceed only after the status shows device.

#Verify

When adb devices -l shows the target device with the device status, verify the environment in Playground. Open Get started, where Android and Playground are already selected, choose how to connect a model, and follow the steps to run one aiAct action and one aiAssert assertion. A passing assertion confirms that the adb connection, screen capture, touch input, and model calls work together.

#FAQ

#The device is missing, unauthorized, or offline

Run adb devices -l and follow the troubleshooting steps in Verify the device connection to check the USB cable, debugging authorization, and connection status. Make sure the target device status is device before investigating screenshot or input issues.

#Why can't I control the device even though it is connected (INJECT_EVENTS error)?

If tapping or swiping fails with a permission error like the following:

Error:
Exception occurred while executing 'tap':
java.lang.SecurityException: Injecting input events requires the caller (or the source of the instrumentation, if any) to have the INJECT_EVENTS permission.

Some Android OEM ROMs (such as Xiaomi / MIUI / HyperOS, vivo, and OPPO) require an additional toggle in Developer options to allow simulated input. Make sure both of the following options are enabled:

1. USB debugging
2. USB debugging (Security settings) or Disable permission monitoring (if present)

#Text input is cleared or disappears after typing

If text is entered properly but disappears when the soft keyboard closes, the input field may be clearing its value in response to the ESC key event sent to dismiss the keyboard. You can customize the keyboard dismissal strategy; see the Android JavaScript SDK guide for options and examples.

#English text is altered by the Android input method

Compare the input value recorded in the execution report with the text actually entered into the field. If the value in the report is correct but characters in the input field are reordered, missing, or intercepted by Pinyin composition, check the active input method and autocorrect settings on the device. See the Android JavaScript SDK guide for details on input channels.

#Screenshots of secure pages (such as password inputs) are black

First switch to a regular page and check whether screenshots work normally. If only password screens or sensitive pages appear black, the app likely enables system screenshot protection (FLAG_SECURE) rather than adb being disconnected. See the Android JavaScript SDK guide for screenshot mechanisms and limitations.

#Why does scrcpy fall back to ADB screenshots?

To prevent the AI model from acting on stale frames, Midscene strictly validates scrcpy frame timestamps: any frame that cannot be proven to have been captured after the most recent completed action—or that exceeds the maximum frame age limit—is rejected. If the active scrcpy stream fails to deliver a fresh frame in time, Midscene automatically restarts scrcpy once, and falls back to adb screenshots only if the restarted stream also fails.

This commonly occurs when an Android hardware encoder pauses frame output on a completely static screen, or during stream startup and transient USB transport hiccups. Note that a frame-freshness warning alone does not indicate insufficient bandwidth or an excessive video bitrate—do not adjust the video bitrate solely in response to this warning.

#How do I configure the scrcpy video bitrate?

See the Android JavaScript SDK guide.

#How do I access Android Playground remotely from another machine on the LAN?

By default, both the Playground server and its built-in scrcpy streaming server bind only to 127.0.0.1. To access Android Playground from another computer on your network, explicitly bind both servers to a reachable interface via environment variables (for example, on macOS or Linux):

MIDSCENE_PLAYGROUND_HOST=0.0.0.0 MIDSCENE_SCRCPY_HOST=0.0.0.0 pnpm dlx @midscene/android-playground

Then open Playground in a browser using the host machine's LAN IP address (or hostname) and the Playground port printed at startup (note that 0.0.0.0 is a wildcard bind address, not a URL you can open in a browser), and ensure your firewall allows incoming connections on both the Playground and scrcpy ports.

MIDSCENE_SCRCPY_HOST applies to the standalone Android Playground CLI and can also be set to a specific local network IP (such as 192.168.1.100, which the auto-opened local preview page will also use). If you instantiate ScrcpyServer directly in code, pass the host option to its constructor instead.

#How do I use a custom adb path or remote adb server?

See the Android JavaScript SDK guide.