Quick experience with Android

By using Midscene.js playground, you can quickly experience the main features of Midscene on Android devices, without needing to write any code.

Preparation

Install Node.js

Install Node.js 18 or above globally.

Prepare an API key

Prepare an API key from a visual-language (VL) model. You will use it later.

You can check the supported models in Choose a model

Install adb

adb is a command-line tool that allows you to communicate with an Android device. There are two ways to install adb:

Verify adb is installed successfully:

adb --version

When you see the following output, adb is installed successfully:

Android Debug Bridge version 1.0.41
Version 34.0.4-10411341
Installed as /usr/local/bin//adb
Running on Darwin 24.3.0 (arm64)

Set environment variable ANDROID_HOME

Reference Android environment variables, set the environment variable ANDROID_HOME.

Verify the ANDROID_HOME variable is set successfully:

echo $ANDROID_HOME

When the command has any output, the ANDROID_HOME variable is set successfully:

/Users/your_username/Library/Android/sdk

Connect Android device with adb

In the developer options of the system settings, enable the 'USB debugging' of the Android device, if the 'USB debugging (secure settings)' exists, also enable it, then connect the Android device with a USB cable

android usb debug

Verify the connection:

adb devices -l

When you see the following output, the connection is successful:

List of devices attached
s4ey59	device usb:34603008X product:cezanne model:M2006J device:cezan transport_id:3

Run the playground

npx --yes @midscene/android-playground

Configure the API key

Click the gear button to enter the configuration page and paste your API key config.

Refer to Config Model and Provider document, config the API Key.

Start experiencing

After the configuration, you can immediately experience Midscene. It provides multiple key operation tabs, including but not limited to:

  • Action: interact with the web page. This is also known as "Auto Planning". For example:
type Midscene in the search box
click the login button
  • Query: extract JSON data from the web page
extract the user id from the page, return in \{ id: string \}
  • Assert: validate the page
the page title is "Midscene"
  • Tap: perform a single tap on the element where you want to click. This is also known as "Instant Action".
the login button

All Agent APIs can be directly debugged and run in the Playground! Interactive, extraction, and verification methods are fully covered, with visual operations and verification that boost your automation development efficiency!

Enjoy !

For the different between "Auto Planning" and "Instant Action", please refer to the API document.

Want to write some code?

After experiencing, you may want to write some code to integrate Midscene. There are multiple ways to do that. Please refer to the documents below:

On this page