See interfaces like a human
E2E testing in natural language

Midscene.js provides a vision-based GUI Agent and a natural-language E2E testing framework for web, mobile, and desktop apps.

🔥 Latest news Starting with v1.16, Midscene supports using Jev 1.4’s visual capabilities to drive E2E testing. See what’s new

15k+Github Stars

MITFully open source

#2No.2 in Github trending

Describe steps and expectations in natural language

Describe what a user wants to do and what they should see. Express the same intent in YAML or in your existing JavaScript tests.

YAML · @midscene/test

Vitest + JavaScript

beforeEach:
  - gotoUrl: https://your-shop.example

cases:
  - name: "Find headphones under $100"
    steps:
      - aiAct: "Search for headphones and filter the results to prices under $100"
      - aiWaitFor: "The filtered search results are visible"
      - aiAssert: "There are products in the search results, and each costs less than $100"

import { test } from 'vitest';
import { PlaywrightAgent } from '@midscene/web/playwright';

// Model configuration, page setup, and cleanup omitted
test("Find headphones under $100", async () => {
  const agent = new PlaywrightAgent(page);
  await agent.aiAct("Search for headphones and filter the results to prices under $100");
  await agent.aiWaitFor("The filtered search results are visible");
  await agent.aiAssert("There are products in the search results, and each costs less than $100");
}, 180_000);

A GUI Agent built for testing

Midscene’s GUI Agent brings visual actions and assertions to your existing JavaScript tests. We recommend Playwright Test for web apps and Vitest for mobile and desktop apps.

See, act, and verify like a human

E2E testing aims to verify complete workflows from the user’s perspective. Midscene uses vision to interact as a user would: observe the screen, find the target, act, and check what the user actually sees. See real-world examples

One API, across platforms

Visual localization does not depend on a particular UI rendering stack. Platform adapters provide screenshots and actions, bringing the same approach to web, mobile, and desktop interfaces, including HTML canvas.

Platform	Package	Description
Web	@midscene/web	Playwright · Puppeteer · Desktop Chrome
Android	@midscene/android	Devices / emulators · ADB
iOS	@midscene/ios	Devices / simulators · WebDriverAgent
HarmonyOS	@midscene/harmony	HarmonyOS NEXT · HDC
Desktop	@midscene/computer	macOS · Windows · Linux, with remote RDP
Custom interfaces	@midscene/core	Connect through screenshots and actions

Model options and cost

Multiple model options

Choose from Doubao, DeepSeek, Qwen, GPT, Gemini, Kimi, and more to fit your tasks and budget.

View model configuration

Low model API cost

In this AppControlBench run, Doubao Seed 2.1 Turbo passed 58 of 60 tasks at a total model API cost of $0.59.

View benchmark report

Benchmarks

AndroidWorld: Pass@1 of 93.1% and Pass@3 of 97.4%. Read the benchmark report
MobileWorld: Pass@1 of 78.6%, completing 92 of 117 tasks. Read the benchmark report
AppControlBench: Pass@1 of 96.7%, passing 58 of 60 tasks. Read the benchmark report

A new E2E testing framework designed for natural language

Midscene Test (@midscene/test) is a natural-language E2E testing framework. In a single YAML case, you can describe UI actions and expected outcomes in natural language while encapsulating business logic that requires precise control in TypeScript. Cases have a clear, readable structure that both people and AI agents can write and maintain, allowing test intent and underlying implementation to evolve independently. Get started with Midscene Test

AI and business operations in one YAML file

Each YAML step calls a Node: an AI operation, a platform operation, or a custom business operation. Compose these Nodes into test cases that people and AI agents can read, write, and review.

cases/refund.yaml

1beforeEach:2  - order.prepare: { status: paid }Business3  - gotoUrl: ${shopUrl}/orders/latestBusiness45cases:6  - name: A paid order supports a full refund7    steps:8      - aiAct: Apply for a full refund and submitAI9      - aiAssert: The page shows that the refund was submittedAI10      - order.checkRefund: { status: refunding }Business1112afterEach:13  - clearCookies: {}Business

Test cases maintained by people and AI

As your product evolves, you can encapsulate reusable business logic into custom Nodes, keeping YAML cases focused on test intent while maintaining business implementations independently. Midscene Test automatically generates a Markdown Node Spec from the Nodes registered in the project, listing all available operations and their input schemas. Both QA engineers and AI agents can reference this spec directly to write and update test cases without reading the underlying TypeScript code. Learn how to develop custom Nodes

nodes/order.ts

defineNode({
  name: 'order.prepare',
  description: "Create a test order",
  inputSchema: z.strictObject({
    status: z.literal('paid'),
  }),
  async execute({ input }) {
    await orders.create(input);
  },
});

midscene-node-spec.md

### order.prepare
Create a test order
- status: "paid"

case.yaml

steps:
  - order.prepare:
      status: paid

Who is using Midscene

Next steps

Understand the basics: Learn how the GUI Agent works and explore its core capabilities in The Basics
Try visual interaction: Follow the quick start to try natural-language actions and assertions
Create a test project: Write and run YAML test cases with Midscene Test
Write tests with code: Integrate with Playwright Test for web apps or Vitest for mobile and desktop apps
Run standalone automation workflows: Write scripts with the JavaScript SDK

To let AI coding agents operate your UI, explore Midscene Skills.

#See interfaces like a humanE2E testing in natural language

#Describe steps and expectations in natural language

#A GUI Agent built for testing

#See, act, and verify like a human

#One API, across platforms

#Model options and cost