#Midscene.js - GUI Agent for E2E Testing

AI-powered vision. Cross-platform. Batteries included.

Midscene is an open-source GUI Agent and testing kit for UI testing and automation across web, mobile, and desktop apps.

#See it in action

The same Agent APIs let you operate and verify interfaces across platforms. The Playwright example below shows a web test; configure a model and open your app in an existing Playwright page to get started:

import { PlaywrightAgent } from '@midscene/web/playwright';

const agent = new PlaywrightAgent(page);

// Let the Agent carry out a flow, then verify the result.
await agent.aiAct('Search for headphones, then filter the results to under $100');
await agent.aiWaitFor('The filtered search results are displayed');
await agent.aiAssert('Every product in the search results has a price below $100');

Open the generated HTML report to inspect screenshots, actions, and assertion results. Follow the Playwright guide for model setup, a complete example, and test runner integration.

#GUI Agent

Midscene models both UI actions and assertions on how people use software: look at the screen, act on what you see, and check the visible result. Describe the task and expected outcome in natural language; Midscene uses screenshots to decide where to interact and whether the interface meets your expectations.

#Visual understanding and cross-platform actions

Like a person finding a control on screen, Midscene locates elements by their appearance and position, then clicks, types, or scrolls to carry out your instructions. You can target icon-only buttons, custom controls, <canvas>, and elements inside cross-origin iframes without writing selectors or adding semantic annotations.

The same Agent APIs work across Web, Android, iOS, HarmonyOS, and desktop apps. You can also connect a custom interface by providing screenshot and action capabilities.

#Verify what users see

Assertions follow the same visual approach: Midscene inspects the screen as a human tester would to judge whether the expected result is visible. Describe the expected appearance in natural language to check colors, selection highlights, layout, and visual feedback — including content drawn on <canvas> or displayed in native apps.

await agent.aiAssert('The selected plan has a blue border and a checkmark');
await agent.aiAssert('The error message is visible below the email field');

#Benchmark performance

Each report includes the run configuration and task results; see AndroidWorld's report for its environment and validator adjustments.

#Cost and model choice

Screenshot-based UI actions avoid sending large DOM trees to the model. In the reported AppControlBench run, Midscene with Doubao Seed 2.1 Turbo completed the 60-task evaluation with $0.59 in total model cost, achieving 58 passes. The report provides per-task costs and comparisons across models.

Midscene supports multimodal models such as Qwen3.x, Doubao-Seed-2.1, GLM-4.6V, gemini-3.5-flash, and UI-TARS, including open-source options you can self-host. Start with one model, or combine planning and vision models for your workload. For data extraction and page understanding, you can opt in to include DOM. See Model Strategy and Supported models and setup.

#Showcases

Register the GitHub form autonomously in a web browser and pass all field validations:

See more real-world examples across iOS, Android, desktop, and custom interfaces in Showcases.

#Testing Kit

Batteries included: Midscene provides the test framework, observability, and integration APIs needed to turn GUI automation into a maintainable E2E test project.

#Midscene Test: an E2E framework for the AI era

Midscene Test (@midscene/test, Beta) separates declarative test intent from programmable engineering. Write UI flows and expectations in YAML, and wrap API calls, data preparation, and cleanup in reusable TypeScript Nodes. A refund test can prepare an order through an API, request a refund through the UI, and verify the result in one workflow.

The framework includes project scaffolding, platform presets, lifecycle hooks, retries, and execution-project isolation and concurrency. It also generates a Markdown reference from registered Nodes and their parameter schemas, so people and AI Agents can discover the same capabilities and co-maintain test cases. See Create and extend a project and Write and run tests.

#Built-in observability

Interactive HTML reports show screenshots, element locations, the AI decision process, and action and assertion results. Midscene Test records the inputs, outputs, duration, and status of each AI step and custom business operation. Reports and runtime logs give both developers and AI Agents the context to investigate failures. Use the Playground to try and refine instructions against your interface.

#Rich APIs that fit your existing stack

Use aiAct for autonomous flows, aiTap and aiInput for individual actions, aiAssert for assertions, and aiQuery for structured data extraction. Combine these Agent APIs with your existing code, fixtures, and assertions through Playwright, Puppeteer, or the JavaScript SDK. You can adopt Midscene's visual capabilities within your current testing framework. See The Basics for how to choose between autonomous actions and JavaScript orchestration. AI coding agents can also operate interfaces through Midscene Skills.

#Get started

• Try Midscene in the Playground — experiment with natural-language actions, data extraction, and visual assertions before writing scripts. Start with the Chrome extension, or launch a Playground for mobile or desktop.
• Write tests with the SDK or YAML — start with Playwright, Puppeteer, or the Midscene Test.
• Let your AI agent operate the UI — install Midscene Skills.
• Test on another platform — follow the guides for Android, iOS, HarmonyOS, or desktop.

#Resources & community

• Sample projects: midscene-example
• GitHub: web-infra-dev/midscene
• Discord · X · Lark group (飞书交流群)

#Credits

Midscene builds on many excellent open-source projects — including UI-TARS, Qwen, Playwright, Puppeteer, scrcpy, appium, WebDriverAgent, YADB, and libnut-core. See the README for the full list.

#License

Midscene.js is MIT licensed.