Midscene.js - Joyful Automation by AI

Open-source AI Operator for Web, Mobile App, Automation & Testing

Features

Write Automation with Natural Language

  • Describe your goals and steps, and Midscene will plan and operate the user interface for you.
  • Use Javascript SDK or YAML to write your automation script.

Web or Mobile App

Tools

  • Visual Reports for Debugging: Through our test reports and Playground, you can easily understand, replay and debug the entire process.
  • Caching for Efficiency: Replay your script with cache and get the result faster.
  • MCP: Allows other MCP Clients to directly use Midscene's capabilities.

Three kinds of APIs

Showcases

We've prepared some showcases for you to learn the use of Midscene.js.

  1. Use JS code to drive task orchestration, collect information about Jay Chou's concert, and write it into Google Docs (By UI-TARS model)
  1. Control Maps App on Android (By Qwen-2.5-VL model)
  1. Using midscene mcp to browse the page (https://www.saucedemo.com/), perform login, add products, place orders, and finally generate test cases based on mcp execution steps and playwright example

Zero-code Quick Experience

Model Choices

Midscene.js supports both multimodal LLMs like gpt-4o, and visual-language models like Qwen2.5-VL, Doubao-1.5-thinking-vision-pro, gemini-2.5-pro and UI-TARS.

Visual-language models are recommended for UI automation.

Read more about Choose a model

Two Styles of Automation

Auto Planning

Midscene will automatically plan the steps and execute them. It may be slower and heavily rely on the quality of the AI model.

await aiAction('click all the records one by one. If one record contains the text "completed", skip it');

Workflow Style

Split complex logic into multiple steps to improve the stability of the automation code.

const recordList = await agent.aiQuery('string[], the record list')
for (const record of recordList) {
  const hasCompleted = await agent.aiBoolean(`check if the record contains the text "completed"`)
  if (!hasCompleted) {
    await agent.aiTap(record)
  }
}

For more details about the workflow style, please refer to Blog - Use JavaScript to Optimize the AI Automation Code

Comparing to other projects

There are so many UI automation tools out there, and each one seems to be all-powerful. What's special about Midscene.js?

  • Debugging Experience: You will soon realize that debugging and maintaining automation scripts is the real challenge. No matter how magical the demo looks, ensuring stability over time requires careful debugging. Midscene.js offers a visualized report file, a built-in playground, and a Chrome Extension to simplify the debugging process. These are the tools most developers truly need, and we're continually working to improve the debugging experience.

  • Open Source, Free, Deploy as you want: Midscene.js is an open-source project. It's decoupled from any cloud service and model provider, you can choose either public or private deployment. There is always a suitable plan for your business.

  • Integrate with Javascript: You can always bet on Javascript 😎

Resources

Community

Credits

We would like to thank the following projects:

  • Rsbuild for the build tool.
  • UI-TARS for the open-source agent model UI-TARS.
  • Qwen2.5-VL for the open-source VL model Qwen2.5-VL.
  • scrcpy and yume-chan allow us to control Android devices with browser.
  • appium-adb for the javascript bridge of adb.
  • YADB for the yadb tool which improves the performance of text input.
  • Puppeteer for browser automation and control.
  • Playwright for browser automation and control and testing.

License

Midscene.js is MIT licensed.

On This Page