#Model Strategy

Midscene's model strategy serves two goals. First, it uses pure vision to understand the interface that users actually see, keeping UI automation independent of the rendering stack. Second, it provides the option to combine multiple models for complex scenarios.

#A pure-vision approach based on the visible interface

AI-powered UI automation requires task planning and element localization. The industry mainly uses two localization approaches: combining DOM data with annotated screenshots, or using screenshots directly for pure-vision localization. Midscene uses pure vision, with the model analyzing the UI screenshot and locating target elements directly. UI actions and element localization do not depend on DOM data or extra annotations.

This choice makes the visible interface the source of truth for automation and gives Midscene several advantages:

• It uses a consistent approach across browser Canvas, Android, iOS, desktop applications, and other types of interfaces.
• It can validate what users actually see, including colors, highlighted states, and layout.
• It is independent of the UI rendering stack and does not require selectors or extra UI annotations.
• Token consumption depends only on page resolution and task complexity, and does not inflate as the page structure (such as the number of DOM elements) grows.

Pure vision is not a local optimization for one platform. It is the shared foundation of Midscene's cross-platform interaction capabilities. The same task descriptions and interaction patterns can extend across interfaces and devices while staying close to how real users operate software.

The vision-based approach also has clear limitations. Pure-vision localization requires models with visual understanding capabilities — only designated models that are stable for GUI operations can be used, not any arbitrary LLM. Midscene accepts higher model capability requirements in exchange for cross-platform consistency and lower UI maintenance costs.

Data extraction and page-understanding workloads can still include DOM data when needed. See the API Reference for the relevant options.

#Combine models for complex scenarios

Midscene uses a multimodal Default model for task planning, element localization, page understanding, and the rest of the automation workflow. This default approach covers most UI automation scenarios and lets users get started with minimal configuration.

When complex planning, data extraction, or page understanding requires specialized model capabilities, users can add Planning and Insight models to the Default model. Each model contributes its strengths to the same automation workflow:

This collaboration extends Midscene's ability to handle complex tasks, but it can also increase task latency and token usage. Start with the Default model, then introduce specialized models only for a clear capability bottleneck.

#Next steps

This page focuses on Midscene's model philosophy and selection principles. For setup instructions, see Optional: configure multiple models. If task quality is unstable, use Model debugging and observability to identify the problem.