--- url: /advanced/bdd-style-scripts-with-gherkin.md --- # BDD-style scripts with Gherkin > This feature is available starting in Midscene 1.10. > > BDD-related capabilities are still in Beta. The API may change in future releases. Gherkin is a plain-text syntax for describing behavior examples. It is commonly used in BDD (Behavior-Driven Development) workflows. A BDD scenario usually contains three kinds of steps: `Given` describes the precondition, `When` describes the user action, and `Then` describes the expected result. For example, when you need to test adding a todo item, you can write the case as the following Gherkin script. ```gherkin Scenario: Add a todo item Given the todo page is open When I add a todo item named "Buy milk" Then the todo list should contain "Buy milk" ``` Longer GUI cases can also be written in one `Scenario`. You can organize the steps by business phase, and each phase can use a group of `Given`, `When`, and `Then`. ```gherkin Scenario: Place an order with a saved address Given the shopping cart contains "Midscene Mug" When I open the cart page Then the cart should show "Midscene Mug" And the subtotal should be visible Given the checkout page is open When I choose the saved shipping address Then the shipping address section should show the selected address Given the payment section is visible When I choose the saved credit card and submit the order Then the order confirmation page should be displayed And the confirmation page should show an order number ``` This style is useful when one business scenario naturally contains several UI phases, such as cart review, shipping selection, payment, and final confirmation. In AI-driven GUI automation, Gherkin is a natural fit for describing operation cases. It keeps the readability of natural language while providing a stable step structure. This structure also helps models generate more consistent cases. See the official [Cucumber Gherkin reference](https://cucumber.io/docs/gherkin/reference) for the full Gherkin syntax. ## Supported rules Midscene supports only a subset of Gherkin. This subset is designed around a single `Scenario`, and is suitable for describing a complete GUI operation flow. ### Step mapping * `Given`: calls `aiAct`, and prepares the precondition. * `When`: calls `aiAct`, and performs the user action. * `Then`: calls `aiAssert`, and verifies the page result. * `And`, `But`: reuse the previous primary keyword. For example, an `And` after `Then` also calls `aiAssert`. Keyword matching is case-insensitive. Therefore, `AND`, `and`, and `And` are all recognized. ### Scenario limits Only one `Scenario:` is supported each time. If the input contains multiple `Scenario:` blocks, Midscene throws an error. You can also omit `Scenario:` and write the steps directly. ```gherkin Given the todo page is open When I add a todo item named "Buy milk" Then the todo list should contain "Buy milk" ``` ### Notes Midscene splits each Gherkin step into an independent `aiAct` or `aiAssert` call. Each step should be complete, and should clearly describe the target, the action, and the expected result. Do not depend on omitted information or vague references from previous steps. For example, do not write only "click it" or "check the result". Instead, write the target to click and the result to check. The following example cannot run correctly. The steps are isolated from each other, so later steps do not know what "it" and "the result" refer to. ```gherkin Given the todo page is open When I add a todo item named "Buy milk" And click it Then check the result ``` Write it this way instead. ```gherkin Given the todo page is open When I add a todo item named "Buy milk" And I click the "Buy milk" todo item Then the "Buy milk" todo item should be marked as completed ``` ### Not supported The following Gherkin features are not supported: * `Feature` * `Background` * `Scenario Outline` or `Scenario Template` * `Examples` * `Rule` * data tables * doc strings * variable replacement, placeholder expansion, or template syntax * multiple `Scenario:` blocks in the same input Midscene does not extend Gherkin into a scripting language. It only reads `Scenario` and step keywords, then passes each step to the Agent. Loops, conditions, variable replacement, and batch generation should happen before calling Midscene. If you need reusable templates, implement a Gherkin generator in your project. The generator should output the final plain-text `Scenario`, then pass it to `runGherkinScenario`. ## Use in JavaScript In JavaScript or TypeScript scripts, call `agent.runGherkinScenario()` directly. It receives a Gherkin text string and executes the steps in order. The following example uses the Playwright fixture to get an Agent. ```ts import { test } from '@midscene/web/playwright'; test('add a todo item', async ({ page, agentForPage }) => { await page.goto('http://localhost:3000/todos'); const agent = await agentForPage(page); await agent.runGherkinScenario(` Scenario: Add a todo item Given the todo page is open When I add a todo item named "Buy milk" Then the todo list should contain "Buy milk" And the todo count should be 1 `); }); ``` The second argument accepts runtime options. For example, you can provide temporary context for this run. ```ts await agent.runGherkinScenario( ` Scenario: Add a todo item Given the todo page is open When I add a todo item named "Buy milk" Then the todo list should contain "Buy milk" `, { context: 'This is a todo demo app. Use the input box and the add button on the page.', }, ); ``` ## Use in YAML Use the `runGherkinScenario` step in a YAML flow. The value should be a block string containing one scenario. ```yaml web: url: http://localhost:3000/todos tasks: - name: Add a todo item flow: - runGherkinScenario: | Scenario: Add a todo item Given the todo page is open When I add a todo item named "Buy milk" Then the todo list should contain "Buy milk" And the todo count should be 1 ``` During YAML execution, Midscene runs the scenario step by step. `Given` and `When` steps call `aiAct`, while `Then` and the following `And` step call `aiAssert`. ## Remarks `runGherkinScenario` disables cache inside the scenario. This means that after `Given` and `When` are mapped to `aiAct`, they do not use the `aiAct` planning cache. This makes each Gherkin step interpreted from the current page state, and avoids incorrect behavior caused by reusing stale steps. --- url: /android-world-benchmark-report.md --- # Midscene AndroidWorld Benchmark Report import { BenchmarkReportPreview } from '@theme'; This is Midscene's test report for the AndroidWorld benchmark. In this run, Midscene achieved **Pass@1 93.10%**, **Pass@2 95.69%**, and **Pass@3 97.41%**. :::info About AndroidWorld [AndroidWorld](https://github.com/google-research/android_world) is an Android agent benchmark from Google Research. It runs on a live Android emulator and evaluates agents on 116 programmatic tasks across 20 real-world Android apps, with task initialization and validation handled by the benchmark. ::: ## Run Configuration | Field | Value | | --- | --- | | Model Name | `Gemini-3.5-Flash` | | Midscene version | `1.9.5` | | DeepThink | on | | `MIDSCENE_REPLANNING_CYCLE_LIMIT` | 120 | | AndroidWorld setup | Stability fixes were applied to the AndroidWorld project to reduce flaky benchmark runs. See examples below. | | Validation notes | A small number of AndroidWorld validators were aligned with the task intent. The affected cases are listed below. | ## Stability Improvements The following changes did not change the task intent. They made benchmark execution more stable by reducing browser rendering races, stale accessibility reads, and setup timing issues. | Change | Affected cases | | --- | --- | | Force canvas pixels to flush after drawing, then use thicker rounded strokes so target colors remain stable in the final canvas pixels. | `BrowserDraw` | | Retry reading the `Success!` text from the accessibility tree after browser tasks finish. | `BrowserMaze`
`BrowserMultiply`
`BrowserDraw` | | Before SMS tasks start, prepare the required incoming messages and contacts; now verify those messages are visible in the inbox and those contacts are visible in Contacts before the agent runs. | `SimpleSmsReplyMostRecent`
`SimpleSmsSendReceivedAddress` | | Wait for Pro Expense to create its database tables before validators write test data. | `ExpenseAddMultiple`
`ExpenseAddMultipleFromGallery`
`ExpenseAddMultipleFromMarkor`
`ExpenseAddSingle`
`ExpenseDeleteDuplicates`
`ExpenseDeleteDuplicates2`
`ExpenseDeleteMultiple`
`ExpenseDeleteMultiple2`
`ExpenseDeleteSingle` | | Preload OsmAnd offline map files into the app data directory and wait for OsmAnd to extract its built-in basemap before map tasks run. | `OsmAndFavorite`
`OsmAndMarker`
`OsmAndTrack` | ## Validation Condition Updates The following AndroidWorld validation checks were changed on the `main` branch used for this benchmark: | Change | Affected cases | | --- | --- | | Calendar "after start time" now validates against an event one minute after the boundary, avoiding ambiguity across models about whether `after` includes the boundary time. | `SimpleCalendarFirstEventAfterStartTime` | | Expense notes imported from Markor accept the extra `Reimbursable.` suffix that Markor can include; note comparison ignores that suffix and terminal period differences. | `ExpenseAddMultipleFromMarkor` | | Markor merged notes accept either single-newline or blank-line separation, and also accept Markor's default `.md` extension when it is auto-added. | `MarkorMergeNotes` | | Recipe quantity fields allow omitted units while still rejecting wrong amounts or incompatible units. | `RecipeAddSingleRecipe`
`RecipeAddMultipleRecipes`
`RecipeAddMultipleRecipesFromMarkor`
`RecipeAddMultipleRecipesFromMarkor2`
`RecipeAddMultipleRecipesFromImage`
`NotesRecipeIngredientCount` | | Minimum brightness is validated against Android's actual minimum setting value, `0`, instead of `1`. | `SystemBrightnessMin`
`SystemBrightnessMinVerify` | ## Report Files Detailed reports are listed below for reference.
Round 1 (115 reports · 108 PASS · 7 FAIL) | # | Task | Status | Report | | --- | --- | --- | --- | | 1 | AudioRecorderRecordAudio | PASS | report | | 2 | AudioRecorderRecordAudioWithFileName | PASS | report | | 3 | BrowserDraw | PASS | report | | 4 | BrowserMaze | PASS | report | | 5 | BrowserMultiply | PASS | report | | 6 | CameraTakePhoto | PASS | report | | 7 | CameraTakeVideo | PASS | report | | 8 | ClockStopWatchPausedVerify | PASS | report | | 9 | ClockStopWatchRunning | PASS | report | | 10 | ClockTimerEntry | PASS | report | | 11 | ContactsAddContact | PASS | report | | 12 | ContactsNewContactDraft | PASS | report | | 13 | ExpenseAddMultiple | PASS | report | | 14 | ExpenseAddMultipleFromGallery | PASS | report | | 15 | ExpenseAddMultipleFromMarkor | FAIL | report | | 16 | ExpenseAddSingle | PASS | report | | 17 | ExpenseDeleteDuplicates | PASS | report | | 18 | ExpenseDeleteDuplicates2 | PASS | report | | 19 | ExpenseDeleteMultiple | PASS | report | | 20 | ExpenseDeleteMultiple2 | PASS | report | | 21 | ExpenseDeleteSingle | PASS | report | | 22 | FilesDeleteFile | PASS | report | | 23 | FilesMoveFile | PASS | report | | 24 | MarkorAddNoteHeader | PASS | report | | 25 | MarkorChangeNoteContent | PASS | report | | 26 | MarkorCreateFolder | PASS | report | | 27 | MarkorCreateNote | PASS | report | | 28 | MarkorCreateNoteAndSms | PASS | report | | 29 | MarkorCreateNoteFromClipboard | PASS | report | | 30 | MarkorDeleteAllNotes | PASS | report | | 31 | MarkorDeleteNewestNote | PASS | report | | 32 | MarkorDeleteNote | PASS | report | | 33 | MarkorEditNote | PASS | report | | 34 | MarkorMergeNotes | PASS | report | | 35 | MarkorMoveNote | PASS | report | | 36 | MarkorTranscribeReceipt | PASS | report | | 37 | MarkorTranscribeVideo | FAIL | report | | 38 | OpenAppTaskEval | PASS | report | | 39 | OsmAndFavorite | PASS | report | | 40 | OsmAndMarker | FAIL | report | | 42 | RecipeAddMultipleRecipes | PASS | report | | 43 | RecipeAddMultipleRecipesFromImage | FAIL | report | | 44 | RecipeAddMultipleRecipesFromMarkor | PASS | report | | 45 | RecipeAddMultipleRecipesFromMarkor2 | PASS | report | | 46 | RecipeAddSingleRecipe | PASS | report | | 47 | RecipeDeleteDuplicateRecipes | PASS | report | | 48 | RecipeDeleteDuplicateRecipes2 | FAIL | report | | 49 | RecipeDeleteDuplicateRecipes3 | FAIL | report | | 50 | RecipeDeleteMultipleRecipes | PASS | report | | 51 | RecipeDeleteMultipleRecipesWithConstraint | PASS | report | | 52 | RecipeDeleteMultipleRecipesWithNoise | PASS | report | | 53 | RecipeDeleteSingleRecipe | PASS | report | | 54 | RecipeDeleteSingleWithRecipeWithNoise | PASS | report | | 55 | RetroCreatePlaylist | PASS | report | | 56 | RetroPlayingQueue | PASS | report | | 57 | RetroPlaylistDuration | PASS | report | | 58 | RetroSavePlaylist | PASS | report | | 59 | SaveCopyOfReceiptTaskEval | PASS | report | | 60 | SimpleCalendarAddOneEvent | PASS | report | | 61 | SimpleCalendarAddOneEventInTwoWeeks | PASS | report | | 62 | SimpleCalendarAddOneEventRelativeDay | PASS | report | | 63 | SimpleCalendarAddOneEventTomorrow | PASS | report | | 64 | SimpleCalendarAddRepeatingEvent | PASS | report | | 65 | SimpleCalendarDeleteEvents | PASS | report | | 66 | SimpleCalendarDeleteEventsOnRelativeDay | PASS | report | | 67 | SimpleCalendarDeleteOneEvent | PASS | report | | 68 | SimpleDrawProCreateDrawing | PASS | report | | 69 | SimpleSmsReply | PASS | report | | 70 | SimpleSmsReplyMostRecent | PASS | report | | 71 | SimpleSmsResend | PASS | report | | 72 | SimpleSmsSend | PASS | report | | 73 | SimpleSmsSendClipboardContent | PASS | report | | 74 | SimpleSmsSendReceivedAddress | PASS | report | | 75 | SystemBluetoothTurnOff | PASS | report | | 76 | SystemBluetoothTurnOffVerify | PASS | report | | 77 | SystemBluetoothTurnOn | PASS | report | | 78 | SystemBluetoothTurnOnVerify | PASS | report | | 79 | SystemBrightnessMax | PASS | report | | 80 | SystemBrightnessMaxVerify | PASS | report | | 81 | SystemBrightnessMin | PASS | report | | 82 | SystemBrightnessMinVerify | PASS | report | | 83 | SystemCopyToClipboard | FAIL | report | | 84 | SystemWifiTurnOff | PASS | report | | 85 | SystemWifiTurnOffVerify | PASS | report | | 86 | SystemWifiTurnOn | PASS | report | | 87 | SystemWifiTurnOnVerify | PASS | report | | 88 | TurnOffWifiAndTurnOnBluetooth | PASS | report | | 89 | TurnOnWifiAndOpenApp | PASS | report | | 90 | VlcCreatePlaylist | PASS | report | | 91 | VlcCreateTwoPlaylists | PASS | report | | 92 | NotesIsTodo | PASS | report | | 93 | NotesMeetingAttendeeCount | PASS | report | | 94 | NotesRecipeIngredientCount | PASS | report | | 95 | NotesTodoItemCount | PASS | report | | 96 | SimpleCalendarAnyEventsOnDate | PASS | report | | 97 | SimpleCalendarEventOnDateAtTime | PASS | report | | 98 | SimpleCalendarEventsInNextWeek | PASS | report | | 99 | SimpleCalendarEventsInTimeRange | PASS | report | | 100 | SimpleCalendarEventsOnDate | PASS | report | | 101 | SimpleCalendarFirstEventAfterStartTime | PASS | report | | 102 | SimpleCalendarLocationOfEvent | PASS | report | | 103 | SimpleCalendarNextEvent | PASS | report | | 104 | SimpleCalendarNextMeetingWithPerson | PASS | report | | 105 | SportsTrackerActivitiesCountForWeek | PASS | report | | 106 | SportsTrackerActivitiesOnDate | PASS | report | | 107 | SportsTrackerActivityDuration | PASS | report | | 108 | SportsTrackerLongestDistanceActivity | PASS | report | | 109 | SportsTrackerTotalDistanceForCategoryOverInterval | PASS | report | | 110 | SportsTrackerTotalDurationForCategoryThisWeek | PASS | report | | 111 | TasksCompletedTasksForDate | PASS | report | | 112 | TasksDueNextWeek | PASS | report | | 113 | TasksDueOnDate | PASS | report | | 114 | TasksHighPriorityTasks | PASS | report | | 115 | TasksHighPriorityTasksDueOnDate | PASS | report | | 116 | TasksIncompleteTasksOnDate | PASS | report |
Round 2 (7 reports · 3 PASS · 4 FAIL) | # | Task | Status | Report | | --- | --- | --- | --- | | 37 | MarkorTranscribeVideo | FAIL | report | | 40 | OsmAndMarker | FAIL | report | | 41 | OsmAndTrack | PASS | report | | 43 | RecipeAddMultipleRecipesFromImage | PASS | report | | 48 | RecipeDeleteDuplicateRecipes2 | FAIL | report | | 49 | RecipeDeleteDuplicateRecipes3 | FAIL | report | | 83 | SystemCopyToClipboard | PASS | report |
Round 3 (5 reports · 2 PASS · 3 FAIL) | # | Task | Status | Report | | --- | --- | --- | --- | | 15 | ExpenseAddMultipleFromMarkor | PASS | report | | 37 | MarkorTranscribeVideo | FAIL | report | | 40 | OsmAndMarker | PASS | report | | 48 | RecipeDeleteDuplicateRecipes2 | FAIL | report | | 49 | RecipeDeleteDuplicateRecipes3 | FAIL | report |
--- url: /app-control-bench-report.md --- # Midscene AppControlBench Benchmark Report import { AppControlBenchComparison, AppControlBenchReport } from '@theme'; This is Midscene's benchmark report for AppControlBench. In this evaluation, each model ran the same set of 60 tasks, with the following results: > Note: Model costs are calculated using OpenRouter pricing, with CNY converted at ¥6.8 per $1. **The results show that the Midscene + GUI vision approach is highly competitive in both cost and pass rate.** :::info About AppControlBench [AppControlBench](https://github.com/software-mansion/app-control-bench) evaluates how well an Agent can operate real iOS apps in isolated Simulator runs. The final screenshot is graded against a task-specific solved-screen description. Each run covers 30 Bluesky tasks and 30 Element iOS tasks. ::: ## Run configuration | Field | Value | | --- | --- | | Test date | 2026-08-25 to 2026-08-27 | | Model Name | Doubao Seed 2.1 Turbo, Qwen3.7 Plus, DeepSeek V4 Flash Vision Exp | | Midscene version | `1.12.0` | | DeepThink | off | | Device | iOS Simulator | | Bluesky version | `1.122.0` | | Element iOS version | `1.11.40` | | Number of tasks | 60 per model | | AppControlBench project | The Midscene adapter delegated Agent execution to `@midscene/ios` while AppControlBench retained task reset, Simulator lifecycle, final screenshot capture, and grading. | | Evaluation rules | The original target states and judge rules were retained; the action intent for 1 case was calibrated. | ## Case Intent Calibration For this evaluation, the action-route description of 1 case was calibrated against the actual UI of the pinned Element iOS version. The calibration only removes a stale or unavailable UI-entry description; it does not change the final target state or judge rules of the case. | Case | Before | After | Reason for change | | --- | --- | --- | --- | | `element-07` | Open the "Project Phoenix" room and open its room-options menu (the '...' in the header). | Open the "Project Phoenix" room info/options screen. | The pinned Element iOS version does not provide a header `...`; only the action route was calibrated, while the target state and judge rules remain unchanged. | ## Report files --- url: /automate-with-scripts-in-yaml.md --- # Automate with scripts in YAML In most cases, developers write automation scripts just to perform some smoke tests, like checking for the appearance of certain content or verifying that a key user path is accessible. In such situations, maintaining a large test project is unnecessary. Midscene offers a way to perform automation using `.yaml` files, which helps you focus on the script itself rather than the testing framework. This allows any team member to write automation scripts without needing to learn any API. Here is an example. By reading its content, you should be able to understand how it works. ```yaml page: url: https://www.bing.com tasks: - name: Search for weather flow: - ai: Search for "today's weather" - sleep: 3000 - name: Check results flow: - aiAssert: The results show weather information ``` :::info Sample Project You can find a sample project that uses YAML scripts for automation here: * [Web](https://github.com/web-infra-dev/midscene-example/tree/main/yaml-scripts-demo) * [Android](https://github.com/web-infra-dev/midscene-example/tree/main/android/yaml-scripts-demo) * [Computer (Mac/Windows/Linux)](https://github.com/web-infra-dev/midscene-example/tree/main/computer/yaml-scripts-demo) ::: ## Set up API keys for model The following example uses `qwen3.7-plus` through Alibaba Cloud: ```bash export MIDSCENE_MODEL_BASE_URL="https://dashscope.aliyuncs.com/compatible-mode/v1" export MIDSCENE_MODEL_API_KEY="your-api-key" export MIDSCENE_MODEL_NAME="qwen3.7-plus" export MIDSCENE_MODEL_FAMILY="qwen3" ``` Replace `your-api-key` with your API Key. > To use another model, such as Doubao, GLM, Gemini, or GPT-5, see [Supported models and setup](/model-common-config.md). For all configuration options, see [Model configuration](/model-config.md). To execute YAML workflows from the command line, install the Midscene CLI. See [YAML script runner](/yaml-script-runner.md) for installation guidance, `.env` usage, and details on the `midscene` runner. ## Script file structure Script files use YAML format to describe automation tasks. It defines the target to be manipulated (like a webpage or an Android app) and the series of steps to perform. A standard `.yaml` script file includes a `page`, `browser`, `web`, `android`, `ios`, `harmony`, or `computer` section to configure the environment, an optional `agent` section to configure AI agent behavior, and a `tasks` section to define the automation tasks. ```yaml page: url: https://www.bing.com # The tasks section defines the series of steps to be executed tasks: - name: Search for weather flow: - ai: Search for "today's weather" - sleep: 3000 - aiAssert: The results show weather information ``` Use `page:` for a page-level Agent. Use `browser:` when one Agent should manage a browser and its active page. `web:` remains supported as a compatibility entry; `web.mode: browser` maps to BrowserAgent and plain `web:` maps to PageAgent. Do not combine `page`, `browser`, `web`, or the deprecated `target` in the same script. ### The `agent` part The `agent` section configures AI agent behavior and test report options. All fields are optional. ```yaml # AI agent configuration agent: # Test identifier, used for reporting and cache identification, optional testId: # Report group name, optional groupName: # Report group description, optional groupDescription: # Whether to generate test reports, optional, defaults to true generateReport: # Whether to automatically print report messages, optional, defaults to true autoPrintReportMsg: # Custom report file name, optional reportFileName: # Maximum AI replanning cycle limit, optional, defaults to 20 (40 for UI-TARS model) replanningCycleLimit: # Background knowledge to send to the AI model when calling aiAct, optional aiActContext: # Legacy alias (aiActionContext) remains for backward compatibility, but avoid using it in new scripts # Cache configuration, optional cache: # Cache strategy, optional, values: 'read-only' | 'read-write' | 'write-only' strategy: # Cache ID, required id: ``` :::info Agent Configuration Notes * **Applicable environments**: Web, iOS, and Android environments all support `agent` configuration * **testId priority**: CLI parameter > YAML agent.testId > filename * **aiActContext**: Provides background knowledge to the AI model, like how to handle popups, business introduction, etc. A legacy alias remains for backward compatibility (see inline comment) but should not be used in new scripts. * **Cache configuration**: For detailed usage, refer to the [Caching documentation](/caching.md) ::: #### Usage example ```yaml # agent configuration, applies to all environments agent: testId: "checkout-test" groupName: "E2E Test Suite" groupDescription: "Complete checkout flow testing" generateReport: true autoPrintReportMsg: false reportFileName: "checkout-report" replanningCycleLimit: 30 aiActContext: "If any popup appears, click agree. If login page appears, skip it." cache: id: "checkout-cache" strategy: "read-write" # iOS environment configuration ios: launch: https://www.bing.com wdaPort: 8100 # Or Android environment configuration android: deviceId: s4ey59 launch: https://www.bing.com tasks: - name: Search for weather flow: - ai: Search for "today's weather" - aiAssert: The results show weather information ``` ### The web target part Recommended page-level target: ```yaml page: url: https://example.com ``` Recommended browser-level target: ```yaml browser: url: https://example.com autoFollowNewPage: true ``` Compatibility form: ```yaml web: mode: browser url: https://example.com autoFollowNewPage: true ``` Shared options: ```yaml page: # The URL to visit, required. If `serve` is provided, provide the relative path. url: # Serve a local path as a static server, optional. serve: # The browser user agent, optional. userAgent: # The browser viewport width, optional, defaults to 1440. viewportWidth: # The browser viewport height, optional, defaults to 800. viewportHeight: # The browser's device pixel ratio, optional, defaults to the system's value. deviceScaleFactor: # Path to a JSON format browser cookie file, optional. cookie: # Chrome download directory (Puppeteer only), optional. # Relative paths are resolved from the current working directory. # Not supported in bridge mode. downloadPath: # Extra HTTP headers sent with every request (Puppeteer only), optional. # Useful when the server validates custom request headers. # Values must be strings; quote values YAML would treat as a boolean or number, e.g. "true". extraHTTPHeaders: X-Custom-Token: my-token Accept-Language: en-US # The strategy for waiting for network idle in Puppeteer mode, optional. # `timeout` applies to the initial opening of `web.url` defined in YAML and to later actions such as `aiTap` and `aiInput`. # `continueOnNetworkIdleError` only applies to the initial opening of `web.url` defined in YAML. waitForNetworkIdle: # The timeout in milliseconds for each network idle wait, optional, defaults to 2000ms. timeout: # Whether to continue if the initial opening of `web.url` defined in YAML times out while waiting for network idle, optional, defaults to true. # Later action-time waits always continue even if they time out. continueOnNetworkIdleError: # The path to the JSON file for outputting aiQuery/aiAssert results, optional. output: # Whether to save log content to a JSON file, optional, defaults to `false`. If true, saves to `unstableLogContent.json`. If a string, saves to the specified path. The log content structure may change in the future. unstableLogContent: # Whether to restrict page navigation to the current tab, optional, defaults to true. # Page mode only. Do not use it with `browser:` or `web.mode: browser`. forceSameTabNavigation: # Whether BrowserAgent should automatically continue in newly opened pages, optional, defaults to false. # Browser mode only. Use `browser:` or `web.mode: browser`. autoFollowNewPage: # CDP endpoint, optional. Connects to an existing browser instance via CDP instead of launching a new one. # Mutually exclusive with bridgeMode. cdpEndpoint: ws://localhost:9222/devtools/browser # The bridge mode, optional, defaults to false. Can be 'newTabWithUrl' or 'currentTab'. See below for more details. bridgeMode: false | 'newTabWithUrl' | 'currentTab' # Whether to close newly created tabs when the bridge disconnects, optional, defaults to false. closeNewTabsAfterDisconnect: # Whether to ignore HTTPS certificate errors, optional, defaults to false. acceptInsecureCerts: # Custom Chrome launch arguments (Puppeteer only, not supported in bridge mode), optional. # Use this to customize Chrome browser behavior, such as disabling third-party cookie blocking. # ⚠️ Security Warning: Some arguments (e.g., --no-sandbox, --disable-web-security) may reduce browser security. # Use only in controlled testing environments. chromeArgs: - '--disable-features=ThirdPartyCookiePhaseout' - '--disable-features=SameSiteByDefaultCookies' - '--window-size=1920,1080' ``` ### The `android` part ```yaml android: # The device ID, optional, defaults to the first connected device. deviceId: # The launch URL, optional, defaults to the device's current page. launch: # The path to the JSON file for outputting aiQuery/aiAssert results, optional. output: # All other options supported by the AndroidDevice constructor # For example: androidAdbPath, remoteAdbHost, remoteAdbPort, # imeStrategy, displayId, autoDismissKeyboard, keyboardDismissStrategy, # keyboardTypeDelay, minScreenshotBufferSize, alwaysRefreshScreenInfo, etc. # See the AndroidDevice constructor documentation for the complete list ``` :::info View Complete Android Configuration Options YAML scripts now support all configuration options from the `AndroidDevice` constructor. For the complete list of options, see [`AndroidDevice`](/reference.md#androiddevice) in the Android API reference. ::: #### Android Platform-Specific Actions **`runAdbShell` - Execute ADB Shell Commands** Execute ADB shell commands on Android devices. Pass only the shell command itself, without the `adb shell` prefix. To set a command timeout, keep `timeout` as a shallow sibling of `runAdbShell`. ```yaml android: deviceId: 'test-device' tasks: - name: Clear app data flow: - runAdbShell: 'pm clear com.example.app' - name: Get battery info flow: - runAdbShell: 'dumpsys battery' - name: Tap screen coordinates flow: - runAdbShell: 'input tap 100 200' - name: Run command with timeout flow: - runAdbShell: 'dumpsys activity services' timeout: 60000 ``` **Common ADB Shell Commands:** * `pm clear ` - Clear app data * `dumpsys battery` - Get battery information * `dumpsys window` - Get window information * `settings get secure android_id` - Get device ID * `input tap ` - Tap screen coordinates * `input keyevent ` - Send key events **`launch` - Launch App or URL** Launch Android apps or open URLs. ```yaml android: deviceId: 'test-device' tasks: - name: Launch Settings app flow: - launch: com.android.settings - name: Open webpage flow: - launch: https://www.example.com ``` **`terminate` - Terminate App** Terminate (force-stop) a running Android app by package name. ```yaml android: deviceId: 'test-device' tasks: - name: Terminate Settings app flow: - terminate: com.android.settings ``` ### The `ios` part ```yaml ios: # WebDriverAgent port, optional, defaults to 8100. wdaPort: # WebDriverAgent host address, optional, defaults to localhost. wdaHost: # Whether to auto dismiss keyboard, optional, defaults to false. autoDismissKeyboard: # Launch URL or app bundle ID, optional, defaults to the device's current page. launch: # The path to the JSON file for outputting aiQuery/aiAssert results, optional. output: # Whether to save log content to a JSON file, optional, defaults to `false`. If true, saves to `unstableLogContent.json`. If a string, saves to the specified path. The log content structure may change in the future. unstableLogContent: # All other options supported by the IOSDevice constructor # See the IOSDevice constructor documentation for the complete list ``` :::info View Complete iOS Configuration Options YAML scripts now support all configuration options from the `IOSDevice` constructor. For the complete list of options, see [`IOSDevice`](/reference.md#iosdevice) in the iOS API reference. ::: #### iOS Platform-Specific Actions **`runWdaRequest` - Execute WebDriverAgent API Requests** Execute WebDriverAgent API requests directly on iOS devices. ```yaml ios: launch: 'com.apple.mobilesafari' tasks: - name: Press home button via WDA flow: - runWdaRequest: method: POST endpoint: /session/test/wda/pressButton data: name: home - name: Get device information flow: - runWdaRequest: method: GET endpoint: /wda/device/info ``` **Parameters:** * `method` (string, required): HTTP method (GET, POST, DELETE, etc.) * `endpoint` (string, required): WebDriverAgent API endpoint * `data` (any, optional): Request body data **Common WebDriverAgent Endpoints:** * `/wda/screen` - Get screen information * `/wda/device/info` - Get device information * `/session/{sessionId}/wda/pressButton` - Press hardware buttons * `/session/{sessionId}/wda/apps/launch` - Launch apps * `/session/{sessionId}/wda/apps/terminate` - Terminate apps * `/session/{sessionId}/wda/apps/activate` - Activate apps **`launch` - Launch App or URL** Launch iOS apps or open URLs. ```yaml ios: wdaPort: 8100 tasks: - name: Launch Settings app flow: - launch: com.apple.Preferences - name: Open webpage flow: - launch: https://www.example.com ``` **`terminate` - Terminate App** Terminate (close) a running iOS app by its bundle ID. ```yaml ios: wdaPort: 8100 tasks: - name: Terminate Settings app flow: - terminate: com.apple.Preferences ``` ### The `harmony` part The `harmony` section is used for HarmonyOS device automation via HDC. The device is connected with `hdc`, so make sure `hdc list targets` reports your device before running. ```yaml harmony: # The HarmonyOS device ID to connect to, optional, defaults to the first connected device. deviceId: # The app to launch, optional, defaults to the device's current screen. launch: # The path to the HDC executable, optional. hdcPath: # Whether to auto dismiss keyboard after input, optional, defaults to false. autoDismissKeyboard: # Custom mapping of app names to bundle names or explicit bundle/ability targets, optional. # User-provided mappings take precedence over defaults. appNameMapping: : # The path to the JSON file for outputting aiQuery/aiAssert results, optional. output: # All other options supported by the HarmonyDevice constructor ``` #### HarmonyOS Platform-Specific Actions **`launch` - Launch App** Launch a HarmonyOS app by its bundle name or an explicit `bundle/Ability` target. ```yaml harmony: deviceId: 'test-device' appNameMapping: Video: com.example.video/PhoneAbility tasks: - name: Launch app flow: - launch: Video ``` **`terminate` - Terminate App** Terminate (force-stop) a running HarmonyOS app by bundle name or mapped app name. ```yaml harmony: deviceId: 'test-device' tasks: - name: Terminate app flow: - terminate: com.example.app ``` **`runHdcShell` - Execute HDC Shell Command** Execute an HDC shell command on the HarmonyOS device. ```yaml harmony: deviceId: 'test-device' tasks: - name: Dump window info flow: - runHdcShell: command: 'hidumper -s WindowManagerService -a' ``` ### The `computer` part The `computer` section is used for PC desktop automation. It allows you to control the desktop environment, including mouse movements, keyboard inputs, and screen queries. ```yaml computer: # The display ID to use, optional, defaults to the primary display. displayId: # The path to the JSON file for outputting aiQuery/aiAssert results, optional. output: ``` #### Usage example ```yaml computer: {} tasks: - name: Open browser and search flow: - aiAct: press Cmd+Space - sleep: 500 - aiAct: type "Safari" and press Enter - sleep: 2000 - aiAct: press Cmd+L to focus address bar - aiAct: type "https://www.bing.com" - aiAct: press Enter - sleep: 3000 - aiAct: type "weather today" in the search box and press Enter - aiAssert: The results show weather information ``` :::info Platform Notes The demo scripts above use macOS commands. For Windows, modify the scripts to use: * `press Windows key` instead of `press Cmd+Space` * `type "Chrome"` instead of `type "Safari"` * `press Ctrl+L` instead of `press Cmd+L` ::: ### The `tasks` part The `tasks` part is an array that defines the steps of the script. Remember to add a `-` before each step to indicate it's an array item. The interfaces in the `flow` section are almost identical to the [API](/reference.md#common), with some differences in parameter nesting levels. ```yaml tasks: - name: continueOnError: # Optional, whether to continue to the next task on error, defaults to false. flow: # Auto Planning (.ai) # ---------------- # Perform an interaction. `ai` is a shorthand for `aiAct`. - ai: cacheable: # Optional, whether to cache the result of this API call when the [caching feature](./caching.mdx) is enabled. Defaults to True. deepThink: # Optional, guide aiAct to focus on task decomposition and separate planning from UI element locating. Defaults to false. deepLocate: # Optional, enable Deep Locate for UI element locating during aiAct execution. Defaults to false. # This usage is the same as `ai`. # Note: In earlier versions, this was also written as `aiAction`. The current version supports both names. - aiAct: cacheable: # Optional, whether to cache the result of this API call when the [caching feature](./caching.mdx) is enabled. Defaults to True. deepThink: # Optional, guide aiAct to focus on task decomposition and separate planning from UI element locating. Defaults to false. deepLocate: # Optional, enable Deep Locate for UI element locating during aiAct execution. Defaults to false. # Instant Action (.aiTap, .aiHover, .aiInput, .aiKeyboardPress, .aiScroll) # ---------------- # Tap an element described by a prompt. - aiTap: deepLocate: # Optional, whether to enable Deep Locate for this element. Defaults to False. xpath: # Optional, the xpath of the target element for the operation. If provided, Midscene will prioritize this xpath to find the element before using the cache and the AI model. Defaults to empty. cacheable: # Optional, whether to cache the result of this API call when the [caching feature](./caching.mdx) is enabled. Defaults to True. fileChooserAccept: | [, ] # Optional, file path(s) to upload when the tap triggers a file chooser; only available on web # Hover over an element described by a prompt. - aiHover: deepLocate: # Optional, whether to enable Deep Locate for this element. Defaults to False. xpath: # Optional, the xpath of the target element for the operation. If provided, Midscene will prioritize this xpath to find the element before using the cache and the AI model. Defaults to empty. cacheable: # Optional, whether to cache the result of this API call when the [caching feature](./caching.mdx) is enabled. Defaults to True. # Input text into an element described by a prompt. - aiInput: # The element to input text into. value: deepLocate: # Optional, whether to enable Deep Locate for this element. Defaults to False. xpath: # Optional, the xpath of the target element for the operation. If provided, Midscene will prioritize this xpath to find the element before using the cache and the AI model. Defaults to empty. cacheable: # Optional, whether to cache the result of this API call when the [caching feature](./caching.mdx) is enabled. Defaults to True. # Press a key (e.g., Enter, Tab, Escape) on an element described by a prompt. - aiKeyboardPress: # The element to press the key on. keyName: deepLocate: # Optional, whether to enable Deep Locate for this element. Defaults to False. xpath: # Optional, the xpath of the target element for the operation. If provided, Midscene will prioritize this xpath to find the element before using the cache and the AI model. Defaults to empty. cacheable: # Optional, whether to cache the result of this API call when the [caching feature](./caching.mdx) is enabled. Defaults to True. # Scroll globally or on an element described by a prompt. - aiScroll: # Optional, the element to scroll on. scrollType: 'singleAction' # or 'scrollToBottom' | 'scrollToTop' | 'scrollToRight' | 'scrollToLeft'. Defaults to 'singleAction'. direction: 'down' # or 'up' | 'left' | 'right'. Defaults to 'down'. Only effective when scrollType is singleAction. distance: # Optional, the scroll distance in pixels. Use null to let Midscene decide automatically. deepLocate: # Optional, whether to enable Deep Locate for this element. Defaults to False. xpath: # Optional, the xpath of the target element for the operation. If provided, Midscene will prioritize this xpath to find the element before using the cache and the AI model. Defaults to empty. cacheable: # Optional, whether to cache the result of this API call when the [caching feature](./caching.mdx) is enabled. Defaults to True. # Log the current screenshot with a description in the report file. - recordToReport: # Optional, the title of the screenshot. If not provided, the title will be 'untitled'. content: <content> # Optional, the description of the screenshot. # Data Extraction # ---------------- # Perform a query that returns a JSON object. - aiQuery: <prompt> # Remember to describe the format of the result in the prompt. name: <name> # The key for the query result in the JSON output. # More APIs # ---------------- # Wait for a condition to be met, with a timeout (in ms, optional, defaults to 30000). - aiWaitFor: <prompt> timeout: <ms> # Perform an assertion. - aiAssert: <prompt> errorMessage: <error-message> # Optional, the error message to print if the assertion fails. name: <name> # Optional, give the assertion a name, which will be used as a key in the JSON output. - aiBoolean: <prompt> name: <name> # Optional, give the boolean result a name for JSON output. # Run one Gherkin Scenario. This is a Beta feature available starting in Midscene 1.10. - runGherkinScenario: | Scenario: Add a todo item Given the todo page is open When I add a todo item named "Buy milk" Then the todo list should contain "Buy milk" # Wait for a specified amount of time. - sleep: <ms> # Execute a piece of JavaScript code in the web page context. - javascript: <javascript> name: <name> # Optional, assign a name to the return value, which will be used as a key in the JSON output. - name: <name> flow: # ... ``` `runGherkinScenario` is a Beta feature available starting in Midscene 1.10. For supported rules and limitations, see [BDD-style scripts with Gherkin](/advanced/bdd-style-scripts-with-gherkin.md). #### Step Result Names Steps that write a result with `name` save that value into the YAML run result and the JSON output. Use `name` to label values that should appear in the run output. ```yaml tasks: - name: Save extracted data flow: - aiString: Get the product id shown on the page name: product_id - aiQuery: Get the search result after submitting the product id name: search_result ``` #### Upload Files With `aiTap` When clicking a button opens a file chooser, you can set `fileChooserAccept` directly on the `aiTap` step. It accepts either a single path or an array of paths. ```yaml tasks: - name: upload single file flow: - aiTap: Choose file button fileChooserAccept: ./fixtures/document.pdf - name: upload multiple files flow: - aiTap: Upload images button fileChooserAccept: - ./fixtures/image1.jpg - ./fixtures/image2.png ``` If you are already using a `locate` object for `prompt`, `images`, or other locate options, keep `fileChooserAccept` at the same level as `locate`. Do not nest it inside `locate` or inside the `aiTap` object: ```yaml tasks: - name: upload file with locate flow: - aiTap: locate: prompt: Click the upload button fileChooserAccept: ./fixtures/document.pdf ``` Notes: * `fileChooserAccept` is only available for web pages (Playwright, Puppeteer, or Chrome extension Bridge mode). * Relative paths are resolved from the current command working directory, not from the YAML file directory. * If a file does not exist, the script throws before the tap is executed. * In Chrome extension Bridge mode, local file uploads require the Midscene extension's "Allow access to file URLs" permission. Enable it in `chrome://extensions` > Midscene > "Details", then reconnect Bridge mode from the target `http(s)://` page. * Chrome extension Bridge mode does not support directory upload inputs (`webkitdirectory` / `directory`). Use Playwright for directory uploads. #### Prompting with images For steps whose prompt accepts images, you can attach images to the prompt by setting the `images` field to an array of objects, each containing a `name` and a `url`. (see the [API reference](/reference.md#prompting-with-images)), replace the string value with an object that contains: * `prompt`: The text prompt. * `images`: (Optional) The reference images used in the prompt. Each image needs a `name` and a `url`. * `convertHttpImage2Base64`: (Optional) Converts HTTP image links to Base64 before sending them to the model, which is useful when the link is not publicly accessible. Image URLs can be local paths, Base64 strings, or remote links. When using image links that cannot be accessed for the model, set `convertHttpImage2Base64: true` so Midscene will download the image and send the base64 string to the model. For interactions like `aiTap`, `aiHover`, `aiDoubleClick`, `aiRightClick`, put the text and images in the `locate` field as a sibling of the action key. ```yaml tasks: - name: Verify branding flow: - aiHover: locate: prompt: Move the cursor to the region containing the GitHub logo. images: - name: GitHub logo url: https://github.githubassets.com/assets/GitHub-Mark-ea2971cee799.png convertHttpImage2Base64: true - aiTap: locate: prompt: Tap the region containing the GitHub logo. images: - name: GitHub logo url: https://github.githubassets.com/assets/GitHub-Mark-ea2971cee799.png convertHttpImage2Base64: true ``` > The legacy nested format (where `locate` is indented under the action key, e.g. `aiTap: \n locate: ...`) is still supported but not recommended. For `aiAct` (and its `ai` shorthand), and for insight steps like `aiAsk`, `aiQuery`, `aiBoolean`, `aiNumber`, `aiString`, and `aiAssert`, you can set the `prompt` and `images` fields directly under the action key. ```yaml tasks: - name: Verify branding flow: - aiAssert: prompt: Check whether the image appears on the page. images: - name: target logo url: https://github.githubassets.com/assets/GitHub-Mark-ea2971cee799.png convertHttpImage2Base64: true ``` ## Notes ### `agent.runYaml` only parses the `tasks` field When using the `agent.runYaml()` API, only the `tasks` field in the YAML file is parsed and executed. At that point the agent has already been initialized in the JS script, so it cannot be reinitialized based on the `agent` configuration in the YAML file. --- url: /awesome-midscene.md --- # Awesome Midscene A curated list of community projects that extend Midscene.js capabilities across different platforms and programming languages. ## Community projects ### iOS automation * **[midscene-ios](https://github.com/lhuanyu/midscene-ios)** - iOS Mirror automation support for Midscene * Enables automated testing and interaction with iOS applications * Extends Midscene's cross-platform capabilities to Apple's mobile ecosystem ### PC automation * **[midscene-pc](https://github.com/Mofangbao/midscene-pc)** - PC operation device for Windows, macOS, and Linux * Enables automated testing and interaction with desktop applications across all major platforms * Supports both local and remote operation capabilities * **[midscene-pc-docker](https://github.com/Mofangbao/midscene-pc-docker)** - Docker container image with Midscene-PC server pre-installed * Based on Ubuntu 20 with GNOME desktop for maximum application compatibility * Includes built-in VNC service for browser-based desktop monitoring * Deploy automation client directly on standard servers with a single command ### Python SDK * **[Midscene-Python](https://github.com/Python51888/Midscene-Python)** - Python SDK for Midscene automation * Brings Midscene's AI-powered automation capabilities to Python developers * Allows integration with existing Python testing and automation workflows ### Java SDK * **[midscene-java](https://github.com/Master-Frank/midscene-java)** by @Master-Frank - Java SDK for Midscene automation * Offers a JVM-friendly way to script Midscene experiences similar to the Python SDK * Fits easily into existing Java automation or testing pipelines * **[midscene-java](https://github.com/alstafeev/midscene-java)** by @alstafeev - Java SDK for Midscene automation * Provides a JVM-native interface for scripting Midscene * Integrates seamlessly into established Java testing frameworks and automation workflows ## Contributing Have you created a project that extends Midscene.js? We'd love to feature it here! To add your project to this list, please submit an issue to the [Midscene repository](https://github.com/web-infra-dev/midscene), and tell us your awesome midscene project. ## Criteria for inclusion Projects featured in Awesome Midscene should: * Extend or integrate with Midscene.js functionality * Be actively maintained * Have clear documentation and usage examples * Provide value to the Midscene community *** *Don't see your favorite platform or language supported yet? Consider creating a community project or contributing to existing ones!* --- url: /basics.md --- # The Basics This page provides an overview of Midscene's core concepts, including Agent usage, architecture, abstractions, and capability boundaries. You will learn how an Agent connects AI models to a target interface and how to perform interactions, assertions, and other operations. :::info You can try every API introduced below in the Playground without writing code and see the results immediately. See [Quick Start](./quick-start#chrome-extension) to get started. ::: ## Plan and interact ### `aiAct` [`aiAct`](./reference/#agentaiact) accepts a goal described in natural language. It observes the interface, plans the next steps, locates the target elements, and executes the actions until the goal is complete. The prompt can also include assertions. Midscene verifies these assertions during execution and throws an error promptly if one fails. `aiAct` is flexible and autonomous, so it works well when a task has multiple steps, conditional branches, or an uncertain execution path. During execution, `aiAct` continuously uses AI to plan from the latest interface state, which means it usually takes more time and tokens than an instant interaction. Typical usage: ```typescript await agent.aiAct( 'Search for headphones, add the first item to the cart, and confirm that the cart count changes to 1', ); ``` To give every subsequent `aiAct` call more business context, use [`agent.setAIActContext()`](./reference/#agentsetaiactcontext): ```typescript agent.setAIActContext( 'Close the cookie consent dialog first if it appears. Prices are shown in USD.', ); ``` `aiAct` provides the following per-call options: - `deepThink`: focuses more on task decomposition and uses separate model calls for planning and element localization. It can make complex tasks more stable, but increases model calls and latency. - `deepLocate`: uses an additional model call to improve element localization accuracy. Enable it when a target is small or difficult to distinguish from nearby elements. - `context`: provides business knowledge or other background for this call only. For `aiAct`, it overrides the Agent-level `aiActContext`, including when it is explicitly set to an empty string. ```typescript await agent.aiAct('Complete the checkout form and stop before placing the order', { deepThink: true, deepLocate: true, context: 'If an address confirmation dialog appears, select the default shipping address.', }); ``` ## Instant interactions Instant interaction APIs perform one specified action. Their main job is to locate a UI element and execute a fixed operation on it. These APIs do not plan a sequence of steps. A request such as “close the popup if it appears, then click the checkout button” should use `aiAct`; an instant interaction treats its prompt as a description of the target element, not as a workflow. ### `aiTap` [`aiTap`](./reference/#agentaitap) locates and taps or clicks one element. Typical usage: ```typescript await agent.aiTap('The checkout button in the shopping cart'); ``` When the target is small or visually ambiguous, enable `deepLocate` (multi-pass deep localization): ```typescript await agent.aiTap('The cart icon in the upper-right corner', { deepLocate: true, }); ``` ### `aiInput` [`aiInput`](./reference/#agentaiinput) locates an input field and enters a specified value. Its default `replace` mode clears the existing content before entering the new value. Typical usage: ```typescript await agent.aiInput('The email address input', { value: 'user@example.com', }); ``` Other input modes are `typeOnly`, which preserves the existing content, and `clear`, which only clears the field. Other instant interaction APIs include `aiHover`, `aiClearInput`, `aiKeyboardPress`, `aiScroll`, `aiPinch`, `aiLongPress`, `aiDoubleClick`, and `aiRightClick`. Platform support varies; see [Planning and interaction](./reference/#planning-interaction) in the API reference. ## Insight Insight APIs observe the interface and return an analysis result without interacting with it. They use the current screenshot by default. On web pages, you can also pass `domIncluded` when the task requires DOM information that is not visible in the screenshot. ### `aiAssert` [`aiAssert`](./reference/#agentaiassert) checks a condition described in natural language. It resolves when the condition is true. When the condition is false, it throws an error that includes the reason returned by the model. Typical usage: ```typescript await agent.aiAssert('The shopping cart contains one item and shows a subtotal'); ``` ### `aiQuery` [`aiQuery`](./reference/#agentaiquery) extracts structured data from the interface. Describe both the required data and its expected type or shape in the prompt. Typical usage: ```typescript const items = await agent.aiQuery< Array<{ name: string; price: number }> >('The products in the shopping cart, {name: string, price: number}[]'); // Example items value: [{ name: 'Wireless headphones', price: 99.9 }] ``` ### `aiBoolean` [`aiBoolean`](./reference/#agentaiboolean) answers a question about the interface and returns a boolean. Typical usage: ```typescript const loginDialogVisible = await agent.aiBoolean( 'Is the login dialog visible?', ); // Example loginDialogVisible value: true ``` Related convenience methods include [`aiNumber`](./reference/#agentainumber) for numbers and [`aiString`](./reference/#agentaistring) or [`aiAsk`](./reference/#agentaiask) for strings. ## Orchestrate workflows with JavaScript {#javascript-orchestration} Midscene offers two basic ways to orchestrate automation: use `aiAct` or use JavaScript orchestration. The following code shows both approaches completing the same task. Using `aiAct`: ```typescript await agent.aiAct( 'Check every record in the list and mark any incomplete record as completed', ); ``` Using JavaScript orchestration: ```typescript const recordNames = await agent.aiQuery<string[]>('All record names in the list'); for (const recordName of recordNames) { const completed = await agent.aiBoolean( `Is the record named "${recordName}" marked as completed?`, ); if (!completed) { await agent.aiTap(`The record named "${recordName}"`); } } ``` As the examples above show, in `aiAct` mode, the Agent plans the execution path. With JavaScript orchestration, developers must define conditions, loops, and step order in code. JavaScript orchestration has an explicit execution path, allowing developers to control the behavior of each branch precisely. This makes the approach highly deterministic. However, JavaScript orchestration can respond only to UI changes already handled in code. For example, a resolution change may require additional scrolling. If the code does not handle such changes, the workflow will fail. Use the following guidelines when choosing an orchestration approach: 1. Use `aiAct` by default to execute operation goals. The Agent determines the specific steps based on the latest interface state, so it can better adapt to UI changes. 2. Use JavaScript orchestration only when the operation flow is well understood and stable. 3. If JavaScript orchestration code becomes increasingly difficult to maintain, or its success rate continues to decline, switch to `aiAct`. --- url: /blog-introducing-instant-actions-and-deep-think.md --- # Introducing Instant Actions and Deep Think From Midscene v0.14.0, we have introduced two new features: Instant Actions and Deep Think. ## Instant Actions - a more predictable way to perform actions You may have already been familiar with our `.ai` interface. It's an auto-planning interface to interact with web pages. For example, when performing a search, you can do this: ```typescript await agent.ai('type "Headphones" in search box, hit Enter'); ``` Behind the scene, Midscene will call the LLM to plan the steps and execute them. You can see the report file to see the process. It's a very common way for AI agents to these kinds of tasks. ![](/blog/report-planning.png) In the meantime, there are many testing engineers who want a faster way to perform actions. When using AI models with complex prompts, some of the LLM models may find it hard to plan the proper steps, or the coordinates of the elements may not be accurate. It could be frustrating for debugging the unpredictable process. To solve this problem, we have introduced the `aiTap()`, `aiHover()`, `aiInput()`, `aiKeyboardPress()`, `aiScroll()` interfaces. They are call the **"instant actions"**. These interfaces will directly perform the specified action as the interface name suggests, while the AI model is responsible for the easier tasks such as locating elements. The whole process can be obviously faster and more reliable after using them. For example, the search action above can be rewritten as: ```typescript await agent.aiInput('Headphones', 'search-box'); await agent.aiKeyboardPress('Enter'); ``` The typical workflow in the report file is like this, as you can see there is no planning process in the report file: ![](/blog/report-instant-action.png) The scripts with instant actions seems a little bit redundant (or not 'ai-style'), but we believe these structured interfaces are a good way to save time debugging when the action is already clear. ## Deep Think - a more accurate way to locate elements When using Midscene with some complex widgets, the LLM may find it hard to locate the target element. We have introduced a new option named `deepThink` to the instant actions. The signature of the instant actions with `deepThink` is like this: ```typescript await agent.aiTap('target', { deepThink: true }); ``` `deepThink` is a strategy of locating elements. It will first find an area that contains the target element, then "focus" on this area to search the element again. By this way, the coordinates of the target element will be more accurate. Let's take the workflow editor page of Coze.com as an example. There are many customized icons on the sidebar. This is usually hard for LLMs to distinguish the target element from its surroundings. ![](/blog/coze-sidebar.png) After using `deepThink` in instant actions, the yaml scripts will be like this (of course, you can also use the javascript interface): ```yaml tasks: - name: edit input panel flow: - aiTap: the triangle icon on the left side of the text "Input" deepThink: true - aiTap: the first checkbox in the Input form deepThink: true - aiTap: the expand button on the second row of the Input form (on the right of the checkbox) deepThink: true - aiTap: the delete button on the second last row of the Input form deepThink: true - aiTap: the add button on the last row of the Input form （second button from the right） deepThink: true ``` By viewing the report file, you can see Midscene has found every target element in the area. ![](/blog/report-coze-deep-think.png) Just like the example above, the highly-detailed prompt for `deepThink` is the key to keeping results stable. `deepThink` is only available with the models that support visual grounding like qwen2.5-vl. If you are using LLM models like gpt-4o, it won't work. --- url: /bridge-mode.md --- # Bridge mode by Chrome extension import { PackageManagerTabs } from '@theme'; The bridge mode in the Midscene Chrome extension is a tool that allows you to use local scripts to control the desktop version of Chrome. Your scripts can connect to either a new tab or the currently active tab. Using the desktop version of Chrome allows you to reuse all cookies, plugins, page status, and everything else you want. You can work with automation scripts to complete your tasks. This mode is commonly referred to as 'man-in-the-loop' in the context of automation. ![bridge mode](/midscene-bridge-mode.png) :::info Demo Project check the demo project of bridge mode: [https://github.com/web-infra-dev/midscene-example/blob/main/bridge-mode-demo](https://github.com/web-infra-dev/midscene-example/blob/main/bridge-mode-demo) ::: ## Set up API keys for model The following example uses `qwen3.7-plus` through Alibaba Cloud: ```bash export MIDSCENE_MODEL_BASE_URL="https://dashscope.aliyuncs.com/compatible-mode/v1" export MIDSCENE_MODEL_API_KEY="your-api-key" export MIDSCENE_MODEL_NAME="qwen3.7-plus" export MIDSCENE_MODEL_FAMILY="qwen3" ``` Replace `your-api-key` with your API Key. > To use another model, such as Doubao, GLM, Gemini, or GPT-5, see [Supported models and setup](/model-common-config.md). For all configuration options, see [Model configuration](/model-config.md). > In bridge mode, the AI models configs should be set in the Node.js side instead of the browser side. ## Get started ### Step 1. Install Midscene extension from Chrome web store Install [Midscene extension from Chrome web store](https://chromewebstore.google.com/detail/midscene/gbldofcpkknbggpkmbdaefngejllnief) :::info File upload permission If your Bridge mode script needs to upload local files, open `chrome://extensions`, find the Midscene extension, click "Details", and turn on "Allow access to file URLs". Then switch back to the target `http(s)://` page before reconnecting Bridge mode. ::: ### Step 2. Install dependencies <PackageManagerTabs command="install @midscene/web tsx --save-dev" /> ### Step 3. Write scripts Write and save the following code as `./demo-new-tab.ts`. ```typescript import { AgentOverChromeBridge } from "@midscene/web/bridge-mode"; const sleep = (ms) => new Promise((r) => setTimeout(r, ms)); Promise.resolve( (async () => { const agent = new AgentOverChromeBridge(); // This will connect to a new tab on your desktop Chrome await agent.connectNewTabWithUrl("https://www.bing.com"); // these are the same as normal Midscene agent await agent.ai('type "AI 101" and hit Enter'); await sleep(3000); await agent.aiAssert("there are some search results"); await agent.destroy(); })() ); ``` ### Step 4. Run the script Run your scripts ```bash tsx demo-new-tab.ts ``` After running the script, the Chrome extension will pop up a confirmation dialog asking whether to allow the connection. Click "Allow" to allow the current connection, or click "Always Allow" to automatically allow all future connection requests (can be reset in the Bridge Mode panel). You should then see a new tab opened in your desktop Chrome, controlled by your scripts. <p align="center"> <img src="/bridge_in_extension.png" alt="bridge in extension" width="400" /> </p> :::info The extension listens for connection requests in the background by default, no manual action is needed. The extension icon will display a status badge: yellow dot for listening, green dot for connected. ::: ## Use bridge mode in YAML script [Yaml scripts](/automate-with-scripts-in-yaml.md) is a way for developers to write automation scripts in yaml format, which is easy to read and write comparing to javascript. To use bridge mode in yaml script, set the `bridgeMode` property in the `web` section. If you want to use the current tab, set it to `currentTab`, otherwise set it to `newTabWithUrl`. Set `closeNewTabsAfterDisconnect` to true if you want to close the newly created tabs when the bridge is destroyed. This is optional and the default value is false. For example, the following script will open a new tab by Chrome extension bridge: ```diff web: url: https://www.bing.com + bridgeMode: newTabWithUrl + closeNewTabsAfterDisconnect: true tasks: ``` Run the script: ```bash midscene ./bing.yaml ``` After the script starts, click "Allow" in the confirmation dialog to proceed. ### Unsupported options In bridge mode, these options will be ignored (they will follow your desktop browser's settings): * userAgent * viewportWidth * viewportHeight * deviceScaleFactor * waitForNetworkIdle * cookie * extraHTTPHeaders * downloadPath * chromeArgs ## Remote Access Configuration By default, the Bridge Server only listens on `127.0.0.1`, allowing only local Chrome extension connections. If you need cross-machine communication (e.g., server on machine A, browser on machine B), you can enable remote access: **Server Side (Node.js Script):** ```typescript // Enable remote access (recommended) const agent = new AgentOverChromeBridge({ allowRemoteAccess: true // Listen on 0.0.0.0:3766 }); // Or specify a specific network interface const agent = new AgentOverChromeBridge({ host: '192.168.1.100', // Listen on specific IP port: 3766 }); ``` **Client Side (Chrome Extension):** 1. Open the Chrome extension's Bridge Mode page 2. Fill in the server address in the "Bridge Server URL" input field * Local mode: `ws://localhost:3766` (default) * Remote mode: `ws://192.168.1.100:3766` (replace with your server IP) 3. Run your script and click "Allow" in the confirmation dialog <p align="center"> <img src="/bridge_remote_config.png" alt="bridge remote config" width="400" /> </p> :::warning Security Notice When remote access is enabled, the Bridge Server will be exposed on the network. Please ensure: * Only use in trusted network environments * Use firewall to restrict access * Do not use in public network environments to avoid security risks ::: ## FAQ * Where should I config the model parameters (like `MIDSCENE_MODEL_API_KEY`), in the browser or in the terminal? When using bridge mode, configure the model parameters in the terminal. For supported models and setup examples, see [Supported models and setup](/model-common-config.md). ## More * For every Agent method, check the [API reference](/reference.md#interaction-methods). * For the full Chrome Bridge API surface, see [API reference (Web)](/reference.md#chrome-bridge-agent). * Demo project * Bridge mode demo: [https://github.com/web-infra-dev/midscene-example/blob/main/bridge-mode-demo](https://github.com/web-infra-dev/midscene-example/blob/main/bridge-mode-demo) --- url: /caching.md --- # Caching AI Planning and DOM Localization Midscene supports caching two kinds of data: AI Planning steps and matched element localization information. The former can be used across automation workflows to reduce AI model calls and improve execution efficiency; for web automation, the DOM localization part (XPath) in the latter can significantly reduce repeated locating overhead, but it is currently web-only and has [certain limitations](#limitations-of-xpath-in-caching-element-location). **Effect** With caching hit, time cost is significantly reduced. For example, in the following case, execution time was reduced from 51 seconds to 28 seconds. * **before** ![](/cache/no-cache-time.png) * **after** ![](/cache/use-cache-time.png) ## Cache files and storage Midscene's caching mechanism is based on input stability and output reusability. When the same task instructions are repeatedly executed in similar page environments, Midscene will prioritize using cached results to avoid repeated AI model calls, significantly improving execution efficiency. The core caching mechanisms include: * **Task instruction caching**: For planning operations (such as `ai`, `aiAct`), Midscene uses the prompt instruction as the cache key to store the execution plan returned by AI * **Element location caching (web only)**: For location operations (such as `aiLocate`, `aiTap`), the system uses the location prompt as the cache key to store element XPath information, and verifies whether the XPath is still valid on the next execution * **Invalidation mechanism**: When cache becomes invalid, the system automatically falls back to AI model for re-analysis * **Plan cache fallback**: If a cached `aiAct` plan fails at runtime, for example because an optional popup no longer appears, Midscene falls back to normal AI planning for the current run and clears the stale cache record's flow * **Never cache query results**: The query results like `aiBoolean`, `aiQuery`, `aiAssert` will never be cached. When this fallback path is used, Midscene does not write the fallback-generated flow back to the original prompt's plan cache, even if the current run succeeds. The cached steps may have already changed the page state before failing, so the fallback plan may only describe the remaining work and may not be valid from the initial page state. On the next run with the same prompt, the empty flow is treated as an unavailable cache entry and Midscene regenerates a complete plan cache from the initial state. Cache contents will be saved in the `./midscene_run/cache` directory with the `.cache.yaml` as the extension name. ## Cache strategies By configuring the `cache` option, you can enable caching for your agent. ### Disable cache Configuration: `cache: false` or not configuring the `cache` option Completely disable cache functionality, always call AI model for every operation. Suitable when you need real-time results or for debugging. By default, if you don't configure the `cache` option, caching is disabled. ```javascript // Direct Agent creation const agent = new PuppeteerAgent(page, { cache: false, }); ``` ```yaml # YAML configuration agent: cache: false ``` ### Read-write mode Configuration: `cache: { id: "my-cache-id" }` or `cache: { strategy: "read-write", id: "my-cache-id" }` Automatically read existing cache and update cache files during execution. The default value of `strategy` is `read-write`. ```javascript // Direct Agent creation - explicit cache ID const agent = new PuppeteerAgent(page, { cache: { id: "my-cache-id" }, }); // Explicitly specify strategy const agent = new PuppeteerAgent(page, { cache: { strategy: "read-write", id: "my-cache-id" }, }); ``` ```yaml # YAML configuration - explicit cache ID agent: cache: id: "my-cache-test" # Explicitly specify strategy agent: cache: id: "my-cache-test" strategy: "read-write" ``` YAML mode also supports `cache: true` to automatically use the file name as the cache ID. ### Read-only, manual write Configuration: `cache: { strategy: "read-only", id: "my-cache-id" }` Only read cache, no automatic writing to cache files. Requires manual `agent.flushCache()` call to write cache files. Suitable for production environments to ensure cache consistency. ```javascript // Direct Agent creation const agent = new PuppeteerAgent(page, { cache: { strategy: "read-only", id: "my-cache-id" }, }); // Manual cache write required await agent.flushCache(); ``` ```yaml # YAML configuration agent: cache: id: "my-cache-test" strategy: "read-only" ``` ### Write-only mode Configuration: `cache: { strategy: "write-only", id: "my-cache-id" }` Only write to cache, do not read existing cache contents. Always call AI model for each execution and write results to cache file. Suitable for initially building cache or updating cache. ```javascript // Direct Agent creation const agent = new PuppeteerAgent(page, { cache: { strategy: "write-only", id: "my-cache-id" }, }); ``` ```yaml # YAML configuration agent: cache: id: "my-cache-test" strategy: "write-only" ``` ### Compatibility mode (not recommended) Configuration via `MIDSCENE_CACHE=1` environment variable with cacheId, equivalent to read-write mode. ```javascript // Old way, requires MIDSCENE_CACHE=1 environment variable and cacheId const agent = new PuppeteerAgent(originPage, { cacheId: 'puppeteer-swag-sab' }); ``` ```bash MIDSCENE_CACHE=1 tsx demo.ts ``` ## Using Playwright AI Fixture from `@midscene/web/playwright` When using `PlaywrightAiFixture` from `@midscene/web/playwright`, pass the same `cache` options to control caching behaviour. ### Disable cache ```typescript // fixture.ts in sample code export const test = base.extend<PlayWrightAiFixtureType>( PlaywrightAiFixture({ cache: false, }), ); ``` ### Read-write mode ```typescript // fixture.ts in sample code // Auto-generate cache ID from test metadata export const test = base.extend<PlayWrightAiFixtureType>( PlaywrightAiFixture({ cache: true, }), ); // fixture.ts in sample code // Explicitly provide cache ID export const test = base.extend<PlayWrightAiFixtureType>( PlaywrightAiFixture({ cache: { id: "my-fixture-cache" }, }), ); ``` ### Read-only, manual write ```typescript // fixture.ts in sample code export const test = base.extend<PlayWrightAiFixtureType>( PlaywrightAiFixture({ cache: { strategy: "read-only", id: "readonly-cache" }, }), ); ``` When you run the fixture in read-only mode you need to manually persist the cache after your test steps. Use the `agentForPage` helper provided by the fixture to fetch the underlying agent, then call `agent.flushCache()` at the point where you want to write the cache file: ```typescript test.afterEach(async ({ page, agentForPage }, testInfo) => { // Only flush cache if the test passed if (testInfo.status === 'passed') { console.log('Test passed, flushing Midscene cache...'); const agent = await agentForPage(page); await agent.flushCache(); } else { console.log(`Test ${testInfo.status}, skipping Midscene cache flush.`); } }); test('manual cache flush', async ({ agentForPage, page, aiTap, aiWaitFor }) => { const agent = await agentForPage(page); await aiTap('first highlighted link in the hero section'); await aiWaitFor('the detail page loads completely'); await agent.flushCache(); }); ``` ### Write-only mode ```typescript // fixture.ts in sample code export const test = base.extend<PlayWrightAiFixtureType>( PlaywrightAiFixture({ cache: { strategy: "write-only", id: "write-only-cache" }, }), ); ``` In write-only mode, each test will call the AI model and automatically write results to cache file without reading existing cache. ## Cache Cleaning Midscene supports cleaning unused cache records when flushing cache to file, ensuring cache files stay lean and maintainable. This feature is **completely manual** and requires explicit call to `agent.flushCache({ cleanUnused: true })`. ### Manual Cleaning Mechanism When calling `agent.flushCache({ cleanUnused: true })`, the system will: 1. **Keep used caches**: Cache records that were matched and used in the current run will be retained 2. **Keep new caches**: Cache records newly generated in the current run will be retained 3. **Remove unused caches**: Old cache records that were not accessed will be automatically deleted 4. **Write to file**: The cleaned cache will be written to file ### Usage **Call in test afterEach:** ```javascript describe('test suite', () => { let resetFn: () => Promise<void>; let agent: PuppeteerAgent; afterEach(async () => { // Clean cache and write to file if (agent) { await agent.flushCache({ cleanUnused: true }); } // Then close the page if (resetFn) { await resetFn(); } }); it('test case', async () => { const { originPage, reset } = await launchPage('https://example.com/'); resetFn = reset; agent = new PuppeteerAgent(originPage, { cache: { id: 'my-cache-id' }, }); // ... test logic }); }); ``` **For Playwright AI Fixture users:** ```typescript test.afterEach(async ({ page, agentForPage }) => { const agent = await agentForPage(page); await agent.flushCache({ cleanUnused: true }); }); ``` ### Cleanup Behavior by Mode * **read-write mode**: Calling `flushCache({ cleanUnused: true })` will clean and write to file * **read-only mode**: Calling `flushCache({ cleanUnused: true })` will also clean and write to file (manual flush overrides read-only restriction) * **write-only mode**: No cleanup (does not read cache) **Note**: If you don't pass `cleanUnused: true` parameter, `flushCache()` will only write to file without cleaning unused caches. ## FAQ ### No cache file is generated Please ensure you have correctly configured caching: 1. **Direct Agent creation**: Set `cache: { id: "your-cache-id" }` in the constructor 2. **Playwright AI Fixture mode**: Set `cache: true` or `cache: { id: "your-cache-id" }` in fixture configuration 3. **YAML script mode**: Set `agent.cache.id` in the YAML file 4. **Read-only mode**: Ensure you called the `agent.flushCache()` method 5. **Legacy approach**: Set `cacheId` and enable `MIDSCENE_CACHE=1` environment variable ### How to check if the cache is hit? You can view the report file. If the cache is hit, you will see the `cache` tip and the time cost is obviously reduced. ### Why the cache is missed on CI? You need to commit the cache files to the repository in CI and recheck the cache hit conditions. ### Does it mean that AI services are no longer needed after using cache? No. Caching is the way to accelerate the execution, but it's not a tool for ensuring long-term script stability. We have noticed many scenarios where the cache may miss when the DOM structure changes. AI services are still needed to reevaluate the task when the cache miss occurs. ### How to manually remove the cache? You can remove the cache file in the `./midscene_run/cache` directory, or edit the contents in the cache file. ### How to disable the cache for a single API? You can use the `cacheable` option to disable the cache for a single API. Please refer to the documentation of the corresponding [API](/reference.md#common) for details. ### Limitations of XPath in caching element location Midscene uses [XPath](https://developer.mozilla.org/en-US/docs/Web/XML/XPath) to cache the element location. ⁠We are using a relatively strict strategy to prevent false matches. In these situations, the cache will not be accessed: 1. The text content of the new element at the same XPath is different from the cached element. 2. The DOM structure of the page is changed from the cached one. Additionally, since element location caching relies on DOM structure, caching is not available in the following scenarios: 1. **Canvas elements**: Graphics content inside Canvas does not have DOM nodes and cannot be located via XPath. 2. **Cross-origin iframes**: Browser security policies restrict access to the internal DOM of cross-origin iframes. 3. **Shadow DOM (closed mode)**: Closed Shadow DOM cannot be accessed from outside. 4. **WebGL / Dynamic SVG content**: Dynamically generated graphics may not have a stable DOM structure. When the cache is not hit or not available, the process will fall back to using AI services to find the element. ### Get more debug logs for caching You can set the `DEBUG=midscene:cache:*` environment variable to get more debug logs for caching. --- url: /changelog.md --- # Changelog ## v1.12 - DeepSeek V4, New Test Runner, and Report Timing Summaries v1.12 adds DeepSeek V4 vision model support, introduces the new Test Runner (Beta), and adds timing summaries to reports. ### DeepSeek V4 Vision Model Support * Added support for `deepseek-v4-flash-vision-exp`. See [Common Model Configuration](/model-common-config.md#deepseek) for setup and [Midscene 1.12: Support for DeepSeek V4 Flash Vision Exp Multimodal Model](https://medium.com/@midscene/midscene-1-12-support-for-deepseek-v4-flash-vision-exp-multimodal-model-f87395075114?postPublishedType=initial) for details. ### New Test Runner (Beta) * Added `@midscene/test` for natural-language tests in YAML with TypeScript Node extensions. The Test Runner will gradually replace the legacy YAML automation solution; its protocol and APIs remain in Beta. See the [Test Runner overview](/test-runner-overview.md). ### Reports and Reliability * The Report sidebar now shows total elapsed time and model-call time. Markdown reports also summarize time and token usage by model. * The screenshot pipeline now saves observation frames as JPEG, reducing the storage footprint of run artifacts. * Fixed resource resolution for external yadb and scrcpy binaries in Electron ASAR packages. ## v1.11 - Cross-Platform Reliability Improvements v1.11 improves model verification, mobile input, and report generation reliability. ### Reliability and Documentation * Fixed the model verification callback losing its Playground SDK method context. * Fixed HarmonyOS explicit app launches failing to resolve declared abilities or mapped launch targets. * Android and HarmonyOS now select all text before clearing an input, improving compatibility across input methods and controls. * Playwright report filenames are now bounded and collision-resistant, preventing `ENAMETOOLONG` failures from long titles. * Added execution cost data to the Doubao Seed Android showcase. ## v1.10 - BDD-Style Scripts, Doubao-Seed-2.1, and MCP Retirement v1.10 adds BDD-style Gherkin script execution, upgrades the recommended Doubao model to Doubao-Seed-2.1, and formally retires MCP server packages. Going forward, use Skills and the platform CLIs when AI agents need to drive Midscene. ### BDD-Style Scripts with Gherkin * Added `agent.runGherkinScenario()` for running Gherkin scenarios directly from JavaScript / TypeScript. * Added the `runGherkinScenario` step for YAML flows, so natural-language scenarios can be written as `Given` / `When` / `Then` steps and executed in order. * `Given` / `When` map to `aiAct`, while `Then` and following `And` / `But` steps map to `aiAssert`. This keeps test cases readable in natural language while giving them a stable step structure. * Midscene currently supports a single-`Scenario` subset of Gherkin. BDD-related capabilities are still in Beta. See: [BDD-Style Scripts with Gherkin](/advanced/bdd-style-scripts-with-gherkin.md) ### New Model Support * Recommended and supported `Doubao-Seed-2.1-turbo`, which has very fast localization speed and strong localization quality in our current private evaluation set. * The Doubao Seed family now uses `MIDSCENE_MODEL_FAMILY="doubao-seed"`, while the legacy `doubao-vision` family remains compatible. See: [Common Model Configuration](/model-common-config.md#doubao-seed-model), [Model Strategy](/model-strategy.md) ### MCP Retirement * Midscene no longer ships MCP server packages, including `@midscene/web-bridge-mcp`, `@midscene/android-mcp`, `@midscene/ios-mcp`, `@midscene/harmony-mcp`, `@midscene/computer-mcp`, and `@midscene/mcp`. * Use [Skills](/skills.md) and the platform CLIs when AI coding agents need to operate browsers, mobile devices, or desktop apps. * If you still depend on MCP servers, pin Midscene to `1.9.8`. This is the final version that includes MCP support. See: [MCP Integration Has Been Retired](/mcp.md) ## v1.9 - New Model Support, YAML Automation, and AndroidWorld Benchmark v1.9 expands model support, improves YAML automation, and makes reports, Android automation, Web input, and desktop automation more reliable. ### AndroidWorld Benchmark Midscene now includes an AndroidWorld benchmark report. With v1.9.5, Midscene achieved **Pass@1 93.10%**, **Pass@2 95.69%**, and **Pass@3 97.41%** in this benchmark. See: [AndroidWorld Benchmark Report](/android-world-benchmark-report.md) ### New Model Support * Added Kimi and Xiaomi MiMo model support. See: [Common Model Configuration](/model-common-config.md) ### Model and Planning Updates * `aiAct` now supports image prompts. * `MIDSCENE_MODEL_REASONING_ENABLED` now supports `default`, so Midscene can follow each model family's default thinking behavior. * Gemini thinking content and GPT-5 reasoning configuration are now handled more completely. * `aiAct` falls back to model planning when a cached plan becomes invalid, and clears the corresponding cache entry. * AI request errors now include retry-attempt details, parse failures preserve raw model responses, and parsed locate results are validated before use. * Model dumps now expose more model response metadata, including raw choice messages and response model names in usage data. * The `deepLocate` search area is now displayed in reports. ### Chrome Extension * Chrome extension Bridge mode now supports file upload. * Bridge-mode file uploads now honor file chooser accept filters and WSL file paths. ### YAML, CLI, and MCP * `1.9.8` is the final Midscene version that includes MCP support. Later versions retire MCP server packages in favor of Skills and the platform CLIs. * Platform CLIs now accept agent behavior init args. * YAML scripts now support the HarmonyOS target in the CLI, so HarmonyOS automation can be driven through the same script runner workflow as Web, Android, iOS, and Computer. See: [Automate with Scripts in YAML](/automate-with-scripts-in-yaml.md), [HarmonyOS API](/reference.md#harmonyos) * YAML Web config supports custom HTTP headers for browser automation. * YAML Web config supports `downloadPath` for browser downloads. * YAML runs now surface real execution errors instead of ending as a silent "not executed" case. * YAML batch runs can retry failed cases. * Successful YAML runs now print the report path. * Explicit YAML report file names are now honored. * CLI / MCP / Skill flows can expose `deepLocate` / `deepThink` controls through shared flags. * Assert CLI / MCP tooling forwards custom failure messages to make assertion failures clearer. * The CLI resolves `@rstest/core` from the CLI package itself and lazy-loads Rstest core, making framework runs more stable from external launch paths. ### Reports * `recordToReport` now supports custom screenshots. * Report exports keep image paths aligned with exported screenshots. * Reports include a JSON tree view for inspecting structured task and model data. * Report screenshots, labels, playground server origin handling, and context spacing were refined. ### Studio and Recorder * Studio recorder descriptions and preview input coalescing were stabilized. * Studio now handles invalid model environment configuration more safely. * Recorder workflows can generate Markdown replay output. ### Android Automation * Android action controls and planning guidance were improved for native mobile automation flows. ### Computer Automation * Computer desktop automation now ships Intel packaging. * Libnut scrolling now emits one full wheel delta per scroll tick. ### Bug Fixes * Fixed `longPress` duration in Web integration being capped at 600ms. * Fixed dropped characters when Web input fields re-render during typing. * Fixed HarmonyOS MCP startup failures caused by `photon` / `sharp` WASM initialization in some environments. * Fixed blank first-frame screenshots in Computer RDP sessions. * Self-healed missing execute permission on the Computer phased-scroll helper. * Added a warning for elevated Windows app input drops. * Added IPv6 RDP host support for Computer automation. ### Documentation Updates * Documented Azure OpenAI-compatible endpoint setup. ## v1.8 - Midscene Studio Desktop App & Multi-Platform Enhancements v1.8 introduces **Midscene Studio**, a brand-new desktop application, alongside new interaction APIs (long press / clear input), refined model planning behavior, and broad upgrades across device integrations, the report system, and the MCP toolset. ### Midscene Studio — A New Desktop Application (Beta) Midscene Studio is an Electron-based desktop app that brings every Midscene playground into a single native shell. It is currently in **Beta** — download the current Studio build from the [latest release page](https://github.com/web-infra-dev/midscene/releases/latest) by choosing a `midscene-studio-beta-*` asset, try it out, and let us know what works (and what doesn't): * **Multi-platform playground**: Switch seamlessly between Web, Android, iOS, HarmonyOS, and Computer playgrounds inside the same Studio app * **Interactive device preview**: Manually drive Android / iOS / HarmonyOS device previews with mouse and touch input; Web preview supports live streaming #### Coming next: record-to-replay Midscene scripts inside Studio We're building a full record → script → replay loop into Studio: drive a real device inside Studio, automatically capture the interaction as a structured Midscene script, then replay, debug, and export it without leaving the app. The capability will roll out across upcoming releases — stay tuned. ### YAML Workflow Enhancements * **Android `runAdbShell` timeout**: A `timeout` option is now accepted in both the JavaScript API and YAML scripts. See: [Android API](/reference.md#android), [Automate with Scripts in YAML](/automate-with-scripts-in-yaml.md) ### New Interaction APIs * **`agent.aiLongPress()`**: Long-press a target element to trigger long-press menus and similar gestures. See: [API Reference](/reference.md#agentailongpress) * **`agent.aiClearInput()`**: Clear the contents of an input field, ideal when clearing needs to be its own step. See: [API Reference](/reference.md#agentaiclearinput) ### Device and Platform Integrations * **iOS external WDA sessions**: iOS now supports connecting to existing WebDriverAgent sessions, making it easier to reuse an external WDA setup * **Pluggable iOS device implementation**: The iOSDevice implementation can be overridden, enabling deeper extension or customization * **Computer remote desktop**: RDP connection options are now exposed through Computer MCP / CLI connection tools, letting you take over a remote Windows desktop directly * **`agentForComputer` rename**: `agentForComputer` is introduced as the primary API; the original `agentFromComputer` is kept as a deprecated alias * **Puppeteer CLI viewport options**: The Puppeteer CLI now accepts viewport configuration, so you can size the browser at launch from the command line ### Model and Planning Behavior * **Usage intent decoupled from config slot**: Model usage intent is now tracked separately from resolved config slots, making multi-model planning, locating, and report display clearer * **Native thinking off by default**: For supported model families, Midscene now disables model-native thinking by default to improve execution speed and stability. See: [Model-Native Thinking Mode](/model-config.md#model-native-reasoning) * **Doubao fast tier**: Doubao fast tier configuration is now supported through `MIDSCENE_MODEL_EXTRA_BODY_JSON={"service_tier":"fast"}`. See: [Common Model Configuration](/model-common-config.md) * **GLM-5V-Turbo support**: Added support for Zhipu GLM-5V-Turbo. See: [Common Model Configuration](/model-common-config.md) * **Better scrollable select planning**: Planning for scrollable selects (dropdowns, scroll wheels) is more reliable in complex scenarios ### MCP and Platform CLIs * **New `assert` MCP tool**: The MCP server now exposes an assertion tool backed by `aiAssert`, so AI assistants can run assertions directly. See: [MCP](/mcp.md) * **Image prompts in assert**: The assert CLI / MCP tool now accepts images as prompt input, so you can assert against a reference image * **Bare init args in platform CLIs**: Platform CLIs simplify argument passing and now accept platform Agent constructor options directly * **Playwright fixture passes Agent options through**: `PlaywrightAiFixture` now forwards `PlaywrightAgent` constructor options, so you can customize the Agent while keeping the fixture ### Report System * **CLI report merging**: A new `report-tool merge` subcommand lets you combine multiple report files into one for centralized review * **Screenshot tool calls recorded in reports**: `take_screenshot` invocations now appear in reports, easing diagnosis of screenshot-related issues ### Chrome Extension * **Automated Chrome Web Store publishing**: The Chrome Web Store extension release pipeline is now fully automated, shortening release cycles ### Bug Fixes * Fixed `aiAct` completion being emitted before actions actually run * Fixed insight prompts prioritizing reference images over the current screenshot in some cases * Fixed Bridge attachment after new-tab navigation * Fixed Playground tap projection on iOS / HarmonyOS / Computer * Fixed HarmonyOS per-call `autoDismissKeyboard` * Fixed excessive Android Playground video streaming memory usage * Fixed Computer scroll default distance not matching Web * Fixed out-of-range bbox handling for models that return normalized `[0,1000]` coordinates * Fixed `aiAct` options not being inherited in Bridge mode * Fixed Action API return values diverging from the documented shape * Fixed `maxTokens` not aligning with intent model config * Fixed server port probing on a non-`0.0.0.0` host that did not match the actual listen host * Fixed `deepThink` dump flag being lost in reports * Fixed sporadic dropped characters when typing on iOS * Fixed unwanted HarmonyOS system action delay overrides ## v1.7 - Flexible Report File Consumption, with Qwen 3.6 Support ### Flexible Report File Consumption Starting in v1.7.0, you can extract raw screenshots and JSON data from report files, or convert reports into Markdown so other tools can continue consuming that content. ## Example You can parse a report file into a Markdown file like this: ```md # Act - Search for and play videos related to Midscene - Execution start: 2026-04-08T02:13:04.795Z - Task count: 21 ## 1. Plan - Click the top search box to activate input - Status: finished - Start: 2026-04-08T02:13:04.845Z - End: 2026-04-08T02:13:15.296Z - Cost(ms): 10451 - Screen size: 2880 x 1536 ![task-1](./screenshots/execution-1-task-1-f9fc3bf9-bdf6-48dd-abea-f8f29874d8c1.jpeg) ..... ``` You can then use the [Remotion Skill](https://www.remotion.dev/docs/ai/skills?utm_source=midscenejs) to parse that Markdown file and generate a customized replay video. The generated video looks like this: <video src="https://lf3-static.bytednsdoc.com/obj/eden-cn/vhaeh7vhabf/midscene-replay.mp4" height="300" controls /> Midscene supports parsing report files through either the CLI or the JavaScript SDK. See: [Consume Report Files](/consume-report-file.md) ### Qwen 3.6 Model Support Added support for the Qwen 3.6 model, so you can use the latest Qwen model in Midscene. See: [Model Config](/model-config.md) ### Chrome Extension Recording Language Setting A new YAML output language option is available in the Chrome extension's recording settings, supporting English, Chinese, Japanese, and more. You can also set it to Auto to follow the system language. ### Android / HarmonyOS Improvements * Added `terminate` action on Android and HarmonyOS to force-stop apps, useful for resetting app state during testing. See: [Android API](/reference.md#android), [HarmonyOS API](/reference.md#harmonyos) * Fixed placeholder text unexpectedly retained when typing on X/Twitter on Android * Fixed Android Playground LAN access issues ### Debugging Improvements * Execution logs can now be saved to disk for post-hoc troubleshooting * Playground config page now supports running a connectivity test when saving model config, catching configuration errors early * Skill CLI `run` command now supports passing images as prompts via the `--image` flag ### Bug Fixes * Fixed unclear error messages when file chooser encounters missing files * Fixed incomplete error collection in CLI batch runner summary * Fixed `aiScroll` indentation formatting errors in YAML scripts * Fixed inaccurate `aiLocate` element bounding box * Fixed missing fallback when screenshot capture fails * Fixed incorrect handling of empty model responses in some cases * Fixed tab reuse issues in CDP connection mode * Fixed `aiQuery` result missing under certain data structures * Fixed AutoGLM app launch parameter format issue * Fixed dropdown display issues in Playground * Fixed custom request header aliases not working in model config ## v1.6 - CDP Connection, Pinch/Zoom & Multi-Model Enhancements v1.6 adds CDP browser connection mode, cross-platform pinch/zoom gestures, GPT-5/GPT-5.4 model support, along with improvements to element locating, report system, Chrome extension, and more. ### CDP Browser Connection Mode You can now connect to an existing browser instance via CDP (Chrome DevTools Protocol) for automation, without having Midscene launch a new browser. This is useful when you need to reuse an existing browser session. See: [Skills - Browser Automation](/skills.md), [YAML Script Runner - CDP Mode](/yaml-script-runner.md#use-cdp-connection-mode) ### Cross-Platform Pinch/Zoom Gestures Pinch/zoom gestures are now supported on Android, iOS, and HarmonyOS, enabling use cases like map zooming and image preview. See: [API Reference - aiPinch](/reference.md#agentaipinch) ### GPT-5 / GPT-5.4 & Codex App-Server Provider Support Added support for GPT-5, GPT-5.4 models and Codex app-server provider. You can now use the latest OpenAI models for visual understanding and automation. See: [Model Config](/model-config.md), [Model Strategy](/model-strategy.md) ### Custom extraBody for Model Requests A new `extraBody` configuration allows you to pass additional custom parameters in model API requests, useful for specific model deployments or environments. See the [`MIDSCENE_MODEL_EXTRA_BODY_JSON` environment variable](/model-config.md#advanced-settings-optional) in Advanced Settings. ### `deepThink` Renamed to `deepLocate` The `deepThink` parameter in element-locating APIs has been renamed to `deepLocate` to better describe its "deep locate" purpose. The original `deepThink` parameter still works but we recommend migrating. See: [API Reference](/reference.md#common) ### Skill CLI & Platform Tool Enhancements * **Skill CLI Custom Interfaces**: Skill CLI now supports custom interfaces for more flexible Skill extensibility. See: [Skills](/skills.md) * **Unified MCP Tool Export**: All platform packages (Web, Android, iOS, etc.) now export MidsceneTools uniformly, making MCP integration simpler. See: [MCP](/mcp.md) * **iOS App Termination by bundleId**: You can now terminate iOS apps by bundleId, making it easier to reset app state during testing. See: [iOS API Reference](/reference.md#ios) * **CLI Version Display**: CLI now shows version info in health checks, helping you troubleshoot environment issues ### Task Cancellation Support `aiAct` now supports `AbortSignal`, allowing you to cancel operations mid-execution instead of waiting for them to complete. See: [API Reference - aiAct](/reference.md#agentaiact) ### Element Locating Optimization The `deepLocate` flow has been optimized for better efficiency and accuracy when locating elements in complex interfaces. ### Report & Playback Improvements * **Faster Large Report Loading**: Screenshots in reports are now lazy-loaded, significantly speeding up reports with many steps * **Mobile Reports with Device Shells**: Report playback now shows device shells, giving you a more realistic view of mobile automation * **More Precise Timing**: AI call and action execution timing in reports is now more precise, helping you identify performance bottlenecks ### Stability Improvements * AI planning now automatically retries once on intermittent parse failures, reducing test interruptions from network issues * Device health check now includes monitor detection, helping diagnose display issues in headless environments ### Chrome Extension Improvements * Fixed crash when generating scripts after stopping recording * Fixed lag during long recording sessions caused by message serialization performance * Bridge mode now has start/stop control buttons, and fixed connection drops during confirmation ### Bug Fixes * Fixed MCP server becoming a zombie process with 100% CPU usage in some cases * Fixed screenshots not retrying when page navigation is in progress * Fixed text input loss on Android in certain scenarios * Fixed `aiNumber` returning incorrect results for some formats * Fixed `aiScroll` throwing errors when called without arguments * Fixed AutoGLM back/home actions not working correctly across different platforms * Fixed report display issues when model names contain `/` * Fixed report player not resetting properly after playback completes * Fixed Deep Think toggle not reading environment variable configuration in Playground * Fixed cursor size displaying incorrectly in high-resolution device screenshots * Fixed Chrome launch path resolution failing on Linux * Fixed inaccurate element locating on pages with iframes * Fixed HarmonyOS screen info parsing errors at certain render resolutions * Fixed device orientation displaying incorrectly after canceling tasks in Playground ## v1.5 - HarmonyOS Support v1.5 adds HarmonyOS automation support, Qwen3.5 and doubao-seed 2.0 model support, along with multiple improvements to desktop automation, report system, Chrome extension, and more. ### HarmonyOS Automation Support New `@midscene/harmony` package officially supports HarmonyOS platform automation. Midscene's automation capabilities now extend from Web, Android, iOS, and Desktop to the HarmonyOS ecosystem. ### Qwen3.5 & doubao-seed 2.0 Model Support Added support for Qwen3.5 and doubao-seed 2.0 models, allowing developers to leverage newer models for better visual understanding. ### Generic Model Reasoning Configuration New `MIDSCENE_MODEL_REASONING_EFFORT` environment variable provides a generic model reasoning effort configuration, enabling developers to uniformly control reasoning behavior across different models. ### Desktop Automation Improvements * **Xvfb virtual display support**: Support Xvfb virtual display for headless Linux environments, enabling desktop automation on CI/CD servers without GUI * **Connection health check**: Added health check during desktop automation connection for improved reliability * **macOS input optimization**: All text input on macOS now uses clipboard to avoid IME issues * **Mouse control failure detection**: Automatically detects mouse control failure and warns about admin privilege requirements * **Stop execution optimization**: Checks destroyed state to abort screenshot operations promptly when stopping execution ### Screenshot & Display Optimization * **Custom screenshot shrink**: Support custom screenshot shrink ratio to optimize performance while maintaining recognition accuracy * **Android scalingRatio decoupling**: Decoupled scalingRatio from size() method for improved flexibility ### Report System Improvements * **More detailed timing**: Finer-grained timing information in reports helps developers analyze performance bottlenecks more precisely * **Directory mode support for mergeReports**: `mergeReports` now supports directory mode report files ### Chrome Extension Improvements * **Always decline option**: Chrome extension adds "always decline" option with confirm race condition fix * **Close Bridge server after CLI**: Bridge server automatically closes after CLI command completes, preventing lingering processes ### Bug Fixes * Fixed `z.preprocess` handling in input mode schema for correct form rendering * Fixed Android swipe parameter passing * Fixed web size calculation * Fixed `BASE_URL_FIX_SCRIPT` closing tag not recognized by HTML parser * Fixed undefined page guard in PlaywrightAgent/PuppeteerAgent constructors ## v1.4 - Skills: Let AI Assistants Control Your Devices v1.4 introduces Midscene Skills — a set of installable skill packs for AI assistants like Claude Code and OpenClaw, enabling them to directly control browsers, desktops, Android, and iOS devices. This release also includes a standalone desktop MCP service, independent CLI entry points for each platform package, enhanced AI planning, and more. ### Midscene Skills — Device Control Skills for AI Assistants Midscene Skills is a set of skill packs that can be installed into AI assistants like Claude Code and OpenClaw. Once installed, AI assistants can control browsers, desktops, Android, and iOS devices using natural language. Each platform package (`@midscene/android`, `@midscene/ios`, `@midscene/web`, etc.) now exposes an independent CLI entry point, which is the foundation that Skills is built upon. **Supported Platforms:** * Browser (Puppeteer headless mode) * Chrome Bridge (user's own desktop Chrome) * Desktop (macOS, Windows, Linux) * Android (via ADB) * iOS (via WebDriverAgent) See: [Midscene Skills](https://github.com/web-infra-dev/midscene-skills) ### Standalone Desktop Automation MCP Package New `@midscene/computer-mcp` package provides PC desktop automation as a standalone MCP service. Developers can use desktop automation capabilities directly in MCP-compatible tools like Cursor and Trae without additional integration. See docs: [PC Desktop Automation](/platforms/desktop.md) ### Chrome Extension MCP Background Connection The Chrome extension now supports background Bridge mode MCP connection, exposing the desktop browser as an MCP tool to AI assistants, further expanding the MCP ecosystem. ### AI Planning Enhancements * **New `deepLocate` option for `aiAct`**: Enable deep locating during action execution, improving element locating accuracy in complex interfaces * **Swipe vs DragAndDrop semantic distinction**: The model can now more precisely distinguish between swipe and drag-and-drop operations, reducing gesture planning errors * **LLM planning page navigation restrictions**: Prevents the model from generating unreasonable page navigation during planning, improving task execution stability * **AppleScript keyboard input on macOS**: Improved keyboard input stability and compatibility in desktop automation * **Cursor move action**: New cursor move action support ### YAML Scripts & File Upload Enhancements * **YAML `aiTap` supports `fileChooserAccept`**: Handle file upload dialogs directly in YAML scripts * **Directory upload support**: Web supports `webkitdirectory` type folder selection upload ### Chrome Extension Bridge Mode Caching Bridge mode now supports caching, reusing existing AI planning results to reduce repeated calls and improve debugging efficiency. ### Android Improvements * Optimized text input logic for improved input stability ### iOS Improvements * **Playground live screen stream**: iOS Playground now features live screen preview for real-time device monitoring during debugging. ## v1.3 - PC Desktop Automation Support v1.3 introduces brand new PC desktop automation capabilities, significantly improves Android screenshot performance, and brings multiple enhancements to report system and stability. ### New PC Desktop Automation Support Midscene now supports PC desktop automation on Windows, macOS, and Linux, driving native keyboard and mouse controls. Whether it's Electron, Qt, WPF, or native desktop applications, they can all be automated through the visual model approach. <video src="https://lf3-static.bytednsdoc.com/obj/eden-cn/nupipfups/Midscene/pc-twitter2.mp4" controls /> **Core Capabilities:** * **Mouse Operations**: click, double-click, right-click, mouse move, drag and drop * **Keyboard Input**: text input, key combinations (Cmd/Ctrl/Alt/Shift) * **Screenshots**: capture screenshots from any monitor * **Multi-monitor Support**: operate across multiple displays simultaneously **Usage Methods:** * Zero-code trial via Computer Playground * JavaScript SDK for scripting * YAML automation scripts and CLI tools * HTML report playback for all operation paths See documentation: [PC Desktop Automation](/platforms/desktop.md) ### Major Android Screenshot Performance Improvement With Scrcpy screenshot mode enabled, screenshot time drops from 500–2000ms to **100–200ms**, significantly improving Android automation response speed. This is particularly useful for remote device debugging and high frame rate scenarios. See documentation: [Scrcpy Screenshot Mode](/reference.md#scrcpy) ### Deep Thinking Mode Enhancement The `aiAct` deep thinking (deepThink) mode now not only helps with element location but also optimizes overall task planning, achieving better execution results in complex forms and multi-step workflows. ### Report Experience Optimization * **Timeline Collapse**: New collapse toggle button for easier viewing of long task flows * **Time Unit Changed to Seconds**: More readable * **Step Sync Highlighting**: Sidebar step highlighting syncs in real-time with player playback * **Reduced Memory Usage**: Optimized report generation mechanism to effectively reduce runtime memory usage ### Mobile Platform Improvements #### Android * More stable special character and Unicode input * More relaxed app package name matching for Launch action (ignores case and spaces) * Auto-retry when screenshot anomalies occur on certain devices #### iOS * More relaxed Bundle ID matching (ignores case and spaces) ### Web Automation Improvements * Fixed issue where Puppeteer could hang when taking screenshots of inactive tabs * Fixed inaccurate window size in headed mode * `shareBrowserContext` mode now supports preserving localStorage and sessionStorage * Playwright multi-project configuration automatically distinguishes test cases by browser in reports * Fixed `typeOnly` mode not working in YAML script input actions ### Other Improvements * Image processing performance improved * SVG icon cache issue fixed * Playground now displays specific reasons for model configuration errors ## v1.2 - Zhipu AI Open-Source Model Support and File Upload Support v1.2 introduces support for Zhipu AI open-source models, adds file upload functionality, and fixes several issues affecting user experience, making automated testing more reliable. ### New Zhipu AI Open-Source Model Support #### Zhipu GLM-V Vision Model * Zhipu GLM-V series models are open-source vision models launched by Zhipu AI, available in multiple parameter versions, supporting both cloud deployment and local deployment. * See: [GLM-V Model Configuration](/model-common-config.md#glm-v) #### Zhipu AutoGLM Mobile Automation Model * Zhipu AutoGLM is an open-source mobile automation model launched by Zhipu AI. It can understand mobile screen content based on natural language instructions, and combined with intelligent planning capabilities, generate operation processes to complete user needs. * See: [AutoGLM Model Configuration](/model-common-config.md#auto-glm) ### File upload feature File upload is a common requirement in Web automation scenarios. v1.2 adds file upload capability for the web, supporting natural language operations for file input boxes, making form automation more complete. See: [aiTap file upload](/reference.md#agentaitap) ### Cache mechanism optimization Fixed the issue where cache wasn't updated after DOM changes. When page DOM changes cause cache validation to fail, the system now automatically updates the cache, avoiding operation failures due to stale cache and improving automation script stability. ### Report and Playground improvements #### Deep thinking tag optimization * Fixed the issue where deepThink tags weren't displayed correctly in reports when using `.aiAct()` method with deep thinking. Now you can clearly see which operations used deep thinking capability in reports * Improved the style of summary rows in reports for better readability #### Playground stability improvements * Fixed the issue where Playground didn't properly create agent instances in `getActionSpace` when using agentFactory mode, ensuring normal operation across various usage modes * Optimized Playground output display to prevent overly long reportHTML content from affecting the interface ### Model configuration updates Updated configuration parameters for Qwen model's deep thinking functionality to ensure compatibility with the latest model version. ## v1.1 - `aiAct` deep thinking and extensible MCP SDK v1.1 optimizes model planning capabilities and MCP extensibility, making automation more stable in complex scenarios while providing more flexible solutions for enterprise MCP service deployments. ### `aiAct` can enable deep thinking (deepThink) When deep thinking is enabled in `aiAct`, the model will interpret intent more thoroughly and optimize its planning results. This is suited for complex forms, multi-step flows, and similar scenarios. It improves accuracy but increases planning latency. Currently supported: Qwen3-vl on Alibaba Cloud and Doubao-vision on Volcano Engine. See [Model strategy](/model-strategy.md) for details. Example usage: ```typescript await agent.aiAct('If the UI shows an "Add shipping address" button, expand the existing "Shipping address" list and select the last item', { deepThink: true }); ``` ### MCP extension and SDK exposure Developers can use the MCP SDK exposed by Midscene to flexibly deploy a public MCP service. This capability applies to Agent instances on any platform. Typical application scenarios: * Run MCP in enterprise intranet to control private device pools * Package Midscene capabilities as internal microservices for multiple teams * Extend custom automation toolchains See documentation: [MCP Services](/mcp.md) ### Chrome extension improvements * Fixed potential event loss during recording, improving recording stability * Optimized coordinate passing in `describeElement` for better element description accuracy ### CLI and configuration enhancements * **File parameter support**: Fixed CLI issue where `--files` parameter wasn't properly handled when `--config` was specified; now they can be flexibly combined * **Dynamic configuration**: Fixed Playground not reading the `MIDSCENE_REPLANNING_CYCLE_LIMIT` environment variable properly ### iOS Agent compatibility improvements * Optimized `getWindowSize` method to automatically fall back to legacy endpoint when newer API is unavailable, improving compatibility with WebDriverAgent versions ### Report and Playground improvements * Fixed issue where report wasn't properly initialized before accessing screen properties * Fixed abnormal behavior of stop function in Playground * Improved error handling during video export to avoid crashes caused by frame cancel Thanks to contributors: @FriedRiceNoodles ## v1.0 - Midscene v1.0 is here! Midscene v1.0 is here! Try it out today and see how it can help you automate your workflows. ### See our new [Showcases](/showcases.md) page for real-world examples Register the GitHub form autonomously in a web browser and pass all field validations: <video src="https://lf3-static.bytednsdoc.com/obj/eden-cn/nupipfups/Midscene/1.0-showcases/github2.mp4" height="300" controls /> Plus these real-world showcases: * [iOS Automation - Meituan coffee order](/showcases.md#ios) * [iOS Automation - Auto-like the first @midscene\_ai tweet](/showcases.md#ios) * [Android Automation - DCar: Xiaomi SU7 specs](/showcases.md#android) * [Android Automation - Booking a hotel for Christmas](/showcases.md#android) * [MCP Integration - Midscene MCP UI prepatch release](/showcases.md#mcp) Some community developers have successfully built on Midscene's capability to [integrate with any interface](/integrate-with-any-interface.md), extending it with a robotic arm plus vision and voice models for in-vehicle large-screen testing scenarios. See the video below. <video src="https://lf3-static.bytednsdoc.com/obj/eden-cn/vhaeh7vhabf/AI_Vision_Powered_Robotic_Arm.mp4" height="300" controls /> ### 🚀 Pure vision path Starting in v1.0, Midscene fully adopts a visual-understanding approach to deliver more stable UI automation. Visual models provide: * **Stable results**: Leading vision models (Doubao Seed 1.6, Qwen3-VL, etc.) are reliable enough for most production needs * **UI workflow planning**: Vision models generally excel at planning UI flows and can handle many complex task sequences * **Works everywhere**: Automation no longer depends on the rendering stack—Android, iOS, desktop apps, or a browser `<canvas>`: if you can capture a screenshot, Midscene can interact with it * **Developer-friendly**: Dropping selectors and DOM keeps prompts simpler; teammates unfamiliar with rendering tech can become productive quickly * **Far fewer tokens**: Removing DOM extraction cuts token usage by about 80%, lowering cost and speeding up local runs * **Open-source options**: Open-source vision models like Qwen3-VL 8B/30B perform well, enabling self-hosted deployments See our updated [Model strategy](/model-strategy.md) for details. ### 🚀 Better results by multiple model combinations Beyond the default interaction intent, Midscene defines Planning and Insight intents, and developers can enable dedicated models for each. For example, you can use a GPT model for planning while using the default Qwen3-VL model for element localization. Multi-model combinations let you scale up handling of complex requirements as needed. ### 🚀 Runtime architecture optimization * Reuse portions of context to reduce device-info calls and improve runtime performance * Optimize Action Space combinations for Web and mobile environments to provide models with a more useful toolset ### 🚀 Report improvements * Parameter view: mark interaction parameter locations, merge screenshot context, and identify model planning results faster * Style updates: dark mode report rendering for better readability * Token usage display: summarize token consumption by model to analyze cost across scenarios ### 🚀 MCP architecture redesign We redefined Midscene MCP services around vision-driven UI operations. Each Action in the iOS / Android / Web Action Space is exposed as an MCP tool, providing atomic operations so developers can focus on higher-level Agents without worrying about UI operation details, while keeping success rates high. See [MCP architecture](/mcp.md). ### 🚀 Mobile automation #### iOS improvements * Added compatibility across WebDriverAgent 5.x–7.x * Added WebDriver Clear API support to solve dynamic input field issues * Improved device compatibility #### Android improvements * Added screenshot polling fallback to improve remote device stability * Added automatic screen-orientation adaptation (displayId screenshots) * Added YAML script support for `runAdbShell` #### Cross-platform * Expose system operation helpers on the Agent instance, including Home, Back, RecentApp, and more ### 🚧 API changes Method renames (backward compatible) * Renamed `aiAction()` → `aiAct()` (old method kept with deprecation warning) * Renamed `logScreenshot()` → `recordToReport()` (old method kept with deprecation warning) Environment variable renames (backward compatible) * Renamed `OPENAI_API_KEY` → `MIDSCENE_MODEL_API_KEY` (new variable preferred, old variable as fallback) * Renamed `OPENAI_BASE_URL` → `MIDSCENE_MODEL_BASE_URL` (new variable preferred, old variable as fallback) ### ⬆️ Upgrade to the latest version Upgrade only the packages you use. For example: `npm install @midscene/web@latest` `npm install @midscene/android@latest` `npm install @midscene/ios@latest` If you use the CLI installed globally: `npm i -g @midscene/cli` ## v0.30 - Cache management upgrade and mobile experience optimization ### More flexible cache strategy v0.30 improves the cache system, allowing you to control cache behavior based on actual needs: * **Multiple cache modes available**: Supports read-only, write-only, and read-write strategies. For example, use read-only mode in CI environments to reuse cache, and use write-only mode in local development to update cache * **Automatic cleanup of unused cache**: Agent can automatically clean up unused cache records when destroyed, preventing cache files from accumulating * **Simplified unified configuration**: Cache configuration parameters for CLI and Agent are now unified, no need to remember different configurations ### Report management convenience * **Support for merging multiple reports**: In addition to playwright scenarios, all scenarios now support merging multiple automation execution reports into a single file for centralized viewing and sharing of test results ### Mobile automation optimization #### iOS platform improvements * **Real device support improvement**: Removed simctl check restriction, making iOS real device automation smoother * **Auto-adapt device display**: Implemented automatic device pixel ratio detection, ensuring accurate element positioning on different iOS devices #### Android platform enhancements * **Flexible screenshot optimization**: Added `screenshotResizeRatio` option, allowing you to customize screenshot size while ensuring visual recognition accuracy, reducing network transmission and storage overhead * **Screen info cache control**: Use `alwaysRefreshScreenInfo` option to control whether to fetch screen information each time, allowing cache reuse in stable environments to improve performance * **Direct ADB command execution**: AndroidAgent added `runAdbCommand` method for convenient execution of custom device control commands #### Cross-platform consistency * **ClearInput support on all platforms**: Solves the problem of AI being unable to accurately plan clear input operations across platforms ### Feature enhancements * **Failure classification**: CLI execution results can now distinguish between "skipped failures" and "actual failures", helping locate issue causes * **aiInput append mode**: Added `append` option to append input while preserving existing content, suitable for editing scenarios * **Chrome extension improvements**: * Popup mode preference saved to localStorage, remembering your choice on next open * Bridge mode supports auto-connect, reducing manual operations * Support for GPT-4o and non-visual language models ### Type safety improvements * **Zod schema validation**: Introduced type checking for action parameters, detecting parameter errors during development to avoid runtime issues * **Number type support**: Fixed `aiInput` support for number type values, making type handling more robust ### Bug fixes * Fixed potential issues caused by Playwright circular dependencies * Fixed issue where `aiWaitFor` as the first statement could not generate reports * Improved video recorder delay logic to ensure the last frame is captured * Optimized report display logic to view both error information and element positioning information simultaneously * Fixed issue where `cacheable` option in `aiAction` subtasks was not properly passed ### Community * Awesome Midscene section added [midscene-java](/awesome-midscene.md) community project ## v0.29 - iOS platform support added ### iOS platform support added The biggest highlight of v0.29 is the official introduction of iOS platform support! Now you can connect and automate iOS devices through WebDriver, extending Midscene's powerful AI automation capabilities to the Apple ecosystem, details: [Support iOS automation](/platforms/ios.md). ### Qwen3-VL model adaptation We've adapted the latest Qwen `Qwen3-VL` model, giving developers faster and more accurate visual understanding capabilities. See [Model strategy](/model-strategy.md). ### AI core capability enhancement * **UI-TARS Model Performance Optimization**: Optimized aiAct planning, improved dialogue history management, and provided better context awareness capabilities * **AI Assertion and Action Optimization**: We updated the prompt for `aiAssert` and optimized the internal implementation of `aiAct`, making AI-driven assertions and action execution more precise and reliable ### Reporting and debugging experience optimization * **URL Parameter Playback Control**: To improve debugging experience, you can now directly control the default behavior of report playback through URL parameters ### Documentation * Updated documentation deployment cache strategy to ensure users can access the latest documentation content in time ## v0.28 - Build your own GUI automation agent by integrating with your own interface (preview feature) ### Support for integration with any interface (preview feature) v0.28 introduces the capability to integrate with your own interface. Define an interface controller class that conforms to the `AbstractInterface` definition, and you can get a fully-featured Midscene Agent. The typical use case for this feature is to build a GUI automation Agent for your own interface, such as IoT devices, in-house applications, car displays, etc.! Combined with the universal Playground architecture and SDK enhancement features, developers can conveniently debug custom devices. For more information, please refer to [Integrate with Any Interface (Preview Feature)](/integrate-with-any-interface.md) ### Android platform optimization * **Planning Cache Support**: Added planning cache functionality for Android platform, improving execution efficiency * **Input Strategy Enhancement**: Optimized input clearing strategy based on IME settings, improving Android platform input experience * **Scroll Calculation Improvement**: Optimized scroll endpoint calculation algorithm for Android platform ### Gesture operation extension * **Double-Click Operation Support**: Added support for double-click actions * **Long Press and Swipe Gestures**: Added support for long press and swipe gestures ### Core function enhancement * **Agent Configuration Isolation**: Implemented model configuration isolation between different agents, avoiding configuration conflicts * **Execution Option Extension**: Added useCache and replanningCycleLimit configuration options for Agent, providing more fine-grained control * **YAML Script Support**: Support for running universal custom devices through YAML scripts, enhancing automation capabilities ### Bug fixes * Fixed Qwen model search region size issues * Optimized deepThink parameter handling and rectangle size calculation * Resolved issues related to Playwright double-click operations * Improved TEXT action type processing logic ### Documentation and community * Added custom interface documentation to help developers better extend functionality * Added [Awesome Midscene](/awesome-midscene.md) section in README to showcase community projects ## v0.27 - Core module refactoring, assertions and reports functionally enhanced ### Core module refactoring Based on the introduction of [Rslib](https://github.com/web-infra-dev/rslib) in v0.26 to improve development experience and reduce contribution thresholds, v0.27 takes it a step further by refactoring the core modules on a large scale. This makes it extremely easy to extend new devices and add new AI operations, and we sincerely welcome community developers to contribute! **Due to the wide scope of this refactoring, please feel free to report any issues you encounter after upgrading, and we will address them promptly.** ### API enhancement * **`aiAssert` Functionally Enhanced** * New `name` field allows naming different assertion tasks, making it easier to identify and parse in JSON output results * New `domIncluded` and `screenshotIncluded` options allow flexible control over whether to send DOM snapshots and page screenshots to AI ### Chrome extension playground upgrade * All Agent APIs can be directly debugged and run in the Playground! Interactive, extraction, and verification cover three major categories of methods, with visual operations and verification that boost your automation development efficiency! Come experience the truly versatile AI automation platform! 🚀 ### Report function optimization * **New Marking Layer Switch**: The report player has added a switch to hide the marking layer, allowing users to view the original page view without obstruction when playing back. ### Bug fixes * Fixed the problem that `aiWaitFor` sometimes caused the report to not be generated * Reduced memory consumption of Playwright plugin ## v0.26 - Toolchain fully integrated [Rslib](https://github.com/web-infra-dev/rslib), greatly improving development experience and reducing contribution threshold ### Web integration optimization * Support freezing page context([freezePageContext](/reference.md#agentfreezepagecontext)/[unfreezePageContext](/reference.md#agentunfreezepagecontext)), so that all subsequent operations reuse the same page snapshot, avoiding repeated page status acquisition * Add all agent APIs to Playwright fixture, simplify test script writing, and solve the problem of not generating reports when using agentForPage ### Android automation enhancement * New keyboard hiding strategy([keyboardDismissStrategy](/reference.md#androiddevice)), allowing you to specify the way to automatically hide the keyboard ### Report function optimization * Report content lazy parsing, solving the problem of report crash when the report is large * Report player adds automatic zoom switch, making it easier to view the global perspective playback * Support aiAssert / aiQuery tasks in report playback, to fully show the entire page change process * Fix the problem that the sidebar status is not displayed as a failure icon when the assertion fails * Fix the problem that the drop-down filter in the report cannot be switched ### Build and engineering * Build tool migration to [Rslib](https://github.com/web-infra-dev/rslib) library development tool, improving build efficiency and development experience * Full repository source code jump, making it easier for developers to view source code * MCP npm package product volume optimization, from 56M to 30M, greatly improving loading speed ### Bug fixes * CLI automatically opens headed mode when keepWindow is true * Fix the implementation problem of getGlobalConfig, solve the problem of abnormal environment variable initialization * Ensure that the mime-type in base64 encoding is correct * Fix the return value type of aiAssert task ## v0.25 - Support using images as AI prompt input ### Core function enhancement * New worker runtime support, support running in worker environment * Support using images as AI prompt input, see [Prompting with images](/reference.md#prompting-with-images) * Image processing upgrade, using Photon & Sharp for efficient image cropping ### Web integration optimization * Get XPath by coordinates, improve cache reproducibility * Cache file moves plan module to the front, improving readability * Chrome Recorder supports exporting all events to markdown documents * agent supports specifying HTML report name, see [reportFileName](/reference.md#common) ### Android automation enhancement * Long press gesture support * Pull-to-refresh support ### Bug fixes * Use global config to handle environment variables, avoid issues caused by multiple packaging * Manually construct error information when error object serialization fails * Fix playwright report type dependency declaration order issue * Fix MCP packaging issue ### Documentation AI-friendly * [LLMs.txt](/llm-txt.md) is now available in both Chinese and English, making it easier for AI to understand * Each document now has a copy-to-markdown button, making it easier to feed to AI ### Other function enhancement * Chrome Recorder supports aiScroll function * Refactor aiAssert to be consistent with aiBoolean ## v0.24 - MCP for Android automation ### MCP for Android automation * You can now use Midscene MCP to automate Android apps, just like you use it for web apps. Read more: [MCP for Android Automation](/mcp.md#android-device-management) ### Optimization * For Mac platform Puppeteer, a double input clearing mechanism has been added to ensure that the input box is cleared before input ### Development experience * Simplify the way to build `htmlElement.js` to avoid report template build issues caused by circular dependencies * Optimize development workflow, just use `npm run dev` to enter midscene project development ## v0.23 - New report style and YAML script ability enhancement ### Report system upgrade #### New report style * New report style design, providing clearer and more beautiful test result display * Optimize report layout and visual effects, improve user reading experience * Enhance report readability and information hierarchy structure ![](https://lf3-static.bytednsdoc.com/obj/eden-cn/nupipfups/Midscene/new%20report.png) ### YAML script ability enhancement #### Support multiple YAML files batch execution * New config mode support, support configure Yaml file running order, browser reuse strategy, parallelism * Support getting JSON format running results ![](https://lf3-static.bytednsdoc.com/obj/eden-cn/nupipfups/Midscene/Tuji_20250722_161353.338.png) ### Test coverage enhancement #### Android test enhancement * New Android platform related test cases, improve code quality and stability * Improve test coverage, ensure the reliability of Android features ## v0.22- Chrome extension recording function released ### Web integration enhancement #### New recording function * Chrome extension adds recording function, which can record user operations on the page and generate automation scripts * Support recording click, input, scroll and other common operations, greatly reducing the threshold for writing automation scripts * The recorded operations can be directly played back and debugged in the Playground #### Upgrade to IndexedDB for storage * Chrome extension's Playground and Bridge now use IndexedDB for data storage * Compared to the previous storage scheme, it provides larger storage capacity and better performance * Support storing more complex data structures, laying the foundation for future feature extensions #### Customize replanning cycle limit * Set the `MIDSCENE_REPLANNING_CYCLE_LIMIT` environment variable to customize the maximum number of re-planning cycles allowed when executing operations (aiAct). * The default value is 10. When the AI needs to re-plan more than this limit, an error will be thrown and suggest splitting the task. * Provide more flexible task execution control, adapting to different automation scenarios ```bash export MIDSCENE_REPLANNING_CYCLE_LIMIT=10 # default value is 10 ``` ### Android interaction optimization #### New screenshot path generation * Generate a unique file path for each screenshot to avoid file overwrite issues * Improve stability in concurrent test scenarios ## v0.21 - Chrome extension UI upgrade ### Web integration enhancement #### New chat-style user interface * New chat-style user interface design for better user experience <video src="https://lf3-static.bytednsdoc.com/obj/eden-cn/nupipfups/Midscene/recording_2025-07-07_08-16-16.mp4" controls /> #### Flexible timeout configuration * Supports overriding timeout settings from test fixture, providing more flexible timeout control * Applicable scenarios: Different test cases require different timeout settings #### Unified Puppeteer and Playwright configuration * New `waitForNavigationTimeout` and `waitForNetworkIdleTimeout` parameters for Playwright * Unified timeout options configuration for Puppeteer and Playwright, providing consistent API experience, reducing learning costs #### New data export callback mechanism * New `agent.onDumpUpdate` callback function, can get real-time notification when data is exported * Refactored the post-task processing flow to ensure the correct execution of asynchronous operations * Applicable scenarios: Monitoring or processing exported data ### Android interaction optimization #### Input experience improvement * Changed click input to slide operation, improving interaction response and stability * Reduced operation failures caused by inaccurate clicks ## v0.20 - Support for assigning XPath to locate elements ### Web integration enhancement #### New `aiAsk` method * Allows direct questioning of the AI model to obtain string-formatted answers for the current page. * Applicable scenarios: Tasks requiring AI reasoning such as Q\&A on page content and information extraction. * Example: ```typescript await agent.aiAsk('any question') ``` #### Support for passing XPath to locate elements * Location priority: Specified XPath > Cache > AI model location. * Applicable scenarios: When the XPath of an element is known and the AI model location needs to be skipped. * Example: ```typescript await agent.aiTap('submit button', { xpath: '//button[@id="submit"]' }) ``` ### Android improvement #### Playground tasks can be cancelled * Supports interrupting ongoing automation tasks to improve debugging efficiency. #### Enhanced `aiLocate` API * Returns the Device Pixel Ratio, which is commonly used to calculate the real coordinates of elements. ### Report generation optimization Improve report generation mechanism, from batch storage to single append, effectively reducing memory usage and avoiding memory overflow when the number of test cases is large. ## v0.19 - Support for getting complete execution process data ### New API for getting Midscene execution process data Add the `_unstableLogContent` API to the agent. Get the execution process data of Midscene, including the time of each step, the AI tokens consumed, and the screenshot. The report is generated based on this data, which means you can customize your own report using this data. Read more: [API documentation](/reference.md#agent_unstablelogcontent) ### CLI support for adjusting Midscene env variable priority By default, `dotenv` does not override the global environment variables in the `.env` file. If you want to override, you can use the `--dotenv-override` option. Read more: [Use YAML-based Automation Scripts](/automate-with-scripts-in-yaml.md#use-env-file-to-override-global-environment-variables) ### Reduce report file size Reduce the size of the generated report by trimming redundant data, significantly reducing the report file size for complex pages. The typical report file size for complex pages has been reduced from 47.6M to 15.6M! ## v0.18 - Enhanced reporting features 🚀 Midscene has another update! It makes your testing and automation processes even more powerful: ### Custom node in report * Add the `recordToReport` API to the agent. Take a screenshot of the current page as a report node, and support setting the node title and description to make the automated testing process more intuitive. Applicable for capturing screenshots of key steps, error status capture, UI validation, etc. * Example: ```javascript test('login github', async ({ ai, aiAssert, aiInput, recordToReport }) => { if (CACHE_TIME_OUT) { test.setTimeout(200 * 1000); } await ai('Click the "Sign in" button'); await aiInput('quanru', 'username'); await aiInput('123456', 'password'); // log by your own await recordToReport('Login page', { content: 'Username is quanru, password is 123456', }); await ai('Click the "Sign in" button'); await aiAssert('Login success'); }); ``` ### Support for downloading reports as videos * Support direct video download from the report player, just by clicking the download button on the player interface. ![](/blog/export-video.png) * Applicable scenarios: Share test results, archive reproduction steps, and demonstrate problem reproduction. ### More Android configurations exposed * Optimize input interactions in Android apps and allow connecting to remote Android devices * `autoDismissKeyboard?: boolean` - Optional parameter. Whether to automatically dismiss the keyboard after entering text. The default value is true. * `androidAdbPath?: string` - Optional parameter. Used to specify the path of the adb executable file. * `remoteAdbHost?: string` - Optional parameter. Used to specify the remote adb host. * `remoteAdbPort?: number` - Optional parameter. Used to specify the remote adb port. * Examples: ```typescript await agent.aiInput('Search Box', 'Test Content', { autoDismissKeyboard: true }) ``` ```typescript const agent = await agentFromAdbDevice('s4ey59', { autoDismissKeyboard: false, // Optional parameter. Whether to automatically dismiss the keyboard after entering text. The default value is true. androidAdbPath: '/usr/bin/adb', // Optional parameter. Used to specify the path of the adb executable file. remoteAdbHost: '192.168.10.1', // Optional parameter. Used to specify the remote adb host. remoteAdbPort: '5037' // Optional parameter. Used to specify the remote adb port. }) ``` Upgrade now to experience these powerful new features! * [Custom Report Node API documentation](/reference/index.md#log-screenshot) * [API documentation for more Android configuration items](/reference/index.md#androiddevice) ## v0.17 - Let AI see the DOM of the page ### Data query API enhanced To meet more automation and data extraction scenarios, the following APIs have been enhanced with the `options` parameter, supporting more flexible DOM information and screenshots: * `agent.aiQuery(dataDemand, options)` * `agent.aiBoolean(prompt, options)` * `agent.aiNumber(prompt, options)` * `agent.aiString(prompt, options)` #### New `options` parameter * `domIncluded`: Whether to pass the simplified DOM information to AI model, default is off. This is useful for extracting attributes that are not visible on the page, like image links. * `screenshotIncluded`: Whether to pass the screenshot to AI model, default is on. #### Code example ```typescript // Extract all contact information (including hidden avatarUrl attributes) const contactsData = await agent.aiQuery( "{name: string, id: number, company: string, department: string, avatarUrl: string}[], extract all contact information including hidden avatarUrl attributes", { domIncluded: true } ); // Check if the id attribute of the first contact is 1 const isId1 = await agent.aiBoolean( "Is the first contact's id is 1?", { domIncluded: true } ); // Get the ID of the first contact (hidden attribute) const firstContactId = await agent.aiNumber("First contact's id?", { domIncluded: true }); // Get the avatar URL of the first contact (invisible attribute on the page) const avatarUrl = await agent.aiString( "What is the Avatar URL of the first contact?", { domIncluded: true } ); ``` ### New right-click ability Have you ever encountered a scenario where you need to automate a right-click operation? Now, Midscene supports a new `agent.aiRightClick()` method! #### Function Perform a right-click operation on the specified element, suitable for scenarios where right-click events are customized on web pages. Please note that Midscene cannot interact with the browser's native context menu after right-click. #### Parameter description * `locate`: Describe the element you want to operate in natural language * `options`: Optional, supports `deepThink` (AI fine-grained positioning) and `cacheable` (result caching) #### Example ```typescript // Right-click on a contact in the contacts application, triggering a custom context menu await agent.aiRightClick("Alice Johnson"); // Then you can click on the options in the menu await agent.aiTap("Copy Info"); // Copy contact information to the clipboard ``` ### A complete example In this report file, we show a complete example of using the new `aiRightClick` API and new query parameters to extract contact data including hidden attributes. Report file: [puppeteer-2025-06-04\_20-34-48-zyh4ry4e.html](https://lf3-static.bytednsdoc.com/obj/eden-cn/nupipfups/Midscene/puppeteer-2025-06-04_20-34-48-zyh4ry4e.html) The corresponding code can be found in our example repository: [puppeteer-demo/extract-data.ts](https://github.com/web-infra-dev/midscene-example/blob/main/puppeteer-demo/extract-data.ts) ### Refactor cache Use xpath cache instead of coordinates, improve cache hit rate. Refactor cache file format from json to yaml, improve readability. ## v0.16 - Support MCP ### Midscene MCP 🤖 Use Cursor / Trae to help write test cases. 🕹️ Quickly implement browser operations akin to the Manus platform. 🔧 Integrate Midscene capabilities swiftly into your platforms and tools. <video src="https://lf3-static.bytednsdoc.com/obj/eden-cn/ozpmyhn_lm_hymuPild/ljhwZthlaukjlkulzlp/midscene/en-midscene-mcp-Sauce-Demo.mp4" controls /> Read more: [MCP](/mcp.md#choosing-your-environment) ### Support structured API for agent APIs: `aiBoolean`, `aiNumber`, `aiString`, `aiLocate` Read more: [Use JavaScript to Optimize the AI Automation Code](/basics.md#javascript-orchestration) ## v0.15 - Android automation unlocked! ### Android automation unlocked! 🤖 AI Playground: natural‑language debugging 📱 Supports native, Lynx & WebView apps 🔁 Replayable runs 🛠️ YAML or JS SDK ⚡ Auto‑planning & Instant Actions APIs Read more: [Android automation](/platforms/android.md) ### More features * Allow custom midscene\_run dir * Enhance report filename generation with unique identifiers and support split mode * Enhance timeout configurations and logging for network idle and navigation * Adapt for gemini-2.5-pro ## v0.14 - Instant actions "Instant Actions" introduces new atomic APIs, enhancing the accuracy of AI operations. Read more: [Instant Actions](/blog-introducing-instant-actions-and-deep-think.md) ## v0.13 - DeepThink mode ### Atomic AI interaction methods * Supports aiTap, aiInput, aiHover, aiScroll, and aiKeyboardPress for precise AI actions. ### DeepThink mode * Enhances click accuracy with deeper contextual understanding. ![](/blog/0.13.0.jpeg) ## v0.12 - Integrate Qwen 2.5 VL ### Integrate Qwen 2.5 VL's native capabilities * Keeps output accuracy. * Supports more element interactions. * Cuts operating cost by over 80%. ## v0.11.0 - UI-TARS model caching ### UI-TARS model support caching * Enable caching by document 👉 : [Enable Caching](/caching.md) * Enable effect <video src="https://lf3-static.bytednsdoc.com/obj/eden-cn/nupipfups/Midscene/antd-form-cache.mp4" controls /> ![](/blog/0.11.0.png) ### Optimize DOM tree extraction strategy * Optimize the information ability of the dom tree, accelerate the inference process of models like GPT 4o ![](/blog/0.11.0-2.png) ## v0.10.0 - UI-TARS model released UI-TARS is a Native GUI agent model released by the **Seed** team. It is named after the [TARS robot](https://interstellarfilm.fandom.com/wiki/TARS) in the movie [Star Trek](https://en.wikipedia.org/wiki/Star_Trek), which has high intelligence and autonomous thinking capabilities. UI-TARS **takes images and human instructions as input information**, can correctly perceive the next action, and gradually approach the goal of human instructions, leading to the best performance in various benchmark tests of GUI automation tasks compared to open-source and closed-source commercial models. ![](/blog/0.10.0.png) UI-TARS: Pioneering Automated GUI Interaction with Native Agents - Figure 1 ![](/blog/0.10.0-2.png) UI-TARS: Pioneering Automated GUI Interaction with Native - Figure 4 ### Model advantage UI-TARS has the following advantages in GUI tasks: * **Target-driven** * **Fast inference speed** * **Native GUI agent model** * **Private deployment without data security issues** ## v0.9.0 - Bridge mode released With the Midscene browser extension, you can now use scripts to link with the desktop browser for automated operations! We call it "Bridge Mode". Compared to previous CI environment debugging, the advantages are: 1. You can reuse the desktop browser, especially Cookie, login state, and front-end interface state, and start automation without worrying about environment setup. 2. Support manual and script cooperation to improve the flexibility of automation tools. 3. Simple business regression, just run it locally with Bridge Mode. ![](/blog/0.9.0.png) Documentation: [Use Chrome Extension to Experience Midscene](/bridge-mode.md) ## v0.8.0 - Chrome extension ### New Chrome extension, run Midscene anywhere Through the Midscene browser extension, you can run Midscene on any page, without writing any code. Experience it now 👉: [Use Chrome Extension to Experience Midscene](/quick-start.md#chrome-extension) <video src="https://lf3-static.bytednsdoc.com/obj/eden-cn/nupipfups/Midscene/Midscene_extension.mov" controls /> ## v0.7.0 - Playground ability ### Playground ability, debug anytime Now you don't have to keep re-running scripts to debug prompts! On the new test report page, you can debug the AI execution results at any time, including page operations, page information extraction, and page assertions. <video src="https://lf3-static.bytednsdoc.com/obj/eden-cn/nupipfups/Midscene/midscene-playground.mov" controls /> ## v0.6.0 - Doubao model support ### Doubao model support * Support for calling Doubao models, reference the environment variables below to experience. ```bash MIDSCENE_OPENAI_INIT_CONFIG_JSON='{"baseURL":"https://xxx.net/api/v3","apiKey":"xxx"}' MIDSCENE_MODEL_NAME='ep-20240925111815-mpfz8' MIDSCENE_MODEL_TEXT_ONLY='true' ``` Summarize the availability of Doubao models: * Currently, Doubao only has pure text models, which means "seeing" is not available. In scenarios where pure text is used for reasoning, it performs well. * If the use case requires combining UI analysis, it is completely unusable Example: ✅ The price of a multi-meat grape (can be guessed from the order of the text on the interface) ✅ The language switch text button (can be guessed from the text content on the interface: Chinese, English text) ❌ The left-bottom play button (requires image understanding, failed) ### Support for GPT-4o structured output, cost reduction By using the gpt-4o-2024-08-06 model, Midscene now supports structured output (structured-output) features, ensuring enhanced stability and reduced costs by 40%+. Midscene now supports hitting GPT-4o prompt caching features, and the cost of AI calls will continue to decrease as the company's GPT platform is deployed. ### Test report: support animation playback Now you can view the animation playback of each step in the test report, quickly debug your running script <video src="https://lf3-static.bytednsdoc.com/obj/eden-cn/nupipfups/Midscene/midscene-play-all.mp4" controls /> ### Speed up: merge plan and locate operations, response speed increased by 30% In the new version, we have merged the Plan and Locate operations in the prompt execution to a certain extent, which increases the response speed of AI by 30%. > Before ![](/blog/0.6.0.png) > after ![](/blog/0.6.0-2.png) ### Test report: the accuracy of different models * GPT 4o series models, 100% correct rate * doubao-pro-4k pure text model, approaching usable state ![](/blog/0.6.0-3.png) ![](/blog/0.6.0-4.png) ### Problem fix * Optimize the page information extraction to avoid collecting obscured elements, improving success rate, speed, and AI call cost 🚀 > before ![](/blog/0.6.0-5.png) > after ![](/blog/0.6.0-6.png) ## v0.5.0 - Support GPT-4o structured output ### New features * Support for gpt-4o-2024-08-06 model to provide 100% JSON format limit, reducing Midscene task planning hallucination behavior ![](/blog/0.5.0.png) * Support for Playwright AI behavior real-time visualization, improve the efficiency of troubleshooting ![](/blog/0.5.0-2.png) * Cache generalization, cache capabilities are no longer limited to playwright, pagepass, puppeteer can also use cache ```diff - playwright test --config=playwright.config.ts # Enable cache + MIDSCENE_CACHE=true playwright test --config=playwright.config.ts ``` * Support for azure openAI * Support for AI to add, delete, and modify the existing input ### Problem fix * Optimize the page information extraction to avoid collecting obscured elements, improving success rate, speed, and AI call cost 🚀 * During the AI interaction process, unnecessary attribute fields were trimmed, reducing token consumption. * Optimize the AI interaction process to reduce the likelihood of hallucination in KeyboardPress and Input events * For pagepass, provide an optimization solution for the flickering behavior that occurs during the execution of Midscene ```javascript // Currently, pagepass relies on a too low version of puppeteer, which may cause the interface to flicker and the cursor to be lost. The following solution can be used to solve this problem const originScreenshot = puppeteerPage.screenshot; puppeteerPage.screenshot = async (options) => { return await originScreenshot.call(puppeteerPage, { ...options, captureBeyondViewport: false }); }; ``` ## v0.4.0 - Support CLI usage ### New features * Support for Cli usage, reducing the usage threshold of Midscene ```bash # Headed mode (visible browser) access baidu.com and search "weather" npx @midscene/cli --headed --url https://www.baidu.com --action "input 'weather', press enter" --sleep 3000 # Visit GitHub status page and save the status to ./status.json npx @midscene/cli --url https://www.githubstatus.com/ \ --query-output status.json \ --query '{serviceName: string, status: string}[], github page status, return service name' ``` * Support for AI to wait for a certain time to continue the subsequent task execution * Playwright AI task report shows the overall time and aggregates AI tasks by test group ### Problem fix * Optimize the AI interaction process to reduce the likelihood of hallucination in KeyboardPress and Input events ## v0.3.0 - Support AI report HTML ### New features * Generate html format AI report, aggregate AI tasks by test group, facilitate test report distribution ### Problem fix * Fix the problem of AI report scrolling preview ## v0.2.0 - Control Puppeteer by natural language ### New features * Support for using natural language to control puppeteer to implement page automation 🗣️💻 * Provide AI cache capabilities for playwright framework, improve stability and execution efficiency * AI report visualization, aggregate AI tasks by test group, facilitate test report distribution * Support for AI to assert the page, let AI judge whether the page meets certain conditions ## v0.1.0 - Control Playwright by natural language ### New features * Support for using natural language to control puppeteer to implement page automation 🗣️💻 * Support for using natural language to extract page information 🔍🗂️ * AI report visualization, AI behavior, AI thinking visualization 🛠️👀 * Direct use of GPT-4o model, no training required 🤖🔧 --- url: /common/get-cdp-url.md --- #### Getting a CDP WebSocket URL You can get a CDP WebSocket URL from various sources, for example: * **BrowserBase**: Sign up at https://browserbase.com and get your CDP URL * **Browserless**: Use https://browserless.io or run your own instance * **Local Chrome**: Run Chrome with `--remote-debugging-port=9222` and use `ws://localhost:9222/devtools/browser/...` * **Docker**: Run Chrome in a Docker container with debugging port exposed --- url: /common/prepare-android.md --- ## Preparation ### Install Node.js Install [Node.js 18 or higher](https://nodejs.org/en/download/). ### Prepare API Key Prepare an API Key for a Vision Language (VL) model. See [Supported models and setup](/model-common-config.md) for the models and configurations supported by Midscene.js. ### Install adb `adb` is a command-line tool that allows you to communicate with Android devices. There are two ways to install `adb`: * Method 1: Install using [Android Studio](https://developer.android.com/studio) * Method 2: Install using [Android Command Line Tools](https://developer.android.com/studio#command-line-tools-only) Verify that `adb` is installed successfully: ```bash adb --version ``` When you see the following output, it means `adb` is installed successfully: ```log Android Debug Bridge version 1.0.41 Version 34.0.4-10411341 Installed as /usr/local/bin//adb Running on Darwin 24.3.0 (arm64) ``` ### Set the `ANDROID_HOME` environment variable Refer to [Android Environment Variables](https://developer.android.com/tools/variables) to set the `ANDROID_HOME` environment variable. Verify that the `ANDROID_HOME` variable is set successfully: ```bash echo $ANDROID_HOME ``` When the above command has output, it means the `ANDROID_HOME` variable is set successfully: ```log /Users/your_username/Library/Android/sdk ``` ### Connect Android Device In the developer options of your Android device, enable 'USB debugging'. If 'USB debugging (Security settings)' exists, enable it as well. Then connect your Android device using a USB cable. <p align="center"> <img src="/android-usb-debug-en.png" alt="android usb debug" width="400" /> </p> Verify the connection: ```bash adb devices -l ``` When you see the following output, it means the connection is successful: ```log List of devices attached s4ey59 device usb:34603008X product:cezanne model:M2006J device:cezan transport_id:3 ``` --- url: /common/prepare-ios.md --- #### Install Node.js Install [Node.js 18 or higher](https://nodejs.org/en/download/). #### Set up WebDriverAgent Before getting started, you need to set up the iOS development environment: * macOS (required for iOS development) * Xcode and Xcode command line tools * iOS Simulator or real device **Configure WebDriverAgent** Before using Midscene iOS, you need to prepare the WebDriverAgent service. :::note Version Requirement WebDriverAgent version must be **>= 7.0.0** ::: Please refer to the official documentation for setup: * **Simulator Configuration**: [Run Prebuilt WDA](https://appium.github.io/appium-xcuitest-driver/latest/guides/run-prebuilt-wda/) * **Real Device Configuration**: [Real Device Configuration](https://appium.github.io/appium-xcuitest-driver/latest/getting-started/device-setup/) **Verify WebDriverAgent** After completing the configuration, you can verify whether the service is working properly by accessing WebDriverAgent's status endpoint: **Access URL**: `http://localhost:8100/status` **Correct Response Example**: ```json { "value": { "build": { "version": "10.1.1", "time": "Sep 24 2025 18:56:41", "productBundleIdentifier": "com.facebook.WebDriverAgentRunner" }, "os": { "testmanagerdVersion": 65535, "name": "iOS", "sdkVersion": "26.0", "version": "26.0" }, "device": "iphone", "ios": { "ip": "10.91.115.63" }, "message": "WebDriverAgent is ready to accept commands", "state": "success", "ready": true }, "sessionId": "BCAD9603-F714-447C-A9E6-07D58267966B" } ``` If you can successfully access this endpoint and receive a similar JSON response as shown above, it indicates that WebDriverAgent is properly configured and running. --- url: /common/setup-env.md --- The following example uses `qwen3.7-plus` through Alibaba Cloud: ```bash export MIDSCENE_MODEL_BASE_URL="https://dashscope.aliyuncs.com/compatible-mode/v1" export MIDSCENE_MODEL_API_KEY="your-api-key" export MIDSCENE_MODEL_NAME="qwen3.7-plus" export MIDSCENE_MODEL_FAMILY="qwen3" ``` Replace `your-api-key` with your API Key. > To use another model, such as Doubao, GLM, Gemini, or GPT-5, see [Supported models and setup](/model-common-config.md). For all configuration options, see [Model configuration](/model-config.md). --- url: /consume-report-file.md --- # Consume Report Files Midscene HTML report files capture the full execution history of a single Agent, making them useful for replay and debugging. Starting in v1.7.0, you can extract raw screenshots and JSON data from a report file, or convert the report into Markdown so other tools can consume it. ## Example You can parse a report file into a Markdown file like this: ```md # Act - Search for and play videos related to Midscene - Execution start: 2026-04-08T02:13:04.795Z - Task count: 21 ## 1. Plan - Click the top search box to activate input - Status: finished - Start: 2026-04-08T02:13:04.845Z - End: 2026-04-08T02:13:15.296Z - Cost(ms): 10451 - Screen size: 2880 x 1536 ![task-1](./screenshots/execution-1-task-1-f9fc3bf9-bdf6-48dd-abea-f8f29874d8c1.jpeg) ### Recorder - #1 type=screenshot, ts=2026-04-08T02:13:15.296Z, timing=after-calling ![task-1](./screenshots/execution-1-task-1-c521b130-5037-4ed2-b70f-705e181d981a.jpeg) ## 2. Locate - The search input with the placeholder text "Li Weigang's Daily Life" at the top - Status: finished - Start: 2026-04-08T02:13:15.305Z - End: 2026-04-08T02:13:15.306Z - Cost(ms): 1 - Screen size: 2880 x 1536 - Locate center: (1489, 71) ..... ``` You can then combine it with the [Remotion Skill](https://www.remotion.dev/docs/ai/skills?utm_source=midscenejs) to parse the Markdown file and generate a customized replay video. The generated video looks like this: <video src="https://lf3-static.bytednsdoc.com/obj/eden-cn/vhaeh7vhabf/midscene-replay.mp4" height="300" controls></video> ## Parse With The CLI The report parsing tool is included in each platform CLI package, such as `@midscene/web` and `@midscene/android`. The subcommand is `report-tool`. Extract report contents as JSON and export the related screenshots into the `output-data` directory: ```shell npx @midscene/web report-tool --action split --htmlPath ./midscene_run/report/puppeteer-2026/index.html --outputDir ./output-data ``` Convert the report file into Markdown and write the result into the `output-markdown` directory: ```shell npx @midscene/web report-tool --action to-markdown --htmlPath ./midscene_run/report/puppeteer-2026/index.html --outputDir ./output-markdown ``` Merge multiple report files into a single combined report: ```shell npx @midscene/web report-tool --action merge-html \ --htmlReport ./midscene_run/report/case-a/index.html \ --htmlReport ./midscene_run/report/case-b.html \ --outputDir ./merged --outputName all-cases ``` Repeat `--htmlReport` once per source report. `--outputDir` and `--outputName` are optional; when omitted, the merged file is written to the default Midscene report directory with an auto-generated name. Pass `--overwrite` to replace an existing merged file. ## Parse With The JavaScript SDK If you prefer to control report parsing in code, use `splitReportFile`, `reportFileToMarkdown`, and `mergeReportFiles` from `@midscene/core`. ```ts import { mergeReportFiles, reportFileToMarkdown, splitReportFile, } from '@midscene/core'; const splitResult = splitReportFile({ htmlPath: './midscene_run/report/puppeteer-2026/index.html', outputDir: './output-data', }); console.log(splitResult.executionJsonFiles); const markdownResult = await reportFileToMarkdown({ htmlPath: './midscene_run/report/puppeteer-2026/index.html', outputDir: './output-markdown', }); console.log(markdownResult.markdownFiles); const mergedResult = mergeReportFiles({ htmlPaths: [ './midscene_run/report/case-a/index.html', './midscene_run/report/case-b.html', ], outputDir: './merged', outputName: 'all-cases', }); console.log(mergedResult.mergedReportPath); ``` `splitReportFile`, `reportFileToMarkdown`, and `mergeReportFiles` serve different outputs: * `splitReportFile` generates JSON files for the original structured objects (one `*.execution.json` per execution). The JSON keeps the raw `ReportActionDump`-style data and exports screenshots alongside it. The returned `executionJsonFiles` and `screenshotFiles` are lists of generated file paths. * `reportFileToMarkdown` converts the same report into human-readable Markdown and exports the screenshots referenced by that Markdown. The returned `markdownFiles` contains the generated Markdown file paths. * `mergeReportFiles` combines several report files into one merged HTML report. It is a thin wrapper over [`ReportMergingTool`](/reference.md#new-reportmergingtool) that derives `testTitle`/`testDescription` from each source report's `groupName` automatically. Use it when you run multiple CLI actions or tests and want to consolidate their reports. ## About Fields In JSON And Markdown The parsed JSON and Markdown structures may change as Midscene evolves. Use the actual conversion result as the source of truth. --- url: /data-privacy.md --- # Data privacy ⁠Midscene.js is an open-source project (GitHub: [Midscene](https://github.com/web-infra-dev/midscene/)) under the MIT license. You can see all the codes in the public repository. When using Midscene.js, your page data (including the screenshot) is sent directly to the AI model provider you choose. No third-party platform will have access to this data. All you need to be concerned about is the data privacy policy of the model provider. If you prefer building Midscene.js and its Chrome Extension in your own environment instead of using the published versions, you can refer to the [Contributing Guide](https://github.com/web-infra-dev/midscene/blob/main/CONTRIBUTING.md) to find building instructions. --- url: /extend-test-runner.md --- # Extend and maintain Test Runner `@midscene/test` supports registering business Nodes, managing runtime resources, and defining execution lifecycles. Framework maintainers can use these capabilities to integrate browsers, Agents, external tools, and business APIs into a customized testing foundation for their teams. For an overview of the design, see [Test Runner overview](/test-runner-overview.md). ## Quick start The following example shows how to set up a simple test project that switches the browser's user-agent language. ### 1. Install dependencies Create an empty project and install the Test Runner's core dependencies and driver tools: ```bash pnpm add -D @midscene/test @midscene/web playwright ``` > **Note**: Before using a Midscene Agent, follow [Model configuration](/model-config.md) to set the required model environment variables, such as your API Key. ### 2. Create the project files We recommend the following basic directory structure: ```text team-test-runner/ ├── cases/ │ └── midscene.yaml └── midscene.config.ts ``` ### 3. Configure the Test Project and register Nodes Create `midscene.config.ts` in the project root. This file registers reusable Nodes and defines the execution environment (Project): ```ts filename=midscene.config.ts import { defineNode, z } from '@midscene/test'; import { defineProjectSetup, defineTestProject, } from '@midscene/test/config'; import { createMidsceneNodes } from '@midscene/test/midscene'; import { createPlaywrightNodes } from '@midscene/test/playwright'; import { PlaywrightAgent } from '@midscene/web/playwright'; import { chromium, type Browser, type Page } from 'playwright'; interface ProjectContext { browser: Browser; page: Page; agent?: PlaywrightAgent; } const geoInputSchema = z.strictObject({ latitude: z.number().describe('Latitude of the mocked location.'), longitude: z.number().describe('Longitude of the mocked location.'), }); // 1. Register a custom Node that mocks a GPS location const mockLocation = defineNode< typeof geoInputSchema, unknown, ProjectContext >({ name: 'browser.mockLocation', title: 'Mock location', description: "Mock the browser's GPS location.", inputSchema: geoInputSchema, async execute({ input, context }) { const browserContext = context.page.context(); // Grant geolocation permission and set the coordinates await browserContext.grantPermissions(['geolocation']); await browserContext.setGeolocation({ latitude: input.latitude, longitude: input.longitude, }); }, }); // Integrate Midscene's built-in AI Nodes, such as aiAct and aiAssert const midsceneNodes = createMidsceneNodes<ProjectContext>({ getAgent: ({ context }) => { context.agent ??= new PlaywrightAgent(context.page); return context.agent; }, includeLaunch: false, }); const playwrightNodes = createPlaywrightNodes<ProjectContext>({ getPage: ({ context }) => context.page, }); // 2. Define browser environment setup and cleanup const playwrightSetup = defineProjectSetup<ProjectContext>({ name: 'playwright', platform: 'web', async setup({ onTeardown }) { const browser = await chromium.launch({ headless: true }); onTeardown(() => browser.close()); const browserContext = await browser.newContext(); const page = await browserContext.newPage(); return { browser, page }; }, }); // 3. Export the project configuration export default defineTestProject<ProjectContext>({ projects: [ { name: 'chromium', platform: 'web', setup: playwrightSetup, files: { include: ['cases/**/*.{yaml,yml}'] }, }, ], nodes: [mockLocation, ...midsceneNodes, ...playwrightNodes], }); ``` ### 4. Write and run a test case Create `cases/midscene.yaml`: ```yaml filename=cases/midscene.yaml cases: - name: Mock a location and show stores in that area tags: [smoke] steps: - browser.mockLocation: latitude: 35.6762 longitude: 139.6503 # Mock a location in Tokyo - gotoUrl: url: https://yoursite.com/stores - aiAssert: prompt: The page successfully loads and displays recommended stores in Tokyo or nearby areas message: The mocked location did not take effect ``` Run the test from the project root: ```bash pnpm exec midscene-test ``` The runner automatically loads `midscene.config.ts`, finds matching test cases, and runs them. ## Register custom business Nodes Use `defineNode()` to encapsulate complex API calls, database operations, cleanup tasks, or specialized browser interactions as named Nodes. Test case authors can then call these Nodes directly from their test cases. ### Basic business Node example The following example wraps an HTTP API that creates a test order: ```ts filename=midscene.config.ts import { defineNode, z } from '@midscene/test'; const orderInputSchema = z.strictObject({ sku: z.string().min(1).describe('SKU of the product to order.'), quantity: z.number().int().positive().describe('Quantity to order.'), }); interface ProjectContext { apiBaseUrl: string; } const createOrder = defineNode< typeof orderInputSchema, { id: string }, ProjectContext >({ name: 'order.create', title: 'Create order', description: 'Create a product order through the test API.', inputSchema: orderInputSchema, async execute({ input, context, signal }) { const response = await fetch(`${context.apiBaseUrl}/test/orders`, { method: 'POST', headers: { 'content-type': 'application/json', }, body: JSON.stringify({ sku: input.sku, quantity: input.quantity, }), signal, }); if (!response.ok) { throw new Error(`Failed to create order: HTTP ${response.status}`); } const order = (await response.json()) as { id: string }; return { summary: `Created order ${order.id}`, data: order, }; }, }); ``` After adding the Node to the configuration's `nodes` array, test case authors can use it directly: ```yaml cases: - name: Order workflow test steps: - order.create: sku: midscene-mug quantity: 1 ``` ### Define and strictly validate input with Zod Although `inputSchema` is optional, we strongly recommend defining it. 1. **Type inference and validation**: the runner validates the input with Zod before entering `execute()`. Invalid input immediately throws a `NodeInputValidationError`, while TypeScript provides compile-time type inference without a separate interface. 2. **Unknown parameter rejection**: use `z.strictObject()` so the runner rejects unexpected fields in a test case. 3. **Automatic reference generation**: information from each field's `.describe()` call is included in the generated Node reference for AI Agents and human test case authors. ```ts const refundInputSchema = z.strictObject({ orderId: z.string().min(1).describe('ID of the order to refund.'), reason: z.string().optional().describe('Reason for the refund.'), }); const refundOrder = defineNode({ name: 'order.refund', description: 'Request a refund for an existing order.', inputSchema: refundInputSchema, async execute({ input }) { // input is inferred as { orderId: string; reason?: string } await refund(input.orderId, input.reason); }, }); ``` ### Node execution context The `ctx` passed to `execute(ctx)` contains these commonly used fields: * `input`: business parameters passed from YAML and validated by Zod. * `$`: general Step properties controlled by the runner, such as normalized `timeout` and `continue-on-error` values. * `signal`: an `AbortSignal` triggered by a timeout or cancellation. Use it in asynchronous requests or long-running tasks to exit early and cleanly. * `context`: Project-level runtime resources returned by `defineProjectSetup()` and shared within the Project. * `history`: deeply read-only, JSON-compatible history of executed Nodes. AI Agent Nodes automatically read this field to understand context. * `onTeardown()`: registers cleanup functions for resources created by the current Node. Cleanup can use attempt or Document scope and runs in LIFO order. * `scope`: identifies the current Node execution boundary as either `case` or `document`. * `case` or `document`: detailed runtime information for the current execution position. ## Share state across Nodes In real-world tests, multiple Nodes often need to share state. For example, an order refund test can create an order and store its ID in `beforeEach`, access the ID from `steps`, and clean up the data in `afterEach`. Define state properties in a custom `ProjectContext` to enable this coordination: ```ts filename=midscene.config.ts import { defineNode, z } from '@midscene/test'; import type { PlaywrightAgent } from '@midscene/web/playwright'; import type { Page } from 'playwright'; interface ProjectContext { agent: PlaywrightAgent; appBaseUrl: string; page: Page; orderId?: string; // Share test state across Nodes orderService: { create(input: { status: 'paid' }): Promise<{ id: string }>; remove(orderId: string): Promise<void>; }; } const emptyInputSchema = z.strictObject({}); const prepareOrderInputSchema = z.strictObject({ status: z.literal('paid').describe('Status of the order to create.'), }); const getOrderId = (context: ProjectContext) => { if (!context.orderId) { throw new Error('The test order has not been created.'); } return context.orderId; }; // 1. Prepare the order environment const prepareOrder = defineNode< typeof prepareOrderInputSchema, { orderId: string }, ProjectContext >({ name: 'order.prepare', description: 'Call the order service to create a test order.', inputSchema: prepareOrderInputSchema, async execute({ input, context }) { const order = await context.orderService.create(input); context.orderId = order.id; // Save the ID in the context return { summary: `Created test order ${order.id}`, data: { orderId: order.id }, }; }, }); // 2. Access the saved order state const openRefundPage = defineNode< typeof emptyInputSchema, unknown, ProjectContext >({ name: 'browser.openRefundPage', description: "Open the current test order's refund page.", inputSchema: emptyInputSchema, async execute({ context }) { const orderId = getOrderId(context); await context.page.goto(`${context.appBaseUrl}/orders/${orderId}/refund`); }, }); // 3. Clean up the business environment const cleanupOrder = defineNode< typeof emptyInputSchema, unknown, ProjectContext >({ name: 'order.cleanup', description: 'Delete the current test order.', inputSchema: emptyInputSchema, async execute({ context }) { const orderId = getOrderId(context); await context.orderService.remove(orderId); delete context.orderId; // Restore a clean context }, }); export const refundNodes = [prepareOrder, openRefundPage, cleanupOrder]; ``` ## Integrate a Midscene Agent `@midscene/test/midscene` exports six built-in system Nodes: `aiAct`, `aiAssert`, `recordToReport`, `launch`, `wait`, and `agent`. Call `createMidsceneNodes()` with a `getAgent` callback to integrate them into your Test Runner configuration: ```ts import { createMidsceneNodes } from '@midscene/test/midscene'; const midsceneNodes = createMidsceneNodes<ProjectContext>({ getAgent: ({ context }) => { context.agent ??= new PlaywrightAgent(context.page); return context.agent; }, // Web uses gotoUrl instead of the legacy launch Node. includeLaunch: false, }); ``` `createMidsceneNodes()` keeps `launch` for compatibility with existing Agent integrations. Android and iOS projects should use the lifecycle Nodes owned by their platform preset and set `includeLaunch: false` here to avoid registering `launch` twice. ## Register platform preset Nodes The Test Runner publishes platform preset factories as separate entry points. Each factory receives getters instead of assuming property names in your Project Context. For Playwright, register `gotoUrl`, `setCookies`, `clearCookies`, and `setViewportSize`. The `playwright` package is an optional peer dependency of `@midscene/test`; install it in projects that use this preset: ```bash pnpm add -D playwright ``` Then create the preset Nodes: ```ts import { createPlaywrightNodes } from '@midscene/test/playwright'; const playwrightNodes = createPlaywrightNodes<ProjectContext>({ getPage: ({ context }) => context.page, getBaseUrl: ({ context }) => context.baseUrl, getEnv: () => process.env, }); ``` `setCookies` does not accept cookie values in YAML. Test Runner persists every Node input in the run result and workflow history. An inline cookie would be copied into those records. Use exactly one of `cookiesEnv`, `profile`, or `storageStatePath` as a cookie reference. The Node resolves the actual cookies only at execution time and passes them directly to the Playwright BrowserContext. Its result contains only the reference name and cookie count. Cookie names, values, and scopes are not written to the run result or workflow history. An environment variable may contain a Cookie header, a JSON cookie array, or Playwright storage-state JSON. Relative storage-state paths resolve from the current working directory by default; use `resolveStorageStatePath` when a project needs a different root. References prevent Test Runner from persisting the cookies, but the environment variable, profile, or storage-state file must still be protected. Do not commit storage-state files containing real cookies. ```yaml beforeEach: - clearCookies: {} - setCookies: cookiesEnv: E2E_COOKIES url: https://example.com - setViewportSize: width: 1440 height: 900 - gotoUrl: url: /chat waitUntil: domcontentloaded ``` `gotoUrl` follows Playwright's navigation semantics. When navigation completes, HTTP 4xx and 5xx responses are returned as successful Node results with their status code, so later steps can assert the error page. Network errors and navigation timeouts still fail the Node. For Android, the platform preset registers `launch`, `terminate`, and `runAdbShell`. Its Agent contract requires all three capabilities: ```ts import { createAndroidNodes } from '@midscene/test/android'; const androidNodes = createAndroidNodes<ProjectContext>({ getAgent: ({ context }) => context.agent, }); ``` ```yaml beforeEach: - runAdbShell: command: pm clear com.example.app - launch: uri: com.example.app ``` For iOS, the platform preset registers `launch`, `terminate`, and `runWdaRequest`. Its Agent contract requires all three capabilities: ```ts import { createIOSNodes } from '@midscene/test/ios'; const iosNodes = createIOSNodes<ProjectContext>({ getAgent: ({ context }) => context.agent, }); ``` ```yaml steps: - launch: uri: com.example.app - runWdaRequest: method: GET endpoint: /status - terminate: uri: com.example.app ``` `runAdbShell` and `runWdaRequest` preserve their complete response in the Node result and workflow history. Test Runner limits only the history representation passed to later Midscene Agent calls: oversized values become bounded previews with their original character count, and recent entries take priority when the total context is too large. Use command-side filtering when the complete output is not needed in the run result. `launch` and `gotoUrl` are intentionally not aliases. `launch` manages an app, URL, or URI through a device Agent. `gotoUrl` navigates the current Playwright Page and supports Web-specific `baseUrl`, lifecycle, and HTTP response semantics. ## Generate a Node reference The runner provides the `describe-nodes` tool so test case authors, including AI Agents, can clearly discover the registered Nodes and their parameter schemas. It compiles each Node's `title`, `description`, and Zod `inputSchema` into a standard Markdown reference. Run the following command to generate a reference for your team: ```bash pnpm exec midscene-test describe-nodes > midscene-nodes.md ``` You can also specify a test directory or a custom configuration file: ```bash pnpm exec midscene-test describe-nodes ./e2e --config ./config/midscene.config.ts ``` The generated reference includes every Node registered in the active Test Project, including the six built-in system Nodes returned by `createMidsceneNodes()`, sorted by name. The runner automatically converts each Zod `inputSchema` to standard JSON Schema. ## Configure and manage a Test Project `defineTestProject()` is the main configuration entry point for the testing foundation and supports one or more Execution Projects: ```ts filename=midscene.config.ts export default defineTestProject<ProjectContext>({ projects: [ { name: 'android-smoke', platform: 'android', setup: doraAndroidSetup, files: { include: ['cases/**/*.{yaml,yml}'], exclude: ['cases/**/*.draft.yaml'], }, tags: { include: ['smoke'], exclude: ['manual'] }, retry: 1, variables: { appUri: 'com.example.app' }, }, ], test: { maxConcurrency: 1, // Maximum number of active Execution Projects bail: 0, // Stop scheduling new tasks after this many failed cases when > 0 testTimeout: 120_000, }, output: { reportDir: './midscene_run/report', }, nodes: [createOrder, ...midsceneNodes], }); ``` ### Key configuration strategies 1. **Environment and resource isolation**: each Execution Project has an independent `setup` environment, such as a separate browser or a specific test device. 2. **Multi-Project concurrency and lifecycle slots**: `test.maxConcurrency` controls the number of active Projects. The default is `1`. * One concurrency slot covers the entire lifecycle from Project setup through teardown. * Within a single Project, all Workflow Documents, Cases, and Steps still run **strictly in sequence** to ensure deterministic tests. * To drive multiple mobile devices or browser instances at the same time, declare multiple entries in `projects` and increase the concurrency value. 3. **Lifecycle cleanup**: use `defineProjectSetup()` to define environment preparation. Register cleanup hooks with `onTeardown()` so that long-lived resources are safely released in reverse LIFO order even if a test is interrupted or fails partway through. ## Programmatic APIs Most teams only need the project configuration, YAML files, and CLI. To embed the runner in another tool, such as a local GUI test panel, use these exported programmatic APIs: * **`loadTestProject()`**: asynchronously loads the TypeScript project configuration from `midscene.config.ts`. The runner does not support synchronous loading. * **`runTestProject()`**: asynchronously discovers, runs, and summarizes the entire project. It is exported from `@midscene/test/config`. * **`CaseRunner` / `createCaseRunner()`**: directly runs one test case represented as a plain object, without file parsing or lifecycle management. * **`runWorkflowDocument()`**: runs the complete lifecycle and every Case in one document. After configuring the project, continue to [Write and run test cases](/use-test-runner.md) for test case syntax and parameters. --- url: /faq.md --- # FAQ ## Platform-Specific FAQ The following platform-specific FAQs are maintained in their respective documentation: * [Web Browser - Playwright](/integrate-with-playwright.md#faq) * [Web Browser - Puppeteer](/integrate-with-puppeteer.md#faq) * [Web Browser - Chrome Extension](/quick-start.md#chrome-extension-faq) * [Web Browser - Bridge Mode](/bridge-mode.md#faq) * [Android](/platforms/android.md#faq) * [iOS](/platforms/ios.md#faq) * [HarmonyOS](/platforms/harmonyos.md#faq) * [PC Desktop](/platforms/desktop.md#faq) ## What data is sent to AI model? The screenshot will be sent to the AI model. In some cases, like setting the `domIncluded` option to `true` when calling `aiAsk` or `aiQuery`, the DOM information will also be sent. ⁠If you are worried about data privacy issues, please refer to [Data Privacy](/data-privacy.md) ## My model provider requires adding specific headers to requests You can use `defaultHeaders` in the `MIDSCENE_MODEL_INIT_CONFIG_JSON` environment variable to specify headers to include in the request. For example: ```bash # Add a header with key "foo" and value "bar" to the request MIDSCENE_MODEL_INIT_CONFIG_JSON='{"defaultHeaders":{"foo":"bar"}}' ``` If your provider documentation calls this field `extra_headers` or `extraHeaders`, Midscene also accepts those aliases and normalizes them to `defaultHeaders`. When multiple aliases are present, the priority is: `defaultHeaders` > `extra_headers` > `extraHeaders`. You can generate the JSON string with JSON serialization to avoid mistakes when writing it by hand: ```javascript JSON.stringify({ defaultHeaders: { foo: 'bar' } }) ``` ## How do I use Azure OpenAI Service? When using Azure OpenAI Service, first choose the model and fill in the regular configuration from [Supported models and setup](/model-common-config.md). Azure only requires changing the model service URL and API Key to the Azure form: ```bash MIDSCENE_MODEL_BASE_URL="https://<your-resource>.services.ai.azure.com/openai/v1" # Or https://<your-resource>.openai.azure.com/openai/v1 MIDSCENE_MODEL_API_KEY="<your-azure-api-key>" ``` In other words, other settings such as `MIDSCENE_MODEL_NAME` and `MIDSCENE_MODEL_FAMILY` should still follow the corresponding model section in [Supported models and setup](/model-common-config.md). Azure is only a model provider with different authentication, not a special model. This uses the normal OpenAI-compatible path and sends `POST /openai/v1/chat/completions` with `Authorization: Bearer ...`. Do not append `/chat/completions` to `MIDSCENE_MODEL_BASE_URL`. For most `/openai/v1` endpoints you do not need `api-version`. If your resource still rejects the request with `400 Missing required query parameter: api-version`, the `/openai/v1` surface on that specific resource has not GA'd yet. Inject the query parameter through `defaultQuery`: ```bash MIDSCENE_MODEL_INIT_CONFIG_JSON='{"defaultQuery":{"api-version":"preview"}}' ``` Use the `api-version` value your resource expects (`preview`, or a dated version like `2025-01-01-preview` shown in the Azure portal). This turns every request into `.../openai/v1/chat/completions?api-version=preview`. If an Azure-compatible gateway only accepts the `api-key` header, use this fallback: ```bash MIDSCENE_MODEL_API_KEY="placeholder" MIDSCENE_MODEL_INIT_CONFIG_JSON='{"defaultHeaders":{"api-key":"<your-azure-api-key>"}}' ``` In this fallback, `MIDSCENE_MODEL_API_KEY="placeholder"` only satisfies the OpenAI SDK constructor check. The real key is sent through `defaultHeaders.api-key`. These two fallbacks can be combined when a resource needs both `api-version` and the `api-key` header: ```bash MIDSCENE_MODEL_API_KEY="placeholder" MIDSCENE_MODEL_INIT_CONFIG_JSON='{"defaultQuery":{"api-version":"preview"},"defaultHeaders":{"api-key":"<your-azure-api-key>"}}' ``` Azure AD / keyless auth (`DefaultAzureCredential`) is not supported. Use an API key. ## Clicks are offset when using Azure OpenAI With a GPT-5 family model, you may find that the same script clicks the correct spot on the official OpenAI API but a consistently offset spot on Azure OpenAI. The offset scales with resolution: it appears at large screenshots (e.g. `1920x1080`) and disappears at small ones (e.g. `1280x600`). The cause is image handling on the Azure side. GPT-5 returns absolute coordinates based on the screenshot it actually sees, and Midscene sends the screenshot with `"detail": "original"` so the model sees the full-resolution image (see the [GPT-5 notes](/model-common-config.md#gpt)). Azure does not honor `"detail": "original"`, so it downscales large images server-side (the short side is capped at 768). The model then answers in the downscaled coordinate space while Midscene maps coordinates against the original resolution, producing a proportional offset. You can confirm `original` is not taking effect by checking token usage: when `original` works, image token consumption is noticeably higher. There are two ways to work around it: 1. Use the official OpenAI GPT-5, or configure a separate model dedicated to grounding (localization) and keep the Azure GPT-5 only as the planning model. 2. Pre-shrink the screenshot with the `screenshotShrinkFactor` agent option so the image stays under Azure's downscale threshold and no server-side resizing happens. See [`screenshotShrinkFactor`](/reference.md#common). ## How to improve the running time? There are several ways to improve the running time: 1. Use instant action interface like `agent.aiTap('Login Button')` instead of `agent.ai('Click Login Button')`. 2. Use a lower resolution if possible, this will reduce the input token cost. 3. Change to a faster model service 4. Use caching to accelerate the debug process. Read more about it in [Caching](/caching.md). ## How do I configure the midscene\_run directory? Midscene saves runtime artifacts (reports, logs, cache, etc.) in the `midscene_run` directory. By default, this directory is created in the current working directory. You can customize the directory location using the `MIDSCENE_RUN_DIR` environment variable, which accepts both relative and absolute paths: ```bash # Using a relative path export MIDSCENE_RUN_DIR="./my_custom_dir" # Using an absolute path export MIDSCENE_RUN_DIR="/tmp/midscene_output" ``` The directory contains the following subdirectories: * `report/` - Test report files (HTML format) * `log/` - Debug log files * `cache/` - Cache files (see [Caching](/caching.md)) For global runtime options, see [Runtime configuration](/reference.md#runtime-configuration). ## How do I control the report player's default replay style via a link? You can override the default values of the **Focus on cursor** and **Show element markers** toggles by adding query parameters to the report URL, which determines whether the report highlights the cursor position and element markers. Use `focusOnCursor` and `showElementMarkers` with values such as `true`, `false`, `1`, or `0`. For example: `...?focusOnCursor=false&showElementMarkers=true`. ## How do I embed the report as a bare player? When embedding the report in another page (for example, in an `iframe`), add the `player-only=1` query parameter to strip all the surrounding chrome (top bar, sidebar, timeline, and detail side) and keep only the replay player. Two more flags tune the player: * `play-control=1` — in player-only mode, show the bottom playback control bar (hidden by default). Opt-in; enabled only by `=1`. * `auto-play` — whether playback starts automatically on load. This is independent of `player-only` and applies to every report player. It is **on by default**; add `auto-play=0` to disable it. A typical embed looks like `...?player-only=1&play-control=1`. It also composes with the `#task-<id>` hash anchor, so you can deep-link to a specific step and show only its player: `...?player-only=1#task-0-5`. To open any report (embedded or not) without autoplay, use `...?auto-play=0`. ## Inaccurate Element Positioning If you encounter inaccurate element positioning when using Midscene, follow these steps to troubleshoot and resolve the issue: ### 1. Upgrade to the Latest Version Make sure you are using the latest version of Midscene, as new versions typically include optimizations and improvements for positioning accuracy. ```bash # Web automation npm install @midscene/web@latest # iOS automation npm install @midscene/ios@latest # CLI tool npm install @midscene/cli@latest # Or other packages corresponding to your platform ``` ### 2. Use Better Vision Models Midscene's element positioning capability relies on the AI model's visual understanding ability, so be sure to choose models that support visual capabilities. Generally, newer versions and models with larger parameters perform better than older versions and smaller models. For example, Qwen3-VL performs better than Qwen2.5-VL, and its plus version performs better than the flash version. For current model recommendations, see [Supported models and setup](/model-common-config.md). ### 3. Check Model Family Configuration Verify that the `MIDSCENE_MODEL_FAMILY` parameter is set correctly in your model configuration. Incorrect `MIDSCENE_MODEL_FAMILY` configuration will affect Midscene's adaptation logic for the model. See [Model Configuration](/model-config.md) for details. ### 4. Optimize prompts with visual features and position information If the positioning result randomly lands on unrelated elements and varies significantly between runs, the model usually cannot understand the semantics behind the icon button. For example, `aiTap('profile center')` is a functional description, and the model may not know the specific appearance of a profile icon. In contrast, `aiTap('person avatar icon')` is a visual description, so the model can locate the element based on its visual characteristics. Solution: optimize prompts by combining visual features and position information to describe the element. ```typescript // ❌ Using only a functional description await agent.aiTap('profile center'); // ✅ Using a visual description await agent.aiTap('person avatar icon'); // ✅ Combining visual features and position information await agent.aiTap('person avatar icon in the top right corner of the page'); ``` ### 5. Enable `deepLocate` If the positioning result lands near the target element but is still off by a few pixels, the model has probably identified the right target but still has some positioning deviation. Solution: enabling `deepLocate` can significantly improve positioning accuracy. ```typescript await agent.aiTap('Login button', { deepLocate: true }); ``` For more information about `deepLocate`, please refer to the [API documentation](/reference/index.md#deep-locate-deeplocate). ### 6. Increase the browser DPR to 2 on web If you are running Midscene in a web browser, you can try increasing the DPR to `2`. In CI environments, the default DPR is often `1`. Raising it to `2` makes the page clearer, which usually improves positioning for small elements. Keep in mind that this will consume more tokens. ## Does the Doubao phone use Midscene under the hood? No. --- url: /index.md --- --- url: /integrate-with-any-interface.md --- # Integrate with any interface You can use Midscene Agent to control any interface—such as IoT devices, in-house apps, and in-vehicle displays—by implementing a UI operation class that conforms to `AbstractInterface`. After implementing the UI operation class, you get the full capabilities of Midscene Agent: * the TypeScript GUI Automation Agent SDK, supporting integration with any interface * the playground for debugging * controlling the interface with YAML scripts * Skills support through CLI commands ## Demo and community project We have prepared a demo project for you to learn how to define your own interface class. It's highly recommended to check it out. * [Demo Project](https://github.com/web-infra-dev/midscene-example/tree/main/custom-interface) - A simple demo project that shows how to define your own interface class * [Android (adb) Agent](https://github.com/web-infra-dev/midscene/blob/main/packages/android/src/device.ts) - This is the Android (adb) Agent for Midscene that implements this feature * [iOS (WebDriverAgent) Agent](https://github.com/web-infra-dev/midscene/blob/main/packages/ios/src/device.ts) - This is the iOS (WebDriverAgent) Agent for Midscene that implements this feature There are also some community projects that use this feature: * [midscene-ios](https://github.com/lhuanyu/midscene-ios) - A project driving the OSX "iPhone Mirroring" app with Midscene ## Set up API keys for model The following example uses `qwen3.7-plus` through Alibaba Cloud: ```bash export MIDSCENE_MODEL_BASE_URL="https://dashscope.aliyuncs.com/compatible-mode/v1" export MIDSCENE_MODEL_API_KEY="your-api-key" export MIDSCENE_MODEL_NAME="qwen3.7-plus" export MIDSCENE_MODEL_FAMILY="qwen3" ``` Replace `your-api-key` with your API Key. > To use another model, such as Doubao, GLM, Gemini, or GPT-5, see [Supported models and setup](/model-common-config.md). For all configuration options, see [Model configuration](/model-config.md). ## Implement your own interface class ### Key concepts * The `AbstractInterface` class: a predefined abstract class that can connect to the Midscene Agent * The **action space**: a set of actions that describe the actions that can be performed on the interface. This will affect how the AI model plans the actions and executes them ### Step 1. Clone and setup from the demo project We provide a demo project that runs all the features of this document below. It's the fastest way to get started. ```bash # prepare the environment git clone https://github.com/web-infra-dev/midscene-example.git cd midscene-example/custom-interface npm install npm run build # run the demo npm run demo ``` ### Step 2. Implement your interface class Define a class that extends the `AbstractInterface` class, and implement the required methods. You can get the sample implementation from the [`./src/sample-device.ts`](https://github.com/web-infra-dev/midscene-example/blob/main/custom-interface/src/sample-device.ts) file. Let's take a glance at it. ```typescript import type { DeviceAction, Size } from '@midscene/core'; import { getMidsceneLocationSchema, z } from '@midscene/core'; import { type AbstractInterface, defineAction, defineActionTap, defineActionInput, // ... other action imports } from '@midscene/core/device'; export interface SampleDeviceConfig { deviceName?: string; width?: number; height?: number; } /** * SampleDevice - A template implementation of AbstractInterface */ export class SampleDevice implements AbstractInterface { interfaceType = 'sample-device'; private config: Required<SampleDeviceConfig>; constructor(config: SampleDeviceConfig = {}) { this.config = { deviceName: config.deviceName || 'Sample Device', width: config.width || 1920, height: config.height || 1080, }; } /** * Required: Take a screenshot and return base64 string */ async screenshotBase64(): Promise<string> { // TODO: Implement actual screenshot capture console.log('📸 Taking screenshot...'); return 'data:image/png;base64,...'; // Your screenshot implementation } /** * Required: Get interface dimensions * The width and height here refer to the logical size of the interface, not considering the device pixel ratio (dpr). The coordinates obtained from actions like defineActionTap are also based on this logical coordinate system. You can convert logical coordinates to physical coordinates in your action implementations if needed. */ async size(): Promise<Size> { return { width: this.config.width, height: this.config.height, }; } /** * Required: Define available actions for AI model */ actionSpace(): DeviceAction[] { return [ // Basic tap action defineActionTap(async (param) => { // TODO: Implement tap at param.locate.center coordinates await this.performTap(param.locate.center[0], param.locate.center[1]); }), // Text input action defineActionInput(async (param) => { // TODO: Implement text input await this.performInput(param.locate.center[0], param.locate.center[1], param.value); }), // Custom action example defineAction({ name: 'CustomAction', description: 'Your custom device-specific action', paramSchema: z.object({ locate: getMidsceneLocationSchema(), // ... custom parameters }), call: async (param) => { // TODO: Implement custom action }, }), ]; } async destroy(): Promise<void> { // TODO: Cleanup resources } // Private implementation methods private async performTap(x: number, y: number): Promise<void> { // TODO: Your actual tap implementation } private async performInput(x: number, y: number, text: string): Promise<void> { // TODO: Your actual input implementation } } ``` The key methods that you need to implement are: * `screenshotBase64()`, `size()`: help the AI model to get the context of the interface * `actionSpace()`: an array of `DeviceAction` objects defining the actions that can be performed on the interface. AI model will use these actions to perform the actions. Midscene has provided a set of predefined action spaces for the most common interfaces and devices. And there is also a method to define any custom action. Use these commands to run the agent: * `npm run build` to rebuild the agent * `npm run demo` to run the agent with javascript * `npm run demo:yaml` to run the agent with yaml script ### Step 3. Test the agent with the playground Attach a playground server to the agent, and you can test the agent in the web browser. ```ts import 'dotenv/config'; // read Midscene environment variables from .env file import { playgroundForAgent } from '@midscene/playground'; const sleep = (ms) => new Promise((r) => setTimeout(r, ms)); // instantiate device and agent const device = new SampleDevice(); await device.launch(); const agent = new Agent(device); // launch playground const server = await playgroundForAgent(agent).launch(); // close playground await sleep(10 * 60 * 1000); await server.close(); console.log('Playground closed!'); ``` ### Step 4. Add Skill support (optional) [Agent Skills](https://github.com/anthropics/skills) let AI coding assistants (Claude Code, Cline, etc.) drive your custom interface through CLI commands. Learn more in the [Skills documentation](/skills.md). Add a CLI entry file to your npm package (e.g., `./src/cli.ts`): ```ts #!/usr/bin/env node import { runSkillCLI } from '@midscene/core/skill'; import { SampleDevice } from './sample-device'; runSkillCLI({ DeviceClass: SampleDevice, scriptName: 'my-device', }); ``` Then add a `bin` field to your `package.json`: ```json { "bin": { "my-device": "./dist/cli.js" } } ``` Once published, AI coding assistants can control your custom interface via `npx my-device`. For Skill authoring guidelines and more examples, see the [midscene-skills](https://github.com/web-infra-dev/midscene-skills) repository. ### Step 5. Release the npm package, and let your users use it The agent and interface class have been exported in `./index.ts` file. Now you can publish it to npm. Fill the `name` and `version` in the `package.json` file, and then run the following command: ```bash npm publish ``` A typical usage of your npm package is like this: ```typescript import 'dotenv/config'; // read Midscene environment variables from .env file import { playgroundForAgent } from '@midscene/playground'; const sleep = (ms) => new Promise((r) => setTimeout(r, ms)); // instantiate device and agent const device = new SampleDevice(); await device.launch(); const agent = new Agent(device); await agent.aiAct('click the button'); ``` ### Step 6. Invoke your class in Midscene CLI and YAML script Write a yaml script with the `interface` section to invoke your class. ```yaml interface: module: 'my-pkg-name' # export: 'MyDeviceClass' # use this if this is a named export config: output: './data.json' ``` This config works same as this: ```typescript import MyDeviceClass from 'my-pkg-name'; const device = new MyDeviceClass(); const agent = new Agent(device, { output: './data.json', }); ``` Other fields in the yaml script are the same as the [yaml script](/automate-with-scripts-in-yaml.md). ## API reference ### Agent constructor ```typescript import { Agent } from '@midscene/core/agent'; import type { AbstractInterface } from '@midscene/core'; ``` Create an agent by pairing your custom `AbstractInterface` implementation with the standard constructor: ```typescript const device = new SampleDevice({ deviceName: 'Demo Panel', width: 1920, height: 1080, }); const agent = new Agent(device, { actionContext: 'Dismiss pop-ups before every task.', generateReport: true, customActions: [], // optional additional custom DeviceAction list }); ``` * `device: AbstractInterface` (required): Any class that fulfills `screenshotBase64`, `size`, and `actionSpace`. This is where you translate Midscene actions into real I/O calls for your hardware or desktop app. * `options?: PageAgentOpt`: Shares the same option bag as the browser and mobile agents described in the [API constructors](/reference.md#common-parameters). Commonly used fields include `generateReport`, `reportFileName`, `actionContext`/`aiActionContext`, `cacheId`, `modelConfig`, `createOpenAIClient`, `customActions`, and `onTaskStartTip`. * The resulting agent instantly unlocks the regular automation surfaces: `aiAct`/`aiTap` APIs, YAML runner (`interface` block), [playground](#playgroundforagent-function), Skills CLI support, and reporting pipeline. ### `AbstractInterface` class ```typescript import { AbstractInterface } from '@midscene/core'; ``` `AbstractInterface` is the key class for the agent to control the interface. These are the required methods that you need to implement: * `interfaceType: string`: define a name for the interface, this will not be provided to the AI model * `screenshotBase64(): Promise<string>`: take a screenshot of the interface and return the base64 string with the `'data:image/` prefix * `size(): Promise<Size>`: the size of the interface, which is an object with the `width` and `height` properties * `actionSpace(): DeviceAction[] | Promise<DeviceAction[]>`: the action space of the interface, which is an array of `DeviceAction` objects. Use predefined actions or define any custom action. Type signatures: ```ts import type { DeviceAction, Size, UIContext } from '@midscene/core'; import type { ElementNode } from '@midscene/shared/extractor'; abstract class AbstractInterface { // Required abstract interfaceType: string; abstract screenshotBase64(): Promise<string>; abstract size(): Promise<Size>; abstract actionSpace(): DeviceAction[] | Promise<DeviceAction[]>; // Optional lifecycle/hooks abstract destroy?(): Promise<void>; abstract describe?(): string; abstract beforeInvokeAction?(actionName: string, param: any): Promise<void>; abstract afterInvokeAction?(actionName: string, param: any): Promise<void>; } ``` These are the optional methods that you can implement: * `destroy?(): Promise<void>`: destroy the interface * `describe?(): string`: describe the interface, this may be used for the report and the playground. But it will not be provided to the AI model. * `beforeInvokeAction?(actionName: string, param: any): Promise<void>`: a hook function before invoking an action in action space * `afterInvokeAction?(actionName: string, param: any): Promise<void>`: a hook function after invoking an action ### The action space Action space is the set of actions that can be performed on the interface. AI model will use these actions to perform the actions. All the descriptions and parameter schemas of the actions will be provided to the AI model. To help you easily define the action space, Midscene has provided a set of predefined action spaces for the most common interfaces and devices. And there is also a method to define any custom action. This is how you can import the utils to define the action space: ```typescript import { type ActionTapParam, defineAction, defineActionTap, } from "@midscene/core/device"; ``` #### The predefined actions These are the predefined action spaces for the most common interfaces and devices. You can expose them to the customized interface by implementing the call method of the action. You can find the parameters of the actions in the type definition of these functions. * `defineActionTap()`: define the tap action. This is also the function to invoke for the `aiTap` method. * `defineActionDoubleClick()`: define the double click action * `defineActionInput()`: define the input action. This is also the function to invoke for the `aiInput` method. This is also the function to invoke for the `aiInput` method. * `defineActionKeyboardPress()`: define the keyboard press action. This is also the function to invoke for the `aiKeyboardPress` method. * `defineActionScroll()`: define the scroll action. This is also the function to invoke for the `aiScroll` method. * `defineActionDragAndDrop()`: define the drag and drop action * `defineActionLongPress()`: define the long press action * `defineActionSwipe()`: define the swipe action #### Define a custom action You can define your own action by using the `defineAction()` function. You can also use this method to define more actions for the [PuppeteerAgent](/integrate-with-puppeteer.md), [AgentOverChromeBridge](/bridge-mode.md#constructor), and [AndroidAgent](/platforms/android.md). API Signature: ```typescript import type { ExecutorContext } from "@midscene/core"; import { defineAction } from "@midscene/core/device"; defineAction( { name: string, description: string, paramSchema: z.ZodType<T>; call: ( param: z.infer<z.ZodType<T>>, context?: ExecutorContext, ) => Promise<void>; } ) ``` * `name`: the name of the action, AI model will use this name to invoke the action * `description`: the description of the action, AI model will use this description to understand what the action is doing. For complex actions, you can provide a more detailed example here. * `paramSchema`: the [Zod](https://www.npmjs.com/package/zod) schema of the parameters of the action, AI model will help to fill the parameters according to this schema * `call`: the function to invoke the action, you can get the parameters from the `param` parameter which conforms to the `paramSchema` * `context`: optional execution context. The action's return value populates `task.output` (for your API callers and the report) and is **not** sent to the planner. To pass a concise, planner-facing summary to the next planning round, set `context.task.planningFeedback`; the core planning layer truncates it to keep it within the model context. Example: ```typescript defineAction({ name: 'MyAction', description: 'My action', paramSchema: z.object({ name: z.string(), }), call: async (param) => { console.log(param.name); }, }); ``` If you want to get a param about the location of some element, you can use the `getMidsceneLocationSchema()` function to get the specific zod schema. A more complex example about defining a custom action: ```typescript import { getMidsceneLocationSchema } from "@midscene/core/device"; defineAction({ name: 'LaunchApp', description: 'A an app on screen', paramSchema: z.object({ name: z.string().describe('The name of the app to launch'), locate: getMidsceneLocationSchema().describe('The app icon to be launched'), }), call: async (param) => { console.log(`launching app: ${param.name}, ui located at: ${JSON.stringify(param.locate.center)}`); }, }); ``` ### `playgroundForAgent` function ```typescript import { playgroundForAgent } from '@midscene/playground'; ``` The `playgroundForAgent` function creates a playground launcher for a specific Agent, allowing you to test and debug your custom interface Agent in a web browser. #### Function signature ```typescript function playgroundForAgent(agent: Agent): { launch(options?: LaunchPlaygroundOptions): Promise<LaunchPlaygroundResult> } ``` #### Parameters * `agent: Agent`: The Agent instance to launch the playground for #### Return value Returns an object containing a `launch` method. #### `launch` method options ```typescript interface LaunchPlaygroundOptions { /** * Port to start the playground server on * @default 5800 */ port?: number; /** * Whether to automatically open the playground in browser * @default true */ openBrowser?: boolean; /** * Custom browser command to open playground * @default 'open' on macOS, 'start' on Windows, 'xdg-open' on Linux */ browserCommand?: string; /** * Whether to show server logs * @default true */ verbose?: boolean; /** * Unique identifier for the playground server instance * Same ID shares playground chat history * @default undefined (generates random UUID) */ id?: string; } ``` #### `launch` method return value ```typescript interface LaunchPlaygroundResult { /** * The playground server instance */ server: PlaygroundServer; /** * The server port */ port: number; /** * The server host */ host: string; /** * Function to close the playground */ close: () => Promise<void>; } ``` #### Usage example ```typescript import 'dotenv/config'; import { playgroundForAgent } from '@midscene/playground'; import { SampleDevice } from './sample-device'; import { Agent } from '@midscene/core/agent'; const sleep = (ms) => new Promise((r) => setTimeout(r, ms)); // Create device and agent instances const device = new SampleDevice(); const agent = new Agent(device); // Launch playground const result = await playgroundForAgent(agent).launch({ port: 5800, openBrowser: true, verbose: true }); console.log(`Playground started: http://${result.host}:${result.port}`); // Close playground when needed await sleep(10 * 60 * 1000); // Wait 10 minutes await result.close(); console.log('Playground closed!'); ``` ## FAQ **My interface-controller is general-purpose; can it be included in this document?** Yes, we are happy to gather creative projects and list them in this document. [Raise an issue](https://github.com/web-infra-dev/midscene/issues) to us when it's ready. --- url: /integrate-with-playwright.md --- import { PackageManagerTabs } from '@theme'; # Integrate with Playwright [Playwright.js](https://playwright.com/) is an open-source automation library developed by Microsoft, mainly used for end-to-end testing and web scraping of web applications. There are two ways to integrate with Playwright: * Directly integrate and call the Midscene Agent via script, suitable for quick prototyping, data scraping, and automation scripts. * Integrate Midscene into Playwright test cases, suitable for UI testing scenarios. ## Set up API keys for model The following example uses `qwen3.7-plus` through Alibaba Cloud: ```bash export MIDSCENE_MODEL_BASE_URL="https://dashscope.aliyuncs.com/compatible-mode/v1" export MIDSCENE_MODEL_API_KEY="your-api-key" export MIDSCENE_MODEL_NAME="qwen3.7-plus" export MIDSCENE_MODEL_FAMILY="qwen3" ``` Replace `your-api-key` with your API Key. > To use another model, such as Doubao, GLM, Gemini, or GPT-5, see [Supported models and setup](/model-common-config.md). For all configuration options, see [Model configuration](/model-config.md). ## Direct integration with Midscene agent :::info Example Project You can find an example project of direct Playwright integration here: [https://github.com/web-infra-dev/midscene-example/blob/main/playwright-demo](https://github.com/web-infra-dev/midscene-example/blob/main/playwright-demo) ::: ### Step 1: Install dependencies <PackageManagerTabs command="install @midscene/web playwright @playwright/test tsx --save-dev" /> ### Step 2: Write the script Save the following code as `./demo.ts`: ```typescript import { chromium } from 'playwright'; import { PlaywrightAgent } from '@midscene/web/playwright'; import 'dotenv/config'; // read environment variables from .env file const sleep = (ms) => new Promise((r) => setTimeout(r, ms)); Promise.resolve( (async () => { const browser = await chromium.launch({ headless: true, // 'true' means we can't see the browser window args: ['--no-sandbox', '--disable-setuid-sandbox'], }); const page = await browser.newPage(); await page.setViewportSize({ width: 1280, height: 768, }); await page.goto('https://www.ebay.com'); await sleep(5000); // 👀 init Midscene agent const agent = new PlaywrightAgent(page); // 👀 type keywords, perform a search await agent.aiAct('type "Headphones" in search box, hit Enter'); // 👀 wait for the loading await agent.aiWaitFor('there is at least one headphone item on page'); // or you may use a plain sleep: // await sleep(5000); // 👀 understand the page content, find the items const items = await agent.aiQuery( '{itemTitle: string, price: Number}[], find item in list and corresponding price', ); console.log('headphones in stock', items); const isMoreThan1000 = await agent.aiBoolean( 'Is the price of the headphones more than 1000?', ); console.log('isMoreThan1000', isMoreThan1000); const price = await agent.aiNumber( 'What is the price of the first headphone?', ); console.log('price', price); const name = await agent.aiString( 'What is the name of the first headphone?', ); console.log('name', name); const location = await agent.aiLocate( 'What is the location of the first headphone?', ); console.log('location', location); // 👀 assert by AI await agent.aiAssert('There is a category filter on the left'); // 👀 click on the first item await agent.aiTap('the first item in the list'); await browser.close(); })(), ); ``` For more Agent API details, please refer to [API Reference](/reference.md#interaction-methods). ### Step 3: Run the script Use `tsx` to run, and you will see the product information printed in the terminal: ```bash # run npx tsx demo.ts # The terminal should output something like: # [ # { # itemTitle: 'JBL Tour Pro 2 - True wireless Noise Cancelling earbuds with Smart Charging Case', # price: 551.21 # }, # { # itemTitle: 'Soundcore Space One Wireless Headphones 40H ANC Playtime 2XStronger Voice', # price: 543.94 # } # ] ``` ### Step 4: View the run report After the above command executes successfully, it will output: `Midscene - report file updated: /path/to/report/some_id.html`. Open this file in your browser to view the report. ## Integration in Playwright test cases Here we assume you already have a repository with Playwright integration. :::info Example Project You can find an example project of Playwright test integration here: [https://github.com/web-infra-dev/midscene-example/blob/main/playwright-testing-demo](https://github.com/web-infra-dev/midscene-example/blob/main/playwright-testing-demo) ::: ### Step 1: Add dependencies and update configuration Add dependencies <PackageManagerTabs command="install @midscene/web --save-dev" /> Update playwright.config.ts ```diff export default defineConfig({ testDir: './e2e', + timeout: 90 * 1000, + reporter: [["list"], ["@midscene/web/playwright-reporter", { type: "merged" }]], // type optional, default is "merged", means multiple test cases generate one report, optional value is "separate", means one report for each test case }); ``` Reporter configuration options: * `type`: Report mode, can be `merged` (default) or `separate`. `merged` means multiple test cases generate one merged report, `separate` means each test case generates its own report. * `outputFormat`: Controls how the report is generated. `'single-html'` (default) embeds all screenshots as base64 in a single HTML file. `'html-and-external-assets'` saves screenshots as separate PNG files in a subdirectory, useful when report files are too large. **Note**: When using `'html-and-external-assets'`, reports must be served via HTTP server and cannot be opened directly using `file://` protocol (because browser CORS restrictions block loading local images via relative paths from the file protocol). Navigate to the report directory and run one of the following commands: * Using Node.js: `npx serve` * Using Python: `python -m http.server` or `python3 -m http.server` Then access the report via `http://localhost:3000` (or the port shown in the terminal). ### Step 2: Extend the `test` instance Save the following code as `./e2e/fixture.ts`: ```typescript import { test as base } from '@playwright/test'; import type { PlayWrightAiFixtureType } from '@midscene/web/playwright'; import { PlaywrightAiFixture } from '@midscene/web/playwright'; export const test = base.extend<PlayWrightAiFixtureType>( PlaywrightAiFixture({ waitForNetworkIdleTimeout: 2000, // optional, the timeout for waiting for network idle between each action, default is 2000ms replanningCycleLimit: 30, // optional, override the default aiAct replanning cycle limit }), ); ``` `PlaywrightAiFixture()` accepts all shared `PlaywrightAgent` options, so you can configure agent behavior like `replanningCycleLimit`, `waitAfterAction`, and `modelConfig` directly at fixture creation time. Fixture-managed metadata like `testId`, `reportFileName`, `groupName`, and `groupDescription` is still generated automatically. ### Step 3: Write test cases Review the full catalog of action, query, and utility methods in the [Agent API reference](/reference.md#interaction-methods). When you need lower-level control, you can use `agentForPage` to obtain the underlying `PageAgent` instance and call any API directly: ```typescript test('case demo', async ({ agentForPage, page }) => { const agent = await agentForPage(page); await agent.recordToReport(); const logContent = agent._unstableLogContent(); console.log(logContent); }); ``` #### Example code ```typescript title="./e2e/ebay-search.spec.ts" import { expect } from '@playwright/test'; import { test } from './fixture'; test.beforeEach(async ({ page }) => { page.setViewportSize({ width: 400, height: 905 }); await page.goto('https://www.ebay.com'); await page.waitForLoadState('networkidle'); }); test('search headphone on ebay', async ({ ai, aiQuery, aiAssert, aiInput, aiTap, aiScroll, aiWaitFor, aiRightClick, recordToReport, }) => { // Use aiInput to enter search keyword await aiInput('Headphones', 'search box'); // Use aiTap to click search button await aiTap('search button'); // Wait for search results to load await aiWaitFor('search results list loaded', { timeoutMs: 5000 }); // Use aiScroll to scroll to bottom await aiScroll( { scrollType: 'untilBottom', }, 'search results list', ); // Use aiQuery to get product information const items = await aiQuery<Array<{ title: string; price: number }>>( 'get product titles and prices from search results', ); console.log('headphones in stock', items); expect(items?.length).toBeGreaterThan(0); // Use aiAssert to verify filter functionality await aiAssert('category filter exists on the left side'); // Use recordToReport to capture the current state await recordToReport('Search Results', { content: 'Final search results for headphones', }); }); ``` For more Agent API details, please refer to [API Reference](/reference.md#interaction-methods). ### Step 4. Run test cases ```bash npx playwright test ./e2e/ebay-search.spec.ts ``` ### Step 5. View test report After the command executes successfully, it will output: `Midscene - report file updated: ./current_cwd/midscene_run/report/some_id.html`. Open this file in your browser to view the report. ## Advanced ### About opening in a new tab `PlaywrightAgent` is a page-level Agent: each instance is bound to a single page. To make debugging easier, Midscene intercepts new tabs by default (for example, links with `target="_blank"`) and opens them in the current page. If you want to restore opening in a new tab while keeping the Agent on the original page, set `forceSameTabNavigation` to `false` and create a new Agent instance for each new tab yourself. If one Agent should manage page switching for the whole browser context, use `PlaywrightBrowserAgent`. Enable `autoFollowNewPage` when subsequent actions should automatically continue in the newly opened tab. ```typescript const mid = new PlaywrightBrowserAgent(context, page, { autoFollowNewPage: true, }); ``` Use `new PlaywrightBrowserAgent(context, page, options)` when you explicitly choose the initial active page. Use `PlaywrightBrowserAgent.create(context, options)` when you want Midscene to choose or create the initial active page; the factory uses `initialPage` when provided, otherwise it reuses the first existing context page or creates a new page. ### Browser support Some Midscene web automation features rely on Chrome DevTools Protocol (CDP), which is provided by Chromium-based browsers. These include browser-level events, touch gestures, and CDP fallback paths used by specific interactions. When using Playwright, Chromium is the recommended browser engine. Firefox and WebKit may work for basic Playwright-native operations, but Midscene features that depend on CDP may report errors on those engines. ### Connect Midscene Agent to a Remote Playwright Browser :::info Example Project You can find an example project of remote Playwright integration here: [https://github.com/web-infra-dev/midscene-example/tree/main/remote-playwright-demo](https://github.com/web-infra-dev/midscene-example/tree/main/remote-playwright-demo) ::: Connect to a remote Playwright browser when you already run browsers in your own infra or vendor grid. This keeps the browser close to the target environment, avoids repeated launches, and still lets Midscene drive it with the same AI APIs. #### Prerequisites <PackageManagerTabs command="install playwright @playwright/test @midscene/web --save-dev" /> #### Getting a CDP WebSocket URL You can get a CDP WebSocket URL from various sources, for example: * **BrowserBase**: Sign up at https://browserbase.com and get your CDP URL * **Browserless**: Use https://browserless.io or run your own instance * **Local Chrome**: Run Chrome with `--remote-debugging-port=9222` and use `ws://localhost:9222/devtools/browser/...` * **Docker**: Run Chrome in a Docker container with debugging port exposed #### Code example ```typescript import { chromium } from 'playwright'; import { PlaywrightAgent } from '@midscene/web/playwright'; // CDP WebSocket URL from your remote browser service const cdpWsUrl = 'ws://your-remote-browser.com/devtools/browser/your-session-id'; // Connect and pick a page const browser = await chromium.connectOverCDP(cdpWsUrl); const context = browser.contexts()[0]; const page = context.pages()[0] || await context.newPage(); // Create Midscene agent (usage matches any Playwright agent) const agent = new PlaywrightAgent(page); // Use AI methods as usual await agent.aiAct('navigate to https://example.com'); await agent.aiAct('click the login button'); const result = await agent.aiQuery('get page title: {title: string}'); // Cleanup await agent.destroy(); await browser.close(); ``` Once connected, keep using `PlaywrightAgent` the same way you would with a locally launched browser. ### Provide custom actions Use `defineAction()` to define custom actions. When constructing the Agent, pass these actions through `customActions`. Midscene appends these actions to the planner so the Agent can call the domain-specific actions you define. ```typescript import { getMidsceneLocationSchema, z } from '@midscene/core'; import { defineAction } from '@midscene/core/device'; const ContinuousClick = defineAction({ name: 'continuousClick', description: 'Click the same target repeatedly', paramSchema: z.object({ locate: getMidsceneLocationSchema(), count: z .number() .int() .positive() .describe('How many times to click'), }), async call(param) { const { locate, count } = param; console.log('click target center', locate.center); console.log('click count', count); // carry out your clicking logic using locate + count }, }); const agent = new PlaywrightAgent(page, { customActions: [ContinuousClick], }); await agent.aiAct('click the red button five times'); ``` Check [Integrate with any interface](/integrate-with-any-interface.md#define-a-custom-action) for more details about defining custom actions. ## FAQ ### Playwright browser download takes too long Playwright does not download browser binaries during `npm install` by default. You need to run `npx playwright install` separately, and that step can be slow on limited networks. You can speed it up in two ways: 1. Use a mirror, for example `npmmirror.com` ```bash PLAYWRIGHT_DOWNLOAD_HOST="https://npmmirror.com/mirrors/playwright" npx playwright install ``` 2. Download only the commonly used `chromium` ```bash npx playwright install --with-deps chromium ``` ### Cannot click the dropdown This usually happens when the page uses a native `select` element for the dropdown. In that case, the browser asks the operating system to render the expanded option list with a native control, so the dropdown is not actually rendered inside the webpage and cannot be captured by Playwright screenshots. First, check the screenshot in the report. If the dropdown options do not appear in the report screenshot after the click, this is very likely the cause. Midscene enables `forceChromeSelectRendering` by default, which forces Chrome to render the `select` dropdown so it appears in screenshots and can be recognized by Playwright. The dropdown style will look noticeably different from the operating system's default style. If you need the native rendering back, set `forceChromeSelectRendering: false`. ### The webpage continues to flash when running in headed mode In the local visualization interface, continuous flashing is usually caused by a mismatch between the viewport's `deviceScaleFactor` and the system/browser's pixel ratio (common on high-resolution or Retina screens). This flashing does not affect Midscene's screenshots or automation execution, but it does affect the local preview experience. To resolve this, set `deviceScaleFactor` to match your browser's `window.devicePixelRatio`, or use Puppeteer's auto-adaptation feature. ```typescript // Playwright: Playwright does not support using 0 for auto-adaptation like Puppeteer const page = await browser.newPage({ deviceScaleFactor: 2, // Replace the number 2 with your window.devicePixelRatio }) ``` If you are unsure of your browser's pixel ratio, you can press F12 on any page to open the console and type `window.devicePixelRatio` to check; or paste the following into the Chrome address bar and press Enter to see the value in a popup: ```plain data:text/html,<script>alert(`deviceScaleFactor of your browser: ${devicePixelRatio}`)</script> ``` ### Customize the network timeout When doing interaction or navigation on web page, Midscene automatically waits for the network to be idle. It's a strategy to ensure the stability of the automation. Nothing would happen if the waiting process is timeout. The default timeout is configured as follows: 1. If it's a page navigation, the default wait timeout is 5000ms (the `waitForNavigationTimeout`) 2. If it's a click, input, etc., the default wait timeout is 2000ms (the `waitForNetworkIdleTimeout`) You can also customize or disable the timeout by options: * Use `waitForNetworkIdleTimeout` and `waitForNavigationTimeout` parameters in [Agent](/reference/index.md#constructors). * Use `waitForNetworkIdle` parameter in [Yaml](/automate-with-scripts-in-yaml.md#the-web-part) or [PlaywrightAiFixture](/integrate-with-playwright.md#step-2-extend-the-test-instance). ### `waiting for fonts to load` or `page.screenshot: Timeout ... exceeded` when taking screenshots If you see an error like this in a Playwright-based environment: ```plain page.screenshot: Timeout 10000ms exceeded. Call log: - taking page screenshot - waiting for fonts to load... ``` This is usually not caused by Midscene itself. Playwright waits for fonts to finish loading before taking a screenshot. In some CI, container, or restricted network environments, font resources may load very slowly or never finish, which can eventually cause the screenshot to time out. You can work around it by setting this environment variable: ```bash export PW_TEST_SCREENSHOT_NO_FONTS_READY=1 ``` If you want to set it only for a single command, you can also write: ```bash PW_TEST_SCREENSHOT_NO_FONTS_READY=1 <your-command> ``` For more background, see the Playwright issue: [\[BUG\] Page.screenshot method hangs indefinitely](https://github.com/microsoft/playwright/issues/28995). ## More * For all the methods on the Agent, please refer to [API Reference](/reference.md#interaction-methods). * For the Playwright API reference, see [Playwright Agent API](/reference.md#playwright-agent). * Demo projects * Direct integration: [https://github.com/web-infra-dev/midscene-example/blob/main/playwright-demo](https://github.com/web-infra-dev/midscene-example/blob/main/playwright-demo) * Playwright test integration: [https://github.com/web-infra-dev/midscene-example/blob/main/playwright-testing-demo](https://github.com/web-infra-dev/midscene-example/blob/main/playwright-testing-demo) * Remote Playwright integration: [https://github.com/web-infra-dev/midscene-example/tree/main/remote-playwright-demo](https://github.com/web-infra-dev/midscene-example/tree/main/remote-playwright-demo) --- url: /integrate-with-puppeteer.md --- # Integrate with Puppeteer import { PackageManagerTabs } from '@theme'; [Puppeteer](https://pptr.dev/) is a Node.js library which provides a high-level API to control Chrome or Firefox over the DevTools Protocol or WebDriver BiDi. Puppeteer runs in the headless (no visible UI) by default but can be configured to run in a visible ("headful") browser. :::info Demo Projects you can check the demo project of Puppeteer here: [https://github.com/web-infra-dev/midscene-example/blob/main/puppeteer-demo](https://github.com/web-infra-dev/midscene-example/blob/main/puppeteer-demo) There is also a demo of Playwright with Vitest: [https://github.com/web-infra-dev/midscene-example/tree/main/playwright-with-vitest-demo](https://github.com/web-infra-dev/midscene-example/tree/main/playwright-with-vitest-demo) ::: ## Set up API keys for model The following example uses `qwen3.7-plus` through Alibaba Cloud: ```bash export MIDSCENE_MODEL_BASE_URL="https://dashscope.aliyuncs.com/compatible-mode/v1" export MIDSCENE_MODEL_API_KEY="your-api-key" export MIDSCENE_MODEL_NAME="qwen3.7-plus" export MIDSCENE_MODEL_FAMILY="qwen3" ``` Replace `your-api-key` with your API Key. > To use another model, such as Doubao, GLM, Gemini, or GPT-5, see [Supported models and setup](/model-common-config.md). For all configuration options, see [Model configuration](/model-config.md). ## Integration with Midscene Agent ### Step 1. Install dependencies <PackageManagerTabs command="install @midscene/web puppeteer tsx dotenv --save-dev" /> ### Step 2. Write scripts Write and save the following code as `./demo.ts`. ```typescript title="./demo.ts" import 'dotenv/config'; // load Midscene environment variables from .env if present import puppeteer from "puppeteer"; import { PuppeteerAgent } from "@midscene/web/puppeteer"; const sleep = (ms: number) => new Promise((r) => setTimeout(r, ms)); Promise.resolve( (async () => { const browser = await puppeteer.launch({ headless: false, // here we use headed mode to help debug }); const page = await browser.newPage(); await page.setViewport({ width: 1280, height: 800, deviceScaleFactor: 1, }); await page.goto("https://www.ebay.com"); await sleep(5000); // 👀 init Midscene agent const agent = new PuppeteerAgent(page); // 👀 type keywords, perform a search await agent.aiAct('type "Headphones" in search box, hit Enter'); await sleep(5000); // 👀 understand the page content, find the items const items = await agent.aiQuery( "{itemTitle: string, price: Number}[], find item in list and corresponding price" ); console.log("headphones in stock", items); // 👀 assert by AI await agent.aiAssert("There is a category filter on the left"); await browser.close(); })() ); ``` ### Step 3. Run Using `tsx` to run, you will get the data of Headphones on eBay: ```bash # run npx tsx demo.ts # it should print # [ # { # itemTitle: 'Beats by Dr. Dre Studio Buds Totally Wireless Noise Cancelling In Ear + OPEN BOX', # price: 505.15 # }, # { # itemTitle: 'Skullcandy Indy Truly Wireless Earbuds-Headphones Green Mint', # price: 186.69 # } # ] ``` For the complete catalog of agent methods, see the [API reference](/reference.md#interaction-methods). ### Step 4: View the report After the above command executes successfully, the console will output: `Midscene - report file updated: /path/to/report/some_id.html`. You can open this file in a browser to view the report. <a id="puppeteeragent" /> ## Advanced ### About opening in a new tab `PuppeteerAgent` is a page-level Agent: each instance is bound to a single page. For easier debugging, Midscene intercepts new tabs by default (for example, links with `target="_blank"`) and opens them in the current page. If you want to allow new tabs again while keeping the Agent on the original page, set `forceSameTabNavigation` to `false` and create a new Agent instance for each new tab yourself. If one Agent should manage page switching for the whole browser, use `PuppeteerBrowserAgent`. Enable `autoFollowNewPage` when subsequent actions should automatically continue in the newly opened tab. ```typescript const mid = new PuppeteerBrowserAgent(browser, page, { autoFollowNewPage: true, }); ``` Use `new PuppeteerBrowserAgent(browser, page, options)` when you explicitly choose the initial active page. Use `PuppeteerBrowserAgent.create(browser, options)` when you want Midscene to choose or create the initial active page; the factory uses `initialPage` when provided, otherwise it reuses the first existing browser page or creates a new page. ### Browser support Some Midscene web automation features rely on Chrome DevTools Protocol (CDP), which is provided by Chromium-based browsers. These include browser-level events, touch gestures, and CDP fallback paths used by specific interactions. When using Puppeteer, Chrome, Chromium, or another Chromium-based browser is recommended. Browsers that do not provide compatible CDP support may report errors when Midscene uses CDP-backed features. ### Connect Midscene Agent to a Remote Puppeteer Browser :::info Example Project You can find an example project of remote Puppeteer integration here: [https://github.com/web-infra-dev/midscene-example/tree/main/remote-puppeteer-demo](https://github.com/web-infra-dev/midscene-example/tree/main/remote-puppeteer-demo) ::: Use this approach when you want to reuse a browser that already runs inside your own infrastructure—such as a persistent cloud worker, a third-party browser grid, or an on-prem desktop. Wiring Midscene into that remote Puppeteer instance keeps the browser close to the target environment, cuts repeated startup costs, and lets you centralize management while keeping the same AI automation APIs. In practice you manually: 1. Obtain a CDP WebSocket URL from the remote browser service 2. Use Puppeteer to connect to the remote browser 3. Create a Midscene agent for AI-driven automation #### Prerequisites <PackageManagerTabs command="install puppeteer @midscene/web --save-dev" /> #### Getting a CDP WebSocket URL You can get a CDP WebSocket URL from various sources, for example: * **BrowserBase**: Sign up at https://browserbase.com and get your CDP URL * **Browserless**: Use https://browserless.io or run your own instance * **Local Chrome**: Run Chrome with `--remote-debugging-port=9222` and use `ws://localhost:9222/devtools/browser/...` * **Docker**: Run Chrome in a Docker container with debugging port exposed #### Basic Example ```typescript import puppeteer from 'puppeteer'; import { PuppeteerAgent } from '@midscene/web/puppeteer'; // Assuming you already have a CDP WebSocket URL const cdpWsUrl = 'ws://your-remote-browser.com/devtools/browser/your-session-id'; // Connect to remote browser const browser = await puppeteer.connect({ browserWSEndpoint: cdpWsUrl }); // Get or create page const pages = await browser.pages(); const page = pages[0] || await browser.newPage(); // Create Midscene agent const agent = new PuppeteerAgent(page); // Use AI methods await agent.aiAct('navigate to https://example.com'); await agent.aiAct('click the login button'); const result = await agent.aiQuery('get page title: {title: string}'); // Cleanup await agent.destroy(); await browser.disconnect(); ``` ### Provide custom actions Use `defineAction()` to define custom actions. When constructing the Agent, pass these actions through `customActions`. Midscene appends these actions to the planner so the Agent can call the domain-specific actions you define. ```typescript import { getMidsceneLocationSchema, z } from '@midscene/core'; import { defineAction } from '@midscene/core/device'; const ContinuousClick = defineAction({ name: 'continuousClick', description: 'Click the same target repeatedly', paramSchema: z.object({ locate: getMidsceneLocationSchema(), count: z .number() .int() .positive() .describe('How many times to click'), }), async call(param) { const { locate, count } = param; console.log('click target center', locate.center); console.log('click count', count); // carry out your clicking logic using locate + count }, }); const agent = new PuppeteerAgent(page, { customActions: [ContinuousClick], }); await agent.aiAct('click the red button five times'); ``` Check [Integrate with any interface](/integrate-with-any-interface.md#define-a-custom-action) for more details about defining custom actions. ## FAQ ### Puppeteer installation is slow or gets stuck Puppeteer downloads a browser binary during `postinstall`. If that download is slow or blocked, installation can appear to hang. ```bash # Skip browser download first so dependency installation can finish PUPPETEER_SKIP_DOWNLOAD=true npm install # Then download Chrome from a mirror # This example uses https://registry.npmmirror.com # Different mirror providers may use different base-url rules npx puppeteer browsers install chrome --base-url="https://registry.npmmirror.com/-/binary/chrome-for-testing" ``` ### Cannot click the dropdown If the dropdown options do not appear in the report screenshot after you click the field, the page is usually using a native `select` element. In that case, the expanded dropdown is rendered by the operating system instead of inside the webpage. Midscene enables `forceChromeSelectRendering` by default to force Chrome to render the dropdown; set it to `false` to opt out. For the detailed explanation and how to confirm the issue, see [Playwright FAQ — Cannot click the dropdown](/integrate-with-playwright.md#cannot-click-the-dropdown). ### The webpage continues to flash when running in headed mode This is caused by a mismatch between the viewport's `deviceScaleFactor` and the system pixel ratio. Set `deviceScaleFactor` to `0` to automatically use the device pixel ratio: ```typescript await page.setViewport({ deviceScaleFactor: 0, }); ``` For more details, see [Playwright FAQ — The webpage continues to flash](/integrate-with-playwright.md#the-webpage-continues-to-flash-when-running-in-headed-mode). ### Customize the network timeout Midscene automatically waits for the network to be idle after interactions. You can customize or disable the timeout — see [Playwright FAQ — Customize the network timeout](/integrate-with-playwright.md#customize-the-network-timeout) for details. ## More * For every Agent method, check the [API Reference](/reference.md#interaction-methods). * For the Puppeteer API reference, see [Puppeteer Agent API](/reference.md#puppeteer-agent). * Demo projects * Puppeteer demo: [https://github.com/web-infra-dev/midscene-example/blob/main/puppeteer-demo](https://github.com/web-infra-dev/midscene-example/blob/main/puppeteer-demo) * Playwright + Vitest demo: [https://github.com/web-infra-dev/midscene-example/tree/main/playwright-with-vitest-demo](https://github.com/web-infra-dev/midscene-example/tree/main/playwright-with-vitest-demo) --- url: /introduction.md --- # Midscene.js - GUI Agent for E2E Testing **AI-powered vision. Cross-platform. Batteries included.** Midscene is an open-source SDK for vision-driven UI testing and automation. You describe an operation goal in natural language, and Midscene drives a multimodal model to plan and operate the interface for you — across web, mobile, desktop, and even `<canvas>` surfaces. ## Why Midscene Most UI automation — including AI tools that read the DOM or the accessibility tree — depends on page structure. That structure is fragile and incomplete: selectors break on every refactor; elements without semantic markup (icon-only buttons, custom-rendered controls, `<canvas>`) are invisible to it; native apps and cross-origin iframes are out of reach; and it cannot tell whether something actually *looks* right. Midscene takes a different route: it **works from the screenshot alone**, using a multimodal model, and you describe operation goals and validation conditions in natural language — the way a human tester would. That changes what UI testing feels like: * **Tests stop breaking on every refactor.** There are no selectors to chase when markup or styles change, so the maintenance cost of your suite drops sharply. * **Reach every element and every surface.** If a human can see it, Midscene can target it — even elements with no semantic annotations, `<canvas>`, native apps, and cross-origin iframes that structure-based tools cannot reach. * **Assert on what users actually see.** Verify visual results — colors, highlights, layout, rendered state — not just whether a node exists in the DOM. * **Two ways to test.** Add Midscene to your existing [Playwright](/integrate-with-playwright.md) or Vitest suite, or let an AI agent test your app autonomously through [Skills](/skills.md). * **Failures you can read.** Every run produces a visual report you can replay step by step. > Midscene is built for UI testing first, but the same vision-driven engine handles any UI automation task — use it however fits your work. ## What you can automate Midscene works anywhere you can take a screenshot — web browsers, Android, iOS, HarmonyOS, desktop apps, and [any custom interface](/integrate-with-any-interface.md) — all through one API. Each platform has its own getting-started guide in the sidebar. Write your automation with the JavaScript SDK or in YAML, and look up every method — `aiAct`, `aiQuery`, `aiAssert`, and more — in the [API reference](/reference.md#common). To understand the role of each API and how to choose between `aiAct` and JavaScript orchestration, see [The Basics](/basics.md). ## Driven by Multimodal Models Midscene supports many popular multimodal models with strong UI localization, so you can pick whichever is easiest to access — including open-source options you can self-host: `Qwen3.x`, `Doubao-Seed-2.1`, `GLM-4.6V`, `gemini-3.5-flash`, and `UI-TARS`. See [Supported models and setup](/model-common-config.md) to choose a model and copy its configuration. ## Showcases Register the GitHub form autonomously in a web browser and pass all field validations: <video src="https://lf3-static.bytednsdoc.com/obj/eden-cn/nupipfups/Midscene/1.0-showcases/github2.mp4" height="300" controls /> See more real-world examples across iOS, Android, desktop, and custom interfaces in [Showcases](/showcases.md). ## Benchmark results Midscene achieved the following results on two Android Agent benchmarks: | Benchmark | Result | Evaluation setup | Full report | | --- | --- | --- | --- | | AndroidWorld | Pass@1 **93.10%**, Pass@2 **95.69%**, Pass@3 **97.41%** | Midscene 1.9.5, Gemini-3.5-Flash | [View report](/android-world-benchmark-report.md) | | MobileWorld | Pass@1 **78.63% (92/117)** | Midscene 1.10.3, Gemini-3.6-Flash | [View report](/mobile-world-benchmark-report.md) | The full reports include the run configuration, validation notes, and execution trace for each task. ## Resources & community * Sample projects: [midscene-example](https://github.com/web-infra-dev/midscene-example) * GitHub: [web-infra-dev/midscene](https://github.com/web-infra-dev/midscene) * [Discord](https://discord.gg/2JyBHxszE4) · [X](https://x.com/midscene_ai) · [Lark group (飞书交流群)](https://applink.larkoffice.com/client/chat/chatter/add_by_link?link_token=693v0991-a6bb-4b44-b2e1-365ca0d199ba) ## Credits Midscene builds on many excellent open-source projects — including UI-TARS, Qwen, Playwright, Puppeteer, scrcpy, appium, WebDriverAgent, YADB, and libnut-core. See the [README](https://github.com/web-infra-dev/midscene) for the full list. ## License Midscene.js is [MIT licensed](https://github.com/web-infra-dev/midscene/blob/main/LICENSE). --- url: /llm-txt.md --- # LLMs.txt documentation How to get tools like Cursor, Windstatic, GitHub Copilot, ChatGPT, and Claude to understand Midscene.js. We support LLMs.txt files for making the Midscene.js documentation available to large language models. ## Directory overview The following files are available. * [llms.txt](https://midscenejs.com/llms.txt): The main LLMs.txt file * [llms-full.txt](https://midscenejs.com/llms-full.txt): The complete documentation for Midscene.js ## Usage ### Cursor Use `@Docs` feature in Cursor to include the LLMs.txt files in your project. [Read more](https://docs.cursor.com/context/@-symbols/@-docs) ### Windstatic Reference the LLMs.txt files using `@` or in your `.windsurfrules` files. [Read more](https://docs.windsurf.com/windsurf/getting-started#memories-and-rules) --- url: /mcp.md --- # MCP integration has been retired Midscene no longer ships MCP servers. Use [Skills](/skills.md) to let AI coding agents drive Midscene through the platform CLIs. If you still need the MCP server packages, pin Midscene to `1.9.8`. This is the final version that includes MCP support. Remove any agent configuration that references the retired MCP packages: * `@midscene/web-bridge-mcp` * `@midscene/android-mcp` * `@midscene/ios-mcp` * `@midscene/harmony-mcp` * `@midscene/computer-mcp` * `@midscene/mcp` If your previous MCP configuration set `MIDSCENE_MCP_CHROME_PATH`, move that value to `MIDSCENE_CHROME_PATH` for Skills and CLI usage. The old variable is still accepted as a temporary migration alias. For code-level automation, use the JavaScript SDK, YAML runner, or the platform CLIs listed in [Skills](/skills.md). --- url: /mobile-world-benchmark-report.md --- # Midscene MobileWorld Benchmark Report <style> {` .benchmark-status { display: inline-flex; min-width: 48px; align-items: center; justify-content: center; border-radius: 999px; padding: 2px 8px; font-size: 12px; font-weight: 600; line-height: 18px; } .benchmark-status-pass { color: #15803d; background: rgb(220 252 231 / 70%); border: 1px solid rgb(187 247 208 / 80%); } .benchmark-status-fail { color: #b91c1c; background: rgb(254 226 226 / 75%); border: 1px solid rgb(254 202 202 / 85%); } .dark .benchmark-status-pass { color: #86efac; background: rgb(22 101 52 / 35%); border-color: rgb(34 197 94 / 35%); } .dark .benchmark-status-fail { color: #fca5a5; background: rgb(127 29 29 / 35%); border-color: rgb(248 113 113 / 35%); } .benchmark-round table { width: 100%; table-layout: fixed; } .benchmark-round th, .benchmark-round td { vertical-align: middle; } .benchmark-round th:nth-child(1), .benchmark-round td:nth-child(1) { width: 56px; text-align: center; } .benchmark-round th:nth-child(2), .benchmark-round td:nth-child(2) { width: auto; word-break: break-word; } .benchmark-round th:nth-child(3), .benchmark-round td:nth-child(3) { width: 104px; text-align: center; } .benchmark-round th:nth-child(4), .benchmark-round td:nth-child(4) { width: 112px; } .benchmark-report-link { display: inline-flex; align-items: center; gap: 4px; font-weight: 600; white-space: nowrap; } .benchmark-report-link::after { content: "↗"; font-size: 0.9em; line-height: 1; transform: translateY(-1px); } `} </style> This report presents Midscene results on the MobileWorld benchmark. The evaluation covers 117 tasks, with a **Pass@1 score of 78.63% (92/117)**. :::info About MobileWorld [MobileWorld](https://github.com/Tongyi-MAI/MobileWorld) is an Android Agent benchmark designed for realistic mobile scenarios. It covers cross-app workflows, long-horizon tasks, user interaction, and tool-augmented tasks, with reproducible Android environments used to validate Agent execution results. ::: ## Run configuration | Field | Value | | --- | --- | | Test date | 2026-07-28 | | Model Name | `Gemini-3.6-Flash` | | Midscene version | `1.10.3` | | Device | `DockerEmulator` | | Number of tasks | 117 | | `MIDSCENE_REPLANNING_CYCLE_LIMIT` | 50 | | MobileWorld setup | MobileWorld used a Midscene benchmark adapter that delegates Agent execution through Midscene RPC while retaining MobileWorld task setup and validation. | | Validation notes | A MobileWorld validator was aligned with the task intent. The affected case is listed below. | ## Validation Condition Updates The following MobileWorld validation check was adjusted for this benchmark: | Change | Affected case | | --- | --- | | The previous validator required a full five-day interval between `created_at` and `expires_at`. For example, if the start date is the 16th, “five days later” is the 21st; however, differences in the converted time of day can make the interval shorter than 5 × 24 hours, causing the previous rule to fail. The validator now accepts intervals longer than 4 days and no longer than 5 days. | `MastodonNewFilterTask` | ## Report files The detailed reports for all 117 tasks are listed below. Each report is a compressed, self-contained HTML file. Select “report” to open the corresponding execution trace in a new page. <details open className="benchmark-round"> <summary>Round 1 (117 reports · 92 PASS · 25 FAIL)</summary> | # | Task | Status | Report | | --- | --- | --- | --- | | 1 | AcceptMeetingTask | <span className="benchmark-status benchmark-status-pass">PASS</span> | <a href="https://lf3-static.bytednsdoc.com/obj/eden-cn/luljzkpt/ljhwZthlaukjlkulzlp/benchmark/MobileWorld/20260728/20260728/Task-1-AcceptMeetingTask__group-0-b3c13bb1-7b21-4fbf-8d3f-65fcdeb309e0-Pass.html" className="benchmark-report-link" target="_blank" rel="noreferrer" title="Open report in a new page">report</a> | | 2 | AdjustBrightnessMaximumTask | <span className="benchmark-status benchmark-status-pass">PASS</span> | <a href="https://lf3-static.bytednsdoc.com/obj/eden-cn/luljzkpt/ljhwZthlaukjlkulzlp/benchmark/MobileWorld/20260728/20260728/Task-2-AdjustBrightnessMaximumTask__group-0-b3c13bb1-7b21-4fbf-8d3f-65fcdeb309e0-Pass.html" className="benchmark-report-link" target="_blank" rel="noreferrer" title="Open report in a new page">report</a> | | 3 | AdjustBrightnessMinimumTask | <span className="benchmark-status benchmark-status-pass">PASS</span> | <a href="https://lf3-static.bytednsdoc.com/obj/eden-cn/luljzkpt/ljhwZthlaukjlkulzlp/benchmark/MobileWorld/20260728/20260728/Task-3-AdjustBrightnessMinimumTask__group-0-b3c13bb1-7b21-4fbf-8d3f-65fcdeb309e0-Pass.html" className="benchmark-report-link" target="_blank" rel="noreferrer" title="Open report in a new page">report</a> | | 4 | AdjustFontIconMaximumTask | <span className="benchmark-status benchmark-status-pass">PASS</span> | <a href="https://lf3-static.bytednsdoc.com/obj/eden-cn/luljzkpt/ljhwZthlaukjlkulzlp/benchmark/MobileWorld/20260728/20260728/Task-4-AdjustFontIconMaximumTask__group-0-b3c13bb1-7b21-4fbf-8d3f-65fcdeb309e0-Pass.html" className="benchmark-report-link" target="_blank" rel="noreferrer" title="Open report in a new page">report</a> | | 5 | AdjustFontIconMinimumTask | <span className="benchmark-status benchmark-status-pass">PASS</span> | <a href="https://lf3-static.bytednsdoc.com/obj/eden-cn/luljzkpt/ljhwZthlaukjlkulzlp/benchmark/MobileWorld/20260728/20260728/Task-5-AdjustFontIconMinimumTask__group-0-b3c13bb1-7b21-4fbf-8d3f-65fcdeb309e0-Pass.html" className="benchmark-report-link" target="_blank" rel="noreferrer" title="Open report in a new page">report</a> | | 6 | BidFileRenameTask | <span className="benchmark-status benchmark-status-pass">PASS</span> | <a href="https://lf3-static.bytednsdoc.com/obj/eden-cn/luljzkpt/ljhwZthlaukjlkulzlp/benchmark/MobileWorld/20260728/20260728/Task-6-BidFileRenameTask__group-0-b3c13bb1-7b21-4fbf-8d3f-65fcdeb309e0-Pass.html" className="benchmark-report-link" target="_blank" rel="noreferrer" title="Open report in a new page">report</a> | | 7 | CVEmailTask | <span className="benchmark-status benchmark-status-pass">PASS</span> | <a href="https://lf3-static.bytednsdoc.com/obj/eden-cn/luljzkpt/ljhwZthlaukjlkulzlp/benchmark/MobileWorld/20260728/20260728/Task-7-CVEmailTask__group-0-b3c13bb1-7b21-4fbf-8d3f-65fcdeb309e0-Pass.html" className="benchmark-report-link" target="_blank" rel="noreferrer" title="Open report in a new page">report</a> | | 8 | CancelMeetingTask | <span className="benchmark-status benchmark-status-pass">PASS</span> | <a href="https://lf3-static.bytednsdoc.com/obj/eden-cn/luljzkpt/ljhwZthlaukjlkulzlp/benchmark/MobileWorld/20260728/20260728/Task-8-CancelMeetingTask__group-0-b3c13bb1-7b21-4fbf-8d3f-65fcdeb309e0-Pass.html" className="benchmark-report-link" target="_blank" rel="noreferrer" title="Open report in a new page">report</a> | | 9 | CartInfoNotificationTask | <span className="benchmark-status benchmark-status-pass">PASS</span> | <a href="https://lf3-static.bytednsdoc.com/obj/eden-cn/luljzkpt/ljhwZthlaukjlkulzlp/benchmark/MobileWorld/20260728/20260728/Task-9-CartInfoNotificationTask__group-0-b3c13bb1-7b21-4fbf-8d3f-65fcdeb309e0-Pass.html" className="benchmark-report-link" target="_blank" rel="noreferrer" title="Open report in a new page">report</a> | | 10 | CartManagementTask | <span className="benchmark-status benchmark-status-fail">FAIL</span> | <a href="https://lf3-static.bytednsdoc.com/obj/eden-cn/luljzkpt/ljhwZthlaukjlkulzlp/benchmark/MobileWorld/20260728/20260728/Task-10-CartManagementTask__group-0-b3c13bb1-7b21-4fbf-8d3f-65fcdeb309e0-Fail.html" className="benchmark-report-link" target="_blank" rel="noreferrer" title="Open report in a new page">report</a> | | 11 | ChangeWallpaperTask | <span className="benchmark-status benchmark-status-pass">PASS</span> | <a href="https://lf3-static.bytednsdoc.com/obj/eden-cn/luljzkpt/ljhwZthlaukjlkulzlp/benchmark/MobileWorld/20260728/20260728/Task-11-ChangeWallpaperTask__group-0-b3c13bb1-7b21-4fbf-8d3f-65fcdeb309e0-Pass.html" className="benchmark-report-link" target="_blank" rel="noreferrer" title="Open report in a new page">report</a> | | 12 | CheckCartPriceTask | <span className="benchmark-status benchmark-status-pass">PASS</span> | <a href="https://lf3-static.bytednsdoc.com/obj/eden-cn/luljzkpt/ljhwZthlaukjlkulzlp/benchmark/MobileWorld/20260728/20260728/Task-12-CheckCartPriceTask__group-0-b3c13bb1-7b21-4fbf-8d3f-65fcdeb309e0-Pass.html" className="benchmark-report-link" target="_blank" rel="noreferrer" title="Open report in a new page">report</a> | | 13 | CheckConferenceAndSendSmsTask1 | <span className="benchmark-status benchmark-status-pass">PASS</span> | <a href="https://lf3-static.bytednsdoc.com/obj/eden-cn/luljzkpt/ljhwZthlaukjlkulzlp/benchmark/MobileWorld/20260728/20260728/Task-13-CheckConferenceAndSendSmsTask1__group-0-b3c13bb1-7b21-4fbf-8d3f-65fcdeb309e0-Pass.html" className="benchmark-report-link" target="_blank" rel="noreferrer" title="Open report in a new page">report</a> | | 14 | CheckConferenceAndSendSmsTask2 | <span className="benchmark-status benchmark-status-fail">FAIL</span> | <a href="https://lf3-static.bytednsdoc.com/obj/eden-cn/luljzkpt/ljhwZthlaukjlkulzlp/benchmark/MobileWorld/20260728/20260728/Task-14-CheckConferenceAndSendSmsTask2__group-0-b3c13bb1-7b21-4fbf-8d3f-65fcdeb309e0-Fail.html" className="benchmark-report-link" target="_blank" rel="noreferrer" title="Open report in a new page">report</a> | | 15 | CheckConferenceDurationTask | <span className="benchmark-status benchmark-status-pass">PASS</span> | <a href="https://lf3-static.bytednsdoc.com/obj/eden-cn/luljzkpt/ljhwZthlaukjlkulzlp/benchmark/MobileWorld/20260728/20260728/Task-15-CheckConferenceDurationTask__group-0-b3c13bb1-7b21-4fbf-8d3f-65fcdeb309e0-Pass.html" className="benchmark-report-link" target="_blank" rel="noreferrer" title="Open report in a new page">report</a> | | 16 | CheckConferenceLocationTask | <span className="benchmark-status benchmark-status-fail">FAIL</span> | <a href="https://lf3-static.bytednsdoc.com/obj/eden-cn/luljzkpt/ljhwZthlaukjlkulzlp/benchmark/MobileWorld/20260728/20260728/Task-16-CheckConferenceLocationTask__group-0-b3c13bb1-7b21-4fbf-8d3f-65fcdeb309e0-Fail.html" className="benchmark-report-link" target="_blank" rel="noreferrer" title="Open report in a new page">report</a> | | 17 | CheckDeduplicatedEventsTask | <span className="benchmark-status benchmark-status-pass">PASS</span> | <a href="https://lf3-static.bytednsdoc.com/obj/eden-cn/luljzkpt/ljhwZthlaukjlkulzlp/benchmark/MobileWorld/20260728/20260728/Task-17-CheckDeduplicatedEventsTask__group-0-b3c13bb1-7b21-4fbf-8d3f-65fcdeb309e0-Pass.html" className="benchmark-report-link" target="_blank" rel="noreferrer" title="Open report in a new page">report</a> | | 18 | CheckDepartTimeTask | <span className="benchmark-status benchmark-status-pass">PASS</span> | <a href="https://lf3-static.bytednsdoc.com/obj/eden-cn/luljzkpt/ljhwZthlaukjlkulzlp/benchmark/MobileWorld/20260728/20260728/Task-18-CheckDepartTimeTask__group-0-b3c13bb1-7b21-4fbf-8d3f-65fcdeb309e0-Pass.html" className="benchmark-report-link" target="_blank" rel="noreferrer" title="Open report in a new page">report</a> | | 19 | CheckEventTimeTask | <span className="benchmark-status benchmark-status-pass">PASS</span> | <a href="https://lf3-static.bytednsdoc.com/obj/eden-cn/luljzkpt/ljhwZthlaukjlkulzlp/benchmark/MobileWorld/20260728/20260728/Task-19-CheckEventTimeTask__group-0-b3c13bb1-7b21-4fbf-8d3f-65fcdeb309e0-Pass.html" className="benchmark-report-link" target="_blank" rel="noreferrer" title="Open report in a new page">report</a> | | 20 | CheckGithubInfoTask | <span className="benchmark-status benchmark-status-pass">PASS</span> | <a href="https://lf3-static.bytednsdoc.com/obj/eden-cn/luljzkpt/ljhwZthlaukjlkulzlp/benchmark/MobileWorld/20260728/20260728/Task-20-CheckGithubInfoTask__group-0-b3c13bb1-7b21-4fbf-8d3f-65fcdeb309e0-Pass.html" className="benchmark-report-link" target="_blank" rel="noreferrer" title="Open report in a new page">report</a> | | 21 | CheckInterviewTimesTask | <span className="benchmark-status benchmark-status-fail">FAIL</span> | <a href="https://lf3-static.bytednsdoc.com/obj/eden-cn/luljzkpt/ljhwZthlaukjlkulzlp/benchmark/MobileWorld/20260728/20260728/Task-21-CheckInterviewTimesTask__group-0-b3c13bb1-7b21-4fbf-8d3f-65fcdeb309e0-Fail.html" className="benchmark-report-link" target="_blank" rel="noreferrer" title="Open report in a new page">report</a> | | 22 | CheckInvoiceTask1 | <span className="benchmark-status benchmark-status-pass">PASS</span> | <a href="https://lf3-static.bytednsdoc.com/obj/eden-cn/luljzkpt/ljhwZthlaukjlkulzlp/benchmark/MobileWorld/20260728/20260728/Task-22-CheckInvoiceTask1__group-0-b3c13bb1-7b21-4fbf-8d3f-65fcdeb309e0-Pass.html" className="benchmark-report-link" target="_blank" rel="noreferrer" title="Open report in a new page">report</a> | | 23 | CheckInvoiceTask2 | <span className="benchmark-status benchmark-status-pass">PASS</span> | <a href="https://lf3-static.bytednsdoc.com/obj/eden-cn/luljzkpt/ljhwZthlaukjlkulzlp/benchmark/MobileWorld/20260728/20260728/Task-23-CheckInvoiceTask2__group-0-b3c13bb1-7b21-4fbf-8d3f-65fcdeb309e0-Pass.html" className="benchmark-report-link" target="_blank" rel="noreferrer" title="Open report in a new page">report</a> | | 24 | CheckInvoiceTask3 | <span className="benchmark-status benchmark-status-pass">PASS</span> | <a href="https://lf3-static.bytednsdoc.com/obj/eden-cn/luljzkpt/ljhwZthlaukjlkulzlp/benchmark/MobileWorld/20260728/20260728/Task-24-CheckInvoiceTask3__group-0-b3c13bb1-7b21-4fbf-8d3f-65fcdeb309e0-Pass.html" className="benchmark-report-link" target="_blank" rel="noreferrer" title="Open report in a new page">report</a> | | 25 | CheckPuchasedItem | <span className="benchmark-status benchmark-status-pass">PASS</span> | <a href="https://lf3-static.bytednsdoc.com/obj/eden-cn/luljzkpt/ljhwZthlaukjlkulzlp/benchmark/MobileWorld/20260728/20260728/Task-25-CheckPuchasedItem__group-0-b3c13bb1-7b21-4fbf-8d3f-65fcdeb309e0-Pass.html" className="benchmark-report-link" target="_blank" rel="noreferrer" title="Open report in a new page">report</a> | | 26 | CheckRegistrationTask | <span className="benchmark-status benchmark-status-pass">PASS</span> | <a href="https://lf3-static.bytednsdoc.com/obj/eden-cn/luljzkpt/ljhwZthlaukjlkulzlp/benchmark/MobileWorld/20260728/20260728/Task-26-CheckRegistrationTask__group-0-b3c13bb1-7b21-4fbf-8d3f-65fcdeb309e0-Pass.html" className="benchmark-report-link" target="_blank" rel="noreferrer" title="Open report in a new page">report</a> | | 27 | CheckSetMeetTimeTask | <span className="benchmark-status benchmark-status-pass">PASS</span> | <a href="https://lf3-static.bytednsdoc.com/obj/eden-cn/luljzkpt/ljhwZthlaukjlkulzlp/benchmark/MobileWorld/20260728/20260728/Task-27-CheckSetMeetTimeTask__group-0-b3c13bb1-7b21-4fbf-8d3f-65fcdeb309e0-Pass.html" className="benchmark-report-link" target="_blank" rel="noreferrer" title="Open report in a new page">report</a> | | 28 | ChromeSearchBeijingWeatherTask | <span className="benchmark-status benchmark-status-pass">PASS</span> | <a href="https://lf3-static.bytednsdoc.com/obj/eden-cn/luljzkpt/ljhwZthlaukjlkulzlp/benchmark/MobileWorld/20260728/20260728/Task-28-ChromeSearchBeijingWeatherTask__group-0-b3c13bb1-7b21-4fbf-8d3f-65fcdeb309e0-Pass.html" className="benchmark-report-link" target="_blank" rel="noreferrer" title="Open report in a new page">report</a> | | 29 | CloseFlightModeTask | <span className="benchmark-status benchmark-status-pass">PASS</span> | <a href="https://lf3-static.bytednsdoc.com/obj/eden-cn/luljzkpt/ljhwZthlaukjlkulzlp/benchmark/MobileWorld/20260728/20260728/Task-29-CloseFlightModeTask__group-0-b3c13bb1-7b21-4fbf-8d3f-65fcdeb309e0-Pass.html" className="benchmark-report-link" target="_blank" rel="noreferrer" title="Open report in a new page">report</a> | | 30 | CountFileLinesTask | <span className="benchmark-status benchmark-status-pass">PASS</span> | <a href="https://lf3-static.bytednsdoc.com/obj/eden-cn/luljzkpt/ljhwZthlaukjlkulzlp/benchmark/MobileWorld/20260728/20260728/Task-30-CountFileLinesTask__group-0-b3c13bb1-7b21-4fbf-8d3f-65fcdeb309e0-Pass.html" className="benchmark-report-link" target="_blank" rel="noreferrer" title="Open report in a new page">report</a> | | 31 | DownloadSendReceiptTask | <span className="benchmark-status benchmark-status-pass">PASS</span> | <a href="https://lf3-static.bytednsdoc.com/obj/eden-cn/luljzkpt/ljhwZthlaukjlkulzlp/benchmark/MobileWorld/20260728/20260728/Task-31-DownloadSendReceiptTask__group-0-b3c13bb1-7b21-4fbf-8d3f-65fcdeb309e0-Pass.html" className="benchmark-report-link" target="_blank" rel="noreferrer" title="Open report in a new page">report</a> | | 32 | GoogleMapsAlibabaPhoneContactTask | <span className="benchmark-status benchmark-status-pass">PASS</span> | <a href="https://lf3-static.bytednsdoc.com/obj/eden-cn/luljzkpt/ljhwZthlaukjlkulzlp/benchmark/MobileWorld/20260728/20260728/Task-32-GoogleMapsAlibabaPhoneContactTask__group-0-b3c13bb1-7b21-4fbf-8d3f-65fcdeb309e0-Pass.html" className="benchmark-report-link" target="_blank" rel="noreferrer" title="Open report in a new page">report</a> | | 33 | GoogleMapsAlibabaSouthNeighborTask | <span className="benchmark-status benchmark-status-pass">PASS</span> | <a href="https://lf3-static.bytednsdoc.com/obj/eden-cn/luljzkpt/ljhwZthlaukjlkulzlp/benchmark/MobileWorld/20260728/20260728/Task-33-GoogleMapsAlibabaSouthNeighborTask__group-0-b3c13bb1-7b21-4fbf-8d3f-65fcdeb309e0-Pass.html" className="benchmark-report-link" target="_blank" rel="noreferrer" title="Open report in a new page">report</a> | | 34 | GraduationMassEmailTask | <span className="benchmark-status benchmark-status-pass">PASS</span> | <a href="https://lf3-static.bytednsdoc.com/obj/eden-cn/luljzkpt/ljhwZthlaukjlkulzlp/benchmark/MobileWorld/20260728/20260728/Task-34-GraduationMassEmailTask__group-0-b3c13bb1-7b21-4fbf-8d3f-65fcdeb309e0-Pass.html" className="benchmark-report-link" target="_blank" rel="noreferrer" title="Open report in a new page">report</a> | | 35 | InvoiceReceiptCopyAskUserTask | <span className="benchmark-status benchmark-status-fail">FAIL</span> | <a href="https://lf3-static.bytednsdoc.com/obj/eden-cn/luljzkpt/ljhwZthlaukjlkulzlp/benchmark/MobileWorld/20260728/20260728/Task-35-InvoiceReceiptCopyAskUserTask__group-0-b3c13bb1-7b21-4fbf-8d3f-65fcdeb309e0-Fail.html" className="benchmark-report-link" target="_blank" rel="noreferrer" title="Open report in a new page">report</a> | | 36 | InvoiceReceiptCopyTask | <span className="benchmark-status benchmark-status-pass">PASS</span> | <a href="https://lf3-static.bytednsdoc.com/obj/eden-cn/luljzkpt/ljhwZthlaukjlkulzlp/benchmark/MobileWorld/20260728/20260728/Task-36-InvoiceReceiptCopyTask__group-0-b3c13bb1-7b21-4fbf-8d3f-65fcdeb309e0-Pass.html" className="benchmark-report-link" target="_blank" rel="noreferrer" title="Open report in a new page">report</a> | | 37 | ItemCheckoutTask | <span className="benchmark-status benchmark-status-pass">PASS</span> | <a href="https://lf3-static.bytednsdoc.com/obj/eden-cn/luljzkpt/ljhwZthlaukjlkulzlp/benchmark/MobileWorld/20260728/20260728/Task-37-ItemCheckoutTask__group-0-b3c13bb1-7b21-4fbf-8d3f-65fcdeb309e0-Pass.html" className="benchmark-report-link" target="_blank" rel="noreferrer" title="Open report in a new page">report</a> | | 38 | LocalFileManagementTask | <span className="benchmark-status benchmark-status-pass">PASS</span> | <a href="https://lf3-static.bytednsdoc.com/obj/eden-cn/luljzkpt/ljhwZthlaukjlkulzlp/benchmark/MobileWorld/20260728/20260728/Task-38-LocalFileManagementTask__group-0-b3c13bb1-7b21-4fbf-8d3f-65fcdeb309e0-Pass.html" className="benchmark-report-link" target="_blank" rel="noreferrer" title="Open report in a new page">report</a> | | 39 | LocalFileManagementTask2 | <span className="benchmark-status benchmark-status-pass">PASS</span> | <a href="https://lf3-static.bytednsdoc.com/obj/eden-cn/luljzkpt/ljhwZthlaukjlkulzlp/benchmark/MobileWorld/20260728/20260728/Task-39-LocalFileManagementTask2__group-0-b3c13bb1-7b21-4fbf-8d3f-65fcdeb309e0-Pass.html" className="benchmark-report-link" target="_blank" rel="noreferrer" title="Open report in a new page">report</a> | | 40 | MastodonAddBookmarkTask | <span className="benchmark-status benchmark-status-pass">PASS</span> | <a href="https://lf3-static.bytednsdoc.com/obj/eden-cn/luljzkpt/ljhwZthlaukjlkulzlp/benchmark/MobileWorld/20260728/20260728/Task-40-MastodonAddBookmarkTask__group-1-0a3fa62d-94f0-4f16-ba4d-7ac1fb2a8596-Pass.html" className="benchmark-report-link" target="_blank" rel="noreferrer" title="Open report in a new page">report</a> | | 41 | MastodonAddFeaturedHashtagsTask | <span className="benchmark-status benchmark-status-fail">FAIL</span> | <a href="https://lf3-static.bytednsdoc.com/obj/eden-cn/luljzkpt/ljhwZthlaukjlkulzlp/benchmark/MobileWorld/20260728/20260728/Task-41-MastodonAddFeaturedHashtagsTask__group-1-0a3fa62d-94f0-4f16-ba4d-7ac1fb2a8596-Fail.html" className="benchmark-report-link" target="_blank" rel="noreferrer" title="Open report in a new page">report</a> | | 42 | MastodonAdjustTootsTask | <span className="benchmark-status benchmark-status-pass">PASS</span> | <a href="https://lf3-static.bytednsdoc.com/obj/eden-cn/luljzkpt/ljhwZthlaukjlkulzlp/benchmark/MobileWorld/20260728/20260728/Task-42-MastodonAdjustTootsTask__group-1-0a3fa62d-94f0-4f16-ba4d-7ac1fb2a8596-Pass.html" className="benchmark-report-link" target="_blank" rel="noreferrer" title="Open report in a new page">report</a> | | 43 | MastodonCalendarMultiMemosTask | <span className="benchmark-status benchmark-status-fail">FAIL</span> | <a href="https://lf3-static.bytednsdoc.com/obj/eden-cn/luljzkpt/ljhwZthlaukjlkulzlp/benchmark/MobileWorld/20260728/20260728/Task-43-MastodonCalendarMultiMemosTask__group-1-0a3fa62d-94f0-4f16-ba4d-7ac1fb2a8596-Fail.html" className="benchmark-report-link" target="_blank" rel="noreferrer" title="Open report in a new page">report</a> | | 44 | MastodonChangeHeaderTask | <span className="benchmark-status benchmark-status-pass">PASS</span> | <a href="https://lf3-static.bytednsdoc.com/obj/eden-cn/luljzkpt/ljhwZthlaukjlkulzlp/benchmark/MobileWorld/20260728/20260728/Task-44-MastodonChangeHeaderTask__group-1-0a3fa62d-94f0-4f16-ba4d-7ac1fb2a8596-Pass.html" className="benchmark-report-link" target="_blank" rel="noreferrer" title="Open report in a new page">report</a> | | 45 | MastodonChangeLanguageTask | <span className="benchmark-status benchmark-status-fail">FAIL</span> | <a href="https://lf3-static.bytednsdoc.com/obj/eden-cn/luljzkpt/ljhwZthlaukjlkulzlp/benchmark/MobileWorld/20260728/20260728/Task-45-MastodonChangeLanguageTask__group-1-0a3fa62d-94f0-4f16-ba4d-7ac1fb2a8596-Fail.html" className="benchmark-report-link" target="_blank" rel="noreferrer" title="Open report in a new page">report</a> | | 46 | MastodonConditionalFavoTask | <span className="benchmark-status benchmark-status-pass">PASS</span> | <a href="https://lf3-static.bytednsdoc.com/obj/eden-cn/luljzkpt/ljhwZthlaukjlkulzlp/benchmark/MobileWorld/20260728/20260728/Task-46-MastodonConditionalFavoTask__group-1-0a3fa62d-94f0-4f16-ba4d-7ac1fb2a8596-Pass.html" className="benchmark-report-link" target="_blank" rel="noreferrer" title="Open report in a new page">report</a> | | 47 | MastodonCreateListTask | <span className="benchmark-status benchmark-status-pass">PASS</span> | <a href="https://lf3-static.bytednsdoc.com/obj/eden-cn/luljzkpt/ljhwZthlaukjlkulzlp/benchmark/MobileWorld/20260728/20260728/Task-47-MastodonCreateListTask__group-1-0a3fa62d-94f0-4f16-ba4d-7ac1fb2a8596-Pass.html" className="benchmark-report-link" target="_blank" rel="noreferrer" title="Open report in a new page">report</a> | | 48 | MastodonCreateMemoTask | <span className="benchmark-status benchmark-status-pass">PASS</span> | <a href="https://lf3-static.bytednsdoc.com/obj/eden-cn/luljzkpt/ljhwZthlaukjlkulzlp/benchmark/MobileWorld/20260728/20260728/Task-48-MastodonCreateMemoTask__group-1-0a3fa62d-94f0-4f16-ba4d-7ac1fb2a8596-Pass.html" className="benchmark-report-link" target="_blank" rel="noreferrer" title="Open report in a new page">report</a> | | 49 | MastodonExportFollowsTask | <span className="benchmark-status benchmark-status-pass">PASS</span> | <a href="https://lf3-static.bytednsdoc.com/obj/eden-cn/luljzkpt/ljhwZthlaukjlkulzlp/benchmark/MobileWorld/20260728/20260728/Task-49-MastodonExportFollowsTask__group-1-0a3fa62d-94f0-4f16-ba4d-7ac1fb2a8596-Pass.html" className="benchmark-report-link" target="_blank" rel="noreferrer" title="Open report in a new page">report</a> | | 50 | MastodonFavoriteTootsTask | <span className="benchmark-status benchmark-status-pass">PASS</span> | <a href="https://lf3-static.bytednsdoc.com/obj/eden-cn/luljzkpt/ljhwZthlaukjlkulzlp/benchmark/MobileWorld/20260728/20260728/Task-50-MastodonFavoriteTootsTask__group-1-0a3fa62d-94f0-4f16-ba4d-7ac1fb2a8596-Pass.html" className="benchmark-report-link" target="_blank" rel="noreferrer" title="Open report in a new page">report</a> | | 51 | MastodonFilterLanguageTask | <span className="benchmark-status benchmark-status-pass">PASS</span> | <a href="https://lf3-static.bytednsdoc.com/obj/eden-cn/luljzkpt/ljhwZthlaukjlkulzlp/benchmark/MobileWorld/20260728/20260728/Task-51-MastodonFilterLanguageTask__group-1-0a3fa62d-94f0-4f16-ba4d-7ac1fb2a8596-Pass.html" className="benchmark-report-link" target="_blank" rel="noreferrer" title="Open report in a new page">report</a> | | 52 | MastodonFollowTask | <span className="benchmark-status benchmark-status-pass">PASS</span> | <a href="https://lf3-static.bytednsdoc.com/obj/eden-cn/luljzkpt/ljhwZthlaukjlkulzlp/benchmark/MobileWorld/20260728/20260728/Task-52-MastodonFollowTask__group-1-0a3fa62d-94f0-4f16-ba4d-7ac1fb2a8596-Pass.html" className="benchmark-report-link" target="_blank" rel="noreferrer" title="Open report in a new page">report</a> | | 53 | MastodonGetServerInfoTask | <span className="benchmark-status benchmark-status-fail">FAIL</span> | <a href="https://lf3-static.bytednsdoc.com/obj/eden-cn/luljzkpt/ljhwZthlaukjlkulzlp/benchmark/MobileWorld/20260728/20260728/Task-53-MastodonGetServerInfoTask__group-1-0a3fa62d-94f0-4f16-ba4d-7ac1fb2a8596-Fail.html" className="benchmark-report-link" target="_blank" rel="noreferrer" title="Open report in a new page">report</a> | | 54 | MastodonImportMutedUsersTask | <span className="benchmark-status benchmark-status-pass">PASS</span> | <a href="https://lf3-static.bytednsdoc.com/obj/eden-cn/luljzkpt/ljhwZthlaukjlkulzlp/benchmark/MobileWorld/20260728/20260728/Task-54-MastodonImportMutedUsersTask__group-1-0a3fa62d-94f0-4f16-ba4d-7ac1fb2a8596-Pass.html" className="benchmark-report-link" target="_blank" rel="noreferrer" title="Open report in a new page">report</a> | | 55 | MastodonInviteTask | <span className="benchmark-status benchmark-status-pass">PASS</span> | <a href="https://lf3-static.bytednsdoc.com/obj/eden-cn/luljzkpt/ljhwZthlaukjlkulzlp/benchmark/MobileWorld/20260728/20260728/Task-55-MastodonInviteTask__group-1-0a3fa62d-94f0-4f16-ba4d-7ac1fb2a8596-Pass.html" className="benchmark-report-link" target="_blank" rel="noreferrer" title="Open report in a new page">report</a> | | 56 | MastodonMallPurchaseCommodityTask | <span className="benchmark-status benchmark-status-pass">PASS</span> | <a href="https://lf3-static.bytednsdoc.com/obj/eden-cn/luljzkpt/ljhwZthlaukjlkulzlp/benchmark/MobileWorld/20260728/20260728/Task-56-MastodonMallPurchaseCommodityTask__group-1-0a3fa62d-94f0-4f16-ba4d-7ac1fb2a8596-Pass.html" className="benchmark-report-link" target="_blank" rel="noreferrer" title="Open report in a new page">report</a> | | 57 | MastodonMallShareOrderTask | <span className="benchmark-status benchmark-status-pass">PASS</span> | <a href="https://lf3-static.bytednsdoc.com/obj/eden-cn/luljzkpt/ljhwZthlaukjlkulzlp/benchmark/MobileWorld/20260728/20260728/Task-57-MastodonMallShareOrderTask__group-1-0a3fa62d-94f0-4f16-ba4d-7ac1fb2a8596-Pass.html" className="benchmark-report-link" target="_blank" rel="noreferrer" title="Open report in a new page">report</a> | | 58 | MastodonManageHashtagsTask | <span className="benchmark-status benchmark-status-pass">PASS</span> | <a href="https://lf3-static.bytednsdoc.com/obj/eden-cn/luljzkpt/ljhwZthlaukjlkulzlp/benchmark/MobileWorld/20260728/20260728/Task-58-MastodonManageHashtagsTask__group-1-0a3fa62d-94f0-4f16-ba4d-7ac1fb2a8596-Pass.html" className="benchmark-report-link" target="_blank" rel="noreferrer" title="Open report in a new page">report</a> | | 59 | MastodonManageMultiListTask | <span className="benchmark-status benchmark-status-pass">PASS</span> | <a href="https://lf3-static.bytednsdoc.com/obj/eden-cn/luljzkpt/ljhwZthlaukjlkulzlp/benchmark/MobileWorld/20260728/20260728/Task-59-MastodonManageMultiListTask__group-1-0a3fa62d-94f0-4f16-ba4d-7ac1fb2a8596-Pass.html" className="benchmark-report-link" target="_blank" rel="noreferrer" title="Open report in a new page">report</a> | | 60 | MastodonMattermostPostNoticeTask | <span className="benchmark-status benchmark-status-pass">PASS</span> | <a href="https://lf3-static.bytednsdoc.com/obj/eden-cn/luljzkpt/ljhwZthlaukjlkulzlp/benchmark/MobileWorld/20260728/20260728/Task-60-MastodonMattermostPostNoticeTask__group-1-0a3fa62d-94f0-4f16-ba4d-7ac1fb2a8596-Pass.html" className="benchmark-report-link" target="_blank" rel="noreferrer" title="Open report in a new page">report</a> | | 61 | MastodonMultiInviteTask | <span className="benchmark-status benchmark-status-pass">PASS</span> | <a href="https://lf3-static.bytednsdoc.com/obj/eden-cn/luljzkpt/ljhwZthlaukjlkulzlp/benchmark/MobileWorld/20260728/20260728/Task-61-MastodonMultiInviteTask__group-1-0a3fa62d-94f0-4f16-ba4d-7ac1fb2a8596-Pass.html" className="benchmark-report-link" target="_blank" rel="noreferrer" title="Open report in a new page">report</a> | | 62 | MastodonNewFilterTask | <span className="benchmark-status benchmark-status-fail">FAIL</span> | <a href="https://lf3-static.bytednsdoc.com/obj/eden-cn/luljzkpt/ljhwZthlaukjlkulzlp/benchmark/MobileWorld/20260728/20260728/Task-62-MastodonNewFilterTask__group-1-0a3fa62d-94f0-4f16-ba4d-7ac1fb2a8596-Fail.html" className="benchmark-report-link" target="_blank" rel="noreferrer" title="Open report in a new page">report</a> | | 63 | MastodonNewPostTask | <span className="benchmark-status benchmark-status-pass">PASS</span> | <a href="https://lf3-static.bytednsdoc.com/obj/eden-cn/luljzkpt/ljhwZthlaukjlkulzlp/benchmark/MobileWorld/20260728/20260728/Task-63-MastodonNewPostTask__group-1-0a3fa62d-94f0-4f16-ba4d-7ac1fb2a8596-Pass.html" className="benchmark-report-link" target="_blank" rel="noreferrer" title="Open report in a new page">report</a> | | 64 | MastodonOpenAutomatedDeletionTask | <span className="benchmark-status benchmark-status-pass">PASS</span> | <a href="https://lf3-static.bytednsdoc.com/obj/eden-cn/luljzkpt/ljhwZthlaukjlkulzlp/benchmark/MobileWorld/20260728/20260728/Task-64-MastodonOpenAutomatedDeletionTask__group-1-0a3fa62d-94f0-4f16-ba4d-7ac1fb2a8596-Pass.html" className="benchmark-report-link" target="_blank" rel="noreferrer" title="Open report in a new page">report</a> | | 65 | MastodonPinTootsTask | <span className="benchmark-status benchmark-status-pass">PASS</span> | <a href="https://lf3-static.bytednsdoc.com/obj/eden-cn/luljzkpt/ljhwZthlaukjlkulzlp/benchmark/MobileWorld/20260728/20260728/Task-65-MastodonPinTootsTask__group-1-0a3fa62d-94f0-4f16-ba4d-7ac1fb2a8596-Pass.html" className="benchmark-report-link" target="_blank" rel="noreferrer" title="Open report in a new page">report</a> | | 66 | MastodonPostEditedPhotoTask | <span className="benchmark-status benchmark-status-pass">PASS</span> | <a href="https://lf3-static.bytednsdoc.com/obj/eden-cn/luljzkpt/ljhwZthlaukjlkulzlp/benchmark/MobileWorld/20260728/20260728/Task-66-MastodonPostEditedPhotoTask__group-1-0a3fa62d-94f0-4f16-ba4d-7ac1fb2a8596-Pass.html" className="benchmark-report-link" target="_blank" rel="noreferrer" title="Open report in a new page">report</a> | | 67 | MastodonPostPollTask | <span className="benchmark-status benchmark-status-pass">PASS</span> | <a href="https://lf3-static.bytednsdoc.com/obj/eden-cn/luljzkpt/ljhwZthlaukjlkulzlp/benchmark/MobileWorld/20260728/20260728/Task-67-MastodonPostPollTask__group-1-0a3fa62d-94f0-4f16-ba4d-7ac1fb2a8596-Pass.html" className="benchmark-report-link" target="_blank" rel="noreferrer" title="Open report in a new page">report</a> | | 68 | MastodonRemoveBookmarkTask | <span className="benchmark-status benchmark-status-pass">PASS</span> | <a href="https://lf3-static.bytednsdoc.com/obj/eden-cn/luljzkpt/ljhwZthlaukjlkulzlp/benchmark/MobileWorld/20260728/20260728/Task-68-MastodonRemoveBookmarkTask__group-1-0a3fa62d-94f0-4f16-ba4d-7ac1fb2a8596-Pass.html" className="benchmark-report-link" target="_blank" rel="noreferrer" title="Open report in a new page">report</a> | | 69 | MastodonReplyTask | <span className="benchmark-status benchmark-status-pass">PASS</span> | <a href="https://lf3-static.bytednsdoc.com/obj/eden-cn/luljzkpt/ljhwZthlaukjlkulzlp/benchmark/MobileWorld/20260728/20260728/Task-69-MastodonReplyTask__group-1-0a3fa62d-94f0-4f16-ba4d-7ac1fb2a8596-Pass.html" className="benchmark-report-link" target="_blank" rel="noreferrer" title="Open report in a new page">report</a> | | 70 | MastodonReportTask | <span className="benchmark-status benchmark-status-pass">PASS</span> | <a href="https://lf3-static.bytednsdoc.com/obj/eden-cn/luljzkpt/ljhwZthlaukjlkulzlp/benchmark/MobileWorld/20260728/20260728/Task-70-MastodonReportTask__group-1-0a3fa62d-94f0-4f16-ba4d-7ac1fb2a8596-Pass.html" className="benchmark-report-link" target="_blank" rel="noreferrer" title="Open report in a new page">report</a> | | 71 | MastodonRevisePhotoAltTask | <span className="benchmark-status benchmark-status-pass">PASS</span> | <a href="https://lf3-static.bytednsdoc.com/obj/eden-cn/luljzkpt/ljhwZthlaukjlkulzlp/benchmark/MobileWorld/20260728/20260728/Task-71-MastodonRevisePhotoAltTask__group-1-0a3fa62d-94f0-4f16-ba4d-7ac1fb2a8596-Pass.html" className="benchmark-report-link" target="_blank" rel="noreferrer" title="Open report in a new page">report</a> | | 72 | MastodonRevisePollTask | <span className="benchmark-status benchmark-status-pass">PASS</span> | <a href="https://lf3-static.bytednsdoc.com/obj/eden-cn/luljzkpt/ljhwZthlaukjlkulzlp/benchmark/MobileWorld/20260728/20260728/Task-72-MastodonRevisePollTask__group-1-0a3fa62d-94f0-4f16-ba4d-7ac1fb2a8596-Pass.html" className="benchmark-report-link" target="_blank" rel="noreferrer" title="Open report in a new page">report</a> | | 73 | MastodonSavePhotosTask | <span className="benchmark-status benchmark-status-pass">PASS</span> | <a href="https://lf3-static.bytednsdoc.com/obj/eden-cn/luljzkpt/ljhwZthlaukjlkulzlp/benchmark/MobileWorld/20260728/20260728/Task-73-MastodonSavePhotosTask__group-1-0a3fa62d-94f0-4f16-ba4d-7ac1fb2a8596-Pass.html" className="benchmark-report-link" target="_blank" rel="noreferrer" title="Open report in a new page">report</a> | | 74 | MastodonServerInfoReportTask | <span className="benchmark-status benchmark-status-fail">FAIL</span> | <a href="https://lf3-static.bytednsdoc.com/obj/eden-cn/luljzkpt/ljhwZthlaukjlkulzlp/benchmark/MobileWorld/20260728/20260728/Task-74-MastodonServerInfoReportTask__group-1-0a3fa62d-94f0-4f16-ba4d-7ac1fb2a8596-Fail.html" className="benchmark-report-link" target="_blank" rel="noreferrer" title="Open report in a new page">report</a> | | 75 | MastodonShareLocationTask | <span className="benchmark-status benchmark-status-fail">FAIL</span> | <a href="https://lf3-static.bytednsdoc.com/obj/eden-cn/luljzkpt/ljhwZthlaukjlkulzlp/benchmark/MobileWorld/20260728/20260728/Task-75-MastodonShareLocationTask__group-1-0a3fa62d-94f0-4f16-ba4d-7ac1fb2a8596-Fail.html" className="benchmark-report-link" target="_blank" rel="noreferrer" title="Open report in a new page">report</a> | | 76 | MastodonUnfollowTask | <span className="benchmark-status benchmark-status-pass">PASS</span> | <a href="https://lf3-static.bytednsdoc.com/obj/eden-cn/luljzkpt/ljhwZthlaukjlkulzlp/benchmark/MobileWorld/20260728/20260728/Task-76-MastodonUnfollowTask__group-1-0a3fa62d-94f0-4f16-ba4d-7ac1fb2a8596-Pass.html" className="benchmark-report-link" target="_blank" rel="noreferrer" title="Open report in a new page">report</a> | | 77 | MastodonUpdateContactsTask | <span className="benchmark-status benchmark-status-pass">PASS</span> | <a href="https://lf3-static.bytednsdoc.com/obj/eden-cn/luljzkpt/ljhwZthlaukjlkulzlp/benchmark/MobileWorld/20260728/20260728/Task-77-MastodonUpdateContactsTask__group-1-0a3fa62d-94f0-4f16-ba4d-7ac1fb2a8596-Pass.html" className="benchmark-report-link" target="_blank" rel="noreferrer" title="Open report in a new page">report</a> | | 78 | MattermostBudgetApprovalPipelineTask | <span className="benchmark-status benchmark-status-pass">PASS</span> | <a href="https://lf3-static.bytednsdoc.com/obj/eden-cn/luljzkpt/ljhwZthlaukjlkulzlp/benchmark/MobileWorld/20260728/20260728/Task-78-MattermostBudgetApprovalPipelineTask__group-1-0a3fa62d-94f0-4f16-ba4d-7ac1fb2a8596-Pass.html" className="benchmark-report-link" target="_blank" rel="noreferrer" title="Open report in a new page">report</a> | | 79 | MattermostCreateChannelTask | <span className="benchmark-status benchmark-status-pass">PASS</span> | <a href="https://lf3-static.bytednsdoc.com/obj/eden-cn/luljzkpt/ljhwZthlaukjlkulzlp/benchmark/MobileWorld/20260728/20260728/Task-79-MattermostCreateChannelTask__group-2-6f07b306-f600-41bd-a7b0-f5cd7a21335d-Pass.html" className="benchmark-report-link" target="_blank" rel="noreferrer" title="Open report in a new page">report</a> | | 80 | MattermostCustomerFeedbackAnalysisTask | <span className="benchmark-status benchmark-status-pass">PASS</span> | <a href="https://lf3-static.bytednsdoc.com/obj/eden-cn/luljzkpt/ljhwZthlaukjlkulzlp/benchmark/MobileWorld/20260728/20260728/Task-80-MattermostCustomerFeedbackAnalysisTask__group-2-6f07b306-f600-41bd-a7b0-f5cd7a21335d-Pass.html" className="benchmark-report-link" target="_blank" rel="noreferrer" title="Open report in a new page">report</a> | | 81 | MattermostDeadlineReconciliationTask | <span className="benchmark-status benchmark-status-pass">PASS</span> | <a href="https://lf3-static.bytednsdoc.com/obj/eden-cn/luljzkpt/ljhwZthlaukjlkulzlp/benchmark/MobileWorld/20260728/20260728/Task-81-MattermostDeadlineReconciliationTask__group-2-6f07b306-f600-41bd-a7b0-f5cd7a21335d-Pass.html" className="benchmark-report-link" target="_blank" rel="noreferrer" title="Open report in a new page">report</a> | | 82 | MattermostEmailTask | <span className="benchmark-status benchmark-status-pass">PASS</span> | <a href="https://lf3-static.bytednsdoc.com/obj/eden-cn/luljzkpt/ljhwZthlaukjlkulzlp/benchmark/MobileWorld/20260728/20260728/Task-82-MattermostEmailTask__group-2-6f07b306-f600-41bd-a7b0-f5cd7a21335d-Pass.html" className="benchmark-report-link" target="_blank" rel="noreferrer" title="Open report in a new page">report</a> | | 83 | MattermostIncidentEscalationTask | <span className="benchmark-status benchmark-status-pass">PASS</span> | <a href="https://lf3-static.bytednsdoc.com/obj/eden-cn/luljzkpt/ljhwZthlaukjlkulzlp/benchmark/MobileWorld/20260728/20260728/Task-83-MattermostIncidentEscalationTask__group-2-6f07b306-f600-41bd-a7b0-f5cd7a21335d-Pass.html" className="benchmark-report-link" target="_blank" rel="noreferrer" title="Open report in a new page">report</a> | | 84 | MattermostProjectHandoverTask | <span className="benchmark-status benchmark-status-fail">FAIL</span> | <a href="https://lf3-static.bytednsdoc.com/obj/eden-cn/luljzkpt/ljhwZthlaukjlkulzlp/benchmark/MobileWorld/20260728/20260728/Task-84-MattermostProjectHandoverTask__group-2-6f07b306-f600-41bd-a7b0-f5cd7a21335d-Fail.html" className="benchmark-report-link" target="_blank" rel="noreferrer" title="Open report in a new page">report</a> | | 85 | MattermostProjectStatusReportTask | <span className="benchmark-status benchmark-status-fail">FAIL</span> | <a href="https://lf3-static.bytednsdoc.com/obj/eden-cn/luljzkpt/ljhwZthlaukjlkulzlp/benchmark/MobileWorld/20260728/20260728/Task-85-MattermostProjectStatusReportTask__group-2-6f07b306-f600-41bd-a7b0-f5cd7a21335d-Fail.html" className="benchmark-report-link" target="_blank" rel="noreferrer" title="Open report in a new page">report</a> | | 86 | MattermostReadingGroupTask | <span className="benchmark-status benchmark-status-fail">FAIL</span> | <a href="https://lf3-static.bytednsdoc.com/obj/eden-cn/luljzkpt/ljhwZthlaukjlkulzlp/benchmark/MobileWorld/20260728/20260728/Task-86-MattermostReadingGroupTask__group-2-6f07b306-f600-41bd-a7b0-f5cd7a21335d-Fail.html" className="benchmark-report-link" target="_blank" rel="noreferrer" title="Open report in a new page">report</a> | | 87 | MattermostReplyToMessageTask | <span className="benchmark-status benchmark-status-pass">PASS</span> | <a href="https://lf3-static.bytednsdoc.com/obj/eden-cn/luljzkpt/ljhwZthlaukjlkulzlp/benchmark/MobileWorld/20260728/20260728/Task-87-MattermostReplyToMessageTask__group-2-6f07b306-f600-41bd-a7b0-f5cd7a21335d-Pass.html" className="benchmark-report-link" target="_blank" rel="noreferrer" title="Open report in a new page">report</a> | | 88 | MattermostResourceConflictResolutionTask | <span className="benchmark-status benchmark-status-fail">FAIL</span> | <a href="https://lf3-static.bytednsdoc.com/obj/eden-cn/luljzkpt/ljhwZthlaukjlkulzlp/benchmark/MobileWorld/20260728/20260728/Task-88-MattermostResourceConflictResolutionTask__group-2-6f07b306-f600-41bd-a7b0-f5cd7a21335d-Fail.html" className="benchmark-report-link" target="_blank" rel="noreferrer" title="Open report in a new page">report</a> | | 89 | MattermostSendFileTask | <span className="benchmark-status benchmark-status-fail">FAIL</span> | <a href="https://lf3-static.bytednsdoc.com/obj/eden-cn/luljzkpt/ljhwZthlaukjlkulzlp/benchmark/MobileWorld/20260728/20260728/Task-89-MattermostSendFileTask__group-2-6f07b306-f600-41bd-a7b0-f5cd7a21335d-Fail.html" className="benchmark-report-link" target="_blank" rel="noreferrer" title="Open report in a new page">report</a> | | 90 | MattermostShiftCoverageTask | <span className="benchmark-status benchmark-status-pass">PASS</span> | <a href="https://lf3-static.bytednsdoc.com/obj/eden-cn/luljzkpt/ljhwZthlaukjlkulzlp/benchmark/MobileWorld/20260728/20260728/Task-90-MattermostShiftCoverageTask__group-2-6f07b306-f600-41bd-a7b0-f5cd7a21335d-Pass.html" className="benchmark-report-link" target="_blank" rel="noreferrer" title="Open report in a new page">report</a> | | 91 | MattermostTechnicalDebtTriageTask | <span className="benchmark-status benchmark-status-pass">PASS</span> | <a href="https://lf3-static.bytednsdoc.com/obj/eden-cn/luljzkpt/ljhwZthlaukjlkulzlp/benchmark/MobileWorld/20260728/20260728/Task-91-MattermostTechnicalDebtTriageTask__group-2-6f07b306-f600-41bd-a7b0-f5cd7a21335d-Pass.html" className="benchmark-report-link" target="_blank" rel="noreferrer" title="Open report in a new page">report</a> | | 92 | MattermostVisualInstructionResponseTask | <span className="benchmark-status benchmark-status-fail">FAIL</span> | <a href="https://lf3-static.bytednsdoc.com/obj/eden-cn/luljzkpt/ljhwZthlaukjlkulzlp/benchmark/MobileWorld/20260728/20260728/Task-92-MattermostVisualInstructionResponseTask__group-2-6f07b306-f600-41bd-a7b0-f5cd7a21335d-Fail.html" className="benchmark-report-link" target="_blank" rel="noreferrer" title="Open report in a new page">report</a> | | 93 | OpenFlightModeTask | <span className="benchmark-status benchmark-status-pass">PASS</span> | <a href="https://lf3-static.bytednsdoc.com/obj/eden-cn/luljzkpt/ljhwZthlaukjlkulzlp/benchmark/MobileWorld/20260728/20260728/Task-93-OpenFlightModeTask__group-2-6f07b306-f600-41bd-a7b0-f5cd7a21335d-Pass.html" className="benchmark-report-link" target="_blank" rel="noreferrer" title="Open report in a new page">report</a> | | 94 | PhotoManagementTask | <span className="benchmark-status benchmark-status-fail">FAIL</span> | <a href="https://lf3-static.bytednsdoc.com/obj/eden-cn/luljzkpt/ljhwZthlaukjlkulzlp/benchmark/MobileWorld/20260728/20260728/Task-94-PhotoManagementTask__group-2-6f07b306-f600-41bd-a7b0-f5cd7a21335d-Fail.html" className="benchmark-report-link" target="_blank" rel="noreferrer" title="Open report in a new page">report</a> | | 95 | ReadQwen3PaperTask1 | <span className="benchmark-status benchmark-status-pass">PASS</span> | <a href="https://lf3-static.bytednsdoc.com/obj/eden-cn/luljzkpt/ljhwZthlaukjlkulzlp/benchmark/MobileWorld/20260728/20260728/Task-95-ReadQwen3PaperTask1__group-2-6f07b306-f600-41bd-a7b0-f5cd7a21335d-Pass.html" className="benchmark-report-link" target="_blank" rel="noreferrer" title="Open report in a new page">report</a> | | 96 | ReadQwen3PaperTask2 | <span className="benchmark-status benchmark-status-fail">FAIL</span> | <a href="https://lf3-static.bytednsdoc.com/obj/eden-cn/luljzkpt/ljhwZthlaukjlkulzlp/benchmark/MobileWorld/20260728/20260728/Task-96-ReadQwen3PaperTask2__group-2-6f07b306-f600-41bd-a7b0-f5cd7a21335d-Fail.html" className="benchmark-report-link" target="_blank" rel="noreferrer" title="Open report in a new page">report</a> | | 97 | ReadQwen3PaperTask3 | <span className="benchmark-status benchmark-status-fail">FAIL</span> | <a href="https://lf3-static.bytednsdoc.com/obj/eden-cn/luljzkpt/ljhwZthlaukjlkulzlp/benchmark/MobileWorld/20260728/20260728/Task-97-ReadQwen3PaperTask3__group-2-6f07b306-f600-41bd-a7b0-f5cd7a21335d-Fail.html" className="benchmark-report-link" target="_blank" rel="noreferrer" title="Open report in a new page">report</a> | | 98 | ReadQwen3PaperTask4 | <span className="benchmark-status benchmark-status-pass">PASS</span> | <a href="https://lf3-static.bytednsdoc.com/obj/eden-cn/luljzkpt/ljhwZthlaukjlkulzlp/benchmark/MobileWorld/20260728/20260728/Task-98-ReadQwen3PaperTask4__group-2-6f07b306-f600-41bd-a7b0-f5cd7a21335d-Pass.html" className="benchmark-report-link" target="_blank" rel="noreferrer" title="Open report in a new page">report</a> | | 99 | ReadQwen3PaperTask5 | <span className="benchmark-status benchmark-status-fail">FAIL</span> | <a href="https://lf3-static.bytednsdoc.com/obj/eden-cn/luljzkpt/ljhwZthlaukjlkulzlp/benchmark/MobileWorld/20260728/20260728/Task-99-ReadQwen3PaperTask5__group-2-6f07b306-f600-41bd-a7b0-f5cd7a21335d-Fail.html" className="benchmark-report-link" target="_blank" rel="noreferrer" title="Open report in a new page">report</a> | | 100 | RecentTotalExpenseTask | <span className="benchmark-status benchmark-status-pass">PASS</span> | <a href="https://lf3-static.bytednsdoc.com/obj/eden-cn/luljzkpt/ljhwZthlaukjlkulzlp/benchmark/MobileWorld/20260728/20260728/Task-100-RecentTotalExpenseTask__group-2-6f07b306-f600-41bd-a7b0-f5cd7a21335d-Pass.html" className="benchmark-report-link" target="_blank" rel="noreferrer" title="Open report in a new page">report</a> | | 101 | RequestCarpoolingTask | <span className="benchmark-status benchmark-status-pass">PASS</span> | <a href="https://lf3-static.bytednsdoc.com/obj/eden-cn/luljzkpt/ljhwZthlaukjlkulzlp/benchmark/MobileWorld/20260728/20260728/Task-101-RequestCarpoolingTask__group-2-6f07b306-f600-41bd-a7b0-f5cd7a21335d-Pass.html" className="benchmark-report-link" target="_blank" rel="noreferrer" title="Open report in a new page">report</a> | | 102 | ReviewPaperEmailTask | <span className="benchmark-status benchmark-status-fail">FAIL</span> | <a href="https://lf3-static.bytednsdoc.com/obj/eden-cn/luljzkpt/ljhwZthlaukjlkulzlp/benchmark/MobileWorld/20260728/20260728/Task-102-ReviewPaperEmailTask__group-2-6f07b306-f600-41bd-a7b0-f5cd7a21335d-Fail.html" className="benchmark-report-link" target="_blank" rel="noreferrer" title="Open report in a new page">report</a> | | 103 | SMSManagement | <span className="benchmark-status benchmark-status-pass">PASS</span> | <a href="https://lf3-static.bytednsdoc.com/obj/eden-cn/luljzkpt/ljhwZthlaukjlkulzlp/benchmark/MobileWorld/20260728/20260728/Task-103-SMSManagement__group-2-6f07b306-f600-41bd-a7b0-f5cd7a21335d-Pass.html" className="benchmark-report-link" target="_blank" rel="noreferrer" title="Open report in a new page">report</a> | | 104 | ScheduleCoffeeTimeViaSmsTask | <span className="benchmark-status benchmark-status-pass">PASS</span> | <a href="https://lf3-static.bytednsdoc.com/obj/eden-cn/luljzkpt/ljhwZthlaukjlkulzlp/benchmark/MobileWorld/20260728/20260728/Task-104-ScheduleCoffeeTimeViaSmsTask__group-2-6f07b306-f600-41bd-a7b0-f5cd7a21335d-Pass.html" className="benchmark-report-link" target="_blank" rel="noreferrer" title="Open report in a new page">report</a> | | 105 | ScheduleLunchViaSmsTask | <span className="benchmark-status benchmark-status-pass">PASS</span> | <a href="https://lf3-static.bytednsdoc.com/obj/eden-cn/luljzkpt/ljhwZthlaukjlkulzlp/benchmark/MobileWorld/20260728/20260728/Task-105-ScheduleLunchViaSmsTask__group-2-6f07b306-f600-41bd-a7b0-f5cd7a21335d-Pass.html" className="benchmark-report-link" target="_blank" rel="noreferrer" title="Open report in a new page">report</a> | | 106 | SearchItemAndCheckoutTask | <span className="benchmark-status benchmark-status-pass">PASS</span> | <a href="https://lf3-static.bytednsdoc.com/obj/eden-cn/luljzkpt/ljhwZthlaukjlkulzlp/benchmark/MobileWorld/20260728/20260728/Task-106-SearchItemAndCheckoutTask__group-2-6f07b306-f600-41bd-a7b0-f5cd7a21335d-Pass.html" className="benchmark-report-link" target="_blank" rel="noreferrer" title="Open report in a new page">report</a> | | 107 | SendFormsTask | <span className="benchmark-status benchmark-status-pass">PASS</span> | <a href="https://lf3-static.bytednsdoc.com/obj/eden-cn/luljzkpt/ljhwZthlaukjlkulzlp/benchmark/MobileWorld/20260728/20260728/Task-107-SendFormsTask__group-2-6f07b306-f600-41bd-a7b0-f5cd7a21335d-Pass.html" className="benchmark-report-link" target="_blank" rel="noreferrer" title="Open report in a new page">report</a> | | 108 | SendInterviewEmailTask | <span className="benchmark-status benchmark-status-pass">PASS</span> | <a href="https://lf3-static.bytednsdoc.com/obj/eden-cn/luljzkpt/ljhwZthlaukjlkulzlp/benchmark/MobileWorld/20260728/20260728/Task-108-SendInterviewEmailTask__group-2-6f07b306-f600-41bd-a7b0-f5cd7a21335d-Pass.html" className="benchmark-report-link" target="_blank" rel="noreferrer" title="Open report in a new page">report</a> | | 109 | SendInterviewInvitationTask | <span className="benchmark-status benchmark-status-fail">FAIL</span> | <a href="https://lf3-static.bytednsdoc.com/obj/eden-cn/luljzkpt/ljhwZthlaukjlkulzlp/benchmark/MobileWorld/20260728/20260728/Task-109-SendInterviewInvitationTask__group-2-6f07b306-f600-41bd-a7b0-f5cd7a21335d-Fail.html" className="benchmark-report-link" target="_blank" rel="noreferrer" title="Open report in a new page">report</a> | | 110 | SendWaiverTask | <span className="benchmark-status benchmark-status-pass">PASS</span> | <a href="https://lf3-static.bytednsdoc.com/obj/eden-cn/luljzkpt/ljhwZthlaukjlkulzlp/benchmark/MobileWorld/20260728/20260728/Task-110-SendWaiverTask__group-2-6f07b306-f600-41bd-a7b0-f5cd7a21335d-Pass.html" className="benchmark-report-link" target="_blank" rel="noreferrer" title="Open report in a new page">report</a> | | 111 | SetAlarmTask | <span className="benchmark-status benchmark-status-pass">PASS</span> | <a href="https://lf3-static.bytednsdoc.com/obj/eden-cn/luljzkpt/ljhwZthlaukjlkulzlp/benchmark/MobileWorld/20260728/20260728/Task-111-SetAlarmTask__group-2-6f07b306-f600-41bd-a7b0-f5cd7a21335d-Pass.html" className="benchmark-report-link" target="_blank" rel="noreferrer" title="Open report in a new page">report</a> | | 112 | SharePhotosTask | <span className="benchmark-status benchmark-status-pass">PASS</span> | <a href="https://lf3-static.bytednsdoc.com/obj/eden-cn/luljzkpt/ljhwZthlaukjlkulzlp/benchmark/MobileWorld/20260728/20260728/Task-112-SharePhotosTask__group-2-6f07b306-f600-41bd-a7b0-f5cd7a21335d-Pass.html" className="benchmark-report-link" target="_blank" rel="noreferrer" title="Open report in a new page">report</a> | | 113 | SuggestPaperTask | <span className="benchmark-status benchmark-status-pass">PASS</span> | <a href="https://lf3-static.bytednsdoc.com/obj/eden-cn/luljzkpt/ljhwZthlaukjlkulzlp/benchmark/MobileWorld/20260728/20260728/Task-113-SuggestPaperTask__group-0-b91cc3df-394d-47e4-8a5b-f61c5beddb79-Pass.html" className="benchmark-report-link" target="_blank" rel="noreferrer" title="Open report in a new page">report</a> | | 114 | SumFileLinesTask | <span className="benchmark-status benchmark-status-pass">PASS</span> | <a href="https://lf3-static.bytednsdoc.com/obj/eden-cn/luljzkpt/ljhwZthlaukjlkulzlp/benchmark/MobileWorld/20260728/20260728/Task-114-SumFileLinesTask__group-2-6f07b306-f600-41bd-a7b0-f5cd7a21335d-Pass.html" className="benchmark-report-link" target="_blank" rel="noreferrer" title="Open report in a new page">report</a> | | 115 | TakeSelfieTask | <span className="benchmark-status benchmark-status-pass">PASS</span> | <a href="https://lf3-static.bytednsdoc.com/obj/eden-cn/luljzkpt/ljhwZthlaukjlkulzlp/benchmark/MobileWorld/20260728/20260728/Task-115-TakeSelfieTask__group-2-6f07b306-f600-41bd-a7b0-f5cd7a21335d-Pass.html" className="benchmark-report-link" target="_blank" rel="noreferrer" title="Open report in a new page">report</a> | | 116 | TextArrivalTimeTask | <span className="benchmark-status benchmark-status-fail">FAIL</span> | <a href="https://lf3-static.bytednsdoc.com/obj/eden-cn/luljzkpt/ljhwZthlaukjlkulzlp/benchmark/MobileWorld/20260728/20260728/Task-116-TextArrivalTimeTask__group-2-6f07b306-f600-41bd-a7b0-f5cd7a21335d-Fail.html" className="benchmark-report-link" target="_blank" rel="noreferrer" title="Open report in a new page">report</a> | | 117 | ThanksgivingPrepTask | <span className="benchmark-status benchmark-status-pass">PASS</span> | <a href="https://lf3-static.bytednsdoc.com/obj/eden-cn/luljzkpt/ljhwZthlaukjlkulzlp/benchmark/MobileWorld/20260728/20260728/Task-117-ThanksgivingPrepTask__group-2-6f07b306-f600-41bd-a7b0-f5cd7a21335d-Pass.html" className="benchmark-report-link" target="_blank" rel="noreferrer" title="Open report in a new page">report</a> | </details> --- url: /model-common-config.md --- import { ModelConfigTab, ModelConfigTabs, PackageManagerTabs } from '@theme'; # Supported Models and Setup Use this guide to choose a supported model, complete the initial setup, and verify model connectivity. The model configuration shown on this page consists of environment variables. Provide it according to how you use Midscene: - In Playground, paste the configuration text from this page directly into the settings page. - With the SDK or CLI, load the configuration as described in [Set environment variables](#set-environment-variables). To understand the model roles, see [Model strategy](./model-strategy). For complete parameter definitions, see [Model configuration reference](./model-config). ## Supported models Midscene supports the following multimodal models for operating user interfaces. Each configuration requires a Base URL, API key, model name, and `MIDSCENE_MODEL_FAMILY`. The model family (`MIDSCENE_MODEL_FAMILY`) determines how Midscene adapts to the selected model. ### Doubao Seed Series {#doubao-seed-model} - Common model provider: [Volcano Engine](https://volcengine.com/) <div className="model-series-table"> | Model version | Commonly used model names | `MIDSCENE_MODEL_FAMILY` | Notes | | --- | --- | --- | --- | | 2.x series | `Doubao-Seed-2.1-turbo`, `Doubao-Seed-2.0-Lite` | `doubao-seed` | `Doubao-Seed-2.1-turbo` has the fastest localization speed and strong localization quality in our current private evaluation set. Recommended. | | 1.x series | `Doubao-Seed-1.6-Vision`, `Doubao-Seed-1.8` | `doubao-seed` | The 1.x series is an older generation of Doubao models and is no longer competitive overall. We recommend using the 2.x series instead. For compatibility with existing configurations, `MIDSCENE_MODEL_FAMILY="doubao-vision"` remains supported. New configurations should use `doubao-seed`. | </div> Environment variable configuration example, using `doubao-seed-2.1-turbo`: <ModelConfigTabs> <ModelConfigTab type="default"> ```bash MIDSCENE_MODEL_BASE_URL="https://ark.cn-beijing.volces.com/api/v3" # Volcano Engine endpoint MIDSCENE_MODEL_API_KEY="...." MIDSCENE_MODEL_NAME="doubao-seed-2-1-turbo-260628" MIDSCENE_MODEL_FAMILY="doubao-seed" ``` </ModelConfigTab> <ModelConfigTab type="planning"> ```bash MIDSCENE_PLANNING_MODEL_BASE_URL="https://ark.cn-beijing.volces.com/api/v3" # Volcano Engine endpoint MIDSCENE_PLANNING_MODEL_API_KEY="...." MIDSCENE_PLANNING_MODEL_NAME="doubao-seed-2-1-turbo-260628" MIDSCENE_PLANNING_MODEL_FAMILY="doubao-seed" ``` </ModelConfigTab> <ModelConfigTab type="insight"> ```bash MIDSCENE_INSIGHT_MODEL_BASE_URL="https://ark.cn-beijing.volces.com/api/v3" # Volcano Engine endpoint MIDSCENE_INSIGHT_MODEL_API_KEY="...." MIDSCENE_INSIGHT_MODEL_NAME="doubao-seed-2-1-turbo-260628" MIDSCENE_INSIGHT_MODEL_FAMILY="doubao-seed" ``` </ModelConfigTab> </ModelConfigTabs> If your Volcano Engine account has Fast Tier quota enabled, add the extra request body below to use it. This usually improves model response speed by about 30%-50%. ```bash MIDSCENE_MODEL_EXTRA_BODY_JSON={"service_tier":"fast"} ``` ### Qwen Series {#qwen} <span id="qwen3x" /> <span id="qwen3-vl" /> <span id="qwen25-vl" /> - Common model provider: [Alibaba Cloud](https://www.aliyun.com/) <div className="model-series-table"> | Model version | Commonly used model names | `MIDSCENE_MODEL_FAMILY` | Notes | | --- | --- | --- | --- | | Qwen3.x series | `qwen3.7-plus`, `qwen3.5-plus`, `qwen3.6-plus` | `qwen3` | Based on localization evaluation results, the recommended order is Qwen3.7 > Qwen3.5 > Qwen3.6. The previous `qwen3.5` and `qwen3.6` families remain compatible. | | Qwen3-VL series | `qwen3-vl-plus` | `qwen3-vl` | As an older model generation, it is not recommended. Use the Qwen3.x series instead. | | Qwen2.5-VL series | `qwen-vl-max-latest` | `qwen2.5-vl` | As an older model generation, it is not recommended. Use the Qwen3.x series instead. | </div> Environment variable configuration example, using `qwen3.7-plus`: <ModelConfigTabs> <ModelConfigTab type="default"> ```bash MIDSCENE_MODEL_BASE_URL="https://dashscope.aliyuncs.com/compatible-mode/v1" # Alibaba Cloud endpoint MIDSCENE_MODEL_API_KEY="......" MIDSCENE_MODEL_NAME="qwen3.7-plus" MIDSCENE_MODEL_FAMILY="qwen3" # If you use another Qwen version, replace this with the corresponding model family ``` </ModelConfigTab> <ModelConfigTab type="planning"> ```bash MIDSCENE_PLANNING_MODEL_BASE_URL="https://dashscope.aliyuncs.com/compatible-mode/v1" # Alibaba Cloud endpoint MIDSCENE_PLANNING_MODEL_API_KEY="......" MIDSCENE_PLANNING_MODEL_NAME="qwen3.7-plus" MIDSCENE_PLANNING_MODEL_FAMILY="qwen3" # If you use another Qwen version, replace this with the corresponding model family ``` </ModelConfigTab> <ModelConfigTab type="insight"> ```bash MIDSCENE_INSIGHT_MODEL_BASE_URL="https://dashscope.aliyuncs.com/compatible-mode/v1" # Alibaba Cloud endpoint MIDSCENE_INSIGHT_MODEL_API_KEY="......" MIDSCENE_INSIGHT_MODEL_NAME="qwen3.7-plus" MIDSCENE_INSIGHT_MODEL_FAMILY="qwen3" # If you use another Qwen version, replace this with the corresponding model family ``` </ModelConfigTab> </ModelConfigTabs> ### DeepSeek Series {#deepseek} Midscene supports DeepSeek starting from v1.12.0. - Common model provider: [DeepSeek - Vision](https://api-docs.deepseek.com/guides/vision/) <div className="model-series-table"> | Model version | Commonly used model names | `MIDSCENE_MODEL_FAMILY` | Notes | | --- | --- | --- | --- | | DeepSeek V4 series | `deepseek-v4-flash-vision-exp` | `deepseek` | Currently, only `deepseek-v4-flash-vision-exp` supports the multimodal visual input required by Midscene. Other DeepSeek V4 models, including `deepseek-v4-pro` and `deepseek-v4-flash`, are not suitable for use with Midscene. | </div> Environment variable configuration example, using `deepseek-v4-flash-vision-exp`: <ModelConfigTabs> <ModelConfigTab type="default"> ```bash MIDSCENE_MODEL_BASE_URL="https://api.deepseek.com" # DeepSeek API endpoint MIDSCENE_MODEL_API_KEY="......" MIDSCENE_MODEL_NAME="deepseek-v4-flash-vision-exp" MIDSCENE_MODEL_FAMILY="deepseek" ``` </ModelConfigTab> <ModelConfigTab type="planning"> ```bash MIDSCENE_PLANNING_MODEL_BASE_URL="https://api.deepseek.com" # DeepSeek API endpoint MIDSCENE_PLANNING_MODEL_API_KEY="......" MIDSCENE_PLANNING_MODEL_NAME="deepseek-v4-flash-vision-exp" MIDSCENE_PLANNING_MODEL_FAMILY="deepseek" ``` </ModelConfigTab> <ModelConfigTab type="insight"> ```bash MIDSCENE_INSIGHT_MODEL_BASE_URL="https://api.deepseek.com" # DeepSeek API endpoint MIDSCENE_INSIGHT_MODEL_API_KEY="......" MIDSCENE_INSIGHT_MODEL_NAME="deepseek-v4-flash-vision-exp" MIDSCENE_INSIGHT_MODEL_FAMILY="deepseek" ``` </ModelConfigTab> </ModelConfigTabs> ### Google Gemini Series {#gemini} - Common model provider: [Google Gemini](https://gemini.google.com/) <div className="model-series-table"> | Model version | Commonly used model names | `MIDSCENE_MODEL_FAMILY` | Notes | | --- | --- | --- | --- | | Gemini 3.x series | `gemini-3.5-flash`, `gemini-3-flash-preview` | `gemini` | `gemini-3.5-flash` currently performs best for localization in our private evaluation set. | </div> Environment variable configuration example, using `gemini-3.5-flash`: <ModelConfigTabs> <ModelConfigTab type="default"> ```bash MIDSCENE_MODEL_BASE_URL="https://generativelanguage.googleapis.com/v1beta/openai/" # Google Gemini API endpoint MIDSCENE_MODEL_API_KEY="......" MIDSCENE_MODEL_NAME="gemini-3.5-flash" MIDSCENE_MODEL_FAMILY="gemini" ``` </ModelConfigTab> <ModelConfigTab type="planning"> ```bash MIDSCENE_PLANNING_MODEL_BASE_URL="https://generativelanguage.googleapis.com/v1beta/openai/" # Google Gemini API endpoint MIDSCENE_PLANNING_MODEL_API_KEY="......" MIDSCENE_PLANNING_MODEL_NAME="gemini-3.5-flash" MIDSCENE_PLANNING_MODEL_FAMILY="gemini" ``` </ModelConfigTab> <ModelConfigTab type="insight"> ```bash MIDSCENE_INSIGHT_MODEL_BASE_URL="https://generativelanguage.googleapis.com/v1beta/openai/" # Google Gemini API endpoint MIDSCENE_INSIGHT_MODEL_API_KEY="......" MIDSCENE_INSIGHT_MODEL_NAME="gemini-3.5-flash" MIDSCENE_INSIGHT_MODEL_FAMILY="gemini" ``` </ModelConfigTab> </ModelConfigTabs> ### OpenAI GPT Series {#gpt} - Common model provider: [OpenAI](https://openai.com/) <div className="model-series-table"> | Model version | Commonly used model names | `MIDSCENE_MODEL_FAMILY` | Notes | | --- | --- | --- | --- | | GPT-5 series | `gpt-5.4`, `gpt-5.5`, `gpt-5.6-sol`, `gpt-5.6-terra`, `gpt-5.6-luna` | `gpt-5` | Models before GPT-5.4 do not support visual localization and can only be used as Planning or Insight models. In practical localization tests, GPT-5.5 and GPT-5.6 perform noticeably better than GPT-5.4; we recommend using GPT-5.5 or GPT-5.6 first. | </div> Environment variable configuration example, using `gpt-5.5`: <ModelConfigTabs> <ModelConfigTab type="default"> ```bash MIDSCENE_MODEL_BASE_URL="https://api.openai.com/v1" # OpenAI API endpoint; or your compatible endpoint MIDSCENE_MODEL_API_KEY="sk-..." MIDSCENE_MODEL_NAME="gpt-5.5" MIDSCENE_MODEL_FAMILY="gpt-5" ``` </ModelConfigTab> <ModelConfigTab type="planning"> ```bash MIDSCENE_PLANNING_MODEL_BASE_URL="https://api.openai.com/v1" # OpenAI API endpoint; or your compatible endpoint MIDSCENE_PLANNING_MODEL_API_KEY="sk-..." MIDSCENE_PLANNING_MODEL_NAME="gpt-5.5" MIDSCENE_PLANNING_MODEL_FAMILY="gpt-5" ``` </ModelConfigTab> <ModelConfigTab type="insight"> ```bash MIDSCENE_INSIGHT_MODEL_BASE_URL="https://api.openai.com/v1" # OpenAI API endpoint; or your compatible endpoint MIDSCENE_INSIGHT_MODEL_API_KEY="sk-..." MIDSCENE_INSIGHT_MODEL_NAME="gpt-5.5" MIDSCENE_INSIGHT_MODEL_FAMILY="gpt-5" ``` </ModelConfigTab> </ModelConfigTabs> <span id="use-codex-app-server-oauth-no-api-key" /> **Use Codex App Server (OAuth, no API Key)** If you already signed in with Codex CLI (`codex login`) and want Midscene to use that OAuth session directly, set: ```bash export MIDSCENE_MODEL_BASE_URL="codex://app-server" export MIDSCENE_MODEL_NAME="gpt-5.4" # or another model shown by Codex model/list export MIDSCENE_MODEL_FAMILY="gpt-5" ``` Notes: - `MIDSCENE_MODEL_API_KEY` is not required in this mode. - Midscene will call `codex app-server` through stdio. - Make sure `codex` is available in your PATH and verify auth with `codex login status`. :::warning Known limitation Compared with calling an OpenAI-compatible API directly, we have observed that this route may take longer and consume more tokens. We are still investigating the cause. ::: When using GPT-5, note the following: - For UI localization with GPT, Midscene currently supports `gpt-5.4` and later models. To get the best localization quality, image requests need `"detail": "original"`. According to OpenAI, this option is available on `gpt-5.4` and future models, while smaller GPT-5 variants such as `gpt-5.4-mini` and `gpt-5.4-nano`, as well as older models, do not support `original` and will fail if you send it. See the [Images and Vision guide](https://developers.openai.com/api/docs/guides/images-vision) and the [Computer use guide](https://developers.openai.com/api/docs/guides/tools-computer-use). - According to OpenAI, GPT-5 may still struggle with non-Latin text and with text that is too small in the image. See the [Images and Vision guide](https://developers.openai.com/api/docs/guides/images-vision). - In the computer use guide, OpenAI says they observe good performance around `1440x900` and `1600x900`. We recommend adjusting screenshot sizes accordingly. In Midscene, you can control screenshot compression with `screenshotShrinkFactor` in the agent options. For browser automation, you can also control the page size and scale through the browser `viewport`. - With Azure OpenAI, Azure may not handle `"detail": "original"` correctly, causing click-coordinate offsets. See [Clicks are offset when using Azure OpenAI](./faq#clicks-are-offset-when-using-azure-openai). - If you use an older GPT-5 model, we recommend using it only as the planning model and pairing it with another multimodal model for localization. See the [multi-model combination example](#multi-model-combination-example). ### Moonshot Kimi Series {#kimi} - Common model provider: [Moonshot AI platform](https://platform.moonshot.cn/) <div className="model-series-table"> | Model version | Commonly used model names | `MIDSCENE_MODEL_FAMILY` | Notes | | --- | --- | --- | --- | | K3 series | `kimi-k3` | `kimi3` | According to the Kimi documentation, K3 always has reasoning enabled and cannot be disabled. Its reasoning effort defaults to `max`. | | K2.x series | `kimi-k2.5`, `kimi-k2.6` | `kimi` | — | </div> Environment variable configuration example, using `kimi-k3`: <ModelConfigTabs> <ModelConfigTab type="default"> ```bash MIDSCENE_MODEL_BASE_URL="https://api.moonshot.cn/v1" # Moonshot AI API endpoint MIDSCENE_MODEL_API_KEY="......" MIDSCENE_MODEL_NAME="kimi-k3" MIDSCENE_MODEL_FAMILY="kimi3" # For kimi-k2.6, use "kimi" ``` </ModelConfigTab> <ModelConfigTab type="planning"> ```bash MIDSCENE_PLANNING_MODEL_BASE_URL="https://api.moonshot.cn/v1" # Moonshot AI API endpoint MIDSCENE_PLANNING_MODEL_API_KEY="......" MIDSCENE_PLANNING_MODEL_NAME="kimi-k3" MIDSCENE_PLANNING_MODEL_FAMILY="kimi3" # For kimi-k2.6, use "kimi" ``` </ModelConfigTab> <ModelConfigTab type="insight"> ```bash MIDSCENE_INSIGHT_MODEL_BASE_URL="https://api.moonshot.cn/v1" # Moonshot AI API endpoint MIDSCENE_INSIGHT_MODEL_API_KEY="......" MIDSCENE_INSIGHT_MODEL_NAME="kimi-k3" MIDSCENE_INSIGHT_MODEL_FAMILY="kimi3" # For kimi-k2.6, use "kimi" ``` </ModelConfigTab> </ModelConfigTabs> ### Xiaomi MiMo Series {#xiaomi-mimo} - Common model provider: [Xiaomi MiMo API Open Platform](https://platform.xiaomimimo.com/) <div className="model-series-table"> | Model version | Commonly used model names | `MIDSCENE_MODEL_FAMILY` | Notes | | --- | --- | --- | --- | | V2.x series | `mimo-v2.5` | `xiaomi-mimo` | Only the Omni series supports multimodal input; the Pro series is text-only and cannot be used for Midscene visual tasks. | </div> Environment variable configuration example, using `mimo-v2.5`: <ModelConfigTabs> <ModelConfigTab type="default"> ```bash MIDSCENE_MODEL_BASE_URL="https://api.xiaomimimo.com/v1" # Xiaomi MiMo API endpoint MIDSCENE_MODEL_API_KEY="......" MIDSCENE_MODEL_NAME="mimo-v2.5" MIDSCENE_MODEL_FAMILY="xiaomi-mimo" ``` </ModelConfigTab> <ModelConfigTab type="planning"> ```bash MIDSCENE_PLANNING_MODEL_BASE_URL="https://api.xiaomimimo.com/v1" # Xiaomi MiMo API endpoint MIDSCENE_PLANNING_MODEL_API_KEY="......" MIDSCENE_PLANNING_MODEL_NAME="mimo-v2.5" MIDSCENE_PLANNING_MODEL_FAMILY="xiaomi-mimo" ``` </ModelConfigTab> <ModelConfigTab type="insight"> ```bash MIDSCENE_INSIGHT_MODEL_BASE_URL="https://api.xiaomimimo.com/v1" # Xiaomi MiMo API endpoint MIDSCENE_INSIGHT_MODEL_API_KEY="......" MIDSCENE_INSIGHT_MODEL_NAME="mimo-v2.5" MIDSCENE_INSIGHT_MODEL_FAMILY="xiaomi-mimo" ``` </ModelConfigTab> </ModelConfigTabs> ### Zhipu GLM-V Series {#glm-v} - Common model providers: [Z.AI (Global)](https://z.ai/manage-apikey/apikey-list), [BigModel (CN)](https://bigmodel.cn/usercenter/proj-mgmt/apikeys) <div className="model-series-table"> | Model version | Commonly used model names | `MIDSCENE_MODEL_FAMILY` | Notes | | --- | --- | --- | --- | | GLM-5V series | `glm-5v-turbo` | `glm-v` | — | | GLM-4.6 series | `glm-4.6v` | `glm-v` | `glm-4.6v` is open-source. | </div> Environment variable configuration example, using `glm-5v-turbo`: <ModelConfigTabs> <ModelConfigTab type="default"> ```bash MIDSCENE_MODEL_BASE_URL="https://open.bigmodel.cn/api/paas/v4" # BigModel API endpoint; use https://api.z.ai/api/paas/v4 for Z.AI MIDSCENE_MODEL_API_KEY="......" MIDSCENE_MODEL_NAME="glm-5v-turbo" MIDSCENE_MODEL_FAMILY="glm-v" ``` </ModelConfigTab> <ModelConfigTab type="planning"> ```bash MIDSCENE_PLANNING_MODEL_BASE_URL="https://open.bigmodel.cn/api/paas/v4" # BigModel API endpoint; use https://api.z.ai/api/paas/v4 for Z.AI MIDSCENE_PLANNING_MODEL_API_KEY="......" MIDSCENE_PLANNING_MODEL_NAME="glm-5v-turbo" MIDSCENE_PLANNING_MODEL_FAMILY="glm-v" ``` </ModelConfigTab> <ModelConfigTab type="insight"> ```bash MIDSCENE_INSIGHT_MODEL_BASE_URL="https://open.bigmodel.cn/api/paas/v4" # BigModel API endpoint; use https://api.z.ai/api/paas/v4 for Z.AI MIDSCENE_INSIGHT_MODEL_API_KEY="......" MIDSCENE_INSIGHT_MODEL_NAME="glm-5v-turbo" MIDSCENE_INSIGHT_MODEL_FAMILY="glm-v" ``` </ModelConfigTab> </ModelConfigTabs> **Learn more about the open-source GLM-4.6V model** - Github: [https://github.com/zai-org/GLM-V](https://github.com/zai-org/GLM-V) - Hugging Face: [https://huggingface.co/zai-org/GLM-4.6V](https://huggingface.co/zai-org/GLM-4.6V) ## Set environment variables Midscene reads model configuration from environment variables. Choose the method that matches how you use Midscene, or keep using your project's existing environment-management approach. ### Set environment variables in the current shell The following example works in Bash and Zsh. These variables are available only to the current shell session and its child processes. ```bash # Replace every value with the configuration for your selected model provider export MIDSCENE_MODEL_BASE_URL="https://replace-with-your-model-service-url/v1" export MIDSCENE_MODEL_API_KEY="replace-with-your-api-key" export MIDSCENE_MODEL_NAME="replace-with-your-model-name" export MIDSCENE_MODEL_FAMILY="replace-with-the-family-for-your-model" ``` ### Load `.env` with the CLI Create a `.env` file in the directory where you run the `midscene` command. `@midscene/cli` loads this file automatically. ```bash # Replace every value with the configuration for your selected model provider MIDSCENE_MODEL_BASE_URL="https://replace-with-your-model-service-url/v1" MIDSCENE_MODEL_API_KEY="replace-with-your-api-key" MIDSCENE_MODEL_NAME="replace-with-your-model-name" MIDSCENE_MODEL_FAMILY="replace-with-the-family-for-your-model" ``` Do not add `export` at the beginning of each line. When running YAML tasks, existing variables in the current shell take precedence over values with the same name in `.env`. To let `.env` override them, use `--dotenv-override`. `midscene model verify` is an exception. This command uses values from `.env` to override variables with the same name in the current shell. ### Load `.env` with dotenv for the JavaScript SDK The Midscene JavaScript SDK reads model configuration directly from the Node.js process environment (`process.env`). If the shell, container, or deployment platform already provides these variables, you do not need dotenv. Use [dotenv](https://www.npmjs.com/package/dotenv) only when the configuration is stored in a `.env` file and needs to be loaded into `process.env`. <PackageManagerTabs command="install dotenv" /> Create a `.env` file in the directory where you run the script: ```bash # Replace every value with the configuration for your selected model provider MIDSCENE_MODEL_BASE_URL="https://replace-with-your-model-service-url/v1" MIDSCENE_MODEL_API_KEY="replace-with-your-api-key" MIDSCENE_MODEL_NAME="replace-with-your-model-name" MIDSCENE_MODEL_FAMILY="replace-with-the-family-for-your-model" ``` Import dotenv before creating a Midscene Agent: ```typescript import 'dotenv/config'; ``` If a variable with the same name already exists in the current shell, dotenv keeps the existing value by default. The [Midscene demo project](https://github.com/web-infra-dev/midscene-example) uses this loading method as well. ## Verify the setup After setting the environment variables, run one of the following commands: ```bash # Use the CLI installed in the current project npx midscene model verify # Or use the latest CLI version npx @midscene/cli@latest model verify ``` For failed checks and connectivity problems, see [Model debugging and observability](./model-debugging-observability). ## Optional: configure multiple models {#multi-model-combination-example} Multi-model setup is optional. For most use cases, a Default model is enough for UI localization and actions. Add a Planning or Insight model only when complex planning or page understanding requires a separate model. You can configure either one or both. For guidance on when to combine models, see [Model strategy](./model-strategy). The following example uses Qwen 3.5 as the Default model for visual grounding. GPT-5.4 serves as the Planning and Insight models for complex reasoning. ```bash # Default multimodal model: Qwen 3.5 export MIDSCENE_MODEL_BASE_URL="https://..." # Qwen 3.5 endpoint export MIDSCENE_MODEL_API_KEY="..." # Your Qwen 3.5 API key export MIDSCENE_MODEL_NAME="qwen3.5-plus" export MIDSCENE_MODEL_FAMILY="qwen3.5" # Planning model: GPT-5.4 export MIDSCENE_PLANNING_MODEL_API_KEY="sk-..." # Your GPT-5.4 API key export MIDSCENE_PLANNING_MODEL_BASE_URL="https://..." export MIDSCENE_PLANNING_MODEL_NAME="gpt-5.4" export MIDSCENE_PLANNING_MODEL_FAMILY="gpt-5" # Insight model: GPT-5.4 export MIDSCENE_INSIGHT_MODEL_API_KEY="sk-..." # Your GPT-5.4 API key export MIDSCENE_INSIGHT_MODEL_BASE_URL="https://..." export MIDSCENE_INSIGHT_MODEL_NAME="gpt-5.4" export MIDSCENE_INSIGHT_MODEL_FAMILY="gpt-5" ``` ## Other compatible models The following smaller models are also compatible with Midscene and designed for automation. They require less deployment hardware but may struggle with complex tasks or large screenshots. Evaluate them against your tasks and deployment constraints before choosing one. ### Zhipu AutoGLM Series {#auto-glm} Zhipu AutoGLM is an open-source mobile UI automation model (9B parameters) from Zhipu AI. After obtaining an API key from [Z.AI (Global)](https://z.ai/manage-apikey/apikey-list) or [BigModel (CN)](https://bigmodel.cn/usercenter/proj-mgmt/apikeys), configure: ```bash MIDSCENE_MODEL_BASE_URL="https://api.z.ai/api/paas/v4" # Or https://open.bigmodel.cn/api/paas/v4 MIDSCENE_MODEL_API_KEY="......" MIDSCENE_MODEL_NAME="autoglm-phone" MIDSCENE_MODEL_FAMILY="auto-glm" # Or "auto-glm-multilingual" ``` **About `MIDSCENE_MODEL_FAMILY` Configuration** AutoGLM provides two model versions, distinguished by `MIDSCENE_MODEL_FAMILY`: - `auto-glm` - Corresponds to AutoGLM-Phone-9B, optimized for **Chinese mobile applications** - `auto-glm-multilingual` - Corresponds to AutoGLM-Phone-9B-Multilingual, supports **English and other languages** Choose the appropriate version based on your application language. :::info AutoGLM is best suited for mobile interaction. APIs such as `aiAssert` and `aiQuery` require page understanding. When using these APIs, configure a separate Insight model with the `MIDSCENE_INSIGHT_MODEL_...` environment variables. See [Model strategy](./model-strategy) for details. ::: **Learn more about Zhipu AutoGLM** - Github: [https://github.com/zai-org/Open-AutoGLM](https://github.com/zai-org/Open-AutoGLM) - Hugging Face: [https://huggingface.co/zai-org/AutoGLM-Phone-9B](https://huggingface.co/zai-org/AutoGLM-Phone-9B) ### UI-TARS Series {#ui-tars} Use the deployed `doubao-1.5-ui-tars` on [Volcano Engine](https://volcengine.com): ```bash MIDSCENE_MODEL_BASE_URL="https://ark.cn-beijing.volces.com/api/v3" MIDSCENE_MODEL_API_KEY="...." MIDSCENE_MODEL_NAME="ep-2025..." # Inference endpoint ID or model name from Volcano Engine MIDSCENE_MODEL_FAMILY="vlm-ui-tars-doubao-1.5" ``` **About `MIDSCENE_MODEL_FAMILY`** This variable selects the UI-TARS version. Supported values: - `vlm-ui-tars` – for the 1.0 release - `vlm-ui-tars-doubao` – for the 1.5 release deployed on Volcano Engine (equivalent to `vlm-ui-tars-doubao-1.5`) - `vlm-ui-tars-doubao-1.5` – for the 1.5 release deployed on Volcano Engine :::info The legacy configurations `MIDSCENE_USE_VLM_UI_TARS=DOUBAO` or `MIDSCENE_USE_VLM_UI_TARS=1.5` are still supported but deprecated. Please migrate to `MIDSCENE_MODEL_FAMILY`. Migration mapping: - `MIDSCENE_USE_VLM_UI_TARS=1.0` → `MIDSCENE_MODEL_FAMILY="vlm-ui-tars"` - `MIDSCENE_USE_VLM_UI_TARS=1.5` → `MIDSCENE_MODEL_FAMILY="vlm-ui-tars-doubao-1.5"` - `MIDSCENE_USE_VLM_UI_TARS=DOUBAO` → `MIDSCENE_MODEL_FAMILY="vlm-ui-tars-doubao"` ::: ## Next steps - Learn when to use Default, Planning, and Insight models in [Model strategy](./model-strategy). - Look up every environment variable in [Model configuration reference](./model-config). - Diagnose connectivity and compatibility issues in [Model debugging and observability](./model-debugging-observability). --- url: /model-config.md --- # Model Configuration Reference Use this page to look up every Midscene model setting. For supported model names, model families, and copyable setup examples, see [Supported models and setup](./model-common-config). For model roles and combination guidance, see [Model strategy](./model-strategy). For connection issues, logs, tracing, and call recording, see [Model debugging and observability](./model-debugging-observability). ## Required settings You need to set a default model for Midscene; see [Model strategy](./model-strategy) for details. | Name | Description | |------|-------------| | `MIDSCENE_MODEL_API_KEY` | Model API key for OpenAI-compatible HTTP providers, e.g., `"sk-abcd..."`. Not required when `MIDSCENE_MODEL_BASE_URL="codex://app-server"`; see [Use Codex App Server](./model-common-config#use-codex-app-server-oauth-no-api-key) | | `MIDSCENE_MODEL_BASE_URL` | API endpoint URL, usually ending with a version (e.g., `/v1`); do not append `/chat/completion` here since the underlying sdk will add it automatically | | `MIDSCENE_MODEL_NAME` | Model name | | `MIDSCENE_MODEL_FAMILY` | Model family, determine the way of dealing with the coordinates | ## Advanced settings (optional) If you configure a dedicated Insight or Planning model, model-related `MIDSCENE_MODEL_*` settings in this section only take effect for the Insight or Planning intent when you configure the corresponding `MIDSCENE_INSIGHT_MODEL_*` or `MIDSCENE_PLANNING_MODEL_*` setting. | Name | Description | |------|-------------| | `MIDSCENE_MODEL_TIMEOUT` | Hard timeout for AI API calls in milliseconds (default intent). Defaults to `180000` (180s). Set to `0` to disable the hard timeout and let the request run indefinitely (only a caller-provided `AbortSignal` will cancel it). Note: Midscene controls the full request lifetime, not just the time until the first response header arrives, so stalled body reads can still be terminated cleanly | | `MIDSCENE_MODEL_TEMPERATURE` | Sampling temperature for model responses | | `MIDSCENE_MODEL_RETRY_COUNT` | Number of retries when AI call fails, default 1 (i.e., retry once after failure). Retries occur when the model request encounters an HTTP error or when the model response cannot be structurally parsed | | `MIDSCENE_MODEL_RETRY_INTERVAL` | Interval between retries in milliseconds, default 2000 | | `MIDSCENE_MODEL_REASONING_ENABLED` | Controls whether model-native thinking is enabled. Midscene disables it by default. See [Model-native thinking](#model-native-reasoning) | | `MIDSCENE_MODEL_REASONING_EFFORT` | Controls model-native thinking effort, supported by some models. Common values: `low`, `medium`, `high`. See [Model-native thinking](#model-native-reasoning) | | `MIDSCENE_MODEL_REASONING_BUDGET` | Thinking token budget (number), supported by some models. See [Model-native thinking](#model-native-reasoning) | | `MIDSCENE_MODEL_RESPONSE_FORMAT` | Structured response strategy: `auto` (default) lets Midscene automatically use `response_format` in appropriate scenarios to specify a structured output format (usually JSON), making the model response as suitable for structured parsing as possible; `none` does not set `response_format`, for models that do not support structured output. | | `MIDSCENE_MODEL_HTTP_PROXY` | HTTP/HTTPS proxy, e.g., `http://127.0.0.1:8080` or `https://proxy.example.com:8080`. Takes precedence over `MIDSCENE_MODEL_SOCKS_PROXY` | | `MIDSCENE_MODEL_SOCKS_PROXY` | SOCKS proxy, e.g., `socks5://127.0.0.1:1080` | | `MIDSCENE_MODEL_INIT_CONFIG_JSON` | JSON blob that overrides the OpenAI SDK initialization config. Use `defaultHeaders` for custom auth headers; `extra_headers` and `extraHeaders` are accepted as aliases | | `MIDSCENE_MODEL_EXTRA_BODY_JSON` | JSON blob merged into each chat completion request body. Unlike `MIDSCENE_MODEL_INIT_CONFIG_JSON` (which configures the SDK client), this is spread into every `completion.create()` call sent to the model, e.g. enabling thinking mode in vLLM: `'{"chat_template_kwargs":{"enable_thinking":true}}'` | > Note: Control replanning behavior with the agent option `replanningCycleLimit` (defaults to 20, or 40 for `vlm-ui-tars`), not with environment variables. ### Configure a dedicated Insight model Set the following if the Insight intent needs a different model: | Name | Description | |------|-------------| | `MIDSCENE_INSIGHT_MODEL_API_KEY` | API key | | `MIDSCENE_INSIGHT_MODEL_BASE_URL` | API endpoint URL (omit the trailing `/chat/completion`) | | `MIDSCENE_INSIGHT_MODEL_NAME` | Model name | | `MIDSCENE_INSIGHT_MODEL_FAMILY` | Model family | | `MIDSCENE_INSIGHT_MODEL_TIMEOUT` | Optional; timeout for Insight intent AI API calls in milliseconds | | `MIDSCENE_INSIGHT_MODEL_TEMPERATURE` | Optional; sampling temperature for Insight intent responses | | `MIDSCENE_INSIGHT_MODEL_RETRY_COUNT` | Optional; same effect as `MIDSCENE_MODEL_RETRY_COUNT` | | `MIDSCENE_INSIGHT_MODEL_RETRY_INTERVAL` | Optional; same effect as `MIDSCENE_MODEL_RETRY_INTERVAL` | | `MIDSCENE_INSIGHT_MODEL_HTTP_PROXY` | Optional; same effect as `MIDSCENE_MODEL_HTTP_PROXY` | | `MIDSCENE_INSIGHT_MODEL_SOCKS_PROXY` | Optional; same effect as `MIDSCENE_MODEL_SOCKS_PROXY` | | `MIDSCENE_INSIGHT_MODEL_INIT_CONFIG_JSON` | Optional; same effect as `MIDSCENE_MODEL_INIT_CONFIG_JSON` | | `MIDSCENE_INSIGHT_MODEL_EXTRA_BODY_JSON` | Optional; same effect as `MIDSCENE_MODEL_EXTRA_BODY_JSON` | | `MIDSCENE_INSIGHT_MODEL_RESPONSE_FORMAT` | Optional; controls the structured response strategy in appropriate Insight scenarios | ### Configure a dedicated Planning model Set the following if the Planning intent needs a different model: | Name | Description | |------|-------------| | `MIDSCENE_PLANNING_MODEL_API_KEY` | API key | | `MIDSCENE_PLANNING_MODEL_BASE_URL` | API endpoint URL (omit the trailing `/chat/completion`) | | `MIDSCENE_PLANNING_MODEL_NAME` | Model name | | `MIDSCENE_PLANNING_MODEL_FAMILY` | Model family | | `MIDSCENE_PLANNING_MODEL_TIMEOUT` | Optional; timeout for Planning intent AI API calls in milliseconds | | `MIDSCENE_PLANNING_MODEL_TEMPERATURE` | Optional; sampling temperature for Planning intent responses | | `MIDSCENE_PLANNING_MODEL_RETRY_COUNT` | Optional; same effect as `MIDSCENE_MODEL_RETRY_COUNT` | | `MIDSCENE_PLANNING_MODEL_RETRY_INTERVAL` | Optional; same effect as `MIDSCENE_MODEL_RETRY_INTERVAL` | | `MIDSCENE_PLANNING_MODEL_HTTP_PROXY` | Optional; same effect as `MIDSCENE_MODEL_HTTP_PROXY` | | `MIDSCENE_PLANNING_MODEL_SOCKS_PROXY` | Optional; same effect as `MIDSCENE_MODEL_SOCKS_PROXY` | | `MIDSCENE_PLANNING_MODEL_INIT_CONFIG_JSON` | Optional; same effect as `MIDSCENE_MODEL_INIT_CONFIG_JSON` | | `MIDSCENE_PLANNING_MODEL_EXTRA_BODY_JSON` | Optional; same effect as `MIDSCENE_MODEL_EXTRA_BODY_JSON` | | `MIDSCENE_PLANNING_MODEL_RESPONSE_FORMAT` | Optional; controls the structured response strategy in appropriate Planning scenarios | ### Model-native thinking {#model-native-reasoning} Midscene disables model-native thinking by default for better execution speed and stability. If a model cannot disable native thinking, Midscene reduces it as much as possible by controlling the thinking granularity or budget. The following environment variables provide a unified Midscene abstraction over provider-specific parameters. The parameters sent to the model depend on `MIDSCENE_MODEL_FAMILY`. `MIDSCENE_MODEL_REASONING_ENABLED` explicitly controls whether model-native thinking is enabled: - `false`: Force-disable model-native thinking. This is the Midscene default. - `true`: Force-enable model-native thinking. - `default`: Follow the model's default behavior. Midscene does not send a parameter that enables or disables thinking, and it ignores explicit `MIDSCENE_MODEL_REASONING_BUDGET` and `MIDSCENE_MODEL_REASONING_EFFORT` settings. The following model families currently support `MIDSCENE_MODEL_REASONING_ENABLED`: - Qwen: Maps to `enable_thinking`. - Doubao: Maps to `thinking.type`. - DeepSeek: Maps to `thinking.type`. - Zhipu GLM: Maps to `thinking.type`. - GPT-5: Maps to `reasoning_effort`. Midscene uses `medium` when enabled and `none` when disabled. - Gemini: Maps to `thinking_config.thinking_level`. Midscene uses `medium` when enabled and `minimal` when disabled. - Kimi K2 series: Maps to `thinking.type`. - Xiaomi MiMo: Maps to `thinking.type`. `MIDSCENE_MODEL_REASONING_BUDGET` controls the thinking budget. Qwen currently supports this setting through `thinking_budget`. `MIDSCENE_MODEL_REASONING_EFFORT` controls thinking effort. The following model families currently support it: - Doubao: Maps to `reasoning_effort`. - DeepSeek: Maps to `reasoning_effort`. - Gemini: Maps to `thinking_config.thinking_level`. - GPT-5: Maps to `reasoning_effort`. - Kimi K3 series: Maps to `reasoning_effort`. Provider support and accepted values vary. See each provider's official documentation for supported model versions and values. If the current model does not support an explicit setting, Midscene ignores it instead of guessing a provider-specific private parameter. ## Still-compatible configs (not recommended) The following environment variables are deprecated but still compatible. We recommend migrating to the new configuration approach. ### Legacy model family configuration | Name | Description | New approach | |------|-------------|--------------| | `MIDSCENE_USE_DOUBAO_VISION` | Deprecated. Enables Doubao vision model | Use `MIDSCENE_MODEL_FAMILY="doubao-vision"` | | `MIDSCENE_USE_QWEN3_VL` | Deprecated. Enables Qwen3-VL model | Use `MIDSCENE_MODEL_FAMILY="qwen3-vl"` | | `MIDSCENE_USE_QWEN_VL` | Deprecated. Enables Qwen2.5-VL model | Use `MIDSCENE_MODEL_FAMILY="qwen2.5-vl"` | | `MIDSCENE_USE_GEMINI` | Deprecated. Enables Gemini model | Use `MIDSCENE_MODEL_FAMILY="gemini"` | | `MIDSCENE_USE_VLM_UI_TARS` | Deprecated. Enables UI-TARS model | Use `MIDSCENE_MODEL_FAMILY="vlm-ui-tars"` | ### General configuration | Name | Description | New approach | |------|-------------|--------------| | `OPENAI_API_KEY` | Deprecated but supported | Prefer `MIDSCENE_MODEL_API_KEY` | | `OPENAI_BASE_URL` | Deprecated but supported | Prefer `MIDSCENE_MODEL_BASE_URL` | | `MIDSCENE_OPENAI_INIT_CONFIG_JSON` | Deprecated but supported | Prefer `MIDSCENE_MODEL_INIT_CONFIG_JSON` | | `MIDSCENE_OPENAI_HTTP_PROXY` | Deprecated but supported | Prefer `MIDSCENE_MODEL_HTTP_PROXY` | | `MIDSCENE_OPENAI_SOCKS_PROXY` | Deprecated but supported | Prefer `MIDSCENE_MODEL_SOCKS_PROXY` | ## Debugging and observability settings The following settings enable model diagnostics, tracing, and local call recording. See [Model debugging and observability](./model-debugging-observability) for installation steps, usage examples, troubleshooting, and security guidance. ### Debug logs For supported `DEBUG` selectors and log behavior, see [Runtime configuration: Debug logs](./reference/#debug-logs). ### LangSmith | Name | Description | | --- | --- | | `MIDSCENE_LANGSMITH_DEBUG` | Set to `1` to enable Midscene's automatic LangSmith integration | | `LANGCHAIN_API_KEY` | LangSmith API key | | `LANGCHAIN_TRACING` | Set to `true` to enable LangSmith tracing | | `LANGCHAIN_ENDPOINT` | LangSmith service endpoint | ### Langfuse | Name | Description | | --- | --- | | `MIDSCENE_LANGFUSE_DEBUG` | Set to `1` to enable Midscene's automatic Langfuse integration | | `LANGFUSE_PUBLIC_KEY` | Langfuse public key | | `LANGFUSE_SECRET_KEY` | Langfuse secret key | | `LANGFUSE_BASE_URL` | Langfuse service URL | ### Model-call recording | Name | Description | | --- | --- | | `MIDSCENE_RECORD_MODEL_CALL` | Set to `true` to write model requests, responses, and streaming chunks to local JSONL files | --- url: /model-debugging-observability.md --- # Model Debugging and Observability Use this guide to diagnose model connectivity and compatibility issues, inspect latency and token usage, collect traces, and record model calls. ## Verify model connectivity This section provides two verification methods. First, send a direct request to confirm that the model API is reachable. Then run the Midscene verification command to check model compatibility. ### Send a direct request to the model service The following `curl` request checks whether the Base URL, API key, and model name work. It only verifies basic model API connectivity. It does not check whether the model meets Midscene's compatibility requirements. ```bash MIDSCENE_MODEL_BASE_URL='replace with your baseUrl' MIDSCENE_MODEL_API_KEY='replace with your API key' MIDSCENE_MODEL_NAME='replace with your model name' curl -X POST "${MIDSCENE_MODEL_BASE_URL%/}/chat/completions" \ -H "Authorization: Bearer ${MIDSCENE_MODEL_API_KEY}" \ -H "Content-Type: application/json" \ -d '{ "model": "'"${MIDSCENE_MODEL_NAME}"'", "messages": [ { "role": "user", "content": "What is 1+1?" } ] }' ``` ### Use the Midscene verification command This command checks both model connectivity and Midscene compatibility. Put your model configuration in a `.env` file, then run: ```bash # If the current project has @midscene/cli installed, use the local midscene command npx midscene model verify # If the current project does not have @midscene/cli installed, or you want to use the latest version npx @midscene/cli@latest model verify ``` The command reads the `.env` file in the current working directory. Dotenv debug logging is enabled by default, and values from `.env` override existing shell environment variables. If the `curl` request succeeds but the Midscene verification command fails, the model API is reachable. Continue by checking the model capabilities and Midscene configuration. ## Common configuration errors ### `MIDSCENE_MODEL_FAMILY` is not set to a multimodal model If you see `MIDSCENE_MODEL_FAMILY is not set to a multimodal model with UI localization`, make sure the `MIDSCENE_MODEL_FAMILY` environment variable for the multimodal model is set correctly. Starting with version 1.0, Midscene recommends using `MIDSCENE_MODEL_FAMILY` to specify the multimodal model type. Legacy `MIDSCENE_USE_...` settings remain compatible but are deprecated. See [Supported models and setup](/model-common-config.md) for the correct model family and a complete configuration example. ### Base URL or model name is incorrect Confirm that `MIDSCENE_MODEL_BASE_URL` points to the provider's API endpoint. It commonly ends with a version such as `/v1`. Do not append `/chat/completion`, because the underlying SDK adds the request path. Also confirm that `MIDSCENE_MODEL_NAME` matches a model available from that endpoint. ## Improve model performance If the model connects successfully but localization, planning, or page understanding is unstable, try the following: * Review the [replay report](/consume-report-file.md) to confirm the task timeline is correct and the flow did not enter the wrong page or branch. * Prefer newer officially supported versions within the same model series. * Compare models from different providers with representative tasks, focusing on success rate, latency, and cost. * For complex tasks, configure a separate Planning or Insight model. See [Model strategy](/model-strategy.md) for the responsibilities of each model. ## Debugging capabilities ### Debug logs Set `DEBUG` when you need additional diagnostic output. Common selectors include: * `DEBUG=midscene:ai:profile:stats` prints model latency and Token usage. * `DEBUG=midscene:ai:call` prints AI response details. * `DEBUG=midscene:*` prints all Midscene Debug logs. For the complete selector list, log location, and handling notes, see [Runtime configuration: Debug logs](/reference.md#debug-logs). Usage statistics are also available in generated [report files](/consume-report-file.md). ### Record model calls Set `MIDSCENE_RECORD_MODEL_CALL=true` to write model requests, responses, and streaming chunks to a JSONL file: ```text midscene_run/model-requests/<start-time>-<pid>.jsonl ``` Each process creates one file, with one JSON event per line. Local recording is available only in Node.js and Electron. Browsers and Workers do not write local files. Codex App Server records also include available protocol metadata. Every event has a `type` of `request`, `chunk`, `response`, or `error`. It also includes an `executionId` that groups calls and retries under the same execution ID. For HTTP model requests, this value is also sent in the `x-midscene-execution-id` header. Calls outside a report execution, such as connectivity checks, use a generated ID prefixed with `unscoped-`. :::warning Important notes Files contain request bodies, including custom `extraBody`, response headers and bodies, streaming responses, and possibly base64-encoded screenshots. Request headers are not recorded. These files may be sensitive and large. Enable recording only while troubleshooting, and protect or delete the files afterward. The record format is not stable across versions. ::: ### Request tracing headers Midscene automatically adds the following headers to OpenAI-compatible HTTP model requests: | Header | Value | Purpose | | --- | --- | --- | | `x-midscene-version` | The current `@midscene/core` version | Identify the Midscene version that sent the request | | `x-midscene-execution-id` | The current execution ID | Group all model requests and retries under the same execution ID, such as an `aiAct` call | Calls outside a report execution, such as connectivity checks, use a generated execution ID prefixed with `unscoped-`. These two headers are sent by default; if headers with the same names are customized in `MIDSCENE_*_INIT_CONFIG_JSON`, Midscene overrides them. ## Observability platforms ### LangSmith LangSmith is a platform for debugging large language models. Midscene provides automatic integration support through a dependency and environment variables. **Install the dependency** ```bash npm install langsmith ``` **Set environment variables** ```bash # Enable Midscene's LangSmith auto-integration export MIDSCENE_LANGSMITH_DEBUG=1 # LangSmith configuration export LANGCHAIN_API_KEY="your-langchain-api-key-here" export LANGCHAIN_TRACING=true export LANGCHAIN_ENDPOINT="https://api.smith.langchain.com" # export LANGCHAIN_ENDPOINT="https://eu.api.smith.langchain.com" # If signed up in the EU region ``` After starting Midscene, you should see a log similar to: ```log DEBUGGING MODE: langsmith wrapper enabled ``` Notes: * LangSmith and Langfuse can be enabled simultaneously. * This integration supports Node.js only. Browser environments throw an error. * If you use [`createOpenAIClient`](/reference.md#custom-openai-client), it overrides the environment-variable integration. For finer-grained control, such as enabling LangSmith only for specific tasks, use [`createOpenAIClient`](/reference.md#custom-openai-client) to wrap the client manually. ### Langfuse [Langfuse](https://langfuse.com) is an LLM observability platform. Midscene integrates Langfuse's `observeOpenAI` wrapper to trace OpenAI API calls automatically. Because Langfuse tracing uses OpenTelemetry, initialize the OpenTelemetry SDK when the application starts. **Install the dependencies** ```bash npm install @langfuse/openai @langfuse/otel @opentelemetry/sdk-node ``` **Initialize OpenTelemetry** Add this code at the very top of the application entry file: ```typescript import { NodeSDK } from "@opentelemetry/sdk-node"; import { LangfuseSpanProcessor } from "@langfuse/otel"; const sdk = new NodeSDK({ spanProcessors: [new LangfuseSpanProcessor()], }); sdk.start(); ``` **Set environment variables** ```bash # Enable Midscene's Langfuse auto-integration export MIDSCENE_LANGFUSE_DEBUG=1 # Langfuse configuration export LANGFUSE_PUBLIC_KEY="your-langfuse-public-key-here" export LANGFUSE_SECRET_KEY="your-langfuse-secret-key-here" export LANGFUSE_BASE_URL="https://cloud.langfuse.com" # 🇪🇺 EU region # export LANGFUSE_BASE_URL="https://us.cloud.langfuse.com" # 🇺🇸 US region ``` After starting Midscene, you should see logs similar to: ```log OpenTelemetry SDK initialized for Langfuse tracing DEBUGGING MODE: langfuse wrapper enabled ``` See the [Langfuse OpenAI integration documentation](https://langfuse.com/integrations/model-providers/openai-js) for more configuration options and best practices. Notes: * LangSmith and Langfuse can be enabled simultaneously. * This integration supports Node.js only. Browser environments throw an error. * If you use [`createOpenAIClient`](/reference.md#custom-openai-client), it overrides the environment-variable integration. ## Security considerations * Do not commit `.env` files, traces, debug logs, or model-call records to source control. * Review logs and traces before sharing them because they can contain model inputs, outputs, or screenshots. --- url: /model-strategy.md --- # Model Strategy Midscene's model strategy serves two goals. First, it uses pure vision to understand the interface that users actually see, keeping UI automation independent of the rendering stack. Second, it provides the option to combine multiple models for complex scenarios. ## A pure-vision approach based on the visible interface AI-powered UI automation requires task planning and element localization. The industry mainly uses two localization approaches: combining DOM data with annotated screenshots, or using screenshots directly for pure-vision localization. Midscene uses pure vision, with the model analyzing the UI screenshot and locating target elements directly. UI actions and element localization do not depend on DOM data or extra annotations. This choice makes the visible interface the source of truth for automation and gives Midscene several advantages: * It uses a consistent approach across browser Canvas, Android, iOS, desktop applications, and other types of interfaces. * It can validate what users actually see, including colors, highlighted states, and layout. * It is independent of the UI rendering stack and does not require selectors or extra UI annotations. * Token consumption depends only on page resolution and task complexity, and does not inflate as the page structure (such as the number of DOM elements) grows. Pure vision is not a local optimization for one platform. It is the shared foundation of Midscene's cross-platform interaction capabilities. The same task descriptions and interaction patterns can extend across interfaces and devices while staying close to how real users operate software. The vision-based approach also has clear limitations. Pure-vision localization requires models with visual understanding capabilities — only designated models that are stable for GUI operations can be used, not any arbitrary LLM. Midscene accepts higher model capability requirements in exchange for cross-platform consistency and lower UI maintenance costs. Data extraction and page-understanding workloads can still include DOM data when needed. See the [API Reference](/reference.md#extraction-location-assertion) for the relevant options. <span id="advanced-combining-multiple-models" /> ## Combine models for complex scenarios Midscene uses a multimodal Default model for task planning, element localization, page understanding, and the rest of the automation workflow. This default approach covers most UI automation scenarios and lets users get started with minimal configuration. When complex planning, data extraction, or page understanding requires specialized model capabilities, users can add Planning and Insight models to the Default model. Each model contributes its strengths to the same automation workflow: | Model | Role in the combination | | --- | --- | | Default model | Provides the foundation, handling element localization (Locate) and workloads not assigned to Planning or Insight | | Planning model | Enhances planning for complex goals, multi-step tasks, and branching scenarios | | Insight model | Enhances data extraction, assertions, and page understanding | This collaboration extends Midscene's ability to handle complex tasks, but it can also increase task latency and token usage. Start with the Default model, then introduce specialized models only for a clear capability bottleneck. ## Next steps This page focuses on Midscene's model philosophy and selection principles. For setup instructions, see [Optional: configure multiple models](/model-common-config.md#multi-model-combination-example). If task quality is unstable, use [Model debugging and observability](/model-debugging-observability.md) to identify the problem. --- url: /platforms/android.md --- import { PackageManagerTabs } from '@theme'; # Android Midscene connects to Android devices through adb to automate apps and system interfaces. This guide covers device connection, model configuration, Playground, and JavaScript SDK integration with `@midscene/android`. ## See it in action **Prompt:** Open the Booking app. Search for a hotel in Tokyo for four adults on Christmas, with a score of 8 or above. <video src="https://lf3-static.bytednsdoc.com/obj/eden-cn/nupipfups/Midscene/1.0-showcases/booking2.mp4" poster="https://lf3-static.bytednsdoc.com/obj/eden-cn/nupipfups/Midscene/1.0-showcases/booking.png" height="300" controls /> View the [full report](https://lf3-static.bytednsdoc.com/obj/eden-cn/nupipfups/Midscene/1.0-showcases/booking.html), or explore more [Midscene showcases](/showcases.md). ## Get started ### Prepare your Android device Before scripting, confirm adb can talk to your device and the device trusts your machine. #### Install adb and set `ANDROID_HOME` * Install via [Android Studio](https://developer.android.com/studio) or the [command-line tools](https://developer.android.com/studio#command-line-tools-only) * Verify installation: ```bash adb --version ``` Example output indicates success: ```log Android Debug Bridge version 1.0.41 Version 34.0.4-10411341 Installed as /usr/local/bin//adb Running on Darwin 24.3.0 (arm64) ``` * Set `ANDROID_HOME` as documented in [Android environment variables](https://developer.android.com/tools/variables), then confirm: ```bash echo $ANDROID_HOME ``` Any non-empty output means it is configured: ```log /Users/your_username/Library/Android/sdk ``` #### Enable USB debugging and verify the device In the system settings developer options, enable **USB debugging** (and **USB debugging (Security settings)** if present), then connect the device via USB. <p align="center"> <img src="/android-usb-debug-en.png" alt="android usb debug" width="400" /> </p> Verify the connection: ```bash adb devices -l ``` Example success output: ```log List of devices attached s4ey59 device usb:34603008X product:cezanne model:M2006J device:cezan transport_id:3 ``` ### Launch Playground Playground is the fastest way to validate the connection and try core capabilities such as `aiAct`, `aiQuery`, and `aiAssert` without writing code. It shares the same core as `@midscene/android`, so anything that works here will behave the same once scripted. 1. Launch the Playground CLI: ```bash npx --yes @midscene/android-playground ``` 2. Click the gear icon in the Playground window, then paste your API Key configuration. See [Supported models and setup](/model-common-config.md) if you still need a model configuration. ![](/android-set-env.png) ## Use the JavaScript SDK Once Playground works, move to a repeatable script with the JavaScript SDK. ### Configure the model The following example uses `qwen3.7-plus` through Alibaba Cloud: ```bash export MIDSCENE_MODEL_BASE_URL="https://dashscope.aliyuncs.com/compatible-mode/v1" export MIDSCENE_MODEL_API_KEY="your-api-key" export MIDSCENE_MODEL_NAME="qwen3.7-plus" export MIDSCENE_MODEL_FAMILY="qwen3" ``` Replace `your-api-key` with your API Key. > To use another model, such as Doubao, GLM, Gemini, or GPT-5, see [Supported models and setup](/model-common-config.md). For all configuration options, see [Model configuration](/model-config.md). ### Install dependencies <PackageManagerTabs command="install @midscene/android dotenv --save-dev" /> ### Write a script Save the following code as `./demo.ts`. It opens the browser on the device, searches eBay, and asserts the result list. ```typescript title="./demo.ts" import 'dotenv/config'; // load Midscene environment variables from .env if present import { AndroidAgent, AndroidDevice, getConnectedDevices, } from '@midscene/android'; const sleep = (ms) => new Promise((r) => setTimeout(r, ms)); Promise.resolve( (async () => { const devices = await getConnectedDevices(); const device = new AndroidDevice(devices[0].udid); const agent = new AndroidAgent(device, { aiActionContext: 'If any location, permission, user agreement, etc. popup, click agree. If login page pops up, close it.', }); await device.connect(); await agent.aiAct('open browser and navigate to ebay.com'); await sleep(5000); await agent.aiAct('type "Headphones" in search box, hit Enter'); await agent.aiWaitFor('There is at least one headphone product'); const items = await agent.aiQuery( '{itemTitle: string, price: Number}[], find item in list and corresponding price', ); console.log('headphones in stock', items); await agent.aiAssert('There is a category filter on the left'); })(), ); ``` ### Run the script ```bash npx tsx demo.ts ``` After the script finishes, you should see `Midscene - report file updated: /path/to/report/some_id.html` in the console. Open the generated HTML file in a browser to replay every interaction, query, and assertion. ## Advanced Use this section to customize device behavior, integrate Midscene into your framework, or troubleshoot adb issues. For detailed constructor parameters, see the [Android section of the API reference](/reference.md#android). ### Extend Midscene on Android Use `defineAction()` to define custom actions. When constructing `AndroidDevice`, pass these actions through `customActions`. Midscene appends these actions to the planner so the Agent can call the domain-specific actions you define. ```typescript import { getMidsceneLocationSchema, z } from '@midscene/core'; import { defineAction } from '@midscene/core/device'; import { AndroidAgent, AndroidDevice, getConnectedDevices } from '@midscene/android'; const ContinuousClick = defineAction({ name: 'continuousClick', description: 'Click the same target repeatedly', paramSchema: z.object({ locate: getMidsceneLocationSchema(), count: z.number().int().positive().describe('How many times to click'), }), async call(param) { const { locate, count } = param; console.log('click target center', locate.center); console.log('click count', count); }, }); const devices = await getConnectedDevices(); const device = new AndroidDevice(devices[0].udid, { customActions: [ContinuousClick], }); await device.connect(); const agent = new AndroidAgent(device); await agent.aiAct('click the red button five times'); ``` See [Integrate with any interface](/integrate-with-any-interface.md#define-a-custom-action) for a deeper explanation of custom actions and action schemas. ## FAQ ### Why can't I control the device even though I've connected it? A common error is: ``` Error: Exception occurred while executing 'tap': java.lang.SecurityException: Injecting input events requires the caller (or the source of the instrumentation, if any) to have the INJECT_EVENTS permission. ``` Make sure USB debugging is enabled and the device is unlocked in developer options. <p align="center"> <img src="/android-usb-debug-en.png" alt="android usb debug" width="400" /> </p> ### Text input is cleared or lost after typing After entering text, Midscene automatically dismisses the keyboard. The default behavior sends an **ESC key event**. However, some input fields (especially those inside WebView) listen for the ESC key event, which can cause side effects such as: * Clearing the text just entered * Closing the popup/modal containing the input field * Navigating away from the current page You can try the following solutions in order of priority: **Option 1: Use the BACK key (Android back button) to dismiss the keyboard** Set `keyboardDismissStrategy` to `'back-first'` to use the Android BACK key instead of ESC to dismiss the keyboard: ```typescript const device = new AndroidDevice('device-id', { keyboardDismissStrategy: 'back-first', }); ``` **Option 2: Disable auto keyboard dismiss** If your input field also listens for the BACK key, you can disable auto keyboard dismiss entirely and let the AI Agent or subsequent actions manage the keyboard state: ```typescript const device = new AndroidDevice('device-id', { autoDismissKeyboard: false, }); ``` With auto dismiss disabled, the keyboard will remain visible and may cover a large portion of the screen. You can work around this by: * Using `aiAct` to manually dismiss the keyboard, e.g. `await agent.aiAct('tap the collapse button on the keyboard')` * Installing and switching to [ADBKeyBoard](https://github.com/senzhk/ADBKeyBoard) — a minimal virtual keyboard that takes up very little screen space, so it barely affects screen interactions even when visible ### English text is rewritten by the Android keyboard If the report shows the correct input parameter, but the app receives different text, missing text, or Chinese/pinyin candidates, the active Android input method may be rewriting the text. This can happen when pure ASCII text goes through the native `adb input text` path while a Chinese keyboard or autocorrect keyboard is active. Use the existing `imeStrategy` option and force all text input through yadb: ```typescript const device = new AndroidDevice('device-id', { imeStrategy: 'always-yadb', }); ``` For YAML scripts: ```yaml android: imeStrategy: always-yadb ``` Or set the environment variable: ```bash export MIDSCENE_ANDROID_IME_STRATEGY=always-yadb ``` This is different from text being cleared after typing. If the text is entered correctly and then disappears, check `keyboardDismissStrategy` or `autoDismissKeyboard` instead. ### Screenshot of secure pages (e.g. password input) shows black Some pages (such as password entry screens in banking or payment apps) set `FLAG_SECURE` to block screenshots. When `screencap` captures such a page, the secure region appears black. The yadb tool creates a virtual display via the `secure` parameter of `SurfaceControl.createDisplay`, which can help capture secure content in some setups. Whether it can actually capture `FLAG_SECURE` pages depends on the Android version, ROM, root/hook environment, and device configuration (for example, some setups may require root plus a Magisk hook). Always verify against your actual device. Use the `screenshotStrategy` option to force yadb for screenshots: ```typescript const device = new AndroidDevice('device-id', { screenshotStrategy: 'always-yadb', }); ``` For YAML scripts: ```yaml android: screenshotStrategy: always-yadb ``` Or set the environment variable: ```bash export MIDSCENE_ANDROID_SCREENSHOT_STRATEGY=always-yadb ``` The default is `auto`, which tries `adb.takeScreenshot` first, falls back to shell `screencap`, and uses the yadb tool if `screencap` fails to execute. scrcpy is only tried before these when `scrcpyConfig.enabled` is turned on. `auto` does not analyze screenshot content — even if a method succeeds but returns a black frame, it will not automatically switch to yadb; it only moves to the next method when the previous one fails to execute. For secure pages that produce a valid but black image, set `always-yadb` to bypass the default `auto` flow (`adb.takeScreenshot`, `screencap`, and scrcpy when enabled) and use yadb directly. Yadb can only capture the default display (`displayId=0`), so combining `always-yadb` with a non-zero `displayId` throws an error. ### Why does scrcpy fall back to ADB screenshots? Midscene rejects scrcpy frames that cannot be proven to have been captured after the latest completed action and that do not satisfy the absolute frame-age limit. If an established stream cannot provide a valid frame, Midscene restarts scrcpy once. It falls back to an ADB screenshot only if the new stream also fails. This can happen when an Android encoder stops emitting frames on a static screen, or when stream startup or transport is interrupted. The freshness warning alone does not diagnose link bandwidth or an excessive video bitrate. Do not change the bitrate solely because this warning appears. ### How do I configure the scrcpy video bitrate? Use `--scrcpy-video-bit-rate <bits-per-second>` (or `--scrcpyVideoBitRate`) on an Android CLI command. Providing this option enables scrcpy, so `--use-scrcpy` is optional when the bitrate option is present: ```bash midscene-android tap \ --device-id <device-id> \ --locate '{"prompt":"the target"}' \ --scrcpy-video-bit-rate <bits-per-second> ``` Each CLI command is a separate process, so pass the option to every command that should override the default. For the JavaScript SDK or YAML, set `scrcpyConfig.videoBitRate` once on that device configuration. Changing the bitrate trades encoded bandwidth against screenshot detail. Tune it only when independent transport measurements show a need, then validate screenshot and recognition quality. See the [`AndroidDevice` scrcpy reference](/reference/index.md#scrcpy) for the default and related options. ### How do I use a custom adb path or remote adb server? Set the environment variables first: ```bash export MIDSCENE_ADB_PATH=/path/to/adb export MIDSCENE_ADB_REMOTE_HOST=192.168.1.100 export MIDSCENE_ADB_REMOTE_PORT=5037 ``` You can also provide the same information via the constructor: ```typescript const device = new AndroidDevice('s4ey59', { androidAdbPath: '/path/to/adb', remoteAdbHost: '192.168.1.100', remoteAdbPort: 5037, }); ``` ## More * For every Agent method, check the [API reference (Common)](/reference.md#interaction-methods). * For Android-specific APIs, see [API reference (Android)](/reference.md#android). * Use [YAML automation scripts and command-line tools](/automate-with-scripts-in-yaml.md). * Demo projects * Android JavaScript SDK demo: [https://github.com/web-infra-dev/midscene-example/blob/main/android/javascript-sdk-demo](https://github.com/web-infra-dev/midscene-example/blob/main/android/javascript-sdk-demo) * Android + Vitest demo: [https://github.com/web-infra-dev/midscene-example/tree/main/android/vitest-demo](https://github.com/web-infra-dev/midscene-example/tree/main/android/vitest-demo) --- url: /platforms/desktop.md --- import { PackageManagerTabs } from '@theme'; # Desktop Midscene uses native keyboard and mouse controls to automate desktop applications on Windows, macOS, and Linux. It supports mouse and keyboard input, screenshots, and multiple displays. Use it to test Electron, Qt, and native applications, or to automate workflows across desktop applications. This guide covers platform setup, model configuration, Playground, and JavaScript SDK integration with `@midscene/computer`. ## See it in action **Prompt (macOS):** Open Safari and post a tweet announcing that Midscene supports AutoGLM. Use the AutoGLM video from the Downloads folder. <video src="https://lf3-static.bytednsdoc.com/obj/eden-cn/nupipfups/Midscene/pc-twitter2.mp4" height="300" controls /> View the [full report](https://lf3-static.bytednsdoc.com/obj/eden-cn/nupipfups/Midscene/1.0-showcases/pc-twitter2-midscene_report.html), or explore more [Midscene showcases](/showcases.md). ## Get started ### Prepare your desktop environment #### Node.js Node.js 18.19.0 or higher is required. #### Platform-specific dependencies **macOS**: Accessibility permissions are required for keyboard and mouse control. When you run the script for the first time, macOS will prompt you to grant access. Go to **System Settings > Privacy & Security > Accessibility** and enable permissions for the application running your script (e.g., Terminal, iTerm2, VS Code, WebStorm, or other IDEs). For more details, see [nut.js macOS setup](https://github.com/nut-tree/nut.js#macos). **Windows**: No extra setup is needed for ordinary apps. However, Windows isolates input across privilege levels (UIPI): a non-elevated process **cannot** send mouse or keyboard input to a window that runs **as Administrator** (elevated). The input is silently dropped — the cursor still moves to the right spot, but clicks and keystrokes have no effect. Prefer running the target application without Administrator privileges. If the target application must stay elevated, run the terminal or Node.js that launches Midscene **as Administrator** too, so both processes share the same privilege level. See [Windows: clicks have no effect on some apps](#windows-clicks-have-no-effect-on-some-apps). **Linux**: [ImageMagick](https://imagemagick.org/script/download.php) is required for screenshot functionality. **Headless Linux (CI)**: To run desktop automation on a headless Linux server (e.g. GitHub Actions), install Xvfb and its dependencies, then enable headless mode: ```bash # Install dependencies sudo apt-get install -y xvfb x11-xserver-utils imagemagick ``` ```typescript // Option 1: Pass headless option const agent = await agentForComputer({ headless: true }); // Option 2: Set environment variable // MIDSCENE_COMPUTER_HEADLESS_LINUX=true npx tsx example.ts ``` Xvfb creates a virtual display so that mouse, keyboard, and screenshot operations work without a physical monitor. See [API Reference](/reference.md#desktop) for details. ### Launch Playground Playground is the fastest way to validate the connection and try core capabilities such as `aiAct`, `aiQuery`, and `aiAssert` without writing code. It shares the same core as `@midscene/computer`, so anything that works here will behave the same once scripted. 1. Launch the Playground CLI: ```bash npx --yes @midscene/computer-playground ``` 2. Click the gear icon in the Playground window, then paste your API Key configuration. See [Supported models and setup](/model-common-config.md) if you still need a model configuration. ## Use the JavaScript SDK Once Playground works, move to a repeatable script with the JavaScript SDK. ### Configure the model The following example uses `qwen3.7-plus` through Alibaba Cloud: ```bash export MIDSCENE_MODEL_BASE_URL="https://dashscope.aliyuncs.com/compatible-mode/v1" export MIDSCENE_MODEL_API_KEY="your-api-key" export MIDSCENE_MODEL_NAME="qwen3.7-plus" export MIDSCENE_MODEL_FAMILY="qwen3" ``` Replace `your-api-key` with your API Key. > To use another model, such as Doubao, GLM, Gemini, or GPT-5, see [Supported models and setup](/model-common-config.md). For all configuration options, see [Model configuration](/model-config.md). ### Install dependencies <PackageManagerTabs command="install @midscene/computer" /> ### Write a script Create `example.ts`: ```typescript import { agentForComputer } from '@midscene/computer'; (async () => { // Create an agent const agent = await agentForComputer({ aiActionContext: 'You are controlling a desktop computer.', }); // Take a screenshot and query information const screenInfo = await agent.aiQuery( '{width: number, height: number}, get screen resolution' ); console.log('Screen resolution:', screenInfo); // Move mouse to center await agent.aiAct('move mouse to center of screen'); // Assert screen has content await agent.aiAssert('The screen has visible content'); console.log('Desktop automation completed!'); })(); ``` ### Run the script ```bash npx tsx example.ts ``` After the script finishes, you should see `Midscene - report file updated: /path/to/report/some_id.html` in the console. Open the generated HTML file in a browser to replay every interaction, query, and assertion. ## Custom actions Use `defineAction()` to define custom actions. When constructing the Agent with `agentForComputer()`, pass these actions through `customActions`. Midscene appends these actions to the planner so the Agent can call the domain-specific actions you define. ```typescript import { getMidsceneLocationSchema, z } from '@midscene/core'; import { defineAction } from '@midscene/core/device'; import { agentForComputer } from '@midscene/computer'; const ContinuousClick = defineAction({ name: 'continuousClick', description: 'Click the same target repeatedly', paramSchema: z.object({ locate: getMidsceneLocationSchema(), count: z.number().int().positive().describe('How many times to click'), }), async call(param) { console.log('click target center', param.locate.center); console.log('click count', param.count); // Carry out your clicking logic using locate + count. }, }); const agent = await agentForComputer({ customActions: [ContinuousClick], }); await agent.aiAct('click the red button five times'); ``` ## Connect to a Remote Windows Desktop via RDP `@midscene/computer` can also drive a remote Windows desktop directly over the RDP protocol through the dedicated `agentForRDPComputer()` factory. ### Prerequisites 1. A reachable Windows machine with RDP enabled. 2. [FreeRDP](https://www.freerdp.com/) installed on the machine running your script. ### Example ```typescript import { agentForRDPComputer } from '@midscene/computer'; const agent = await agentForRDPComputer({ aiActionContext: 'You are controlling a remote Windows desktop over the RDP protocol.', host: '10.75.166.249', port: 3389, username: 'Admin', password: 'replace-with-your-password', ignoreCertificate: true, }); await agent.aiWaitFor('The remote Windows desktop is visible'); await agent.aiAct('Click the Windows Start button'); await agent.aiAct('Open Settings'); await agent.aiAssert('The Windows Settings window is visible'); ``` ### Common RDP Options * `host`: Remote Windows host or IP. * `port`: RDP port. Defaults to `3389`. * `username` / `password`: Account credentials for the remote session. * `domain`: Optional Windows domain. * `ignoreCertificate`: Skip certificate validation for self-signed setups. * `desktopWidth` / `desktopHeight`: Request a specific remote desktop resolution. * `adminSession`: Request the remote admin session when the server allows it. RDP sessions are exposed to Midscene as a single remote display. You can still use the same `aiAct`, `aiQuery`, `aiAssert`, and report features as local desktop automation. ## Multi-Display Support If you have multiple displays, you can control a specific one: ```typescript import { ComputerDevice, agentForComputer } from '@midscene/computer'; // List all displays const displays = await ComputerDevice.listDisplays(); console.log('Available displays:', displays); // Connect to a specific display const agent = await agentForComputer({ displayId: displays[0].id, }); ``` ## Example Usage ### Basic Mouse Operations ```typescript // Click at center of screen await agent.aiAct('click mouse at center of screen'); // Move mouse to a specific location await agent.aiAct('move mouse to top-left corner'); // Double-click await agent.aiAct('double-click on the desktop icon'); // Right-click await agent.aiAct('right-click to open context menu'); ``` ### Keyboard Operations ```typescript // Type text await agent.aiAct('type "Hello World"'); // Press keyboard shortcuts if (process.platform === 'darwin') { await agent.aiAct('press Cmd+Space to open Spotlight'); await agent.aiAct('type "Calculator" and press Enter'); } else { await agent.aiAct('press Windows key'); await agent.aiAct('type "Calculator" and press Enter'); } // Press function keys await agent.aiAct('press Escape'); await agent.aiAct('press Enter'); ``` ### Query Information ```typescript // Extract screen information const info = await agent.aiQuery( '{hasDesktop: boolean, visibleApps: string[]}, check if desktop is visible and list visible apps' ); // Locate elements const position = await agent.aiLocate('the File menu'); console.log('File menu position:', position); ``` ### Complex Workflows ```typescript // Open an application and interact with it await agent.aiAct('open Finder'); await agent.aiWaitFor('Finder window is visible'); await agent.aiAct('click on Documents folder'); await agent.aiAct('press Cmd+N to create new folder'); await agent.aiAct('type "My Project"'); await agent.aiAct('press Enter'); await agent.aiAssert('A folder named "My Project" exists'); ``` ## Environment Check You can check if your system is properly configured: ```typescript import { checkComputerEnvironment } from '@midscene/computer'; const env = await checkComputerEnvironment(); console.log('Platform:', env.platform); console.log('Available:', env.available); console.log('Displays:', env.displays); if (!env.available) { console.error('Environment not available:', env.error); } ``` ## FAQ ### macOS: Script cannot control mouse or keyboard macOS requires Accessibility permissions for keyboard and mouse control. Go to **System Settings > Privacy & Security > Accessibility** and enable the toggle for the application running your script (e.g., Terminal, iTerm2, VS Code, or WebStorm). If you have already granted permission but it still doesn't work, try removing the app from the Accessibility list and re-adding it — macOS sometimes caches stale permissions. ### Windows: clicks have no effect on some apps If the cursor moves to the correct position but clicks or key presses do nothing on a particular application — while other apps work fine — check whether the target app is running **as Administrator** (elevated). Windows UIPI blocks input injected from a lower-privilege process into an elevated window and drops it silently, with no error. Prefer lowering the target application's privilege level first, for example by launching it without "Run as Administrator" or disabling any setting that always starts it elevated. If the target app must stay elevated, run the terminal or Node.js that launches Midscene **as Administrator** so it matches the target app's privilege level, then try again. System-level shortcuts such as `Win+Tab` are handled by the shell and keep working even when this happens, which is why keyboard shortcuts may appear to work while in-app clicks do not. > The health check logged at connection time prints this troubleshooting link when Midscene is not running as Administrator on Windows. ### Linux: Screenshots or interactions fail on a headless server A headless Linux environment (e.g. CI) has no physical display. You need to install Xvfb and ImageMagick, and enable headless mode: ```bash sudo apt-get install -y xvfb x11-xserver-utils imagemagick ``` ```typescript const agent = await agentForComputer({ headless: true }); ``` Or set the environment variable: ```bash MIDSCENE_COMPUTER_HEADLESS_LINUX=true npx tsx example.ts ``` ## More * [API Reference](/reference.md#desktop) * [Use YAML format automation scripts](/automate-with-scripts-in-yaml.md) * [YAML script runner](/yaml-script-runner.md) * [Caching for efficiency](/caching.md) * Demo projects * [JavaScript SDK demo](https://github.com/web-infra-dev/midscene-example/tree/main/computer/javascript-sdk-demo) * [Vitest demo](https://github.com/web-infra-dev/midscene-example/tree/main/computer/vitest-demo) * [Remote Windows desktop over RDP](https://github.com/web-infra-dev/midscene-example/tree/main/computer/rdp-demo) * [Obsidian on headless Linux CI](https://github.com/web-infra-dev/midscene-example/tree/main/computer/electron-demo) --- url: /platforms/harmonyos.md --- import { PackageManagerTabs } from '@theme'; # HarmonyOS Midscene connects to HarmonyOS NEXT devices through HarmonyOS Device Connector (HDC) to automate apps and system interfaces. This guide covers device connection, model configuration, Playground, and JavaScript SDK integration with `@midscene/harmony`. ## See it in action **Prompt:** Open Settings, find About phone, and view the device information. <video src="https://lf3-static.bytednsdoc.com/obj/eden-cn/nupipfups/Midscene/1.0-showcases/harmony.mp4" poster="https://lf3-static.bytednsdoc.com/obj/eden-cn/nupipfups/Midscene/1.0-showcases/harmony.png" height="300" controls /> View the [full report](https://lf3-static.bytednsdoc.com/obj/eden-cn/nupipfups/Midscene/1.0-showcases/harmony.html), or explore more [Midscene showcases](/showcases.md). ## Get started ### Prepare your HarmonyOS device Before writing scripts, verify that HDC can connect to your device and the device trusts the current computer. #### Install HDC HDC (HarmonyOS Device Connector) is a command-line tool provided by HarmonyOS for communicating with devices. Installation options: * Via [DevEco Studio](https://developer.huawei.com/consumer/en/deveco-studio/) (recommended) * Via [HarmonyOS command-line tools](https://developer.huawei.com/consumer/en/download/) standalone installation Verify HDC is installed: ```bash hdc version ``` A version number in the output confirms successful installation. :::info Configuring HDC Path If `hdc` is not in your system PATH, set the `HDC_HOME` environment variable to the directory containing HDC: ```bash export HDC_HOME=/path/to/hdc/directory ``` ::: #### Enable Developer Mode and verify the device In your HarmonyOS device settings, go to **Developer Options** and enable **USB Debugging**, then connect via USB cable. Verify the connection: ```bash hdc list targets ``` A device ID in the output confirms a successful connection: ```log 0123456789ABCDEF ``` ### Launch Playground Playground is the fastest way to validate the connection and try core capabilities such as `aiAct`, `aiQuery`, and `aiAssert` without writing code. It shares the same core as `@midscene/harmony`, so anything that works here will behave the same once scripted. 1. Launch the Playground CLI: ```bash npx --yes @midscene/harmony-playground ``` 2. Click the gear button in the Playground window and paste your API Key configuration. See [Supported models and setup](/model-common-config.md) if you still need a model configuration. ## Use the JavaScript SDK Once Playground runs successfully, you can switch to reusable JavaScript scripts. ### Configure the model The following example uses `qwen3.7-plus` through Alibaba Cloud: ```bash export MIDSCENE_MODEL_BASE_URL="https://dashscope.aliyuncs.com/compatible-mode/v1" export MIDSCENE_MODEL_API_KEY="your-api-key" export MIDSCENE_MODEL_NAME="qwen3.7-plus" export MIDSCENE_MODEL_FAMILY="qwen3" ``` Replace `your-api-key` with your API Key. > To use another model, such as Doubao, GLM, Gemini, or GPT-5, see [Supported models and setup](/model-common-config.md). For all configuration options, see [Model configuration](/model-config.md). ### Install dependencies <PackageManagerTabs command="install @midscene/harmony dotenv --save-dev" /> ### Write a script The following example opens the Settings app on the device and performs scrolling operations. ```typescript title="./demo.ts" import 'dotenv/config'; // load Midscene environment variables from .env if present import { HarmonyAgent, HarmonyDevice, getConnectedDevices, } from '@midscene/harmony'; const sleep = (ms) => new Promise((r) => setTimeout(r, ms)); Promise.resolve( (async () => { const devices = await getConnectedDevices(); const device = new HarmonyDevice(devices[0].deviceId, {}); const agent = new HarmonyAgent(device, { aiActionContext: 'This is a HarmonyOS device. The system language is Chinese. If any popup appears, dismiss or agree to it.', }); await device.connect(); // Open Settings app await agent.launch('com.huawei.hmos.settings'); await sleep(2000); // Scroll down await agent.aiAct('scroll down one screen'); // Query page content const items = await agent.aiQuery( 'string[], list all visible setting item names', ); console.log('Settings items', items); // Assert await agent.aiAssert('There is a settings item list on the page'); })(), ); ``` ### Run the script ```bash npx tsx demo.ts ``` After the script finishes, you should see `Midscene - report file updated: /path/to/report/some_id.html` in the console. Open the generated HTML file in a browser to replay every interaction, query, and assertion. ## Advanced Use this section to customize device behavior, integrate Midscene into a standalone framework, or troubleshoot HDC issues. See the [HarmonyOS section of the API reference](/reference.md#harmonyos) for more constructor parameters. ### Extending Midscene on HarmonyOS Use `defineAction()` to define custom actions. When constructing `HarmonyDevice`, pass these actions through `customActions`. Midscene appends these actions to the planner so the Agent can call the domain-specific actions you define. ```typescript import { getMidsceneLocationSchema, z } from '@midscene/core'; import { defineAction } from '@midscene/core/device'; import { HarmonyAgent, HarmonyDevice, getConnectedDevices } from '@midscene/harmony'; const ContinuousClick = defineAction({ name: 'continuousClick', description: 'Click the same target repeatedly', paramSchema: z.object({ locate: getMidsceneLocationSchema(), count: z.number().int().positive().describe('How many times to click'), }), async call(param) { const { locate, count } = param; console.log('click target center', locate.center); console.log('click count', count); }, }); const devices = await getConnectedDevices(); const device = new HarmonyDevice(devices[0].deviceId, { customActions: [ContinuousClick], }); await device.connect(); const agent = new HarmonyAgent(device); await agent.aiAct('click the red button five times'); ``` For more details on custom actions and action schemas, see [Integrate with Any Interface](/integrate-with-any-interface.md#define-a-custom-action). ## FAQ ### Keyboard is not dismissed or the page goes back after typing Midscene automatically dismisses the keyboard after entering text. By default, HarmonyOS uses the ESC key so the current page is less likely to navigate back. If ESC does not close the keyboard in your app, switch to Back first: ```typescript const device = new HarmonyDevice('device-id', { keyboardDismissStrategy: 'back-first', }); ``` If your input field listens for Back and clears or closes in response, disable auto keyboard dismiss: ```typescript const device = new HarmonyDevice('device-id', { autoDismissKeyboard: false, }); ``` With auto dismiss disabled, the keyboard will remain visible. You can use `aiAct` to manually dismiss it, e.g. `await agent.aiAct('dismiss the keyboard')`. ### How to use a custom HDC path? Set the `HDC_HOME` environment variable to point to the HDC directory: ```bash export HDC_HOME=/path/to/hdc/directory ``` Or pass it via the constructor: ```typescript const device = new HarmonyDevice('0123456789ABCDEF', { hdcPath: '/path/to/hdc', }); ``` ## More * View all Agent methods: [API Reference (Common)](/reference.md#interaction-methods) * HarmonyOS-specific parameters and interfaces: [API reference (HarmonyOS)](/reference.md#harmonyos) * Use [YAML automation scripts and command-line tools](/automate-with-scripts-in-yaml.md). * Demo projects * HarmonyOS JavaScript SDK demo: [https://github.com/web-infra-dev/midscene-example/blob/main/harmony/javascript-sdk-demo](https://github.com/web-infra-dev/midscene-example/blob/main/harmony/javascript-sdk-demo) * HarmonyOS + Vitest demo: [https://github.com/web-infra-dev/midscene-example/tree/main/harmony/vitest-demo](https://github.com/web-infra-dev/midscene-example/tree/main/harmony/vitest-demo) --- url: /platforms/index.md --- # More platforms Midscene supports mobile devices, HarmonyOS devices, and desktop applications in addition to Web browsers. Choose a platform based on the target interface and its connection method. Midscene uses a multimodal visual model to understand screenshots, so automation works with the rendered interface instead of depending on the underlying UI structure. The same approach applies to native apps and cross-platform technology stacks. ## Choose a platform | Platform | Package | Connection | Typical use | | --- | --- | --- | --- | | [Android](/platforms/android.md) | `@midscene/android` | adb | Android apps and system interfaces | | [iOS](/platforms/ios.md) | `@midscene/ios` | WebDriverAgent | iOS apps and system interfaces | | [HarmonyOS](/platforms/harmonyos.md) | `@midscene/harmony` | HDC | HarmonyOS NEXT apps and system interfaces | | [Desktop](/platforms/desktop.md) | `@midscene/computer` | Native input or RDP | Windows, macOS, and Linux applications | Each guide covers platform capabilities, environment preparation, Playground, JavaScript integration, examples, and troubleshooting. Use the [API reference](/reference.md) when you need constructor options, platform-specific methods, or shared Agent APIs. --- url: /platforms/ios.md --- import { PackageManagerTabs } from '@theme'; # iOS Midscene connects to iOS devices through WebDriverAgent to automate apps and system interfaces. This guide covers WebDriverAgent setup, model configuration, Playground, and JavaScript SDK integration with `@midscene/ios`. ## See it in action **Prompt:** Open Twitter and like the first post from `@midscene_ai`. <video src="https://lf3-static.bytednsdoc.com/obj/eden-cn/nupipfups/Midscene/1.0-showcases/x2.mp4" poster="https://lf3-static.bytednsdoc.com/obj/eden-cn/nupipfups/Midscene/1.0-showcases/x.png" height="300" controls /> View the [full report](https://lf3-static.bytednsdoc.com/obj/eden-cn/nupipfups/Midscene/1.0-showcases/x.html), or explore more [Midscene showcases](/showcases.md). ## Get started ### Prepare the iOS environment WebDriver is a W3C standard protocol for browser automation. It provides a common API for controlling browsers and applications. The protocol defines how clients and servers communicate, which allows automation tools to control interfaces across platforms. The Appium team and other open source communities maintain tools that expose desktop and mobile automation through WebDriver. These tools include: * **Appium**: a cross-platform mobile automation framework. * **WebDriverAgent**: a service for iOS device automation. * **Selenium**: a Web browser automation tool. * **WinAppDriver**: a Windows application automation tool. Midscene supports the WebDriver protocol. You can use AI models to automate any compatible device. Midscene can understand interface context, perform multistep operations, validate results, and extract data in addition to clicking and typing. On iOS, Midscene connects through WebDriverAgent. You can then use natural-language instructions to control iOS apps and system interfaces. Before continuing, make sure WebDriverAgent can communicate with the device. #### Install Node.js Install [Node.js 18 or higher](https://nodejs.org/en/download/). #### Set up WebDriverAgent Before getting started, you need to set up the iOS development environment: * macOS (required for iOS development) * Xcode and Xcode command line tools * iOS Simulator or real device **Configure WebDriverAgent** Before using Midscene iOS, you need to prepare the WebDriverAgent service. :::note Version Requirement WebDriverAgent version must be **>= 7.0.0** ::: Please refer to the official documentation for setup: * **Simulator Configuration**: [Run Prebuilt WDA](https://appium.github.io/appium-xcuitest-driver/latest/guides/run-prebuilt-wda/) * **Real Device Configuration**: [Real Device Configuration](https://appium.github.io/appium-xcuitest-driver/latest/getting-started/device-setup/) **Verify WebDriverAgent** After completing the configuration, you can verify whether the service is working properly by accessing WebDriverAgent's status endpoint: **Access URL**: `http://localhost:8100/status` **Correct Response Example**: ```json { "value": { "build": { "version": "10.1.1", "time": "Sep 24 2025 18:56:41", "productBundleIdentifier": "com.facebook.WebDriverAgentRunner" }, "os": { "testmanagerdVersion": 65535, "name": "iOS", "sdkVersion": "26.0", "version": "26.0" }, "device": "iphone", "ios": { "ip": "10.91.115.63" }, "message": "WebDriverAgent is ready to accept commands", "state": "success", "ready": true }, "sessionId": "BCAD9603-F714-447C-A9E6-07D58267966B" } ``` If you can successfully access this endpoint and receive a similar JSON response as shown above, it indicates that WebDriverAgent is properly configured and running. ### Launch Playground Playground is the fastest way to validate the connection and try core capabilities such as `aiAct`, `aiQuery`, and `aiAssert` without writing code. It shares the same core as `@midscene/ios`, so anything that works here will behave the same once scripted. 1. Launch the Playground CLI: ```bash npx --yes @midscene/ios-playground ``` 2. Click the gear button to enter the configuration page and paste your API Key configuration. See [Supported models and setup](/model-common-config.md) if you still need a model configuration. ## Use the JavaScript SDK Once Playground works, move to a repeatable script with the JavaScript SDK. ### Configure the model The following example uses `qwen3.7-plus` through Alibaba Cloud: ```bash export MIDSCENE_MODEL_BASE_URL="https://dashscope.aliyuncs.com/compatible-mode/v1" export MIDSCENE_MODEL_API_KEY="your-api-key" export MIDSCENE_MODEL_NAME="qwen3.7-plus" export MIDSCENE_MODEL_FAMILY="qwen3" ``` Replace `your-api-key` with your API Key. > To use another model, such as Doubao, GLM, Gemini, or GPT-5, see [Supported models and setup](/model-common-config.md). For all configuration options, see [Model configuration](/model-config.md). ### Install dependencies <PackageManagerTabs command="install @midscene/ios dotenv --save-dev" /> ### Write a script Save the following code as `./demo.ts`. It opens Safari on the device, searches eBay, and asserts the result list. ```typescript title="./demo.ts" import 'dotenv/config'; // load Midscene environment variables from .env if present import { IOSAgent, IOSDevice, agentFromWebDriverAgent, } from '@midscene/ios'; const sleep = (ms) => new Promise((r) => setTimeout(r, ms)); Promise.resolve( (async () => { // Method 1: Create device and agent directly const page = new IOSDevice({ wdaPort: 8100, wdaHost: 'localhost', }); // 👀 Initialize Midscene agent const agent = new IOSAgent(page, { aiActionContext: 'If any location, permission, user agreement, etc. popup appears, click agree. If login page appears, close it.', }); await page.connect(); // Method 2: Or use convenience function (recommended) // const agent = await agentFromWebDriverAgent({ // wdaPort: 8100, // wdaHost: 'localhost', // aiActionContext: 'If any location, permission, user agreement, etc. popup appears, click agree. If login page appears, close it.', // }); // 👀 Directly open ebay.com webpage (recommended approach) await page.launch('https://ebay.com'); await sleep(3000); // 👀 Enter keywords and perform search await agent.aiAct('Search for "Headphones"'); // 👀 Wait for loading to complete await agent.aiWaitFor('At least one headphone product is displayed on the page'); // Or you can use a simple sleep: // await sleep(5000); // 👀 Understand page content and extract data const items = await agent.aiQuery( '{itemTitle: string, price: Number}[], find product titles and prices in the list', ); console.log('Headphone product information', items); // 👀 Use AI assertion await agent.aiAssert('Multiple headphone products are displayed on the interface'); await page.destroy(); })(), ); ``` ### Run the script ```bash npx tsx demo.ts ``` After the script finishes, you should see `Midscene - report file updated: /path/to/report/some_id.html` in the console. Open the generated HTML file in a browser to replay every interaction, query, and assertion. ## Keyboard dismissal `autoDismissKeyboard` defaults to `true`. After entering text, Midscene locates the standard iOS keyboard accessory toolbar, activates its dismissal control, and waits until WebDriverAgent confirms that the keyboard is hidden. If the toolbar cannot be identified safely or the keyboard does not disappear before the operation deadline, the input action fails instead of continuing with an uncertain keyboard state. When an `Enter`, `Return`, or `Tab` action immediately follows an auto-dismissed input, Midscene can restore that input's focus once before sending the key. A different pointer, touch, system, custom, or text-input action invalidates the pending focus target. :::warning An app-defined input accessory can contain business actions such as Save or Submit. Midscene deliberately rejects accessory toolbars that do not match the standard iOS navigation-and-dismiss structure. For a custom toolbar, disable automatic dismissal and call `hideKeyboard()` with the control's stable accessibility name. ::: ```typescript const device = new IOSDevice({ autoDismissKeyboard: false, }); // After entering text, dismiss a custom keyboard control explicitly. await device.hideKeyboard(['Close Keyboard']); ``` ## Custom actions Use `defineAction()` to define custom actions. When constructing the Agent with `agentFromWebDriverAgent()`, pass these actions through `customActions`. Midscene appends these actions to the planner so the Agent can call the domain-specific actions you define. ```typescript import { getMidsceneLocationSchema, z } from '@midscene/core'; import { defineAction } from '@midscene/core/device'; import { agentFromWebDriverAgent } from '@midscene/ios'; const ContinuousClick = defineAction({ name: 'continuousClick', description: 'Click the same target repeatedly', paramSchema: z.object({ locate: getMidsceneLocationSchema(), count: z.number().int().positive().describe('How many times to click'), }), async call(param) { console.log('click target center', param.locate.center); console.log('click count', param.count); // Carry out your clicking logic using locate + count. }, }); const agent = await agentFromWebDriverAgent({ customActions: [ContinuousClick], }); await agent.aiAct('click the red button five times'); ``` ## API reference and more resources For constructors, helper methods, and platform-specific device APIs, see the [iOS section of the API reference](/reference.md#ios). It includes detailed parameter lists and advanced topics such as custom actions. For APIs shared across platforms, see the [Common section](/reference.md#common). ## FAQ ### Why can't I control my device through WebDriverAgent even though it's connected? Please check the following: 1. **Developer Mode**: Ensure it's enabled in Settings > Privacy & Security > Developer Mode 2. **UI Automation**: Ensure it's enabled in Settings > Developer > UI Automation 3. **Device Trust**: Ensure the device trusts the current Mac ### What are the differences between simulators and real devices? | Feature | Real Device | Simulator | |---------|-------------|-----------| | Port Forwarding | Requires iproxy | Not required | | Developer Mode | Must enable | Auto-enabled | | UI Automation Settings | Must enable manually | Auto-enabled | | Performance | Real device performance | Depends on Mac performance | | Sensors | Real hardware | Simulated data | ### How to use custom WebDriverAgent port and host? You can specify WebDriverAgent port and host through the `IOSDevice` constructor or `agentFromWebDriverAgent`: ```typescript // Method 1: Using IOSDevice const device = new IOSDevice({ wdaPort: 8100, // Custom port wdaHost: '192.168.1.100', // Custom host }); // Method 2: Using convenience function (recommended) const agent = await agentFromWebDriverAgent({ wdaPort: 8100, // Custom port wdaHost: '192.168.1.100', // Custom host }); ``` For remote devices, you also need to set up port forwarding accordingly: ```bash iproxy 8100 8100 YOUR_DEVICE_ID ``` ### How to get smoother live screen preview in Playground? Playground's screen preview supports two modes: * **Polling mode** (default): Captures screenshots one by one via the WDA screenshot API, achieving ~5-10fps. * **Native MJPEG stream** (recommended): Proxies WDA's built-in MJPEG Server directly for higher frame rate and lower latency. To enable the native MJPEG stream, forward the WDA MJPEG Server port (default 9100) to localhost: ```bash # Required for real devices only (simulators don't need this) iproxy 9100 9100 YOUR_DEVICE_ID ``` Playground automatically probes port 9100 on startup. If available, the log will show `MJPEG: streaming via native WDA MJPEG server`; otherwise it falls back to polling mode automatically. ## More * For every Agent method, check the [API reference (Common)](/reference.md#interaction-methods). * For iOS-specific APIs, see [API reference (iOS)](/reference.md#ios). * Use [YAML automation scripts and command-line tools](/automate-with-scripts-in-yaml.md). * Demo projects * iOS JavaScript SDK demo: [https://github.com/web-infra-dev/midscene-example/blob/main/ios/javascript-sdk-demo](https://github.com/web-infra-dev/midscene-example/blob/main/ios/javascript-sdk-demo) * iOS + Vitest demo: [https://github.com/web-infra-dev/midscene-example/tree/main/ios/vitest-demo](https://github.com/web-infra-dev/midscene-example/tree/main/ios/vitest-demo) --- url: /quick-start.md --- import { ChromeExtensionButton } from '@theme'; import SetupEnv from './common/setup-env.mdx'; import ShowcaseWeb from './showcases-web.mdx'; # Quick Start The Chrome extension is Midscene's Playground for the web. Without setting up a project, you can try its core interaction, data extraction, and interface-checking capabilities on web pages. This guide walks you through configuring a model, installing the Chrome extension, and running your first natural-language instruction. After validating an instruction, you can integrate it into automation code through the Agent APIs. At the end, you will also find getting-started guides for Android, iOS, HarmonyOS, and desktop platforms. ## Configure a model Before using the Chrome extension, prepare a multimodal model with UI localization capabilities. <SetupEnv /> After installing the Chrome extension, paste this configuration into its settings. ## Install the Chrome extension {#chrome-extension} 1. Use the button below to install Midscene from the Chrome Web Store: <ChromeExtensionButton linkLabel="Install the Midscene extension from the Chrome Web Store" /> 2. Open **Midscene** from the Chrome extensions list. The Midscene sidebar appears on the right side of the browser. 3. Click the settings icon in the sidebar, paste the complete configuration from [Configure a model](#configure-a-model), and save it. <span id="chrome-extension-faq" /> **FAQ** <details> <summary>Can I install the Chrome extension manually?</summary> If you cannot access the Chrome Web Store, download the installation package from the [GitHub Releases page](https://github.com/web-infra-dev/midscene/releases) and install it manually. Manual installations do not receive automatic updates. </details> <details> <summary>It fails with `Cannot access a chrome-extension:// URL of different extension`</summary> This error usually indicates a conflict between Midscene and another Chrome extension. For example, another Chrome extension may have injected an `<iframe />` or `<script />` into the page. Follow these steps to find the conflicting Chrome extension: 1. Open the page's developer tools, find an `<iframe />` or `<script />` whose URL starts with `chrome-extension://`, and copy the extension ID from the URL. 2. Open `chrome://extensions/`, find the Chrome extension by its ID, and disable it. 3. Refresh the page and try again. </details> <details> <summary>I get a 403 error when using an Ollama model</summary> Set the `OLLAMA_ORIGINS="*"` environment variable to allow the Chrome extension to access the Ollama model. </details> ## Complete your first task Open any web page and enter a natural-language instruction in the Midscene sidebar that matches the current page. For example: - Plan and interact (`aiAct`): `Click the login button`. - Extract structured data (`aiQuery`): `Products on the page, {name: string, price: number}[]`. - Check the interface (`aiAssert`): `A navigation bar appears at the top of the page`. When you run the instruction, Midscene understands the current page and either performs the action or returns the result. The following example shows the Chrome extension filling out a GitHub sign-up form: <ShowcaseWeb /> ## Integrate Playground instructions into code The Chrome extension shares its core capabilities with `@midscene/web`. After validating natural-language instructions in the Playground, use the corresponding Agent APIs to integrate them into UI test scripts: ```typescript // Plan and perform interactions await agent.aiAct('Click the login button'); // Extract structured data const products = await agent.aiQuery<Array<{ name: string; price: number }>>( 'Products on the page, {name: string, price: number}[]', ); // Check the interface await agent.aiAssert('A navigation bar appears at the top of the page'); ``` This example only shows the API call pattern. To create an Agent and run a complete browser script, continue with [Integrate with Playwright](./integrate-with-playwright) or [Integrate with Puppeteer](./integrate-with-puppeteer). To learn when to use each API category, read [The Basics](./basics). For all options, see the [API reference](./reference/#common). ## Use Midscene on other platforms Midscene provides the same complete automation capabilities on Android, iOS, HarmonyOS, and desktop, with a dedicated Playground for each platform. Before using a Playground, prepare the device environment for that platform. For example, Android requires adb to be installed and configured. See the following guides for platform requirements, Playground launch instructions, and troubleshooting. | Platform | Platform guide | | --- | --- | | Android | [Android](./platforms/android) | | iOS | [iOS](./platforms/ios) | | HarmonyOS | [HarmonyOS](./platforms/harmonyos) | | Desktop | [Windows, macOS, and Linux](./platforms/desktop) | --- url: /reference/index.md --- # API reference Use this page to look up shared Agent APIs and platform-specific constructors, options, actions, and helper methods. This page documents API contracts. For installation, end-to-end workflows, and troubleshooting, follow the linked guides. Platform Agents inherit the [Shared Agent APIs](#common) unless a platform section documents a difference. This page retains a small number of complete examples to show how related APIs work together. For end-to-end integration and best practices, follow the guide links in each platform section. | Area | Contents | | --- | --- | | [Shared Agent APIs](#common) | Agent options, interaction, extraction, observation, workflow, reporting, shared types, and report utilities | | [Web](#web) | Puppeteer, Playwright, and Chrome Bridge APIs | | [Android](#android) | Android device, Agent, factory, and utility APIs | | [iOS](#ios) | iOS device, Agent, factory, and utility APIs | | [HarmonyOS](#harmonyos) | HarmonyOS device, Agent, factory, and utility APIs | | [Desktop](#desktop) | Local desktop and RDP APIs | | [Runtime configuration](#runtime-configuration) | Global environment variables for run artifacts, language, Playground networking, and Debug logs | ## Shared Agent APIs {#common} <a id="constructors"></a> ### Agent construction and options {#agent-options} Midscene provides agents for different automation environments. Each constructor accepts a target page or device and the shared options described below, including reporting, caching, AI configuration, and hooks. Platform-specific sections document additional options such as browser navigation controls and Android `adb` settings: - In Puppeteer, use [PuppeteerAgent](#puppeteer-agent) - In Playwright, use [PlaywrightAgent](#playwright-agent) - In Bridge mode, use [AgentOverChromeBridge](#chrome-bridge-agent) - On Android, use [Android API reference](#android) - On iOS, use [iOS API reference](#ios) - For GUI agents integrating with your own interface, refer to [Custom Interface Agent](../integrate-with-any-interface) <a id="common-parameters"></a> **Parameters** All agents share these base options: - `generateReport: boolean` — Whether Midscene generates a report file. Default: `true`. - `persistExecutionDump: boolean` — Whether Midscene writes a JSON dump for each execution alongside the report. Default: `false`. This option requires `generateReport` to be `true`. - `reportFileName: string` — Report output name. By default, Midscene generates the name. Its exact meaning depends on `outputFormat`: - `single-html` (default) — Midscene treats the value as a file name and writes `<reportFileName>.html` under `midscene_run/report/`. If the name already ends in `.html`, Midscene preserves it. - `html-and-external-assets` — Midscene treats the value as a directory name and writes `index.html` and related assets under `midscene_run/report/<reportFileName>/`. - `autoPrintReportMsg: boolean` — Whether Midscene prints report messages. Default: `true`. - `cache?: false | { id: string; strategy?: 'read-only' | 'read-write' | 'write-only'; cacheDir?: string }` — Cache configuration: - `false` — Disable the cache. - `id` — Required cache ID. - `strategy` — Cache strategy. Default: `'read-write'`. - `cacheDir` — Cache directory. When set, Midscene writes cache files here instead of `<MIDSCENE_RUN_DIR>/cache`. Relative paths are resolved from the current working directory, not from `MIDSCENE_RUN_DIR`. This lets you use separate cache, log, and report directories. - `cacheId: string | undefined` (deprecated) — Legacy cache ID for backward compatibility. Prefer `cache.id`. - `aiActContext: string` — Background context sent to the AI model with `agent.aiAct()` calls. For example: `'Close the cookie consent dialog first if it exists.'` Default: `undefined`. This option was previously named `aiActionContext`; the legacy name remains supported. - `modelConfig: Record<string, string | number>` — Model configuration for this Agent. When provided, it replaces model-related system environment variables for this Agent. See the detailed configuration section below. - `replanningCycleLimit: number` — Maximum number of `aiAct` replanning cycles. Default: `20` for standard models, `40` for UI-TARS models, and `100` for AutoGLM models. Prefer this Agent option; `MIDSCENE_REPLANNING_CYCLE_LIMIT` remains available only for backward compatibility. - `waitAfterAction: number` — Delay in milliseconds after each action. This gives the UI time to settle before the next action. Default: `300`. - `useDeviceTime: boolean` — Whether task timestamps use the target device's local time. The target interface must implement `getDeviceLocalTimeString`; otherwise, Midscene logs a warning and uses the runtime system time. Default: `false`. - `onTaskStartTip: (tip: string) => void | Promise<void>` — Optional hook called before each task begins. It receives a human-readable task summary. Default: `undefined`. - `createOpenAIClient: (openai, options) => Promise<OpenAI | undefined>` — Optional factory for wrapping the OpenAI client with observability or custom middleware. See the detailed example below. - `onLLMUsage: (usage: AIUsageInfo) => void` — Optional callback invoked once for each LLM call when usage data becomes available. Use it for real-time cost and usage tracking. - `outputFormat: 'single-html' | 'html-and-external-assets'` — Report output format. `'single-html'` embeds all screenshots as base64 in one HTML file and uses `reportFileName` as the file name. `'html-and-external-assets'` saves screenshots as separate PNG files and uses `reportFileName` as the output directory. Default: `'single-html'`. Reports created with `'html-and-external-assets'` must be served over HTTP; they cannot be opened through the `file://` protocol because browsers block the relative asset requests. For local testing, open the report directory and run one of these commands: - Using Node.js: `npx serve` - Using Python: `python -m http.server` or `python3 -m http.server` Then access the report via `http://localhost:3000` (or the port shown in the terminal). - `screenshotShrinkFactor: number` — Factor used to reduce screenshot dimensions before sending them to the AI model. Default: `1`, which preserves the original size. A value of `2` halves both dimensions and reduces the image area to one quarter. Choose a value that balances image clarity and token usage. - On mobile devices, `2` often reduces token usage while preserving enough detail. Values above `3` may make screenshots too blurry for reliable model interpretation. - For browser automation, prefer Puppeteer or Playwright's `deviceScaleFactor` when possible. Keep `screenshotShrinkFactor` low for highly detailed pages. :::info **Difference between `screenshotShrinkFactor` and `deviceScaleFactor`:** - `screenshotShrinkFactor` is a Midscene option that reduces a screenshot after capture. It can lower token usage and model latency, but excessive reduction can obscure important details. - `deviceScaleFactor` is a Puppeteer and Playwright option that controls how many physical pixels render each CSS pixel. It therefore affects the original screenshot dimensions. A mismatch with the actual display scale can cause flickering in headed browsers. **Can they be used together?** - For browser automation, using both options usually provides little benefit. Prefer `deviceScaleFactor` to control the original screenshot size. - One exception: when you configure `deviceScaleFactor` to avoid browser flickering, but still do not want to send an oversized screenshot to the model. In that case, you can also use `screenshotShrinkFactor` to compress the image before model consumption. - On mobile and other non-web platforms, `deviceScaleFactor` is unavailable. Use `screenshotShrinkFactor` to reduce the screenshot sent to the model. ::: The device CLIs expose the same Agent behavior options for individual calls. Convert camelCase option names to kebab-case flags; for example, `waitAfterAction` becomes `--wait-after-action`. See [Skills](../skills) for platform-specific CLI entry points. **Custom model configuration** Use `modelConfig: Record<string, string | number>` to configure models directly in code instead of environment variables. > If `modelConfig` is provided at agent initialization, **all system environment variables for model config are ignored**. Only the values in this object are used. > The supported keys and values are listed in [Model configuration](../model-config). See [Model strategy](../model-strategy) to understand the Default, Planning, and Insight model roles. **Custom OpenAI client** `createOpenAIClient: (openai, options) => Promise<OpenAI | undefined>` lets you wrap the OpenAI client to integrate observability tools such as LangSmith and Langfuse, or to apply custom middleware. **Parameters** - `openai: OpenAI` — Base OpenAI client created by Midscene with the configured API Key, base URL, proxy, and other settings. - `options: Record<string, unknown>` — OpenAI initialization options, including: - `baseURL?: string` — API endpoint URL. - `apiKey?: string` — API Key. - `dangerouslyAllowBrowser: boolean` — Always `true` in Midscene. - Other OpenAI initialization options. **Return value** - Returns the wrapped OpenAI client, or `undefined` to use the original client. <a id="interaction-methods"></a> ### Planning and interaction {#planning-interaction} Below are the main APIs available for the various Agents in Midscene. `agent.ai()` plans and executes a sequence of actions, while instant-action methods perform a specified action after locating its target. <a id="agentaiact-or-agentai"></a> #### `aiAct()` or `ai()` {#agentaiact} This method accepts UI goals and assertions described in natural language. During execution, Midscene continuously uses AI to plan and act from the latest interface state until the goal is complete. If an assertion in the prompt fails, `aiAct` throws an error promptly. :::info Backward compatibility This method was previously named `aiAction()` in earlier versions. The current version supports both names for backward compatibility. We recommend using the new `aiAct()` method for code consistency. ::: - Type ```typescript function aiAct( prompt: string | object, options?: { cacheable?: boolean; deepThink?: 'unset' | true | false; deepLocate?: boolean; fileChooserAccept?: string | string[]; fileChooserAllowedDir?: string; abortSignal?: AbortSignal; context?: string; }, ): Promise<string | undefined>; function ai(prompt: string, options?: Object): Promise<string | undefined>; // shorthand form ``` - Parameters: - `prompt: string | object` — A natural language description of the UI goal and any optional assertions, or [prompting with images](#prompting-with-images). - `options?: object` — Optional configuration: - `cacheable?: boolean` — Whether this call can use the [cache](../caching.mdx). Default: `true`. - `deepThink?: 'unset' | true | false` — Controls the planning implementation used by Midscene when `aiAct` performs planning. See [`deepThink` planning mode](#aiact-deepthink). - `deepLocate?: boolean` — Whether to enable [Deep Locate](#deep-locate-deeplocate). Default: `false`. - `context?: string` — Additional context for this call. See [Per-call context](#per-call-context). - `fileChooserAccept?: string | string[]` — When a file chooser pops up, specify the file path(s) to accept. Can be a single file path or an array of paths. Only available in web pages (Playwright, Puppeteer, or Chrome extension Bridge mode). This option is not constrained by `fileChooserAllowedDir`. - **Note**: If the file input does not support multiple files (no `multiple` attribute) but multiple files are provided, an error will be thrown. - **Note**: If a file chooser is triggered but no `fileChooserAccept` parameter is provided, the file chooser will be ignored and the page can continue to operate normally. - **Note**: In Chrome extension Bridge mode, local file uploads require the Midscene extension's "Allow access to file URLs" permission. Enable it in `chrome://extensions` > Midscene > "Details", then reconnect Bridge mode from the target `http(s)://` page. - **Note**: Chrome extension Bridge mode does not support directory upload inputs (`webkitdirectory` / `directory`). Use Playwright for directory uploads. - `fileChooserAllowedDir?: string` — Explicitly authorizes the directory this `aiAct` call may access for prompt-driven file uploads. Without it, model-planned file uploads are rejected. Relative paths in the prompt are resolved against this directory and then validated; absolute paths are validated directly against this directory. Symlinks located within this directory are also allowed. We recommend using the test case's `fixtures` directory to reduce the risk of sensitive-file uploads caused by AI hallucinations or page prompt injection. - `abortSignal?: AbortSignal` — Signal used to cancel the `aiAct` call. When aborted, Midscene stops the planning loop and throws an error. Use it to implement timeouts or user-initiated cancellation. - Return value: - Returns the output text produced by the completed plan, or `undefined` when the plan does not produce output. If execution fails, an error is thrown. - Examples: ```typescript // Perform actions and verify the result await agent.aiAct( 'Search for headphones, add the first item to the cart, and confirm that the cart count changes to 1', ); // Using the shorthand .ai form await agent.ai( 'Click the login button at the top of the page, then enter "test@example.com" in the username field', ); // Using abortSignal to set a timeout const controller = new AbortController(); setTimeout(() => controller.abort('timeout'), 30000); // 30s timeout await agent.aiAct('Fill in the form and submit', { abortSignal: controller.signal, }); // For complex tasks, you can enable the deepThink parameter await agent.aiAct('Complete the GitHub account registration form. The region must be set to "Canada". Make sure no fields on the form are missed and all form fields pass validation. Just fill in the form fields without actually submitting the registration. Finally, return the actual content filled in the form fields', { deepThink: true }); ``` ##### `deepThink` planning mode {#aiact-deepthink} `deepThink` controls the planning implementation used by `aiAct`: - By default, `aiAct` plans the next step and locates the target element in the same planning request. - When set to `true`, `aiAct` focuses more on task decomposition and separates task planning from element localization into different model calls. This can improve stability for complex tasks, but it also increases model calls and latency. `deepThink` accepts `'unset' | true | false`. The legacy `'unset'` value behaves the same as `false`. `deepThink` does not control model-native thinking. See [Model-native thinking](../model-config#model-native-reasoning) for the related environment variables. ##### Upload files from an `aiAct` prompt To upload files mentioned in an `aiAct` prompt, explicitly pass `fileChooserAllowedDir` for that call. We recommend using the test case's `fixtures` directory. Midscene resolves both relative and absolute paths before validating that they stay within the selected directory. ```typescript const agent = new PlaywrightAgent(page); await agent.aiAct( 'First click the "Upload avatar" button and upload avatar.png. Then click the "Upload cover" button and upload cover.png. After uploading, confirm that the avatar displays a cat and the cover displays a beach.', { fileChooserAllowedDir: './fixtures' }, ); ``` For multiple uploads in one `aiAct`, name each relative path together with the action that should upload it. A later path replaces the previously configured file chooser path(s). Do not combine prompt-specified paths with `options.fileChooserAccept`: a later file chooser configuration generated during planning can override the option value. This capability is available for web pages (Playwright, Puppeteer, and Chrome extension Bridge mode). Use `fileChooserAllowedDir` when fixtures live outside the current working directory or when the same prompt must work in local and CI environments. When using Chrome extension Bridge mode for local file uploads, enable the Midscene extension's "Allow access to file URLs" permission in `chrome://extensions` > Midscene > "Details", then reconnect Bridge mode from the target `http(s)://` page. :::info Midscene uses an AI model to split the instruction into a sequence of steps, then executes those steps in order. If it cannot complete an action, it throws an error. For optimal results, please provide clear and detailed instructions for `agent.aiAct()`. Related documentation: - [Model strategy](../model-strategy) ::: #### `aiTap()` {#agentaitap} Tap an element. - Type ```typescript function aiTap(locate: string | object, options?: object): Promise<void>; ``` - Parameters: - `locate: string | object` — A natural language description of the element to tap, or [prompting with images](#prompting-with-images). - `options?: object` — Optional configuration: - `deepLocate?: boolean` — Whether to enable [Deep Locate](#deep-locate-deeplocate). The deprecated `deepThink` name remains supported. Default: `false`. - `context?: string` — Additional context for this call. See [Per-call context](#per-call-context). - `xpath?: string` — XPath for the target element. Midscene tries the XPath before consulting the cache or AI model. Default: `undefined`. - `cacheable?: boolean` — Whether this call can use the [cache](../caching.mdx). Default: `true`. - `fileChooserAccept?: string | string[]` — When a file chooser pops up, specify the file path(s) to accept. Can be a single file path or an array of paths. Only available in web pages (Playwright, Puppeteer, or Chrome extension Bridge mode). - **Note**: If the file input does not support multiple files (no `multiple` attribute) but multiple files are provided, an error will be thrown. - **Note**: If a file chooser is triggered but no `fileChooserAccept` parameter is provided, the file chooser will be ignored and the page can continue to operate normally. - **Note**: Chrome extension Bridge mode does not support directory upload inputs (`webkitdirectory` / `directory`). Use Playwright for directory uploads. - Return value: - `Promise<void>`. - Examples: ```typescript await agent.aiTap('The login button at the top of the page'); // Use deepLocate feature to precisely locate the element await agent.aiTap('The login button at the top of the page', { deepLocate: true, }); // File upload: tap the upload button and select files await agent.aiTap('Choose file button', { fileChooserAccept: ['./document.pdf'] }); await agent.aiTap('Upload images', { fileChooserAccept: ['./image1.jpg', './image2.png'] }); ``` #### `aiHover()` {#agentaihover} > Available on web pages and desktop (`@midscene/computer`). Not available on mobile devices (Android, iOS, or HarmonyOS). Move the pointer over an element. - Type ```typescript function aiHover(locate: string | object, options?: object): Promise<void>; ``` - Parameters: - `locate: string | object` — A natural language description of the element to hover over, or [prompting with images](#prompting-with-images). - `options?: object` — Optional configuration: - `deepLocate?: boolean` — Whether to enable [Deep Locate](#deep-locate-deeplocate). The deprecated `deepThink` name remains supported. Default: `false`. - `context?: string` — Additional context for this call. See [Per-call context](#per-call-context). - `xpath?: string` — XPath for the target element. Midscene tries the XPath before consulting the cache or AI model. Default: `undefined`. - `cacheable?: boolean` — Whether this call can use the [cache](../caching.mdx). Default: `true`. - Return value: - `Promise<void>`. - Examples: ```typescript await agent.aiHover('The version number of the current page'); ``` #### `aiInput()` {#agentaiinput} Enter text into an input field. - Type ```typescript // Recommended: locate first, then options with value function aiInput( locate: string | object, opt: { value: string | number; deepLocate?: boolean; xpath?: string; cacheable?: boolean; autoDismissKeyboard?: boolean; keyboardTypeDelay?: number; inputStrategy?: 'legacy' | 'sequential' | 'bulk'; mode?: 'replace' | 'clear' | 'typeOnly'; }, ): Promise<void>; // Backward compatible (legacy) function aiInput( value: string | number, locate: string | object, options?: object, ): Promise<void>; ``` - Parameters: **Recommended usage:** - `locate: string | object` — A natural language description of the target input field, or [prompting with images](#prompting-with-images). - `opt: object` — Configuration: - `value: string | number` — **Required.** Text to enter. - When `mode` is `'replace'`: The text will replace all existing content in the input field. - When `mode` is `'typeOnly'`: The text will be typed directly without clearing the field first. - When `mode` is `'clear'`: The text is ignored and the input field will be cleared. - `deepLocate?: boolean` — Whether to enable [Deep Locate](#deep-locate-deeplocate). The deprecated `deepThink` name remains supported. Default: `false`. - `context?: string` — Additional context for this call. See [Per-call context](#per-call-context). - `xpath?: string` — XPath for the target element. Midscene tries the XPath before consulting the cache or AI model. Default: `undefined`. - `cacheable?: boolean` — Whether this call can use the [cache](../caching.mdx). Default: `true`. - `autoDismissKeyboard?: boolean` — Whether to dismiss the on-screen keyboard after entering text. Available only on Android, iOS, and HarmonyOS. Default: `true`. - `keyboardTypeDelay?: number` — Finite non-negative delay in milliseconds between keystrokes. Whether this option affects `legacy` input depends on the platform's existing input path; see the table below. A `legacy` path that supports delay enters one Unicode code point at a time when the value is greater than `0`. Use this option when an input field drops characters during fast input. - `inputStrategy?: 'legacy' | 'sequential' | 'bulk'` — How Midscene sends the text to the platform. Default: `'legacy'`. - `'legacy'`: Preserve the platform's existing behavior. This is the compatibility-safe default, so existing tests do not change after upgrading. - `'sequential'`: Send one Unicode code point at a time, including when `keyboardTypeDelay` is omitted or `0`. - `'bulk'`: Send the complete text in one platform input operation where the platform supports it. On Web, `replace` selects the existing value and uses `insertText`, producing one insertion `input` event rather than first emitting a clearing input event and then per-character input events. Other platforms make one call from Midscene to their input backend, but the OS or driver may still synthesize multiple application events. - `'bulk'` requires `keyboardTypeDelay` to be omitted or set to `0`. A value greater than `0` causes Midscene to throw an error; negative and non-finite values are invalid for every strategy. An action-level value overrides the Device or Agent default, so use `keyboardTypeDelay: 0` to override a positive default when selecting `'bulk'` for one action. - `mode?: 'replace' | 'clear' | 'typeOnly'` — Input mode. Default: `'replace'`. - `'replace'`: Clear the input field first, then input the text. - `'typeOnly'`: Type the value directly without clearing the field first. - `'clear'`: Clear the input field without entering new text. The default value of `inputStrategy` is `'legacy'` on every platform. This mode preserves Midscene's platform-specific input logic from earlier versions, so its behavior can vary by platform: | Platform | `legacy` behavior | | --- | --- | | Web / Playwright / Puppeteer | In `replace` mode, clear the existing value and wait for the DOM to settle, then pass the complete value to `keyboard.type(value)`. A positive delay is handled by the browser driver. | | Chrome Extension / Bridge | Keep the Web clear-then-type flow and send text through the remote keyboard primitive. Legacy CDP typing keeps its existing zero-delay behavior; Bridge resolves the strategy on the CLI side before forwarding the primitive. | | Android | Non-ASCII or shell-sensitive text normally uses one yadb call and preserves the historical behavior of ignoring delay. The native `input text` path sends a complete segment by default and switches to one call per Unicode code point when delay is greater than `0`. | | iOS | Use one WDA `typeText` call by default; when delay is greater than `0`, use `typeRawKeys` once per Unicode code point. | | HarmonyOS | Use one HDC `inputText` call by default; when delay is greater than `0`, call it once per Unicode code point. | | Local Computer | Use clipboard paste when delay is omitted or `0`; when delay is greater than `0`, use real keyboard events once per Unicode code point. | | RDP Computer | Use one backend `typeText` call when delay is omitted or `0`; when delay is greater than `0`, call the backend once per Unicode code point. | **Backward-compatible usage (deprecated but still supported):** - `value: string | number` — The text content to input. - `locate: string | object` — A natural language description of the element, or [prompting with images](#prompting-with-images). - `options?: object` — Optional configuration. The type is the same as the `opt` type in the recommended usage. - Return value: - `Promise<void>`. - Examples: ```typescript // Recommended await agent.aiInput('The search input box', { value: 'Hello World' }); // Backward compatible (not recommended) await agent.aiInput('Hello World', 'The search input box'); ``` :::note Signature update The recommended signature places the locate prompt first. The legacy signature `aiInput(value, locate, options)` remains supported, but new code should use the recommended signature. ::: #### `aiClearInput()` {#agentaiclearinput} Clear the content of an input field. Useful as a standalone step before typing, or when you need to remove existing text without immediately inputting a new value. - Type ```typescript function aiClearInput( locate: string | object, opt?: { deepLocate?: boolean; xpath?: string; cacheable?: boolean; }, ): Promise<void>; ``` - Parameters: - `locate: string | object` — A natural language description of the input field to clear, or [prompting with images](#prompting-with-images). - `opt?: object` — Optional configuration: - `deepLocate?: boolean` — Whether to enable [Deep Locate](#deep-locate-deeplocate). Default: `false`. - `context?: string` — Additional context for this call. See [Per-call context](#per-call-context). - `xpath?: string` — XPath for the target element. Default: `undefined`. - `cacheable?: boolean` — Whether this call can use the [cache](../caching.mdx). Default: `true`. - Return value: - `Promise<void>`. - Examples: ```typescript // Clear the search box await agent.aiClearInput('the search input field'); // Clear and then type a new value await agent.aiClearInput('the email input'); await agent.aiInput('the email input', { value: 'user@example.com' }); ``` :::info When to use `aiClearInput` vs `aiInput` `aiInput(locate, { value: '...' })` already clears the field by default (via `mode: 'replace'`). Reach for `aiClearInput` when you need clearing as an independent step — for example, to test empty-state validation, or when you want to control clearing and typing as separate actions. ::: #### `aiKeyboardPress()` {#agentaikeyboardpress} Press a keyboard key. - Type ```typescript // Recommended: locate first, then options with keyName function aiKeyboardPress( locate: string | object | undefined, opt: { keyName: string; deepLocate?: boolean; xpath?: string; cacheable?: boolean; }, ): Promise<void>; // Backward compatible (legacy) function aiKeyboardPress( key: string, locate?: string | object, options?: object, ): Promise<void>; ``` - Parameters: **Recommended usage:** - `locate: string | object | undefined` — An optional natural language description of the element to press the key on, or [prompting with images](#prompting-with-images). Pass `undefined` to press the key against the currently focused element without AI location or a preliminary click. - `opt: object` — Configuration: - `keyName: string` — **Required.** Keyboard key to press, such as `'Enter'`, `'Tab'`, or `'Escape'`. Use `aiInput()` to enter text. Web and Computer devices support modifier shortcuts such as `'Control+A'` and `'Shift+Enter'`; use `'+'` to join the keys. See the [shared key-name definitions](https://github.com/web-infra-dev/midscene/blob/main/packages/shared/src/us-keyboard-layout.ts) for candidate names on non-mobile devices; actual support is platform-specific. The built-in Android, iOS, and Harmony devices support single keys only. Harmony supports a conservative set of named keys plus `A`-`Z` and `0`-`9`; output characters that do not have their own key code, such as `'?'`, are not key names. Unsupported keys and mobile key combinations throw an error. Support in custom devices depends on their implementation. - `deepLocate?: boolean` — Whether to enable [Deep Locate](#deep-locate-deeplocate). The deprecated `deepThink` name remains supported. Default: `false`. - `context?: string` — Additional context for this call. See [Per-call context](#per-call-context). - `xpath?: string` — XPath for the target element. Midscene tries the XPath before consulting the cache or AI model. Default: `undefined`. - `cacheable?: boolean` — Whether this call can use the [cache](../caching.mdx). Default: `true`. **Backward-compatible usage (deprecated but still supported):** - `key: string` — Keyboard key to press, such as `'Enter'`, `'Tab'`, or `'Escape'`. - `locate?: string | object` — Optional natural-language description of the element, or [prompting with images](#prompting-with-images). - `options?: object` — Optional configuration. The type is the same as the `opt` type in the recommended usage. - Return value: - `Promise<void>`. - Examples: ```typescript // Recommended await agent.aiKeyboardPress('The search input box', { keyName: 'Enter' }); await agent.aiKeyboardPress('The search input box', { keyName: 'Control+A' }); // Pure keyboard operation on the currently focused element await agent.aiKeyboardPress(undefined, { keyName: 'Control+X' }); // Backward compatible (not recommended) await agent.aiKeyboardPress('Enter', 'The search input box'); ``` :::note Signature update The recommended signature places the optional locate prompt first. Pass `undefined` when the shortcut should operate on the current focus without locating or clicking an element. A valid model configuration is still required and is validated before the action runs, although this targetless form does not send a location request to the model. The legacy signature `aiKeyboardPress(key, locate, options)` remains supported, but new code should use the recommended signature. ::: #### `aiScroll()` {#agentaiscroll} Scroll a page or an element. - Type ```typescript // Recommended: locate first, then options with scroll parameters function aiScroll( locate: string | object | undefined, opt: { scrollType?: 'singleAction' | 'scrollToBottom' | 'scrollToTop' | 'scrollToRight' | 'scrollToLeft'; direction?: 'down' | 'up' | 'left' | 'right'; distance?: number | null; deepLocate?: boolean; xpath?: string; cacheable?: boolean; }, ): Promise<void>; // Backward compatible (legacy) function aiScroll( scrollParam: PlanningActionParamScroll, locate?: string | object, options?: object, ): Promise<void>; ``` - Parameters: **Recommended usage:** - `locate: string | object | undefined` — A natural language description of the scroll target, or [prompting with images](#prompting-with-images). When omitted, Midscene scrolls at the current pointer position. - `opt: object` — Configuration: - `scrollType?: 'singleAction' | 'scrollToBottom' | 'scrollToTop' | 'scrollToRight' | 'scrollToLeft'` — Scroll behavior. Default: `'singleAction'`. - `direction?: 'down' | 'up' | 'right' | 'left'` — Direction of content movement. Used only when `scrollType` is `'singleAction'`. For example, `'down'` reveals content below the current viewport. Default: `'down'`. - `distance?: number | null` — Scroll distance in pixels. Use `null` to let Midscene choose the distance. - `deepLocate?: boolean` — Whether to enable [Deep Locate](#deep-locate-deeplocate). The deprecated `deepThink` name remains supported. Default: `false`. - `context?: string` — Additional context for this call. See [Per-call context](#per-call-context). - `xpath?: string` — XPath for the target element. Midscene tries the XPath before consulting the cache or AI model. Default: `undefined`. - `cacheable?: boolean` — Whether this call can use the [cache](../caching.mdx). Default: `true`. **Backward-compatible usage (deprecated but still supported):** - `scrollParam: PlanningActionParamScroll` — Legacy object containing `scrollType`, `direction`, and `distance`. - `locate?: string | object` — Optional natural-language description of the element, or [prompting with images](#prompting-with-images). - `options?: object` — Optional configuration. The type is the same as the `opt` type in the recommended usage. - Return value: - `Promise<void>`. - Examples: ```typescript // Recommended await agent.aiScroll('The form panel', { scrollType: 'singleAction', direction: 'up', distance: 100, }); // Backward compatible (not recommended) await agent.aiScroll( { scrollType: 'singleAction', direction: 'up', distance: 100 }, 'The form panel', ); ``` :::note Signature update The recommended signature places the locate prompt first. The legacy signature `aiScroll(scrollParam, locate, options)` remains supported, but new code should use the recommended signature. ::: #### `aiPinch()` {#agentaipinch} Perform a two-finger pinch gesture to zoom in or out. This method supports Android, iOS, and Chromium-based web browsers. - Type ```typescript function aiPinch( locate: string | object | undefined, opt: { direction: 'in' | 'out'; distance?: number; duration?: number; deepLocate?: boolean; xpath?: string; cacheable?: boolean; }, ): Promise<void>; ``` - Parameters: - `locate: string | object | undefined` — A natural-language description of the element to pinch, or [prompting with images](#prompting-with-images). When omitted, Midscene pinches at the center of the screen. - `opt: object` — Configuration: - `direction: 'in' | 'out'` — **Required.** Use `'in'` to pinch the fingers together and zoom out. Use `'out'` to spread them apart and zoom in. - `distance?: number` — Distance each finger moves in pixels. Default: one quarter of the shorter screen dimension. - `duration?: number` — Gesture duration in milliseconds. Default: `500`. - `deepLocate?: boolean` — Whether to enable [Deep Locate](#deep-locate-deeplocate). Default: `false`. - `context?: string` — Additional context for this call. See [Per-call context](#per-call-context). - `xpath?: string` — XPath for the target element. Default: `undefined`. - `cacheable?: boolean` — Whether this call can use the [cache](../caching.mdx). Default: `true`. - Return value: - `Promise<void>`. - Examples: ```typescript // Zoom in on the map (spread fingers apart) await agent.aiPinch('the map area', { direction: 'out', distance: 200 }); // Zoom out at screen center (pinch fingers together) await agent.aiPinch(undefined, { direction: 'in' }); // Zoom in with custom duration await agent.aiPinch('the image', { direction: 'out', distance: 300, duration: 1000 }); ``` :::info Platform support - **Android** — Implemented via [yadb](https://github.com/ysbing/yadb) `-pinch` command. - **iOS** — Implemented via W3C Actions API with dual touch pointers. - **Web** — Implemented through CDP touch events. Puppeteer and Playwright require `enableTouchEventsInActionSpace: true`; Playwright support is limited to Chromium-based browsers. - **HarmonyOS** — Not supported. The `uitest` framework does not provide multi-touch APIs. ::: #### `aiLongPress()` {#agentailongpress} Long-press (or click-and-hold) an element. Useful for opening context menus, selecting items, or triggering long-press gestures. - Type ```typescript function aiLongPress( locate: string | object, opt?: { duration?: number; deepLocate?: boolean; xpath?: string; cacheable?: boolean; }, ): Promise<void>; ``` - Parameters: - `locate: string | object` — A natural language description of the element to long-press on, or [prompting with images](#prompting-with-images). - `opt?: object` — Optional configuration: - `duration?: number` — How long to hold the press, in milliseconds. Defaults: Android `2000`, iOS `1000`, and web `500`. HarmonyOS uses the system long-click duration and ignores this option. - `deepLocate?: boolean` — Whether to enable [Deep Locate](#deep-locate-deeplocate). Default: `false`. - `context?: string` — Additional context for this call. See [Per-call context](#per-call-context). - `xpath?: string` — XPath for the target element. Default: `undefined`. - `cacheable?: boolean` — Whether this call can use the [cache](../caching.mdx). Default: `true`. - Return value: - `Promise<void>`. - Examples: ```typescript // Long-press an article to open the context menu await agent.aiLongPress('the first article on the homepage'); // Long-press with a custom duration await agent.aiLongPress('the message bubble', { duration: 2000 }); ``` :::info Platform support - **Android**, **iOS**, **HarmonyOS**, **Web** (Chromium-based browsers via touch events). On HarmonyOS the `duration` option is ignored because the underlying `uitest` API does not expose a custom hold time. ::: #### `aiDoubleClick()` {#agentaidoubleclick} Double-click on an element. - Type ```typescript function aiDoubleClick(locate: string | object, options?: object): Promise<void>; ``` - Parameters: - `locate: string | object` — A natural language description of the element to double-click on, or [prompting with images](#prompting-with-images). - `options?: object` — Optional configuration: - `deepLocate?: boolean` — Whether to enable [Deep Locate](#deep-locate-deeplocate). The deprecated `deepThink` name remains supported. Default: `false`. - `context?: string` — Additional context for this call. See [Per-call context](#per-call-context). - `xpath?: string` — XPath for the target element. Midscene tries the XPath before consulting the cache or AI model. Default: `undefined`. - `cacheable?: boolean` — Whether this call can use the [cache](../caching.mdx). Default: `true`. - Return value: - `Promise<void>`. - Examples: ```typescript await agent.aiDoubleClick('The file name at the top of the page'); // Use deepLocate feature to precisely locate the element await agent.aiDoubleClick('The file name at the top of the page', { deepLocate: true, }); ``` #### `aiRightClick()` {#agentairightclick} > Available on web pages and desktop (`@midscene/computer`). Not available on mobile devices (Android, iOS, or HarmonyOS). Right-click an element. Midscene cannot interact with the browser's native context menu after the click. Use this method for elements that handle their own `contextmenu` events. - Type ```typescript function aiRightClick(locate: string | object, options?: object): Promise<void>; ``` - Parameters: - `locate: string | object` — A natural language description of the element to right-click on, or [prompting with images](#prompting-with-images). - `options?: object` — Optional configuration: - `deepLocate?: boolean` — Whether to enable [Deep Locate](#deep-locate-deeplocate). The deprecated `deepThink` name remains supported. Default: `false`. - `context?: string` — Additional context for this call. See [Per-call context](#per-call-context). - `xpath?: string` — XPath for the target element. Midscene tries the XPath before consulting the cache or AI model. Default: `undefined`. - `cacheable?: boolean` — Whether this call can use the [cache](../caching.mdx). Default: `true`. - Return value: - `Promise<void>`. - Examples: ```typescript await agent.aiRightClick('The file name at the top of the page'); // Use deepLocate feature to precisely locate the element await agent.aiRightClick('The file name at the top of the page', { deepLocate: true, }); ``` <a id="data-extraction"></a> ### Extraction, location, and assertions {#extraction-location-assertion} #### `aiAsk()` {#agentaiask} Ask the AI model a question about the current page. The method returns the model's answer as a string. `aiAsk()` is fully equivalent to `aiString()`. - Type ```typescript function aiAsk(prompt: string | object, options?: object): Promise<string>; ``` - Parameters: - `prompt: string | object` — A natural language description of the question, or [prompting with images](#prompting-with-images). - `options?: object` — Optional configuration: - `domIncluded?: boolean | 'visible-only'` — Whether to include simplified DOM data, which can expose attributes such as image URLs. Use `'visible-only'` to include only visible elements. Default: `false`. - `screenshotIncluded?: boolean` — Whether to include a screenshot. Default: `true`. - `context?: string` — Additional context for this call. See [Per-call context](#per-call-context). - Return value: - Returns a Promise that resolves to the model's answer. - Examples: ```typescript const result = await agent.aiAsk('What should I do to test this page?'); console.log(result); // Output the answer from the AI model ``` Use `aiQuery` when you need structured data instead of a string. #### `aiQuery()` {#agentaiquery} Extract structured data from the current page. Describe the expected shape in `dataDemand`, and Midscene returns a matching value. - Type ```typescript function aiQuery<T>(dataDemand: string | object, options?: object): Promise<T>; ``` - Parameters: - `dataDemand: string | object` — A description of the expected data and its return format. - `options?: object` — Optional configuration: - `domIncluded?: boolean | 'visible-only'` — Whether to include simplified DOM data, which can expose attributes such as image URLs. Use `'visible-only'` to include only visible elements. Default: `false`. - `screenshotIncluded?: boolean` — Whether to include a screenshot. Default: `true`. - `context?: string` — Additional context for this call. See [Per-call context](#per-call-context). - Return value: - Returns a value that matches the shape described in `dataDemand`, such as a string, number, object, or array. - Examples: ```typescript const dataA = await agent.aiQuery({ time: 'The date and time displayed in the top-left corner as a string', userInfo: 'User information in the format {name: string}', tableFields: 'An array of table field names, string[]', tableDataRecord: 'Table records in the format {id: string, [fieldName]: string}[]', }); // You can also describe the expected return format using a string: // dataB will be an array of strings const dataB = await agent.aiQuery('string[], list of task names'); // dataC will be an array of objects const dataC = await agent.aiQuery( '{name: string, age: string}[], table data records', ); // Use domIncluded feature to extract invisible attributes const dataD = await agent.aiQuery( '{name: string, age: string, avatarUrl: string}[], table data records', { domIncluded: true }, ); ``` #### `aiBoolean()` {#agentaiboolean} Extract a boolean value from the UI. - Type ```typescript function aiBoolean(prompt: string | object, options?: object): Promise<boolean>; ``` - Parameters: - `prompt: string | object` — A natural language description of the expected value, or [prompting with images](#prompting-with-images). - `options?: object` — Optional configuration: - `domIncluded?: boolean | 'visible-only'` — Whether to include simplified DOM data, which can expose attributes such as image URLs. Use `'visible-only'` to include only visible elements. Default: `false`. - `screenshotIncluded?: boolean` — Whether to include a screenshot. Default: `true`. - `context?: string` — Additional context for this call. See [Per-call context](#per-call-context). - Return value: - Returns a Promise that resolves to the boolean returned by the AI model. - Examples: ```typescript const boolA = await agent.aiBoolean('Whether there is a login dialog'); // Use domIncluded feature to extract invisible attributes const boolB = await agent.aiBoolean('Whether the login button has a link', { domIncluded: true, }); ``` #### `aiNumber()` {#agentainumber} Extract a number value from the UI. - Type ```typescript function aiNumber(prompt: string | object, options?: object): Promise<number>; ``` - Parameters: - `prompt: string | object` — A natural language description of the expected value, or [prompting with images](#prompting-with-images). - `options?: object` — Optional configuration: - `domIncluded?: boolean | 'visible-only'` — Whether to include simplified DOM data, which can expose attributes such as image URLs. Use `'visible-only'` to include only visible elements. Default: `false`. - `screenshotIncluded?: boolean` — Whether to include a screenshot. Default: `true`. - `context?: string` — Additional context for this call. See [Per-call context](#per-call-context). - Return value: - Returns a Promise that resolves to the number returned by the AI model. - Examples: ```typescript const numberA = await agent.aiNumber('The remaining points of the account'); // Use domIncluded feature to extract invisible attributes const numberB = await agent.aiNumber( 'The value of the remaining points element', { domIncluded: true }, ); ``` #### `aiString()` {#agentaistring} Extract a string value from the UI. `aiString()` is fully equivalent to `aiAsk()`. - Type ```typescript function aiString(prompt: string | object, options?: object): Promise<string>; ``` - Parameters: - `prompt: string | object` — A natural language description of the expected value, or [prompting with images](#prompting-with-images). - `options?: object` — Optional configuration: - `domIncluded?: boolean | 'visible-only'` — Whether to include simplified DOM data, which can expose attributes such as image URLs. Use `'visible-only'` to include only visible elements. Default: `false`. - `screenshotIncluded?: boolean` — Whether to include a screenshot. Default: `true`. - `context?: string` — Additional context for this call. See [Per-call context](#per-call-context). - Return value: - Returns a Promise that resolves to the string returned by the AI model. - Examples: ```typescript const stringA = await agent.aiString('The first item in the list'); // Use domIncluded feature to extract invisible attributes const stringB = await agent.aiString('The link of the first item in the list', { domIncluded: true, }); ``` <a id="more-apis"></a> #### `aiLocate()` {#agentailocate} Locate an element using natural language. - Type ```typescript function aiLocate( locate: string | object, options?: object, ): Promise<{ rect: { left: number; top: number; width: number; height: number; }; center: [number, number]; dpr?: number; // Web only: device pixel ratio }>; ``` - Parameters: - `locate: string | object` — A natural language description of the element to locate, or [prompting with images](#prompting-with-images). - `options?: object` — Optional configuration: - `deepLocate?: boolean` — Whether to enable [Deep Locate](#deep-locate-deeplocate). The deprecated `deepThink` name remains supported. Default: `false`. - `context?: string` — Additional context for this call. See [Per-call context](#per-call-context). - `xpath?: string` — XPath for the target element. Midscene tries the XPath before consulting the cache or AI model. Default: `undefined`. - `cacheable?: boolean` — Whether this call can use the [cache](../caching.mdx). Default: `true`. - Return value: - Returns a Promise that resolves to a location information object. - `rect` usually represents the matched element boundary. - Some models only support point grounding rather than boundary grounding. In those cases, such as AutoGLM, `rect` will be a small `8x8` box containing the element center instead of the true element boundary. - Because `rect` can vary significantly depending on model capability, it is not recommended to rely too heavily on this field for strict boundary semantics. - If you need a stable click target, prefer the `center` field. - `dpr` is a Web-only compatibility field. It is the ratio between physical screenshot pixels and logical CSS pixels. Other Agent types do not guarantee this field. - Examples: ```typescript const locateInfo = await agent.aiLocate( 'The login button at the top of the page', ); console.log(locateInfo); ``` #### `aiAssert()` {#agentaiassert} Specify an assertion in natural language, and the AI determines whether the condition is true. If the assertion fails, the SDK throws an error that includes both the optional `errorMsg` and a detailed reason generated by the AI. - Type ```typescript function aiAssert( assertion: string | object, errorMsg?: string, options?: object, ): Promise<void>; ``` - Parameters: - `assertion: string | object` — The assertion described in natural language, or [prompting with images](#prompting-with-images). - `errorMsg?: string` — An optional error message to append if the assertion fails. - `options?: object` — Optional configuration: - `domIncluded?: boolean | 'visible-only'` — Whether to include simplified DOM data, which can expose attributes such as image URLs. Use `'visible-only'` to include only visible elements. Default: `false`. - `screenshotIncluded?: boolean` — Whether to include a screenshot. Default: `true`. - `context?: string` — Additional context for this call. See [Per-call context](#per-call-context). - Return value: - Returns a Promise that resolves to void if the assertion passes; if it fails, an error is thrown with `errorMsg` and additional AI-provided information. - Example: ```typescript await agent.aiAssert('The price of "Sauce Labs Onesie" is 7.99'); ``` :::info Assertions are critical in test scripts. To reduce the risk of model false positives or false negatives, combine `.aiQuery` with standard JavaScript assertions when you need deterministic checks. For example, you might replace the above code with: ```typescript const items = await agent.aiQuery( '{name: string, price: number}[], return product names and prices', ); const onesieItem = items.find((item) => item.name === 'Sauce Labs Onesie'); expect(onesieItem).toBeTruthy(); expect(onesieItem.price).toBe(7.99); ``` ::: ### Observation and waiting {#observation-waiting} #### `startObserving()` {#agentstartobserving} `startObserving()` records the screen continuously. Use it to check transient UI, such as toasts, banners, and page transitions. The flow is simple: start recording, perform the page actions, and call `stop()`. `stop()` returns a `UIObservation`. You can query or assert against the recorded frames. ```typescript function startObserving(options?: { intervalMs?: number; // sampling interval; default 1,000 ms, minimum 200 ms maxFrames?: number; // maximum number of retained frames; default 30 watchdogMs?: number; // maximum recording time; default 300,000 ms; 0 disables the limit }): Promise<UIObserver>; ``` `UIObserver` manages the recording: - `observer.bufferedFrameCount: number` — Number of frames currently buffered while recording. - `observer.stop(): Promise<UIObservation>` — Stop recording and return a `UIObservation`. `UIObservation` contains the recorded frames. The frames no longer change after you call `stop()`. You can call `aiQuery()`, `aiBoolean()`, `aiNumber()`, `aiString()`, `aiAsk()`, and `aiAssert()`. These methods only read recorded frames. They do not read the current page DOM. For this reason, the methods do not support `domIncluded`. TypeScript reports this mistake. Midscene also throws at runtime if JavaScript passes the option. `UIObservation` also contains `frameCount`, `startedAt`, and `endedAt`. Call `observation.dispose()` to remove temporary images when you finish. `agent.destroy()` also removes any images that remain. ```typescript const observer = await agent.startObserving(); await agent.aiAct('submit the form'); const observation = await observer.stop(); await observation.aiAssert('a success toast appeared during the process'); const toastCount = await observation.aiNumber( 'how many success toasts appeared?', ); await observation.dispose(); ``` Sampling and resource usage: - Midscene uses a continuous frame source when one is available. Android uses scrcpy (`scrcpyConfig.enabled`), iOS uses WDA MJPEG (`wdaMjpegFrameSource.enabled`), and the web uses CDP screencast. Without a continuous source, Midscene takes screenshots at intervals. Sampling is slower on mobile in this mode. - Data URL frames are written to files during sampling. `UIObserver` does not keep every image in memory. Other frame handles are decoded and saved in small batches when recording stops. Midscene reads the images when a query or assertion runs. - The model receives up to `maxFrames` frames and the final screenshot. Increase `intervalMs` to sample less often, or decrease `maxFrames` to limit the frame count. Observed frames appear in the report timeline with an `Observed` label. - On iOS, observed frames come from the lower-quality MJPEG stream. The final screenshot still uses full quality. #### `aiWaitFor()` {#agentaiwaitfor} Wait until a condition described in natural language becomes true. Midscene starts checks at least `checkIntervalMs` apart to avoid unnecessary AI calls. - Type ```typescript function aiWaitFor( assertion: string, options?: { timeoutMs?: number; checkIntervalMs?: number; context?: string; }, ): Promise<void>; ``` - Parameters: - `assertion: string` — The condition described in natural language. - `options?: object` — Optional configuration: - `timeoutMs?: number` — Maximum window in milliseconds for starting a new check. If the previous check began within this window, Midscene may complete it; otherwise the method times out. Default: `15000`. - `checkIntervalMs?: number` — Minimum interval in milliseconds between the start of consecutive checks. Default: `3000`. - `context?: string` — Additional context for this call. See [Per-call context](#per-call-context). - Return value: - Returns a Promise that resolves to void if the condition is met; if not, an error is thrown when the timeout is reached. - Examples: ```typescript // Basic usage await agent.aiWaitFor( 'There is at least one headphone information displayed on the interface', ); // Using custom options await agent.aiWaitFor('The shopping cart icon shows a quantity of 2', { timeoutMs: 30000, // Wait for 30 seconds checkIntervalMs: 5000, // Check every 5 seconds }); ``` :::info Because each check calls an AI model, `.aiWaitFor()` can be slower and more expensive than a fixed delay. Use a simple sleep when you only need to wait for a known amount of time. ::: <a id="runyaml"></a> ### Workflow execution and context {#workflow-context} #### `runYaml()` {#agentrunyaml} Execute an automation script written in YAML. Midscene parses and runs only the `tasks` section. The method returns the results of all `.aiQuery` calls in the script. - Type ```typescript function runYaml(yamlScriptContent: string): Promise<{ result: any }>; ``` - Parameters: - `yamlScriptContent: string` — The YAML-formatted script content. - Return value: - Returns an object with a `result` property that includes the results of all `.aiQuery` calls. - Example: ```typescript const { result } = await agent.runYaml(` tasks: - name: search weather flow: - ai: input 'weather today' in input box, click search button - sleep: 3000 - name: query weather flow: - aiQuery: "the result shows the weather info, {description: string}" `); console.log(result); ``` :::info For more information about YAML scripts, please refer to [Automate with Scripts in YAML](../automate-with-scripts-in-yaml). ::: #### `runGherkinScenario()` {#agentrungherkinscenario} Run a single Gherkin scenario and map its steps to Midscene Agent calls. :::caution Beta This API has been available since Midscene 1.10 and remains in beta. It may change in future releases. ::: - Type ```typescript function runGherkinScenario( scenarioText: string, options?: { context?: string; abortSignal?: AbortSignal; deepThink?: 'unset' | true | false; deepLocate?: boolean; }, ): Promise<void>; ``` - Parameters: - `scenarioText: string` — One Gherkin scenario, or a list of Gherkin steps without the `Scenario:` header. - `options?: object` — Optional runtime options. - `context?: string` — Temporary context for this run. - `abortSignal?: AbortSignal` — An optional signal for aborting the run. - `deepThink?: 'unset' | true | false` — Passed to `aiAct` for `Given` and `When` steps. - `deepLocate?: boolean` — Passed to `aiAct` for `Given` and `When` steps. - Return value: - `Promise<void>` — Resolves after all steps finish. If a step fails, the error message includes the Gherkin line, the original step, and the Midscene semantic action being executed. - Example: ```typescript await agent.runGherkinScenario(` Scenario: Add a todo item Given the todo page is open When I add a todo item named "Buy milk" Then the todo list should contain "Buy milk" `); ``` For supported rules, limitations, cache behavior, and YAML usage, see [BDD-style scripts with Gherkin](../advanced/bdd-style-scripts-with-gherkin). #### `setAIActContext()` {#agentsetaiactcontext} Set the context sent with subsequent `agent.aiAct()` or `agent.ai()` calls. This replaces any existing context. This setting does not affect instant-action APIs such as `aiTap()`. - Type ```typescript function setAIActContext(aiActContext: string): void; ``` - Parameters: - `aiActContext: string` — The background knowledge that should be sent to the AI model. The deprecated `aiActionContext` name is still accepted. - Example: ```typescript await agent.setAIActContext('Close the cookie consent dialog first if it exists'); ``` :::note `agent.setAIActionContext()` is deprecated. Please use `agent.setAIActContext()` instead. The deprecated method remains as an alias for compatibility. ::: #### `evaluateJavaScript()` {#agentevaluatejavascript} > Available only for web agents. Evaluate a JavaScript expression in the web page context. - Type ```typescript function evaluateJavaScript(script: string): Promise<any>; ``` - Parameters: - `script: string` — The JavaScript expression to evaluate. - Return value: - Returns the result of the JavaScript expression. - Example: ```typescript const result = await agent.evaluateJavaScript('document.title'); console.log(result); ``` #### `freezePageContext()` {#agentfreezepagecontext} Freeze the current page context so subsequent operations reuse one snapshot instead of retrieving page state repeatedly. This can improve performance for a large batch of concurrent read operations. Usage constraints: - Use this method only when context retrieval is a confirmed bottleneck. - Call `agent.unfreezePageContext()` when you need live page state again. - Do not perform interaction methods while the context is frozen. The model cannot observe state changes and may act on stale information. - Type ```typescript function freezePageContext(): Promise<void>; ``` - Return value: - `Promise<void>`. - Examples: ```typescript // Freeze the page context await agent.freezePageContext(); // Some queries... const results = await Promise.all([ agent.aiQuery('Username input box value'), agent.aiQuery('Password input box value'), agent.aiLocate('Login button'), ]); console.log(results); // Unfreeze the page context, subsequent operations will use real-time page state await agent.unfreezePageContext(); ``` :::info In the report, operations using frozen context will display a 🧊 icon in the Insight tab. ::: #### `unfreezePageContext()` {#agentunfreezepagecontext} Unfreeze the page context and resume retrieving live page state. - Type ```typescript function unfreezePageContext(): Promise<void>; ``` - Return value: - `Promise<void>`. ### Reporting, metrics, and lifecycle {#reporting-metrics-lifecycle} <a id="log-screenshot"></a> <a id="agentlogscreenshot"></a> #### `recordToReport()` {#agentrecordtoreport} Add a report entry using either a newly captured screenshot or screenshots provided by the caller. - Type ```typescript interface RecordToReportOptions { content?: string; /** @deprecated Use screenshots instead. */ screenshotBase64?: string; screenshots?: { /** * PNG/JPEG data URI, or raw PNG base64 body. */ base64: string; description?: string; }[]; } function recordToReport( title?: string, options?: RecordToReportOptions, ): Promise<void>; ``` - Parameters: - `title?: string` — Optional title for the report entry. Default: `'untitled'`. - `options?: RecordToReportOptions` — Optional configuration: - `content?: string` — Description of the screenshot. - `screenshots?: Array<{ base64: string; description?: string }>` — One or more screenshots to record under the same report entry. When this option is set, Midscene does not capture another screenshot automatically. `base64` accepts a PNG/JPEG data URI such as `data:image/png;base64,...` or a raw base64 body, which Midscene treats as PNG. - Compatibility: `screenshotBase64?: string` is still accepted as a backward-compatible single-screenshot override. Prefer `screenshots: [{ base64 }]` for new code. Provide only one of `screenshots` or `screenshotBase64`. - Return value: - `Promise<void>`. - Examples: ```typescript await agent.recordToReport('Login page', { content: 'User A', }); const before = await page.screenshot({ encoding: 'base64' }); const after = await page.screenshot({ encoding: 'base64' }); await agent.recordToReport('Checkout comparison', { content: 'Compare the state before and after submit.', screenshots: [ { base64: `data:image/png;base64,${before}`, description: 'Before submit', }, { base64: `data:image/png;base64,${after}`, description: 'After submit', }, ], }); ``` #### `_unstableLogContent()` {#agent_unstablelogcontent} Retrieve the log content from the report file. The structure of the log object may change in future versions. - Type ```typescript function _unstableLogContent(): object; ``` - Return value: - Returns an object that contains the log content. - Examples: ```typescript const logContent = agent._unstableLogContent(); console.log(logContent); ``` <a id="llm-usage-metrics"></a> **LLM usage metrics** Midscene records the token usage of every LLM call. You can read the aggregated totals from the agent at runtime, which is useful for cost observability with tools like Langfuse. #### `metrics` {#agentmetrics} A getter that returns a snapshot of the LLM usage accumulated by the agent since it was created. - Type ```typescript interface UsageBucket { promptTokens: number; completionTokens: number; totalTokens: number; calls: number; } interface MidsceneUsageMetrics { totalPromptTokens: number; totalCompletionTokens: number; totalTokens: number; totalCachedInput: number; totalTimeCostMs: number; calls: number; // Breakdown by call intent: `planning`, `insight`, `default`. byIntent: Record<string, UsageBucket>; // Breakdown by model name. byModel: Record<string, UsageBucket>; } ``` - Example ```typescript await agent.aiAct('search for headphones'); const usage = agent.metrics; console.log(usage.totalTokens, usage.byIntent, usage.byModel); ``` #### `onLLMUsage` option For real-time tracking, pass an `onLLMUsage` callback when constructing the agent. It is invoked once per LLM call as soon as the usage is available, with the raw usage info (token counts, model name, intent, request id, etc.). ```typescript const agent = new PuppeteerAgent(page, { onLLMUsage: (usage) => { langfuse.event({ name: usage.intent, value: usage.total_tokens }); }, }); ``` #### `destroy()` {#agentdestroy} Finalize the Agent's report and release resources owned by the Agent. - Type ```typescript function destroy(): Promise<void>; ``` - Behavior: - Stops an active observer, if one exists. - Calls the underlying interface's optional `destroy()` method and waits for it to finish. - Flushes and finalizes the report, then updates `.reportFile` with the final path, or `undefined` when no report was generated. - Does nothing when called again after the first call. - If interface cleanup fails, Midscene still attempts to finalize the report before rejecting with the cleanup error. Treat the Agent as disposed after calling this method. :::warning Cleanup scope `Agent.destroy()` stops the Agent, finalizes its report, and releases the control resources held by Midscene. It generally does not close the target page or disconnect a physical device, although some platforms end their automation session or connection. See each platform's `destroy()` description for the exact behavior. ::: <a id="properties"></a> #### `.reportFile` The path to the current report file. Its type is `string | null | undefined`. The value is unavailable before the first report update. It also remains unavailable when report generation is disabled, the Agent runs in a browser runtime without file-system access, or the Agent produces no executions. After the value becomes available, the report can still receive updates. Call [`await agent.destroy()`](#agentdestroy) before consuming the final report. This flushes and finalizes the report, then updates `.reportFile` with the final path. ## Per-call context Every `agent.ai*` method accepts an optional `context` string in its options. Use it to provide business knowledge, user state, terminology, or other background that is relevant to one request without changing the main prompt. ```typescript await agent.aiAssert('The current page uses the new version styling', undefined, { context: 'In the new version, the Tab bar is at the top and the page theme color is red. In the old version, the Tab bar is at the bottom and the page theme color is green. Treat the page as the new version only when both new-version conditions are met.', }); ``` The context only applies to that method call. For `aiAct`, a per-call `context` takes precedence over the Agent-level `aiActContext` (including an explicitly empty string). Other AI methods do not automatically inherit `aiActContext`. ### Shared types {#shared-types} <a id="deep-locate-deeplocate"></a> **Locate options: Deep Locate (`deepLocate`)** `deepLocate` is available on APIs that locate elements, including `aiAct`, `aiTap`, `aiHover`, `aiInput`, `aiKeyboardPress`, `aiScroll`, `aiDoubleClick`, `aiRightClick`, and `aiLocate`. When enabled, Midscene calls the AI model twice to locate the element more precisely. This can improve accuracy for small targets or elements that are difficult to distinguish from their surroundings. Newer models such as Qwen3.x, Doubao 2.0, and Gemini 3.5 usually benefit less, so enable this option only when needed. - **Default**: `false` ```typescript // Enable deepLocate to precisely locate hard-to-identify elements await agent.aiTap('Shopping cart icon in the top right', { deepLocate: true }); ``` :::note Historically, the name `deepThink` has carried two different meanings across APIs: - In `aiAct()`, `deepThink` has always meant **planning mode**, guiding task decomposition and planning-focused reasoning. See [`deepThink` planning mode](#aiact-deepthink) for details. - In single-step action methods such as `aiTap` and `aiHover`, the old `deepThink` meant **enhanced element location**, equivalent to today's `deepLocate`. To separate these two meanings, `deepThink` has been reserved for planning mode since v1.5.1, while enhanced element location is consistently named `deepLocate`. - For `aiAct()`, you can use both `deepThink` and `deepLocate`: `deepThink` controls planning mode, while `deepLocate` controls the Deep Locate behavior described in this section. - For single-step action methods such as `aiTap` and `aiHover`, use the clearer `deepLocate` parameter when you need to improve location precision. The old `deepThink` parameter remains compatible and is equivalent to `deepLocate`. ::: <a id="prompting-with-images"></a> **Prompt input with images** Include reference images when text alone cannot identify the target clearly. Image-enabled prompts use the following shape: ```javascript { // Prompt text that refers to the attached images prompt: string, // Images referenced by the prompt images?: { // Name used to reference the image in the prompt name: string, // Local path, base64 string, or HTTP URL url: string }[] // Convert HTTP images to base64 before sending them to the LLM. // Use this when the URLs are not publicly accessible. convertHttpImage2Base64?: boolean } ``` - Example 1: use a reference image to identify a tap target. ```javascript await agent.aiTap({ prompt: 'The specific logo', images: [ { name: 'The specific logo', url: 'https://github.githubassets.com/assets/GitHub-Mark-ea2971cee799.png', }, ], }); ``` - Example 2: use images to assert the page content. ```javascript await agent.aiAssert({ prompt: 'Whether there is a specific logo on the page.', images: [ { name: 'The specific logo', url: 'https://github.githubassets.com/assets/GitHub-Mark-ea2971cee799.png', }, ], }); ``` - Example 3: use images to guide an action (`aiAct`). ```javascript await agent.aiAct({ prompt: 'Tap the icon that matches the reference logo', images: [ { name: 'The specific logo', url: 'https://github.githubassets.com/assets/GitHub-Mark-ea2971cee799.png', }, ], }); ``` **Image size** Follow your model provider's image size and dimension limits. Oversized or very small images may be rejected. Check the provider documentation for exact limits. <a id="report-merging-tool"></a> ### Report utilities {#reporting-utilities} Each automation workflow can generate its own report. `ReportMergingTool` combines those reports into one report so you can review all workflows together. The output is either a standalone HTML file or a directory containing `index.html` and its external screenshot assets. #### `new ReportMergingTool()` Create a `ReportMergingTool` instance. - Example: ```typescript import { ReportMergingTool } from '@midscene/core/report'; const reportMergingTool = new ReportMergingTool(); ``` #### `.append()` Add an automation report to the list to be merged, typically right after each workflow finishes. - Type ```typescript type SkippedReportFileAttributes = Omit< ReportFileAttributes, 'testStatus' > & { testStatus: 'skipped'; }; type ReportFileWithAttributes = | { reportFilePath: string; reportAttributes: ReportFileAttributes; } | { reportFilePath?: undefined; reportAttributes: SkippedReportFileAttributes; }; function append(reportInfo: ReportFileWithAttributes): void; ``` - Parameters: - `reportInfo: ReportFileWithAttributes` — Report to append: - `reportFilePath: string | undefined` — Path to the report file, usually `agent.reportFile`. This field can be omitted only when `reportAttributes.testStatus` is `'skipped'`. Otherwise, `append()` throws an error when the path is missing. - `reportAttributes: object` — Report metadata: - `testId: string` — Unique workflow identifier. - `testTitle: string` — Workflow title. - `testDescription: string` — Workflow description. - `testDuration: number` — Workflow duration in milliseconds. - `testStatus: 'passed' | 'failed' | 'timedOut' | 'skipped' | 'interrupted'` — Workflow status. - Return value: - `void` - Example: ```typescript import type { TestStatus } from '@midscene/core'; // Add a report in your test hooks // This example assumes that each test owns its Agent and underlying interface. afterEach(async (ctx) => { let workflowStatus: TestStatus = 'passed'; if (ctx.task.result?.state === 'skip') { workflowStatus = 'skipped'; } else if (ctx.task.result?.errors?.[0]?.message.includes('timed out')) { workflowStatus = 'timedOut'; } else if (ctx.task.result?.state === 'fail') { workflowStatus = 'failed'; } // Finalize the report before reading its final path. await agent.destroy(); const reportFilePath = agent.reportFile; const reportAttributes = { testId: ctx.task.name, testTitle: ctx.task.name, testDescription: 'Automation workflow description', testDuration: Date.now() - startTime, }; if (!reportFilePath) { if (workflowStatus !== 'skipped') { throw new Error('Midscene report was not generated'); } reportMergingTool.append({ reportAttributes: { ...reportAttributes, testStatus: 'skipped', }, }); return; } reportMergingTool.append({ reportFilePath, reportAttributes: { ...reportAttributes, testStatus: workflowStatus, }, }); }); ``` Finalize every report-producing Agent before calling `.mergeReports()`. #### `.mergeReports()` Merge all added reports into one report. - Type ```typescript function mergeReports( reportFileName?: 'AUTO' | string, opts?: { rmOriginalReports?: boolean; overwrite?: boolean; outputDir?: string; }, ): string | null; ``` - Parameters: - `reportFileName?: 'AUTO' | string` — Name of the merged report. Default: `'AUTO'`, which tells Midscene to generate the name. A custom name does not need the `.html` suffix. - `opts?: object` — Optional configuration: - `rmOriginalReports?: boolean` — Whether to delete the original report files after merging. Default: `false`. - `overwrite?: boolean` — Whether to overwrite an existing target file. Default: `false`. - `outputDir?: string` — Directory for the merged report. Relative paths are resolved from the current working directory. Default: `midscene_run/report/`. - Return value: - Returns the merged report's entry HTML path, or `null` if no reports have been added. - If every source report uses `single-html`, the output is `<outputDir>/<reportFileName>.html`. - If any source report uses `html-and-external-assets`, the output is `<outputDir>/<reportFileName>/index.html` plus its screenshot assets. - Examples: ```typescript // Basic usage with an auto-generated file name afterAll(() => { reportMergingTool.mergeReports(); }); // Specify a custom file name afterAll(() => { reportMergingTool.mergeReports('my-automation-report'); }); // Merge and delete the original reports afterAll(() => { reportMergingTool.mergeReports('my-automation-report', { rmOriginalReports: true, }); }); // Overwrite an existing report file afterAll(() => { reportMergingTool.mergeReports('my-automation-report', { overwrite: true, }); }); // Write the merged report to a custom directory afterAll(() => { reportMergingTool.mergeReports('my-automation-report', { outputDir: './test-results', }); }); ``` #### `.clear()` Clear the list of reports to be merged. Use this if you need to reuse the same instance for multiple merge operations. - Type ```typescript function clear(): void; ``` - Return value: - `void` - Example: ```typescript reportMergingTool.mergeReports('first-batch'); reportMergingTool.clear(); // Clear the list // Continue adding new reports... ``` ## Web (`@midscene/web`) {#web} Use this section to configure Midscene's browser agents and review browser-specific constructor options. For shared parameters such as reporting, hooks, and caching, see [Shared Agent APIs](#common). ### Action space `PuppeteerAgent`, `PlaywrightAgent`, and Chrome Bridge agents share the following action space: - `Tap` — Left-click an element. - `RightClick` — Right-click an element. - `DoubleClick` — Double-click an element. - `Hover` — Hover over an element. - `Input` — Enter text with `replace`/`typeOnly`/`clear` modes (`append` is a deprecated alias for `typeOnly`). - `KeyboardPress` — Press a specified key (optionally focusing a target element first). - `Scroll` — Scroll from an element or screen center; supports scroll-to-top/bottom/left/right helpers. - `DragAndDrop` — Drag from one element to another. - `LongPress` — Long-press a target element with optional duration. - `Swipe` — Touch-style swipe gesture (available when `enableTouchEventsInActionSpace` is `true`). - `Pinch` — Two-finger pinch gesture for zoom in/out (available when `enableTouchEventsInActionSpace` is `true`; Chromium-based browsers only for Playwright). - `ClearInput` — Clear the contents of an input field. - `Navigate` — Open a URL in the current tab. - `Reload` — Reload the page. - `GoBack` — Navigate back in history. **Lifecycle and ownership** Puppeteer and Playwright Page/Browser Agents inherit the shared [`destroy()`](#agentdestroy) method. Calling it finalizes the Midscene report and performs Agent-owned cleanup, but does not close the Page, Browser, or BrowserContext used by the Agent. ### PuppeteerPageAgent / PuppeteerAgent {#puppeteer-agent} Use the Puppeteer integration to add AI actions to an existing Puppeteer workflow. `PuppeteerPageAgent` is bound to one Puppeteer `Page`. `PuppeteerAgent` remains an alias for backward compatibility. **Import** ```ts import { PuppeteerPageAgent } from '@midscene/web/puppeteer'; ``` **Constructor** ```ts const agent = new PuppeteerPageAgent(page, { // browser-specific options... }); ``` **Browser-specific options** In addition to the base agent options, Puppeteer exposes: - `forceSameTabNavigation: boolean` — Whether to keep navigation in the current tab. Default: `true`. - `waitForNavigationTimeout: number` — Maximum time in milliseconds to wait when an action triggers navigation. Set to `0` to skip the wait. Default: `5000`. - `waitForNetworkIdleTimeout: number` — Maximum time in milliseconds to wait for network idle between actions. Set to `0` to skip the wait. Default: `2000`. - `enableTouchEventsInActionSpace: boolean` — Whether to add touch gestures such as swipe to the action space. Default: `false`. - `keyboardTypeDelay: number` — Finite non-negative per-character delay in milliseconds. In `'legacy'` mode, Puppeteer's `page.keyboard.type` handles a positive delay. Increase this value only when a controlled input drops characters during fast input. Default: `undefined`, which uses Puppeteer's default. - `inputStrategy: 'legacy' | 'sequential' | 'bulk'` — Default text input strategy for `Input` actions. Use `'bulk'` to insert the complete value in one operation or `'sequential'` to force code-point-by-code-point input. `'bulk'` requires `keyboardTypeDelay` to be omitted or set to `0`; use `'sequential'` for delayed input. Default: `'legacy'`. - `forceChromeSelectRendering: boolean` — Whether to render `select` elements with Chrome's base-select styling so they remain visible in screenshots and element extraction. Requires a Puppeteer version later than `24.6.0`. Default: `true`. - `customActions: DeviceAction[]` — Add additional custom actions so the Agent can call your domain-specific actions. **Usage notes** :::info - Use one page agent per page. With `forceSameTabNavigation: true`, Midscene opens new links in the current tab. Set it to `false` to preserve normal new-tab behavior, then create a separate page agent for each page. - `PuppeteerAgent` and `PuppeteerPageAgent` remain page-scoped for compatibility. Use `PuppeteerBrowserAgent` when one Agent must switch between pages. - For the full list of interaction methods, see [Shared Agent APIs](#interaction-methods). ::: ### PuppeteerBrowserAgent Use `PuppeteerBrowserAgent` when one Midscene Agent must switch between pages in a Puppeteer browser. It binds to a browser instance, tracks one active page, and can follow newly opened pages. ```ts const agent = new PuppeteerBrowserAgent(browser, page, { autoFollowNewPage: true, }); ``` - Constructor: `new PuppeteerBrowserAgent(browser, page, options?)` — Use this when you explicitly choose the initial active page. - Factory: `PuppeteerBrowserAgent.create(browser, options?)` — Let Midscene choose or create the initial active page. It uses `initialPage` when provided; otherwise it reuses the first existing page or creates a new one. - `initialPage: Page` — Initial Puppeteer page for the factory. - `autoFollowNewPage: boolean` — Automatically switch the active page when the browser opens a new page. Default: `false`. - `newPageTimeout: number` — Timeout for `waitForNewPage`. Default: `5000`. - `activePage: Page` — Current page controlled by the Browser Agent. - `pages()` — List pages from the bound browser. - `newPage()` — Create a new page and make it active. - `setActivePage(page: Page)` — Explicitly set which Puppeteer page the Browser Agent controls next. - `waitForNewPage(action?, options?)` — Wait for a newly opened page without implicitly switching the active page. <a id="web-quick-start"></a> <a id="web-connect-to-a-remote-puppeteer-browser"></a> **See also** - [Integrate with Puppeteer](../integrate-with-puppeteer) for installation, fixtures, and remote-CDP guidance. ### PlaywrightPageAgent / PlaywrightAgent {#playwright-agent} Use Midscene inside a Playwright browser for AI-driven testing or automation alongside your Playwright flows. `PlaywrightPageAgent` is bound to one Playwright `Page`. `PlaywrightAgent` remains an alias for backward compatibility. **Import** ```ts import { PlaywrightPageAgent } from '@midscene/web/playwright'; ``` **Constructor** ```ts const agent = new PlaywrightPageAgent(page, { // browser-specific options... }); ``` **Browser-specific options** - `forceSameTabNavigation: boolean` — Whether to keep navigation in the current tab. Default: `true`. - `waitForNavigationTimeout: number` — Maximum time in milliseconds to wait for navigation. Set to `0` to skip the wait. Default: `5000`. - `waitForNetworkIdleTimeout: number` — Maximum time in milliseconds to wait for network idle between actions. Set to `0` to skip the wait. Default: `2000`. - `enableTouchEventsInActionSpace: boolean` — Whether to add touch gestures such as swipe to the action space. Default: `false`. - `keyboardTypeDelay: number` — Finite non-negative per-character delay in milliseconds. In `'legacy'` mode, Playwright's `page.keyboard.type` handles a positive delay. Increase this value only when a controlled input drops characters during fast input. Default: `undefined`, which uses Playwright's default. - `inputStrategy: 'legacy' | 'sequential' | 'bulk'` — Default text input strategy for `Input` actions. Use `'bulk'` to insert the complete value in one operation or `'sequential'` to force code-point-by-code-point input. `'bulk'` requires `keyboardTypeDelay` to be omitted or set to `0`; use `'sequential'` for delayed input. Default: `'legacy'`. - `forceChromeSelectRendering: boolean` — Whether to render `select` elements with Chrome's base-select styling so they remain visible in screenshots and element extraction. Requires Playwright `1.52.0` or later. Default: `true`. - `customActions: DeviceAction[]` — Add additional custom actions so the Agent can call your domain-specific actions. **Usage notes** :::info - Use one page agent per page. With `forceSameTabNavigation: true`, Midscene intercepts new tabs for stability. Set it to `false` to preserve normal new-tab behavior, then create a separate page agent for each page. - `PlaywrightAgent` and `PlaywrightPageAgent` remain page-scoped for compatibility. Use `PlaywrightBrowserAgent` when one Agent must switch between pages in a browser context. - For the full list of interaction methods, see [Shared Agent APIs](#interaction-methods). ::: ### PlaywrightBrowserAgent Use `PlaywrightBrowserAgent` when one Midscene Agent must switch between pages in a Playwright browser context. It binds to the context, tracks one active page, and can follow newly opened pages. ```ts const agent = new PlaywrightBrowserAgent(context, page, { autoFollowNewPage: true, }); ``` - Constructor: `new PlaywrightBrowserAgent(context, page, options?)` — Use this when you explicitly choose the initial active page. - Factory: `PlaywrightBrowserAgent.create(context, options?)` — Let Midscene choose or create the initial active page. It uses `initialPage` when provided; otherwise it reuses the first existing page or creates a new one. - `initialPage: Page` — Initial Playwright page for the factory. - `autoFollowNewPage: boolean` — Automatically switch the active page when the context opens a new page. Default: `false`. - `newPageTimeout: number` — Timeout for `waitForNewPage`. Default: `5000`. - `activePage: Page` — Current page controlled by the Browser Agent. - `pages()` — List pages from the bound browser context. - `newPage()` — Create a new page and make it active. - `setActivePage(page: Page)` — Explicitly set which Playwright page the Browser Agent controls next. - `waitForNewPage(action?, options?)` — Wait for a newly opened page without implicitly switching the active page. <a id="web-playwright-quick-start"></a> <a id="web-extend-playwright-tests-with-midscene-fixtures"></a> **See also** - [Integrate with Playwright](../integrate-with-playwright) for setup, fixtures, and advanced configuration. ### Chrome Bridge Agent {#chrome-bridge-agent} Bridge mode lets Midscene operate the active tab in desktop Chrome through the extension instead of launching a separate automation browser. **Import** ```ts import { AgentOverChromeBridge } from '@midscene/web/bridge-mode'; ``` **Constructor** ```ts const agent = new AgentOverChromeBridge({ allowRemoteAccess: false, // other bridge options... }); ``` **Bridge options** - `closeNewTabsAfterDisconnect?: boolean` — Close any bridge-created tabs when the agent is destroyed. Default: `false`. - `allowRemoteAccess?: boolean` — Whether to allow remote machines to attach. Default: `false`, which binds to `127.0.0.1`. - `host?: string` — Override the interface for the bridge server. Takes precedence over `allowRemoteAccess`. - `port?: number` — TCP port for the bridge server. Default: `3766`. - `enableWaterFlowAnimation?: boolean` — Show the blue animated border and mouse pointer while Midscene controls the page. Defaults to `true`; set it to `false` when taking screenshots without the visual overlay. - `keyboardTypeDelay?: number` — Finite non-negative delay in milliseconds. Legacy Bridge input preserves the existing zero-delay CDP typing behavior. Set `inputStrategy` to `'sequential'` to apply the delay between forwarded Unicode code points. - `inputStrategy?: 'legacy' | 'sequential' | 'bulk'` — Default text input strategy. `'sequential'` forwards one Unicode code point at a time, while `'bulk'` forwards one `insertText` operation. `'bulk'` requires `keyboardTypeDelay` to be omitted or set to `0`. Default: `'legacy'`. See [Bridge mode by Chrome extension](../bridge-mode#constructor) for full installation and capability details. **Usage notes** :::info Call `connectCurrentTab()` or `connectNewTabWithUrl()` before calling other Agent methods. Each `AgentOverChromeBridge` instance can attach to only one tab. Create a new instance after calling `destroy()`. ::: **Bridge methods** <a id="web-connectcurrenttab"></a> **`connectCurrentTab()`** ```ts function connectCurrentTab(options?: { forceSameTabNavigation?: boolean; }): Promise<void>; ``` - `options.forceSameTabNavigation` intercepts new tabs and opens them in the current tab. Set it to `false` to preserve normal new-tab behavior, then create a separate agent for each tab. Default: `true`. - Resolves after a successful handshake with the active tab. Rejects if the extension cannot connect. <a id="web-connectnewtabwithurl"></a> **`connectNewTabWithUrl()`** ```ts function connectNewTabWithUrl( url: string, options?: { forceSameTabNavigation?: boolean }, ): Promise<void>; ``` - `url` — Address to open in a new desktop tab before attaching. - `options` — Same as `connectCurrentTab`. - Resolves when the new tab is opened and the bridge is connected. <a id="web-destroy"></a> **`destroy()`** ```ts function destroy(closeNewTabsAfterDisconnect?: boolean): Promise<void>; ``` - `closeNewTabsAfterDisconnect` — Optional runtime override for the constructor setting; `true` closes bridge-created tabs on teardown. - Includes all [shared Agent cleanup behavior](#agentdestroy), including report finalization. - Resolves after the bridge connection, local server, and Agent report are cleaned up. <a id="web-open-a-new-desktop-tab"></a> <a id="web-attach-to-current-tab"></a> **See also** - [Shared Agent APIs](#interaction-methods) for shared agent methods. - [Bridge mode](../bridge-mode) for extension setup, command sequence, and YAML usage. ## Android (`@midscene/android`) {#android} Use this section to configure Midscene's Android automation and review Android-specific constructor options. For shared parameters such as reporting, hooks, and caching, see [Shared Agent APIs](#common). ### Action space `AndroidDevice` provides the following actions to the Midscene Agent: - `Tap` — Tap an element. - `DoubleClick` — Double-tap an element. - `Input` — Enter text with `replace`/`typeOnly`/`clear` modes (`append` is a deprecated alias for `typeOnly`). Supports optional `autoDismissKeyboard`, `keyboardTypeDelay`, and `inputStrategy` parameters. - `Scroll` — Scroll from an element or screen center in any direction, with helpers to reach the top, bottom, left, or right. - `DragAndDrop` — Drag from one element to another. - `KeyboardPress` — Press a specified key. - `LongPress` — Long-press a target element with optional duration. - `PullGesture` — Pull up or down, for example to refresh, with optional distance and duration. - `Pinch` — Two-finger pinch gesture. Use `scale > 1` to zoom in, `scale < 1` to zoom out. - `ClearInput` — Clear the contents of an input field. - `Launch` — Open a web URL or `package/.Activity` string. - `Terminate` — Force-stop an app by package name. - `RunAdbShell` — Execute raw `adb shell` commands. This action is enabled by default. Set `exposeRunAdbShellAction` to `false` to remove it from the action space. - `AndroidBackButton` — Trigger the system back action. - `AndroidHomeButton` — Return to the home screen. - `AndroidRecentAppsButton` — Open the multitasking/recent apps view. ### AndroidDevice {#androiddevice} Create a connection to a device available through `adb`. **Import** ```ts import { AndroidDevice, getConnectedDevices, getConnectedDevicesWithDetails, } from '@midscene/android'; ``` **Constructor** ```ts const device = new AndroidDevice(deviceId, { // device options... }); ``` **Device options** - `deviceId: string` — Value returned by `adb devices` or `getConnectedDevices()`. - `autoDismissKeyboard?: boolean` — Whether to hide the on-screen keyboard after text input. Default: `true`. - `keyboardDismissStrategy?: 'esc-first' | 'back-first'` — Key order used to dismiss the on-screen keyboard. Default: `'esc-first'`. - `keyboardTypeDelay?: number` — Finite non-negative delay in milliseconds between keystrokes. In `'legacy'` mode, a positive value makes the native `input text` path enter one Unicode code point at a time and remains ignored by yadb, preserving existing behavior. Set `inputStrategy` to `'sequential'` to split yadb input too. - `inputStrategy?: 'legacy' | 'sequential' | 'bulk'` — Default text input strategy for `Input` actions. Use `'sequential'` for one ADB/yadb call per code point or `'bulk'` for one backend text call where the selected IME supports it. `'bulk'` requires `keyboardTypeDelay` to be omitted or set to `0`; use `'sequential'` for delayed input. Default: `'legacy'`. - `androidAdbPath?: string` — Custom path to the adb executable. - `remoteAdbHost?: string` / `remoteAdbPort?: number` — Point to a remote adb server. - `imeStrategy?: 'always-yadb' | 'yadb-for-non-ascii'` — When to invoke [yadb](https://github.com/ysbing/yadb) for text input. Default: `'yadb-for-non-ascii'`. - `'yadb-for-non-ascii'` — Uses yadb for accented Latin characters such as ö, é, and ñ; Chinese and Japanese text; and format specifiers such as `%s` and `%d`. Pure ASCII text uses the faster native `adb input text`. - `'always-yadb'` — Uses yadb for all text input. This provides maximum compatibility but is slightly slower for pure ASCII text. - `screenshotStrategy?: 'auto' | 'always-yadb'` — Controls the screenshot method. Default: `'auto'`. - `'auto'` (default) — Tries `adb.takeScreenshot`, falls back to shell `screencap`, and uses the yadb tool if `screencap` fails. scrcpy is tried first only when `scrcpyConfig.enabled` is set. It does not analyze screenshot content — even if a method succeeds but returns a black frame, it will not automatically switch to yadb; it only moves to the next method when the previous one fails to execute. - `'always-yadb'` — Bypasses the default `auto` flow (`adb.takeScreenshot`, `screencap`, and scrcpy when enabled) and captures directly via [yadb](https://github.com/ysbing/yadb). Use this when `screencap` produces black frames for secure pages (`FLAG_SECURE`) while yadb captures them correctly — whether that is possible depends on the Android version, ROM, root/hook environment, and device configuration, so verify on your actual device. Yadb can only capture the default display (`displayId=0`); combining this strategy with a non-zero `displayId` throws an error. Can also be set via the `MIDSCENE_ANDROID_SCREENSHOT_STRATEGY=always-yadb` environment variable. - `displayId?: number` — Target a specific virtual display if the device mirrors multiple displays. - `exposeRunAdbShellAction?: boolean` — Whether to expose the built-in `RunAdbShell` action in the action space. Default: `true`. Set it to `false` to prevent the AI planner, YAML scripts, and `agent.runAdbShell()` from executing ADB shell commands. - `customActions?: DeviceAction[]` — Add additional custom actions so the Agent can call your domain-specific actions. - `screenshotResizeScale?: number` — **Deprecated.** This option has been removed and no longer has any effect. Use `screenshotShrinkFactor` in `AgentOpt` instead to control screenshot size sent to the AI model. - `minScreenshotBufferSize?: number` — Minimum valid screenshot buffer size in bytes. Smaller buffers are treated as failed or corrupted captures. Set to `0` to skip this size check; Midscene still rejects empty buffers and invalid image formats. Default: `1024` (1 KB). - `alwaysRefreshScreenInfo?: boolean` — Whether to query rotation and screen size before every step. Default: `false`. <a id="scrcpy"></a> - `scrcpyConfig?: object` — High-performance scrcpy screenshot configuration: - `enabled?: boolean` — Whether to enable scrcpy screenshots. Default: `false`. - `maxSize?: number` — Maximum screenshot width or height in pixels. The same limit applies to ADB/yadb fallback screenshots while scrcpy is temporarily unavailable, preventing planning and report images from returning to full device resolution. Use a non-negative integer; `0` disables scaling. Default: `0`. - `videoBitRate?: number` — scrcpy H.264 encoding bitrate in bits per second. Default: `100000000`. Changing it trades encoded bandwidth against screenshot detail. Tune it only from independent transport measurements and validate recognition quality; a freshness-timeout warning alone is not a reason to change it. - `idleTimeoutMs?: number` — Idle time before disconnecting the scrcpy stream. `0` disables automatic disconnection. Default: `30000`. `device.getScrcpyStatus()` returns the current `enabled`, `connected`, `lastError`, and `retryAfter` state. `device.retryScrcpy()` immediately retries the scrcpy connection and returns `Promise<void>`. **Usage notes** - Discover devices with `getConnectedDevices()`; the `udid` matches `adb devices`. - Midscene supports remote `adb` through `remoteAdbHost` and `remoteAdbPort`. Set `androidAdbPath` if `adb` is not on `PATH`. - Use `screenshotShrinkFactor` in `AgentOpt` to reduce screenshot processing cost on high-DPI devices. <a id="android-device-destroy"></a> **`destroy()`** ```ts function destroy(): Promise<void>; ``` Release resources owned by this `AndroidDevice`, including an active scrcpy connection, and clear its ADB state. The method is idempotent. After it resolves, this Device instance cannot execute more ADB commands. The physical device remains connected to the ADB server. Calling [`AndroidAgent.destroy()`](#agentdestroy) invokes this method automatically. Each `AndroidDevice` instance belongs to exactly one `AndroidAgent`. ### AndroidAgent {#androidagent} Connect Midscene's AI planner to an `AndroidDevice`. **Import** ```ts import { AndroidAgent } from '@midscene/android'; ``` **Constructor** ```ts const agent = new AndroidAgent(device, { // common agent options... }); ``` **Android-specific options** - `appNameMapping?: Record<string, string>` — Map friendly app names to package names. When you pass an app name to `launch(target)`, the agent will look up the package name in this mapping. If no mapping is found, it will attempt to launch `target` as-is. User-provided mappings take precedence over default mappings. - All other fields match the [common constructor parameters](#common-parameters), including `generateReport`, `reportFileName`, `aiActContext`, `modelConfig`, `cache`, `createOpenAIClient`, and `onTaskStartTip`. **Usage notes** :::info - Use one agent per device connection. - `customActions` adds additional custom actions to `AndroidDevice`. Pass it to the Device constructor or to `agentFromAdbDevice()`. - Android-only helpers such as `launch`, `terminate`, and `runAdbShell` are also exposed in YAML scripts. See [Android platform-specific actions](../automate-with-scripts-in-yaml#the-android-part). - For shared interaction methods, see [Shared Agent APIs](#interaction-methods). ::: **Android-specific methods** <a id="android-agentlaunch"></a> **`agent.launch()`** Launch a web URL, Android activity, or app package. ```ts function launch(target: string): Promise<void>; ``` - `target: string` — Web URL, `package/.Activity` string such as `com.android.settings/.Settings`, package name, or app name. If `appNameMapping` contains the app name, Midscene resolves it to the mapped package; otherwise, it launches `target` as provided. <a id="android-agentrunadbshell"></a> **`agent.runAdbShell()`** Run a raw `adb shell` command through the connected device. Pass only the shell command itself, without the `adb shell` prefix. ```ts function runAdbShell(command: string, opt?: { timeout?: number }): Promise<string>; ``` - `command: string` — Command passed verbatim to `adb shell`. For example, use `input tap 100 200`, not `adb shell input tap 100 200`. - `opt.timeout?: number` — Optional command execution timeout in milliseconds. This method invokes the `RunAdbShell` action. If `exposeRunAdbShellAction` is set to `false` when creating the `AndroidDevice`, this method is unavailable. ```ts const result = await agent.runAdbShell('dumpsys battery', { timeout: 60 * 1000 }); console.log(result); await agent.runAdbShell('input tap 100 200'); ``` <a id="android-agentterminate"></a> **`agent.terminate()`** Terminate (force-stop) a running Android app. ```ts function terminate(uri: string): Promise<void>; ``` - `uri: string` — Package name, app name in `appNameMapping`, or `package/.Activity` (only the package part is used). ```ts await agent.terminate('com.android.settings'); ``` <a id="android-navigation-helpers"></a> **Navigation helpers** - `agent.back(): Promise<void>` — Trigger the Android system back action. - `agent.home(): Promise<void>` — Return to the launcher. - `agent.recentApps(): Promise<void>` — Open the recent apps screen. ### Factory functions and utilities <a id="android-agentfromadbdevice"></a> **`agentFromAdbDevice()`** Create an `AndroidAgent` from a connected `adb` device. ```ts function agentFromAdbDevice( deviceId?: string, opts?: AndroidAgentOpt & AndroidDeviceOpt, ): Promise<AndroidAgent>; ``` - `deviceId?: string` — Connect to a specific device. Omit this value to use the first available device. - `opts?: AndroidAgentOpt & AndroidDeviceOpt` — Agent options and [AndroidDevice](#androiddevice) settings. When you omit `deviceId`, Midscene discovers a device through adb. Set `androidAdbPath`, `remoteAdbHost`, and `remoteAdbPort` in `opts` to choose the adb configuration for device discovery and connection. <a id="android-getconnecteddevices"></a> **`getConnectedDevices()`** List the `adb` devices that Midscene can drive. ```ts function getConnectedDevices( deviceOptions?: AndroidDeviceOpt, ): Promise<Array<{ udid: string; state: string; port?: number; }>>; ``` `deviceOptions` is optional. Omit it to use Midscene's default adb configuration. Set `androidAdbPath` to choose an adb executable, or set `remoteAdbHost` and `remoteAdbPort` to connect to a remote adb server: ```ts const devices = await getConnectedDevices({ androidAdbPath: '/absolute/path/to/adb', remoteAdbHost: '192.168.1.10', remoteAdbPort: 5038, }); ``` <a id="android-getconnecteddeviceswithdetails"></a> **`getConnectedDevicesWithDetails()`** This function works like `getConnectedDevices()` and also returns the device brand, model, resolution, and screen density. Unavailable fields are `undefined`. ```ts function getConnectedDevicesWithDetails( deviceOptions?: AndroidDeviceOpt, ): Promise<Array<{ udid: string; state: string; port?: number; model?: string; brand?: string; resolution?: string; density?: number; }>>; ``` <a id="android-quick-start"></a> <a id="android-launch-native-packages"></a> **See also** - [Android getting started](../platforms/android) for setup and scripting steps. ## iOS (`@midscene/ios`) {#ios} Use this section to configure iOS device behavior, integrate Midscene with WebDriverAgent workflows, and troubleshoot WDA requests. For shared parameters such as reporting, hooks, and caching, see [Shared Agent APIs](#common). ### Action space `IOSDevice` provides the following actions to the Midscene Agent: - `Tap` — Tap an element. - `DoubleClick` — Double-tap an element. - `Input` — Enter text with `replace`/`typeOnly`/`clear` modes (`append` is a deprecated alias for `typeOnly`). Supports optional `autoDismissKeyboard`, `keyboardTypeDelay`, and `inputStrategy` parameters. - `Scroll` — Scroll from an element or screen center in any direction, including scroll-to-top/bottom/left/right helpers. - `DragAndDrop` — Drag from one element to another. - `KeyboardPress` — Press a specified key. - `LongPress` — Long-press a target element with optional duration. - `Pinch` — Two-finger pinch gesture. Use `scale > 1` to zoom in, `scale < 1` to zoom out. - `ClearInput` — Clear the contents of an input field. - `Launch` — Open a URL, bundle identifier, or URL scheme. - `Terminate` — Close a running iOS app by its bundle identifier. - `RunWdaRequest` — Call WebDriverAgent REST endpoints directly. - `IOSHomeButton` — Trigger the iOS system Home action. - `IOSAppSwitcher` — Open the iOS multitasking view. ### IOSDevice {#iosdevice} Create a WebDriverAgent-backed instance that an IOSAgent can drive. **Import** ```ts import { IOSDevice } from '@midscene/ios'; ``` **Constructor** ```ts const device = new IOSDevice({ // device options... }); ``` **Device options** - `wdaPort?: number` — WebDriverAgent port. Default: `8100`. - `wdaHost?: string` — WebDriverAgent host. Default: `'localhost'`. - `iOSDeviceClassOverride?: string` — Optional npm module path that replaces the default `IOSDevice` when using `agentFromWebDriverAgent()` or iOS Playground. The module must export an `IOSDevice` class or a default class. - `sessionId?: string` — Existing WebDriverAgent session ID to reuse. When provided, Midscene skips creating a new WDA session. During cleanup, Midscene detaches from the externally supplied WebDriver session instead of deleting it. - `wdaMjpegPort?: number` — WDA MJPEG server port for real-time screen streaming. Default: `9100`. - `wdaMjpegFrameSource?: { enabled?: boolean }` — Use WDA's MJPEG stream as the continuous frame source for `agent.startObserving()`. Disabled by default; when disabled, observers fall back to sequential `screenshotBase64()` capture. - `autoDismissKeyboard?: boolean` — Whether to hide the on-screen keyboard after text input. Default: `true`. - `keyboardTypeDelay?: number` — Finite non-negative delay in milliseconds between keystrokes. A positive value makes legacy input enter one Unicode code point at a time through WDA's `/wda/keys` endpoint. Use this option when an input field drops characters during fast input. - `inputStrategy?: 'legacy' | 'sequential' | 'bulk'` — Default text input strategy for `Input` actions. Use `'sequential'` for one WDA call per code point or `'bulk'` for one WDA text call. `'bulk'` requires `keyboardTypeDelay` to be omitted or set to `0`; use `'sequential'` for delayed input. Default: `'legacy'`. - `customActions?: DeviceAction<any>[]` — Add additional custom actions so the Agent can call your domain-specific actions. **Usage notes** - Ensure Developer Mode is enabled and WDA can reach the device; use `iproxy` when forwarding ports from a real device. - Use `wdaHost`/`wdaPort` to target remote devices or custom WDA deployments. - For multi-device concurrency, use distinct `wdaPort` and `wdaMjpegPort` values for each device so WDA commands and MJPEG streams do not conflict. - For shared interaction methods, see [Shared Agent APIs](#interaction-methods). <a id="ios-device-destroy"></a> **`destroy()`** ```ts function destroy(): Promise<void>; ``` Attempt to stop the MJPEG frame source, delete the WebDriverAgent session, and stop the WDA manager. Cleanup failures are logged and do not reject the Promise. The method is idempotent. After it resolves, this `IOSDevice` instance cannot be reused. Calling [`IOSAgent.destroy()`](#agentdestroy) invokes this method automatically. Each `IOSDevice` instance belongs to exactly one `IOSAgent`. ### IOSAgent {#iosagent} Connect Midscene's AI planner to an `IOSDevice` through WebDriverAgent. **Import** ```ts import { IOSAgent } from '@midscene/ios'; ``` **Constructor** ```ts const agent = new IOSAgent(device, { // common agent options... }); ``` **iOS-specific options** - `appNameMapping?: Record<string, string>` — Map friendly app names to bundle identifiers. When you pass an app name to `launch(target)` or `terminate(bundleId)`, the agent will look up the bundle ID in this mapping. If no mapping is found, it will attempt to use `target` as-is. User-provided mappings take precedence over default mappings. - All other fields match the [common constructor parameters](#common-parameters), including `generateReport`, `reportFileName`, `aiActContext`, `modelConfig`, `cache`, `createOpenAIClient`, and `onTaskStartTip`. **Usage notes** :::info - Use one agent per device connection. - `customActions` adds additional custom actions to `IOSDevice`. Pass it to the Device constructor or to `agentFromWebDriverAgent()`. - iOS-only helpers such as `launch`, `terminate`, and `runWdaRequest` are also exposed in YAML scripts. See [iOS platform-specific actions](../automate-with-scripts-in-yaml#the-ios-part). - For shared interaction methods, see [Shared Agent APIs](#interaction-methods). ::: **iOS-specific methods** <a id="ios-agentlaunch"></a> **`agent.launch()`** Launch a web URL, app bundle identifier, or custom URL scheme. ```ts function launch(target: string): Promise<void>; ``` - `target: string` — Web URL, bundle identifier, URL scheme such as `tel:` or `mailto:`, or app name. If `appNameMapping` contains the app name, Midscene resolves it to the mapped bundle identifier; otherwise, it launches `target` as provided. ```ts await agent.launch('https://www.apple.com'); await agent.launch('com.apple.Preferences'); await agent.launch('myapp://profile/user/123'); await agent.launch('tel:+1234567890'); ``` <a id="ios-agentterminate"></a> **`agent.terminate()`** Terminate (close) a running iOS app by its bundle ID. ```ts function terminate(bundleId: string): Promise<void>; ``` - `bundleId: string` — Bundle identifier of the app to terminate, such as `com.apple.Preferences`. If you pass an app name that exists in `appNameMapping`, Midscene resolves it to the mapped Bundle ID. ```ts await agent.terminate('com.apple.Preferences'); await agent.terminate('com.apple.mobilesafari'); ``` <a id="ios-agentrunwdarequest"></a> **`agent.runWdaRequest()`** Send raw requests to WebDriverAgent REST endpoints when you need low-level control. ```ts function runWdaRequest(params: { method: 'GET' | 'POST' | 'DELETE' | 'PUT'; endpoint: string; data?: Record<string, any>; }): Promise<any>; ``` - `params.method` — HTTP verb. Supported values: `GET`, `POST`, `DELETE`, and `PUT`. - `params.endpoint` — WebDriverAgent endpoint path. - `params.data` — Optional JSON body. ```ts const screen = await agent.runWdaRequest({ method: 'GET', endpoint: '/wda/screen', }); await agent.runWdaRequest({ method: 'POST', endpoint: '/session/test/wda/pressButton', data: { name: 'home' }, }); ``` When calling `IOSDevice.runWdaRequest()` directly on the device instance, use the positional signature `runWdaRequest(method, endpoint, data?)`. <a id="ios-navigation-helpers"></a> **Navigation helpers** - `agent.home(): Promise<void>` — Return to the Home screen. - `agent.appSwitcher(): Promise<void>` — Open the app switcher. ### Factory functions and utilities <a id="agentfromwebdriveragent"></a> **`agentFromWebDriverAgent()`** Connect to WebDriverAgent and return a ready-to-use IOSAgent. ```ts function agentFromWebDriverAgent( opts?: IOSAgentOpt & IOSDeviceOpt, ): Promise<IOSAgent>; ``` - `opts?: IOSAgentOpt & IOSDeviceOpt` — Combine iOS Agent options with [`IOSDevice`](#iosdevice) settings. - Set `MIDSCENE_IOS_DEVICE_CLASS_OVERRIDE` to apply the same device class override through the environment. An explicit option takes precedence over the environment variable. ```ts import { agentFromWebDriverAgent } from '@midscene/ios'; const agent = await agentFromWebDriverAgent({ wdaHost: 'localhost', wdaPort: 8100, iOSDeviceClassOverride: '@your-scope/ios-device', aiActContext: 'Accept permission dialogs automatically.', }); ``` <a id="ios-quick-start"></a> <a id="ios-custom-host-and-port"></a> **See also** - [iOS getting started](../platforms/ios) for setup and scripting steps. - [Integrate with any interface](../integrate-with-any-interface#define-a-custom-action) for custom actions and schemas. ## HarmonyOS (`@midscene/harmony`) {#harmonyos} Use this section to configure HarmonyOS device behavior, integrate Midscene with another framework, and troubleshoot HDC issues. For shared parameters such as reporting, hooks, and caching, see [Shared Agent APIs](#common). ### Action space `HarmonyDevice` provides the following actions to the Midscene Agent: - `Tap` — Tap an element. - `DoubleClick` — Double-tap an element. - `Input` — Enter text with `replace`, `typeOnly`, or `clear` mode. - `Scroll` — Scroll from an element or the screen center in any direction, including scroll-to-top, bottom, left, and right helpers. - `DragAndDrop` — Drag from one element to another. - `KeyboardPress` — Press a specific key. - `LongPress` — Long-press a target element with an optional duration. - `ClearInput` — Clear the contents of an input field. - ~~`Pinch`~~ — Not supported. The HarmonyOS `uitest` framework does not provide multi-touch input APIs. - `Launch` — Open a HarmonyOS app (bundle name). - `Terminate` — Force-stop a HarmonyOS app by bundle name. - `RunHdcShell` — Execute a raw `hdc shell` command. - `HarmonyBackButton` — Trigger the system back action. - `HarmonyHomeButton` — Return to the home screen. - `HarmonyRecentAppsButton` — Open the recent apps screen. ### HarmonyDevice {#harmonydevice} Create an HDC-backed device instance that a `HarmonyAgent` can drive. **Import** ```ts import { HarmonyDevice, getConnectedDevices } from '@midscene/harmony'; ``` **Constructor** ```ts const device = new HarmonyDevice(deviceId, { // device options... }); ``` **Device options** - `deviceId: string` — Value from `hdc list targets` or `getConnectedDevices()`. - `hdcPath?: string` — Custom path to the HDC executable. If omitted, Midscene checks the `HDC_HOME` environment variable and common installation paths. - `autoDismissKeyboard?: boolean` — Whether to hide the on-screen keyboard after text input. Default: `true`. - `keyboardDismissStrategy?: 'esc-first' | 'back-first'` — Key used to dismiss the on-screen keyboard. `'esc-first'` sends Escape, while `'back-first'` sends Back. Default: `'esc-first'`. - `keyboardTypeDelay?: number` — Finite non-negative delay in milliseconds between keystrokes. A positive value makes legacy input enter one Unicode code point at a time through `uitest uiInput inputText`. Use this option when an input field drops characters during fast input. - `inputStrategy?: 'legacy' | 'sequential' | 'bulk'` — Default text input strategy for `Input` actions. Use `'sequential'` for one HDC call per code point or `'bulk'` for one HDC text call. `'bulk'` requires `keyboardTypeDelay` to be omitted or set to `0`; use `'sequential'` for delayed input. Default: `'legacy'`. - `screenshotResizeScale?: number` — **Deprecated.** This option has been removed and no longer has any effect. Use `screenshotShrinkFactor` in `AgentOpt` instead to control screenshot size sent to the AI model. - `customActions?: DeviceAction[]` — Add additional custom actions so the Agent can call your domain-specific actions. **Usage notes** - Use `getConnectedDevices()` to discover devices. The returned `deviceId` matches `hdc list targets` output. - If HDC is not in your system PATH, specify it via the `HDC_HOME` environment variable or the `hdcPath` option. <a id="harmony-device-destroy"></a> **`destroy()`** ```ts function destroy(): Promise<void>; ``` Release the HDC state and cached screen information owned by this `HarmonyDevice`. The method is idempotent. After it resolves, this Device instance cannot execute more HDC commands. The physical device remains connected to HDC. Calling [`HarmonyAgent.destroy()`](#agentdestroy) invokes this method automatically. Each `HarmonyDevice` instance belongs to exactly one `HarmonyAgent`. ### HarmonyAgent {#harmonyagent} Connect Midscene's AI planner to a `HarmonyDevice`. **Import** ```ts import { HarmonyAgent } from '@midscene/harmony'; ``` **Constructor** ```ts const agent = new HarmonyAgent(device, { appNameMapping: { Video: 'com.example.video/PhoneAbility', }, }); ``` **HarmonyOS-specific options** - `appNameMapping?: Record<string, string>` — Map friendly app names to bundle names or explicit `bundle/Ability` targets. Bundle-only values use bundle metadata to resolve the declared launch ability; explicit targets bypass that lookup. When you pass an app name to `launch(target)`, the Agent uses the mapped target when available; otherwise, it launches `target` as provided. - All other fields match the [common constructor parameters](#common-parameters), including `generateReport`, `reportFileName`, `aiActContext`, `modelConfig`, `cache`, `createOpenAIClient`, and `onTaskStartTip`. **Usage notes** :::info - Use one Agent per device connection. - `customActions` adds additional custom actions to `HarmonyDevice`. Pass it to the Device constructor or to `agentFromHdcDevice()`. - HarmonyOS-specific helpers like `launch`, `terminate`, and `runHdcShell` can also be used in YAML scripts. See [HarmonyOS platform-specific actions](../automate-with-scripts-in-yaml#the-harmony-part). - For shared interaction methods, see [Shared Agent APIs](#interaction-methods). ::: **HarmonyOS-specific methods** <a id="harmonyos-agentlaunch"></a> **`agent.launch()`** Launch a HarmonyOS app. ```ts function launch(uri: string): Promise<void>; ``` - `uri: string` — App bundle name, explicit `bundle/Ability` target, app name registered in `appNameMapping`, or HTTP/HTTPS URL. Midscene opens HTTP and HTTPS URLs in the browser. ```ts await agent.launch('com.huawei.hmos.settings'); // Open Settings await agent.launch('com.huawei.hmos.camera'); // Open Camera await agent.launch('Video'); // Open the mapped explicit ability ``` <a id="harmonyos-agentrunhdcshell"></a> **`agent.runHdcShell()`** Run a raw `hdc shell` command on the connected device. ```ts function runHdcShell(command: string): Promise<string>; ``` - `command: string` — The command passed directly to `hdc shell`. ```ts const result = await agent.runHdcShell('hidumper -s RenderService -a screen'); console.log(result); ``` <a id="harmonyos-agentterminate"></a> **`agent.terminate()`** Terminate (force-stop) a running HarmonyOS app. ```ts function terminate(uri: string): Promise<void>; ``` - `uri: string` — Bundle name, app name in `appNameMapping`, or `bundle/Ability` (only the bundle part is used). ```ts await agent.terminate('com.huawei.hmos.settings'); ``` <a id="harmonyos-navigation-helpers"></a> **Navigation helpers** - `agent.back(): Promise<void>` — Trigger the HarmonyOS system back action. - `agent.home(): Promise<void>` — Return to the home screen. - `agent.recentApps(): Promise<void>` — Open the recent apps screen. ### Factory functions and utilities <a id="harmonyos-agentfromhdcdevice"></a> **`agentFromHdcDevice()`** Create a `HarmonyAgent` from any connected HDC device. ```ts function agentFromHdcDevice( deviceId?: string, opts?: HarmonyAgentOpt & HarmonyDeviceOpt, ): Promise<HarmonyAgent>; ``` - `deviceId?: string` — Connect to a specific device. Omit this value to use the first available device. - `opts?: HarmonyAgentOpt & HarmonyDeviceOpt` — Merge Agent options and [`HarmonyDevice`](#harmonydevice) settings in a single object. ```ts import { agentFromHdcDevice } from '@midscene/harmony'; const agent = await agentFromHdcDevice('0123456789ABCDEF'); // specific device // Or use the first available device: // const agent = await agentFromHdcDevice(); ``` <a id="harmonyos-getconnecteddevices"></a> **`getConnectedDevices()`** List HDC devices that Midscene can drive. ```ts function getConnectedDevices( hdcPath?: string, ): Promise<Array<{ deviceId: string }>>; ``` ```ts import { getConnectedDevices } from '@midscene/harmony'; const devices = await getConnectedDevices(); console.log(devices); // [{ deviceId: '0123456789ABCDEF' }] ``` <a id="harmonyos-quick-start"></a> <a id="harmonyos-launch-apps"></a> **Related reading** - [HarmonyOS getting started](../platforms/harmonyos) for setup and script examples. ## Desktop (`@midscene/computer`) {#desktop} This section documents the desktop-specific APIs provided by `@midscene/computer`. For common APIs that work across all platforms, see [Common API reference](#common). ### Agent factory functions `agentForComputer(opts?): Promise<ComputerAgent>` Create an agent for local desktop automation. > Backward compatibility: `agentFromComputer` is still available as an alias. `agentForRDPComputer(opts): Promise<ComputerAgent<RDPDevice>>` Create an agent for remote Windows desktop automation over RDP. **Parameters** ```typescript interface BaseComputerAgentOpt { // Agent options (inherited from AgentOpt) aiActContext?: string; cache?: false | CacheConfig; // ... other AgentOpt properties customActions?: DeviceAction<any>[]; keyboardTypeDelay?: number; inputStrategy?: 'legacy' | 'sequential' | 'bulk'; } interface LocalComputerAgentOpt extends BaseComputerAgentOpt { // Local desktop options displayId?: string; keyboardDriver?: 'applescript' | 'libnut'; headless?: boolean; xvfbResolution?: string; } interface RDPComputerAgentOpt extends BaseComputerAgentOpt { host: string; port?: number; username?: string; password?: string; domain?: string; localAddress?: string; adminSession?: boolean; ignoreCertificate?: boolean; securityProtocol?: 'auto' | 'tls' | 'nla' | 'rdp'; desktopWidth?: number; desktopHeight?: number; } ``` **Local desktop options** - `displayId?: string` — Display to control. Use `ComputerDevice.listDisplays()` to list the available displays. Default: the primary display. - `customActions?: DeviceAction<any>[]` — Add additional custom actions so the Agent can call your domain-specific actions. - `keyboardDriver?: 'applescript' | 'libnut'` — On macOS, keyboard event backend. Use `'applescript'` for the default, more compatible behavior or `'libnut'` for faster input when the target application supports it. Default: `'applescript'`. - `headless?: boolean` — On Linux, set this to `true` to start a virtual display with [Xvfb](https://www.x.org/releases/X11R7.6/doc/man/man1/Xvfb.1.xhtml). This enables desktop automation on headless servers and in CI environments. You can also set the `MIDSCENE_COMPUTER_HEADLESS_LINUX=true` environment variable. Default: `false`. - `xvfbResolution?: string` — Resolution of the Xvfb virtual display. Default: `'1920x1080x24'`. **Keyboard input options** - `keyboardTypeDelay?: number` — Finite non-negative minimum delay in milliseconds between keystrokes. In `'legacy'` mode, a positive value makes local and RDP Computer Agents emit text one Unicode code point at a time instead of using the default burst input. A local Computer Agent uses real key events in this mode; when this option is omitted or set to `0`, legacy input uses clipboard paste to avoid interference from the active IME. An action-level `keyboardTypeDelay` passed to `aiInput()` overrides the Agent-level default. Pass `0` to restore clipboard input for one action. Default: `undefined`. - `inputStrategy?: 'legacy' | 'sequential' | 'bulk'` — Default text input strategy. `'sequential'` uses real key events locally and one backend call per code point over RDP. `'bulk'` uses one clipboard paste locally and one backend call over RDP. `'bulk'` requires `keyboardTypeDelay` to be omitted or set to `0`; use `'sequential'` for delayed input. Default: `'legacy'`. The Agent-level option also applies to `Input` actions generated by `agent.ai()`, so the prompt does not need to call `aiInput()` explicitly: ```typescript const agent = await agentForComputer({ keyboardTypeDelay: 80 }); await agent.ai('Type the account details into the form'); // Override the default for one deterministic input action. await agent.aiInput('the notes field', { value: 'Pasted as one value', keyboardTypeDelay: 0, inputStrategy: 'bulk', }); ``` **RDP options** - `host: string` — Remote Windows hostname or IP address. - `port?: number` — RDP port. Default: `3389`. - `username?: string` / `password?: string` — Credentials for the remote session. - `domain?: string` — Windows domain. Default: `undefined`. - `localAddress?: string` — Local source IP address for the RDP TCP connection. Use this when the machine has multiple outbound routes. Default: `undefined`. - `adminSession?: boolean` — Whether to request the remote admin session when the server allows it. Default: `false`. - `ignoreCertificate?: boolean` — Whether to skip certificate validation, which is useful for self-signed certificates. Default: `false`. - `securityProtocol?: 'auto' | 'tls' | 'nla' | 'rdp'` — RDP security protocol. Default: `'auto'`. - `desktopWidth?: number` / `desktopHeight?: number` — Requested remote desktop resolution. When omitted, the RDP client and server negotiate the session size. :::info Example: Test an Electron app on headless Linux See the complete [Obsidian desktop automation demo](https://github.com/web-infra-dev/midscene-example/tree/main/computer/electron-demo) for a CI example that uses `@midscene/computer`. ::: **Example** ```typescript import { agentForComputer } from '@midscene/computer'; // Connect to primary display const agent = await agentForComputer({ aiActContext: 'You are automating a desktop application.', }); // Connect to specific display const displays = await ComputerDevice.listDisplays(); const agent2 = await agentForComputer({ displayId: displays[1].id, }); ``` **Example: connect to a remote Windows desktop over RDP** ```typescript import { agentForRDPComputer } from '@midscene/computer'; const agent = await agentForRDPComputer({ aiActContext: 'You are controlling a remote Windows desktop over the RDP protocol.', host: '10.75.166.249', port: 3389, username: 'Admin', password: 'replace-with-your-password', // Optional: bind the TCP connection to this local source IP. localAddress: '10.75.166.10', ignoreCertificate: true, }); await agent.aiWaitFor('The remote Windows desktop is visible'); await agent.aiAct('Click the Windows Start button'); await agent.aiAct('Open Settings'); ``` :::info Example: Automate a remote Windows desktop over RDP See the runnable [RDP automation demo](https://github.com/web-infra-dev/midscene-example/tree/main/computer/rdp-demo), which connects to a remote Windows machine, opens Settings, navigates to Windows Update, and generates a structured report. ::: Use `localAddress` only when the machine running Midscene has multiple outbound routes and the RDP server must be reached from a specific local source IP. Pass an IP address, not a network interface name. ### ComputerDevice `ComputerDevice.listDisplays(): Promise<DisplayInfo[]>` List all available displays. **Returns** ```typescript interface DisplayInfo { id: string; name: string; primary?: boolean; } ``` **Example** ```typescript import { ComputerDevice } from '@midscene/computer'; const displays = await ComputerDevice.listDisplays(); console.log('Available displays:', displays); // [ // { id: '0', name: 'Built-in Display', primary: true }, // { id: '1', name: 'External Display', primary: false } // ] ``` `checkComputerEnvironment(): Promise<EnvironmentCheck>` Check if the computer environment is properly configured. **Returns** ```typescript interface EnvironmentCheck { available: boolean; error?: string; platform: string; displays: number; } ``` **Example** ```typescript import { checkComputerEnvironment } from '@midscene/computer'; const env = await checkComputerEnvironment(); console.log('Environment check:', env); if (!env.available) { console.error('Environment error:', env.error); } ``` <a id="computer-device-destroy"></a> **`ComputerDevice.destroy()`** ```typescript function destroy(): Promise<void>; ``` Release the local input driver and stop an Agent-owned Xvfb instance, if one exists. The method is idempotent. After it resolves, this `ComputerDevice` instance cannot be reused. <a id="rdp-device-destroy"></a> **`RDPDevice.destroy()`** ```typescript function destroy(): Promise<void>; ``` Disconnect the RDP backend and clear its connection state. The method is idempotent. After it resolves, this `RDPDevice` instance cannot be reused. Calling [`ComputerAgent.destroy()`](#agentdestroy) invokes the matching Device method automatically. This applies to Agents returned by both `agentForComputer()` and `agentForRDPComputer()`. ### ComputerAgent `ComputerAgent` extends `PageAgent<ComputerDevice>` and inherits the shared Agent methods described in [Shared Agent APIs](#common), including: - `aiAct(action: string)` — Perform an AI-planned action. - `aiQuery(query: string)` — Extract structured information. - `aiAssert(assertion: string)` — Assert a condition. - `aiWaitFor(condition: string)` — Wait for a condition. - `aiLocate(description: string)` — Locate an element. Instant actions provide direct control once Midscene locates an element: - `aiTap()`, `aiDoubleClick()`, `aiRightClick()`, and `aiHover()` — Mouse actions. - `aiInput()`, `aiClearInput()`, and `aiKeyboardPress()` — Keyboard actions. - `aiScroll()` — Scroll actions. ### Action space `ComputerDevice` supports the following actions. **Mouse actions** <a id="desktop-tap-click"></a> **Tap (Click)** Single click at the target location. ```typescript await agent.aiAct('click on the File menu'); await agent.aiAct('click at center of screen'); ``` <a id="desktop-doubleclick"></a> **DoubleClick** Double-click at the target location. ```typescript await agent.aiAct('double-click on the desktop icon'); ``` <a id="desktop-rightclick"></a> **RightClick** Right-click to open context menu. ```typescript await agent.aiAct('right-click on the desktop'); await agent.aiAct('right-click on the file'); ``` <a id="desktop-mousemove-hover"></a> **MouseMove (Hover)** Move the pointer to an element, for example to reveal a hover menu or tooltip. ```typescript // Natural-language form (move mouse / hover) await agent.aiAct('move mouse to the menu item'); // Instant action: locate and hover in one call await agent.aiHover('the menu item "Products"'); ``` <a id="desktop-draganddrop"></a> **DragAndDrop** Drag from one location and drop at another. ```typescript await agent.aiAct('drag the file to the folder'); ``` **Keyboard actions** <a id="desktop-keyboardpress"></a> **KeyboardPress** Press keyboard keys with optional modifiers. **Supported keys** - Regular keys: `a-z`, `0-9`, `Enter`, `Escape`, `Space`, `Tab`, etc. - Arrow keys: `ArrowUp`, `ArrowDown`, `ArrowLeft`, `ArrowRight` - Function keys: `F1`-`F12` - Modifiers: `Command`/`Cmd` (macOS), `Control`/`Ctrl`, `Alt`, `Shift`, `Win` (Windows) - Media keys: `VolumeUp`, `VolumeDown`, `Mute`, etc. **Examples** ```typescript // Simple key press await agent.aiAct('press Enter'); await agent.aiAct('press Escape'); // Key combinations (platform-specific) if (process.platform === 'darwin') { // macOS await agent.aiAct('press Cmd+Space'); // Open Spotlight await agent.aiAct('press Cmd+Tab'); // App switcher await agent.aiAct('press Cmd+C'); // Copy await agent.aiAct('press Cmd+V'); // Paste } else { // Windows/Linux await agent.aiAct('press Windows key'); // Start menu await agent.aiAct('press Alt+Tab'); // App switcher await agent.aiAct('press Ctrl+C'); // Copy await agent.aiAct('press Ctrl+V'); // Paste } // Arrow keys await agent.aiAct('press ArrowDown'); await agent.aiAct('press ArrowUp'); // Function keys await agent.aiAct('press F5'); // Refresh ``` <a id="desktop-input"></a> **Input** Type text into an input field. ```typescript await agent.aiAct('type "Hello World" in the search box'); await agent.aiAct('type "my-document.txt"'); ``` <a id="desktop-clearinput"></a> **ClearInput** Clear the content of an input field. ```typescript await agent.aiAct('clear the text field'); ``` **Scroll actions** Scroll the screen or a specific area. ```typescript // Scroll directions await agent.aiAct('scroll down'); await agent.aiAct('scroll up'); await agent.aiAct('scroll left'); await agent.aiAct('scroll right'); // Scroll to positions await agent.aiAct('scroll to top'); await agent.aiAct('scroll to bottom'); ``` **Display actions** <a id="desktop-listdisplays"></a> **ListDisplays** Get information about all connected displays. ```typescript const displays = await ComputerDevice.listDisplays(); ``` With RDP, `ListDisplays` returns the current remote session as a single display. ## Runtime configuration {#runtime-configuration} These environment variables control global runtime behavior rather than model requests. They are not supported in an Agent's `modelConfig` object. | Name | Type | Default | Description | | --- | --- | --- | --- | | `MIDSCENE_RUN_DIR` | string | `midscene_run` | Directory for reports, logs, model-call records, and other run artifacts. Accepts an absolute path or a path relative to the current working directory. | | `MIDSCENE_PREFERRED_LANGUAGE` | string | Automatically determined from the time zone: `Chinese` if the time zone is `Asia/Shanghai`; otherwise `English` | Preferred language for relevant model responses. This setting guides the model through prompts rather than enforcing the language; actual output may not fully comply depending on the model's capabilities and context. | | `MIDSCENE_PLAYGROUND_HOST` | string | `127.0.0.1` | Network interface used by the Playground server. Set it to a reachable interface address when a remote device, virtual machine, container, or another computer needs to connect. | | `DEBUG` | string | Unset | Enables additional debug log namespaces. See [Debug logs](#debug-logs) for supported selectors. | Setting `MIDSCENE_PLAYGROUND_HOST=0.0.0.0` listens on all network interfaces. Use this value only on a trusted network. ### Debug logs {#debug-logs} Set `DEBUG` to one of the following selectors: | Value | Description | | --- | --- | | `midscene:ai:profile:stats` | Prints model latency and Token usage in a comma-separated format. | | `midscene:ai:profile:detail` | Prints detailed Token usage logs. | | `midscene:ai:call` | Prints AI response details. | | `midscene:android:adb` | Prints Android ADB command details. | | `midscene:*` | Prints all Midscene Debug logs. | Midscene saves logs under `<MIDSCENE_RUN_DIR>/log` even when `DEBUG` is unset. Debug logs may contain model inputs, outputs, or screenshots. Review them before sharing, and do not commit them to a source repository. For model connectivity checks, call recording, and Tracing integrations, see [Model debugging and observability](../model-debugging-observability). **See also** - [Common API reference](#common) — APIs that work across all platforms - [Model configuration](../model-config) — Configure AI models - [Caching](../caching) — Improve performance with caching --- url: /showcases-android.md --- **Prompt** : Open the Booking App, search for a hotel in Tokyo for four adults on Christmas, with a score of 8 or above. <video src="https://lf3-static.bytednsdoc.com/obj/eden-cn/nupipfups/Midscene/1.0-showcases/booking2.mp4" poster="https://lf3-static.bytednsdoc.com/obj/eden-cn/nupipfups/Midscene/1.0-showcases/booking.png" height="300" controls /> View the full report of this task: [report.html](https://lf3-static.bytednsdoc.com/obj/eden-cn/nupipfups/Midscene/1.0-showcases/booking.html) --- url: /showcases-computer.md --- **macOS** **Prompt:** Help me post a tweet promoting Midscene's support for AutoGLM through safari, with the following requirements: 1. Text content: Midscene now supports AutoGLM! 2. Media content: Use the AutoGLM video from the download folder! <video src="https://lf3-static.bytednsdoc.com/obj/eden-cn/nupipfups/Midscene/pc-twitter2.mp4" height="300" controls /> View the full report for this task: [report.html](https://lf3-static.bytednsdoc.com/obj/eden-cn/nupipfups/Midscene/1.0-showcases/pc-twitter2-midscene_report.html) **Prompt:** Open Google and query San Jose tomorrow weather temperature <video src="https://lf3-static.bytednsdoc.com/obj/eden-cn/nupipfups/Midscene/mac.mov" height="300" controls /> View the full report for this task: [report.html](https://lf3-static.bytednsdoc.com/obj/eden-cn/nupipfups/Midscene/weather-computer-2026-01-14_11-26-38-9592ecf5.html) **Windows** **Prompt:** Open Sauce Demo e-commerce site, login and add items to cart <video src="https://lf3-static.bytednsdoc.com/obj/eden-cn/nupipfups/Midscene/windows.mov" height="300" controls /> View the full report for this task: [report.html](https://lf3-static.bytednsdoc.com/obj/eden-cn/nupipfups/Midscene/shop-computer-2026-01-14_11-57-36-f8411b8f.html) **Linux** **Prompt:** Open TodoMVC, add multiple tasks and filter them <video src="https://lf3-static.bytednsdoc.com/obj/eden-cn/nupipfups/Midscene/linux.mov" height="300" controls /> View the full report for this task: [report.html](https://lf3-static.bytednsdoc.com/obj/eden-cn/nupipfups/Midscene/todo-computer-2026-01-13_15-40-37-6f45fb0f.html) --- url: /showcases-harmony.md --- **Prompt** : Open Settings, scroll to find "About phone", view device information. <video src="https://lf3-static.bytednsdoc.com/obj/eden-cn/nupipfups/Midscene/1.0-showcases/harmony.mp4" poster="https://lf3-static.bytednsdoc.com/obj/eden-cn/nupipfups/Midscene/1.0-showcases/harmony.png" height="300" controls /> View the full report of this task: [report.html](https://lf3-static.bytednsdoc.com/obj/eden-cn/nupipfups/Midscene/1.0-showcases/harmony.html) --- url: /showcases-ios.md --- **Prompt** : Open Twitter and auto-like the first tweet by @midscene\_ai <video src="https://lf3-static.bytednsdoc.com/obj/eden-cn/nupipfups/Midscene/1.0-showcases/x2.mp4" poster="https://lf3-static.bytednsdoc.com/obj/eden-cn/nupipfups/Midscene/1.0-showcases/x.png" height="300" controls /> View the full report of this task: [report.html](https://lf3-static.bytednsdoc.com/obj/eden-cn/nupipfups/Midscene/1.0-showcases/x.html) --- url: /showcases-web.md --- **Prompt:** Fill out the GitHub sign-up form and pass validation, but do not submit it. <video src="https://lf3-static.bytednsdoc.com/obj/eden-cn/nupipfups/Midscene/1.0-showcases/github2.mp4" poster="https://lf3-static.bytednsdoc.com/obj/eden-cn/nupipfups/Midscene/1.0-showcases/github.png" height="300" controls /> Midscene generates a complete report for every task so developers can review the operation process. See the report for the demo above: [report.html](https://lf3-static.bytednsdoc.com/obj/eden-cn/nupipfups/Midscene/1.0-showcases/github.html) --- url: /showcases.md --- # Showcases This page introduces cross-platform GUI Agent, E2E test, and community showcases built with Midscene. ## Cross-platform GUI Agent Showcases ### Web **Prompt:** Fill out the GitHub sign-up form and pass validation, but do not submit it. <video src="https://lf3-static.bytednsdoc.com/obj/eden-cn/nupipfups/Midscene/1.0-showcases/github2.mp4" poster="https://lf3-static.bytednsdoc.com/obj/eden-cn/nupipfups/Midscene/1.0-showcases/github.png" height="300" controls /> Midscene generates a complete report for every task so developers can review the operation process. See the report for the demo above: [report.html](https://lf3-static.bytednsdoc.com/obj/eden-cn/nupipfups/Midscene/1.0-showcases/github.html) ### iOS **Prompt** : Open Twitter and auto-like the first tweet by @midscene\_ai <video src="https://lf3-static.bytednsdoc.com/obj/eden-cn/nupipfups/Midscene/1.0-showcases/x2.mp4" poster="https://lf3-static.bytednsdoc.com/obj/eden-cn/nupipfups/Midscene/1.0-showcases/x.png" height="300" controls /> View the full report of this task: [report.html](https://lf3-static.bytednsdoc.com/obj/eden-cn/nupipfups/Midscene/1.0-showcases/x.html) ### Android **Prompt** : Open the Booking App, search for a hotel in Tokyo for four adults on Christmas, with a score of 8 or above. <video src="https://lf3-static.bytednsdoc.com/obj/eden-cn/nupipfups/Midscene/1.0-showcases/booking2.mp4" poster="https://lf3-static.bytednsdoc.com/obj/eden-cn/nupipfups/Midscene/1.0-showcases/booking.png" height="300" controls /> View the full report of this task: [report.html](https://lf3-static.bytednsdoc.com/obj/eden-cn/nupipfups/Midscene/1.0-showcases/booking.html) ### HarmonyOS **Prompt** : Open Settings, scroll to find "About phone", view device information. <video src="https://lf3-static.bytednsdoc.com/obj/eden-cn/nupipfups/Midscene/1.0-showcases/harmony.mp4" poster="https://lf3-static.bytednsdoc.com/obj/eden-cn/nupipfups/Midscene/1.0-showcases/harmony.png" height="300" controls /> View the full report of this task: [report.html](https://lf3-static.bytednsdoc.com/obj/eden-cn/nupipfups/Midscene/1.0-showcases/harmony.html) ### Desktop **macOS** **Prompt:** Help me post a tweet promoting Midscene's support for AutoGLM through safari, with the following requirements: 1. Text content: Midscene now supports AutoGLM! 2. Media content: Use the AutoGLM video from the download folder! <video src="https://lf3-static.bytednsdoc.com/obj/eden-cn/nupipfups/Midscene/pc-twitter2.mp4" height="300" controls /> View the full report for this task: [report.html](https://lf3-static.bytednsdoc.com/obj/eden-cn/nupipfups/Midscene/1.0-showcases/pc-twitter2-midscene_report.html) **Prompt:** Open Google and query San Jose tomorrow weather temperature <video src="https://lf3-static.bytednsdoc.com/obj/eden-cn/nupipfups/Midscene/mac.mov" height="300" controls /> View the full report for this task: [report.html](https://lf3-static.bytednsdoc.com/obj/eden-cn/nupipfups/Midscene/weather-computer-2026-01-14_11-26-38-9592ecf5.html) **Windows** **Prompt:** Open Sauce Demo e-commerce site, login and add items to cart <video src="https://lf3-static.bytednsdoc.com/obj/eden-cn/nupipfups/Midscene/windows.mov" height="300" controls /> View the full report for this task: [report.html](https://lf3-static.bytednsdoc.com/obj/eden-cn/nupipfups/Midscene/shop-computer-2026-01-14_11-57-36-f8411b8f.html) **Linux** **Prompt:** Open TodoMVC, add multiple tasks and filter them <video src="https://lf3-static.bytednsdoc.com/obj/eden-cn/nupipfups/Midscene/linux.mov" height="300" controls /> View the full report for this task: [report.html](https://lf3-static.bytednsdoc.com/obj/eden-cn/nupipfups/Midscene/todo-computer-2026-01-13_15-40-37-6f45fb0f.html) ## E2E Test Scenarios ### Dongchedi App: Doubao Seed 2.1 Turbo The following five test cases use `doubao-seed-2-1-turbo-260628` to test the ranking filters in the Android Dongchedi app on a device with a 720 × 1600 resolution. #### Case 1: Sales ranking test * **Test scope:** Verify that the sales ranking correctly applies the combined filters for sedan, the month before last, fuel, and a preset CNY 180,000–250,000 price range, and displays the corresponding results * **Steps:** 8 / 16 (script / model calls) * **Tokens:** Input 126,935 (89,760 cached) / output 3,109 * **Cost:** $0.0299 * **Test report:** [View report](https://lf3-static.bytednsdoc.com/obj/eden-cn/luljzkpt/ljhwZthlaukjlkulzlp/showcases/dongchedi/01-sales-sedan-fuel-preset-price.html) #### Case 2: Sales ranking combined-filter test * **Test scope:** Verify that the sales ranking correctly applies the combined filters for SUV, the previous month, and plug-in hybrid, and displays the corresponding results * **Steps:** 4 / 14 (script / model calls) * **Tokens:** Input 137,310 (100,000 cached) / output 2,472 * **Cost:** $0.0295 * **Test report:** [View report](https://lf3-static.bytednsdoc.com/obj/eden-cn/luljzkpt/ljhwZthlaukjlkulzlp/showcases/dongchedi/02-sales-suv-hybrid.html) #### Case 3: New-energy ranking test * **Test scope:** Verify that the new-energy ranking correctly applies the combined filters for SUV, the last six months, battery electric, and a preset CNY 180,000–250,000 price range, and displays the corresponding results * **Steps:** 8 / 18 (script / model calls) * **Tokens:** Input 149,996 (103,896 cached) / output 6,279 * **Cost:** $0.0415 * **Test report:** [View report](https://lf3-static.bytednsdoc.com/obj/eden-cn/luljzkpt/ljhwZthlaukjlkulzlp/showcases/dongchedi/03-new-energy-suv-pure-electric-preset-price.html) #### Case 4: Price-drop ranking test * **Test scope:** Verify that the price-drop ranking correctly applies the combined filters for MPV, the last year, new energy, and a custom CNY 150,000–300,000 price range, and displays the corresponding results * **Steps:** 13 / 29 (script / model calls) * **Tokens:** Input 253,991 (174,176 cached) / output 8,203 * **Cost:** $0.0658 * **Test report:** [View report](https://lf3-static.bytednsdoc.com/obj/eden-cn/luljzkpt/ljhwZthlaukjlkulzlp/showcases/dongchedi/04-price-drop-mpv-new-energy-custom-price.html) #### Case 5: Ranking switch and filter reset test * **Test scope:** Verify that switching from the sales ranking to the new-energy ranking resets dependent filters correctly, and that subsequent filtering and reset-to-default behavior work as expected * **Steps:** 16 / 28 (script / model calls) * **Tokens:** Input 217,283 (151,848 cached) / output 5,688 * **Cost:** $0.0525 * **Test report:** [View report](https://lf3-static.bytednsdoc.com/obj/eden-cn/luljzkpt/ljhwZthlaukjlkulzlp/showcases/dongchedi/05-switch-and-reset.html) :::info Doubao pricing assumptions Pricing was checked on August 17, 2026 and sourced from [AIHubMix](https://aihubmix.com/model/doubao-seed-2-1-turbo). Input tokens are priced at `$0.423 / M tokens`, cache reads at `$0.08452 / M tokens`, and output tokens at `$2.113 / M tokens`. The five cases cost approximately `$0.2192` in total and average `$0.0438` per case. These figures reflect this test run only; actual costs vary with task complexity, cache hit rate, and model pricing. ::: ### Reddit App: Qwen 3.7 Plus The following two test cases use [`qwen/qwen3.7-plus`](https://openrouter.ai/qwen/qwen3.7-plus) to test community search, joining, and post upvoting in the Android Reddit app on a device with a 720 × 1600 resolution. #### Case 1: Search for and join the Midscene community * **Test scope:** Search Reddit for Midscene, open the exact `r/midscene` community, join it if needed, and verify that the account has joined * **Steps:** 7 / 17 (script / model calls) * **Tokens:** Input 147,127 (34,816 cached) / output 3,418 * **Cost:** $0.0425 * **Test report:** [View report](https://lf3-static.bytednsdoc.com/obj/eden-cn/luljzkpt/ljhwZthlaukjlkulzlp/showcases/reddit/01-search-and-join-midscene-community.html) #### Case 2: Upvote the first post in the Midscene community * **Test scope:** Open the exact `r/midscene` community, upvote the first post in its post list if needed, and verify that it is upvoted * **Steps:** 6 / 12 (script / model calls) * **Tokens:** Input 103,231 (8,704 cached) / output 3,142 * **Cost:** $0.0348 * **Test report:** [View report](https://lf3-static.bytednsdoc.com/obj/eden-cn/luljzkpt/ljhwZthlaukjlkulzlp/showcases/reddit/02-upvote-first-post-in-midscene-community.html) :::info Qwen pricing assumptions Pricing was checked on August 21, 2026 and sourced from [OpenRouter](https://openrouter.ai/qwen/qwen3.7-plus). Input tokens are priced at `$0.32 / M tokens`, cache reads at `$0.064 / M tokens`, and output tokens at `$1.28 / M tokens`. The two cases cost approximately `$0.0774` in total and average `$0.0387` per case. These figures reflect this test run only; actual costs vary with task complexity, cache hit rate, and model pricing. ::: ## Community showcases Some community developers have successfully built on Midscene's capability to [integrate with any interface](/integrate-with-any-interface.md), extending it with a robotic arm plus vision and voice models for in-vehicle large-screen testing scenarios. <video src="https://lf3-static.bytednsdoc.com/obj/eden-cn/vhaeh7vhabf/AI_Vision_Powered_Robotic_Arm.mp4" height="300" controls /> --- url: /skills.md --- # Control any platform with Skills [Agent Skills](https://github.com/anthropics/skills) are a format for extending AI coding agents with specialized capabilities. Midscene provides Agent Skills that let AI coding tools (like Claude Code, Cline, etc.) drive UI automation through CLI commands. Skills work by running CLI commands directly in the terminal. The AI agent acts as the brain: it takes screenshots, analyzes the UI, and decides which actions to perform next. ## Supported platforms | Skill | Package | CLI command | Description | |-------|---------|-------------|-------------| | Browser Automation | `@midscene/web` | `npx @midscene/web` | Browser automation with three modes: default Puppeteer headless, `--bridge` to use your own Chrome, `--cdp <ws-endpoint>` to connect via CDP | | Desktop Computer Automation | `@midscene/computer` | `npx @midscene/computer` | macOS, Windows, Linux desktop control | | Android Device Automation | `@midscene/android` | `npx @midscene/android` | Android device control via ADB | | iOS Device Automation | `@midscene/ios` | `npx @midscene/ios` | iOS device control via WebDriverAgent | | HarmonyOS Device Automation | `@midscene/harmony` | `npx @midscene/harmony` | HarmonyOS device control via HDC | In default Puppeteer mode, you can override the default `1440x800` viewport with `--viewport-width <width>` and `--viewport-height <height>`. These flags are only supported in default Puppeteer mode, not in `--bridge` or `--cdp` mode. In CDP mode, use a separate `--extra-http-header 'Name:Value'` option for each HTTP header added to page requests. Repeat the option on every separate CLI command that may issue requests because each command creates a new CDP session: ```bash npx @midscene/web connect \ --cdp ws://127.0.0.1:9222/devtools/browser \ --extra-http-header 'x-use-ppe:1' \ --extra-http-header 'x-tt-env:ppe_example' \ --url https://example.com npx @midscene/web act \ --cdp ws://127.0.0.1:9222/devtools/browser \ --extra-http-header 'x-use-ppe:1' \ --extra-http-header 'x-tt-env:ppe_example' \ --prompt "click the button" ``` The headers are applied before `connect --url` navigates, so the initial document request includes them. Avoid putting sensitive authentication values directly in shell history. If Chrome is installed in a non-standard location, set `MIDSCENE_CHROME_PATH` to the Chrome executable path. `MIDSCENE_MCP_CHROME_PATH` is still accepted as a temporary migration alias. ## Installation Make sure [Node.js](https://nodejs.org) is installed, then run: ```bash # General installation npx skills add web-infra-dev/midscene-skills # Claude Code npx skills add web-infra-dev/midscene-skills -a claude-code # OpenClaw npx skills add web-infra-dev/midscene-skills -a openclaw ``` Skills repository: [github.com/web-infra-dev/midscene-skills](https://github.com/web-infra-dev/midscene-skills) ## Model configuration Midscene skills require a multimodal model with strong UI localization. Configure the following environment variables — either as system environment variables or in a `.env` file in the current working directory (Midscene loads `.env` automatically): ```bash MIDSCENE_MODEL_API_KEY="your-api-key" MIDSCENE_MODEL_NAME="model-name" MIDSCENE_MODEL_BASE_URL="https://..." MIDSCENE_MODEL_FAMILY="family-identifier" ``` For supported models and configuration details, see [Supported models and setup](/model-common-config.md). ## Use skills Once installed, just describe the task in natural language to your AI coding agent. It picks the right Skill, runs the CLI, reads the screenshots, and decides what to do next — for example: > Open the photo app and tell me the first photo in the album. ## Example: Coding Agent self-verifies after writing code In this example, we ask Claude Code to develop an Electron Todo app, and after writing the code, it uses the `desktop-computer-automation` Skill to launch the app, interact with the UI, and take screenshots to verify the feature works as expected — no manual intervention or test scripts needed. **Prompt:** ``` Build an Electron Todo app with add, toggle, and delete functionality. After development, launch the app and verify with desktop automation: add 3 todos, check one off, delete one, and take a screenshot to confirm the final state is correct. ``` The coding agent autonomously completes the entire workflow: write the Todo component → launch the Electron app → connect to the desktop → take screenshots to understand the UI → interact via natural language → take screenshots to verify results. The developer only describes the intent, and Skills give the agent the ability to "see the screen and move the mouse", letting it verify its own code just like a human would. <video src="https://lf3-static.bytednsdoc.com/obj/eden-cn/nupipfups/Midscene/computer-skill.mp4" height="300" controls /> ## More use cases Skills go beyond local desktop testing. By combining different Skills, you can cover a wide range of automation scenarios: * **Desktop app testing** — Verify functionality of Electron, Qt, WPF and other desktop applications * **Remote computer control** — Operate applications on remote machines via remote desktop connections for remote ops and debugging * **Mobile app testing** — Use `@midscene/android` and `@midscene/ios` Skills to test mobile apps on real devices or simulators * **Cross-app workflows** — Chain operations across multiple apps, e.g. fetch data from browser → paste into Excel → take screenshot and send to Slack * **CI/CD integration** — Run desktop automation in headless mode on Linux CI via Xvfb, no physical display needed * **Daily task automation** — Batch form filling, scheduled screenshot monitoring, automatic file organization, etc. ## More Please refer to the [Skills Repository](https://github.com/web-infra-dev/midscene-skills) for more details. --- url: /test-runner-overview.md --- # Test Runner overview: natural language and extensible Nodes :::info Project status (Beta) This document introduces Midscene's new next-generation, general-purpose Test Runner. It separates declarative test cases from programmable extensions and is intended to replace the previous YAML automation solution. The Test Runner is currently in **Beta**, and its test protocol and APIs continue to evolve. If you have questions or suggestions, we welcome your feedback on GitHub. (If you are still using the previous solution, see [YAML script runner](/yaml-script-runner.md).) ::: After extensive use of Midscene, we believe that **in the AI era, natural language should be the primary medium for automated test cases**. Describing test intent directly in natural language matches how people communicate and provides strong business-level expressiveness. We use declarative YAML as the structural format so that these natural-language workflows are easy for people to write and review, as well as for AI and Agents to understand and maintain. Real-world test workflows also need to call external tools or scripts, such as business APIs, test data setup utilities, and browser resource managers. For this reason, `@midscene/test` establishes a test paradigm designed for long-term evolution: **use natural language to drive the main test flow, with programmable Nodes providing supporting extensions**. ## Core design: natural language first, programmable Nodes second This approach divides the test framework into two clear tracks: * **Primary track (natural language + YAML)**: expresses more than 80% of business test intent. Test cases are assembled declaratively in YAML, with natural language describing UI actions such as `aiAct` and assertions such as `aiAssert`. * **Supporting track (TypeScript Nodes)**: provides the remaining 20% of programmable extension capabilities. Custom Nodes registered in TypeScript handle API calls, data preparation, and resource management. With this design, `@midscene/test` creates a clear collaboration boundary between platform developers and test case authors: * **Test framework maintenance (platform developers)**: use TypeScript to integrate browsers, Agents, external tools, and business APIs, while centrally managing runtime resources and the execution lifecycle. * **Test case maintenance (case authors and Agents)**: people or Agents use YAML to compose the capabilities provided by the framework. They can write and maintain test cases directly in natural language without writing JavaScript or dealing with implementation details. ## Smooth three-party collaboration with generated Node references A testing system often involves three distinct roles: **framework developers who write TypeScript Nodes**, **people who write YAML test cases**, and **AI Agents that automatically operate and maintain tests**. To make collaboration among these roles straightforward, `@midscene/test` provides the `describe-nodes` tool. It exports all registered Node definitions as a Markdown reference: * **Framework developers stay focused**: declare Zod validation rules and descriptions in TypeScript Nodes. The documentation is generated automatically, so there is no separate API reference to write and maintain. * **Test case authors get a clear reference**: people can read the exported Markdown to see which business-specific Nodes, such as `order.prepare`, are available and which parameters they accept. * **AI assistants call Nodes accurately**: the Markdown reference combines explicit type constraints with semantic descriptions, making it effective context for AI Agents. Agents can use it to write, run, and maintain YAML test cases with valid Node calls. **Example of an exported Node reference** For example, the `describe-nodes` tool exports a custom Node named `browser.mockLocation` in the following Markdown structure: ```markdown ## browser.mockLocation - **Title**: Mock location - **Description**: Mock the browser's GPS location. - **Parameter schema (JSON Schema)**: { "type": "object", "properties": { "latitude": { "type": "number", "description": "Latitude of the mocked location." }, "longitude": { "type": "number", "description": "Longitude of the mocked location." } }, "required": ["latitude", "longitude"] } ``` This generated, strongly typed reference connects platform development, test case authoring, and AI-based test operations, making subsequent workflows easier to implement and run. ## Example scenario: verify an order refund flow Suppose an e-commerce team needs to verify the refund flow for paid orders: 1. **Programmable Nodes (supporting)**: call an order API or script before each case to create test data, then clean up the test order when the case finishes. 2. **Natural language (primary)**: use YAML instructions to have the Agent submit a refund request in the UI and verify the result. ```yaml # 1. Programmable Nodes: prepare data and the environment beforeEach: - order.prepare: status: paid - browser.openRefundPage: {} # 2. Natural language: express business test intent cases: - name: A paid order supports a full refund steps: - aiAct: Click Apply for refund, select Full refund, and submit - aiAssert: prompt: The page shows that the refund request was submitted and the refund amount equals the full order amount message: Failed to submit the full refund request - name: A paid order supports a partial refund steps: - aiAct: Click Apply for refund, enter a refund amount of 10, and submit - aiAssert: prompt: The page shows that the refund request was submitted and the refund amount is 10 message: Failed to submit the partial refund request # 3. Programmable Nodes: clean up resources afterEach: - order.cleanup: {} ``` Framework maintainers register custom Nodes such as `order.prepare`, `browser.openRefundPage`, and `order.cleanup` to manage data and pages. Midscene Nodes handle UI actions and assertions. Test intent remains separate from technical implementation, so the two can evolve independently. ## Next steps * Read [Extend and maintain Test Runner](/extend-test-runner.md) to learn how to register Nodes, manage runtime resources, and configure a Test Project. * Read [Write and run test cases](/use-test-runner.md) to learn about Cases, Steps, lifecycles, and test results. --- url: /use-test-runner.md --- # Write and run test cases After framework maintainers configure a Test Project, people or Agents can use its registered Nodes to write test cases. Before writing cases, we recommend running `describe-nodes` to generate a Node reference and confirm the available capabilities and input parameters. If you are not familiar with the overall design, start with [Test Runner overview](/test-runner-overview.md). If your project is not configured yet, see [Extend and maintain Test Runner](/extend-test-runner.md). ## Core concepts `@midscene/test` uses the following concepts to describe a test project: ```text Test Project configuration └── Execution Project (Web / Android / iOS / computer) └── Workflow Document ├── Lifecycle Step └── Case └── Step └── Node ``` * **Test Project**: the top-level configuration defined by `midscene.config.ts`. It registers shared Nodes and configures the default execution target or a `projects` array. * **Execution Project**: an entry within a Test Project that describes one execution target, including its platform, Project setup, file and tag filters, variables, and retry strategy. It is not a peer of the Test Project. * **Workflow Document**: a YAML file selected by an Execution Project. It contains lifecycle hooks and one or more Cases. * **Lifecycle Step**: a Step declared in a Workflow Document's `beforeAll`, `beforeEach`, `afterEach`, or `afterAll` hook. Use these Steps for supporting operations such as data preparation, state reset, and resource cleanup. * **Case**: a named test case composed of multiple Steps. * **Step**: one call to a Node in YAML. * **Node**: an execution capability registered by a platform developer, such as `aiAssert` or `order.create`. Nodes define the execution capabilities available to a team. YAML defines how each test case combines those capabilities. ## Write YAML test cases Every Workflow Document must contain a non-empty `cases` array. Every Case must contain a `name` and a non-empty `steps` array. ```yaml cases: - name: Create an order steps: - order.create: sku: midscene-mug quantity: 1 - aiAssert: The page shows "Order placed" - name: Cancel an order steps: - order.cancel: orderId: example-order-id - aiAssert: The page shows "Order canceled" ``` The runner executes Cases in their declared YAML order. If one Case fails, the runner records the failure and continues with subsequent Cases. ### Write a Step Each Step can call only one Node. If a call only needs a `prompt` parameter, use the string shorthand: ```yaml steps: - aiAct: Click the Submit order button ``` The previous form is equivalent to: ```yaml steps: - aiAct: prompt: Click the Submit order button ``` Business Nodes can accept custom parameters: ```yaml steps: - order.create: sku: midscene-mug quantity: 2 ``` ### Configure timeouts and error handling Use `$` for Step parameters controlled by the runner. The runner does not include these parameters in the Node's `input`. ```yaml steps: - order.create: sku: midscene-mug $: timeout: 30000 continue-on-error: true ``` The following fields are supported: * `timeout`: Step timeout in milliseconds. * `continue-on-error`: when set to `true`, the runner continues with subsequent Steps in the current phase even if this Step fails. The default is `false`. `continue-on-error` controls only whether execution continues. If any Step fails, the Case's final status is `failed`. ### Use Project variables and environment variables The runner recursively resolves Node input before execution: ```yaml steps: - launch: uri: ${appUri} - api.createOrder: baseURL: ${{TEST_API_BASE_URL}} payload: count: ${orderCount} ``` * `${name}` reads a value from the current Execution Project's `variables`. When the placeholder occupies the entire scalar, it preserves the original JSON type. * `${{ENV_NAME}}` reads an environment variable. The result is always a string. * An object or array variable can be used as a complete value, but it cannot be embedded in a longer string. * An undefined variable fails during collection. Variables are resolved only in Node input, not in `$`. Workflow YAML does not provide `set`, `saveAs`, or Step output expressions. Natural-language Nodes can use the runner's read-only execution history to understand previous results. ### Filter Cases with tags ```yaml cases: - name: Android smoke test for placing an order tags: [smoke, android] steps: - aiAct: Complete the order ``` Framework maintainers configure `tags.include` and `tags.exclude` for each Execution Project. Exclusions always take precedence. When the include list is not empty, a Case is selected if it matches any included tag. ## Define the execution lifecycle A Workflow Document can declare lifecycle Steps around `cases`: ```yaml beforeAll: - data.prepare: Prepare the test data required by this file beforeEach: - browser.reset: Reset the page to its initial state cases: - name: Create an order steps: - aiAct: Create an order - aiAssert: The page shows "Order placed" - name: Cancel an order steps: - aiAct: Cancel the latest order - aiAssert: The page shows "Order canceled" afterEach: - report.save: Save execution information for the current Case afterAll: - data.cleanup: Delete the test data created by this file ``` An Execution Project follows this complete execution sequence: ```text Project setup Workflow Document 1 beforeAll Case 1 attempt 1: beforeEach → steps → afterEach Case 1 retry: beforeEach → steps → afterEach Case 2 attempt 1: beforeEach → steps → afterEach afterAll Document-scoped Node cleanup Workflow Document 2 ... Project teardown ``` Each phase has the following responsibilities: * `beforeAll` and `afterAll` run once for each YAML file and handle document-level business setup and cleanup. * `beforeEach` and `afterEach` run once for every Case attempt. * A retry reruns the entire Case with a new run ID, Agent scope, cache scope, and Case history, but it does not rerun `beforeAll`. * A Node can register internal cleanup at attempt or Document scope, such as destroying an Agent or generating a report. Each cleanup runs when its scope ends. `afterEach` still runs if the main body of a Case fails. If `beforeAll` fails, the runner marks the Cases in the current file as `not-run`, but it still runs `afterAll` and all registered Node cleanup. Project setup runs only once before the Project's Workflow Documents. Project teardown always runs in LIFO order. When the runner receives an interrupt signal, it cancels the current Step, while cleanup hooks and registered teardown functions receive a signal that still allows cleanup work to execute. ## Run tests Run YAML test cases from the project root: ```bash pnpm exec midscene-test ``` ### Specify a test case directory or file By default, the runner recursively finds and executes all `.yaml` and `.yml` files under the project root, automatically ignoring `node_modules` and `.git`. Pass a specific directory or test case file as an argument to run only that target: ```bash # Run all test cases in a specific directory pnpm exec midscene-test ./cases/smoke # Run one test case file pnpm exec midscene-test ./cases/order.yaml ``` ### Filter execution targets and configuration files If a project defines multiple platforms or environments, select specific Execution Projects or specify a custom configuration file on the command line: ```bash # Run only the android-smoke and ios-regression Execution Projects pnpm exec midscene-test --project android-smoke --project ios-regression # Run tests with a specific configuration file pnpm exec midscene-test --config ./config/midscene.config.ts ``` The CLI exits with code `1` when a test case fails, a document cannot be parsed, or an error occurs during collection. ## View test results After each test run, the console displays Case statuses and a brief summary. The runner also generates a visual test report locally. ### Midscene visual report The runner records detailed test Steps, UI screenshots, and the AI decision process in an interactive HTML report. By default, reports are saved to: ```text midscene_run/report/ ``` Open an HTML file in this directory in a browser to inspect each `aiAct` and `aiAssert` execution trace, element location result, and complete screenshot history. > **Note**: To change the default report directory, set `output.reportDir` in `midscene.config.ts`. ## Current limitations The current Test Runner has the following limitations: * **No parallel execution within an Execution Project**: all Workflow Documents, Cases, attempts, and Steps within one Execution Project run sequentially. Case-level concurrency is not supported yet. * **No control flow**: DAGs, branches such as If-Else, and loops are not supported. Test cases run linearly in their declared order. * **No cross-document data dependencies**: Workflow Documents are fully isolated and cannot pass or share runtime data with one another. * **Local loading only**: the runner cannot load and execute YAML files directly from remote URLs, npm packages, or Git repositories yet. The runner also statically collects and validates every YAML file before execution. If it finds an unknown top-level field, an unregistered Node, or an invalid Step, it immediately throws a Collection Error and stops the run instead of waiting until that Case begins. --- url: /yaml-script-runner.md --- # YAML script runner Midscene defines a YAML-based scripting format so you can quickly author automation scripts, then run them from the command line without extra setup. For more details on YAML scripts, see [Automate with scripts in YAML](/automate-with-scripts-in-yaml.md). For example, you can write a YAML script like this: ```yaml page: url: https://www.bing.com tasks: - name: Search for weather flow: - ai: Search for "today's weather" - sleep: 3000 - aiAssert: The results show weather information ``` Run it with one command: ```bash midscene ./bing-search.yaml ``` The CLI prints execution progress and generates a visual report when it finishes, while keeping setup simple. ## Configure environment variables with `.env` The Midscene CLI uses [dotenv](https://www.npmjs.com/package/dotenv) to load a `.env` file from the directory where you run the tool. Create a `.env` file and add: ```ini filename=.env MIDSCENE_MODEL_BASE_URL="replace with your model service URL/v1" MIDSCENE_MODEL_API_KEY="replace with your API Key" MIDSCENE_MODEL_NAME="replace with your model name" MIDSCENE_MODEL_FAMILY="replace with your model family" ``` For supported models and complete setup examples, see [Supported models and setup](/model-common-config.md). Notes: * The file is optional; you can also set global environment variables instead. * Do not add an `export` prefix—this is how dotenv expects values. * Place `.env` in the directory where you run the tool, not necessarily next to the YAML file. * These values do **not** override existing global environment variables unless you enable `--dotenv-override` (see below). * Use `--dotenv-debug` if you need to debug how environment variables load. ## Get started ### Install the CLI Before installing the CLI, make sure the terminal that runs `midscene` uses Node.js `20.19+`, `22.12+`, or `24+`. Some CLI execution paths use the Rstest/Rspack toolchain, which rejects older Node 20 patch versions such as `20.17.0`. If you see an `Unsupported Node.js version` message from Rspack, upgrade Node.js and reinstall the global CLI or project dependencies. Install `@midscene/cli` globally (recommended for first-time users): ```bash npm i -g @midscene/cli ``` Or install it per project: ```bash npm i @midscene/cli --save-dev ``` ### Write your first script Create `bing-search.yaml` to drive a web browser: ```yaml page: url: https://www.bing.com tasks: - name: Search for weather flow: - ai: Search for "today's weather" - sleep: 3000 - aiAssert: The results show weather information ``` Drive an Android device connected over adb: ```yaml android: deviceId: s4ey59 # find the device id with `adb devices` tasks: - name: Maps Navigation flow: - ai: Open the Maps app - ai: Input 'West Lake, Hangzhou' in the search bar, and click the search button - ai: Click the first search result, enter the details page - ai: Click "Directions" button, enter the route planning page - ai: Click "Start" button to start navigation ``` Or drive an iOS device with WebDriverAgent configured: ```yaml ios: wdaPort: 8100 tasks: - name: Change System Settings flow: - ai: Open the Settings app - ai: Tap "Display & Brightness" - ai: Turn on "Dark Mode" - aiAssert: Dark Mode is enabled ``` ### Run the script ```bash midscene ./bing-search.yaml # If Midscene is installed in your project npx midscene ./bing-search.yaml ``` The CLI prints execution progress and generates a visual report when it finishes. ## Advanced usage of the command-line tool ### Use environment variables in `.yaml` Reference environment variables in your scripts with `${variable-name}`. Environment-variable interpolation is applied before YAML task execution, including task strings. ```ini filename=.env topic=weather today ``` ```yaml # ... - ai: type ${topic} in input box # ... ``` ### Run multiple scripts `@midscene/cli` supports glob patterns to batch-execute scripts, which is a shorthand for the `--files` argument. ```bash # Run a single script midscene ./bing-search.yaml # Use a glob pattern to run all matching scripts midscene './scripts/**/*.yaml' ``` ### Analyze command-line output After execution, the output directory contains: * A JSON summary specified by `--summary` (defaults to `index.json`) with execution status and statistics for all scripts. * Individual execution results for each YAML file (JSON). * Visual reports for each script (HTML). ### Run in headed mode > Web page scenarios only Headed mode opens the browser window. By default, scripts run headless. ```bash # Run in headed mode midscene /path/to/yaml --headed # Run in headed mode and keep the window after finishing midscene /path/to/yaml --keep-window ``` ### Use CDP connection mode > `web` scenarios only CDP mode lets YAML scripts connect to an existing browser instance via Chrome DevTools Protocol, without launching a new browser. This is useful for reusing an existing browser session, connecting to remote browsers, or cloud browser services. Set `cdpEndpoint` in the `page` section: ```diff page: url: https://www.bing.com + cdpEndpoint: ws://localhost:9222/devtools/browser ``` :::info CDP mode and bridge mode are mutually exclusive. In CDP mode, Midscene will only disconnect from the browser, not close it. ::: ### Use bridge mode > Web page scenarios only Bridge mode lets YAML scripts drive your existing desktop browser so you can reuse cookies, extensions, or state. Install the Chrome extension, then add: ```diff page: url: https://www.bing.com + bridgeMode: newTabWithUrl ``` See [Bridge Mode via Chrome Extension](/bridge-mode.md) for details. ### Run YAML scripts with JavaScript Call the Agent's [`runYaml`](/reference.md#runyaml) method to execute YAML from JavaScript. This runs only the `tasks` section of the script. ## Command-line options The CLI provides parameters to control how scripts run: * `--files <file1> <file2> ...`: List of script files. Executes in order, sequentially by default (`--concurrent` is `1`), or concurrently when `--concurrent` is set. Supports [glob](https://www.npmjs.com/package/glob) patterns; when a glob pattern or directory matches multiple files, matched files are added to the execution list in lexicographic path order. * `--setup <file>`: A setup script that runs before the main `--files` for any supported target. If all setup attempts fail, the batch is aborted and the main scripts are reported as not executed. Puppeteer Web setup requires `--share-browser-context`; each retry then starts with a clean BrowserContext and Page, and the successful context is shared with the main scripts. Every YAML script still runs in its own Page, so page-scoped state such as `sessionStorage` is not carried between scripts. Bridge mode and non-Web targets must omit `--share-browser-context`; their retries create a new player and Agent but do not reset the underlying browser, device, desktop, or external interface state. * `--concurrent <number>`: Number of concurrent executions. Default `1`. * `--continue-on-error`: Continue running remaining scripts even if one fails. Default off. * `--retry <number>`: Number of extra attempts for a failed script. Only failed scripts are retried, which helps with unstable networks or unstable model output. Default `0`. Puppeteer Web setup retries use a clean BrowserContext and Page, while main-script retries preserve the successful setup context. Other targets create a new player and Agent for each retry without resetting their underlying environment. * `--share-browser-context`: Share one Puppeteer BrowserContext (cookies, same-origin `localStorage`, etc.) across scripts while giving every YAML script an independent Page. Page-scoped state such as `sessionStorage`, the DOM, URL, and `window.name` is not shared, even with `--concurrent 1`. Every setup and main script in the batch must use a Puppeteer Web target. Bridge mode and non-Web targets are not supported. Because the browser is created or connected only once, put browser-level options (`cdpEndpoint`, `chromeArgs`, `acceptInsecureCerts`, and `downloadPath`) in the batch config's global Web target, not in an individual setup or main script. Default off. * `--summary <filename>`: Path for the JSON summary report. * `--headed`: Run in a headed browser instead of headless. * `--keep-window`: Keep the browser window after execution; enables `--headed` automatically. * `--config <filename>`: Config file whose values become defaults for CLI arguments. * `--web.userAgent <ua>`: Override `web.userAgent` for all scripts. * `--web.viewportWidth <width>`: Override `web.viewportWidth` for all scripts. * `--web.viewportHeight <height>`: Override `web.viewportHeight` for all scripts. * `--android.deviceId <device-id>`: Override `android.deviceId` for all scripts. * `--ios.wdaPort <port>`: Override `ios.wdaPort` for all scripts. * `--ios.wdaHost <host>`: Override `ios.wdaHost` for all scripts. * `--dotenv-debug`: Enable dotenv debug logs. Default off. * `--dotenv-override`: Allow dotenv to override global environment variables. Default off. Examples: Use `--files` to specify execution order: ```bash midscene --files ./login.yaml ./buy/*.yaml ./checkout.yaml ``` Run multiple independent search scripts with a concurrency of 4 and continue when errors occur: ```bash midscene --files './scripts/search-*.yaml' --concurrent 4 --continue-on-error ``` ### Write command-line arguments in a file You can place arguments in a YAML config file and reference it with `--config`. Command-line arguments take priority over the config file. ```yaml files: - './scripts/search-iphone.yaml' - './scripts/search-laptop.yaml' - './scripts/search-headphones.yaml' - './scripts/search-camera.yaml' concurrent: 4 continueOnError: true retry: 2 ``` Run with: ```bash midscene --config ./config.yaml ``` Set `concurrent: 1` (the default) when scripts must run in the exact order of the `files` list. With a value greater than `1`, execution order is unspecified; scripts must not depend on another script's start or completion order. #### Run a setup before parallel scripts When several independent Puppeteer Web scripts all depend on the same prerequisite (for example a login), put the prerequisite under `setup`. The setup script runs before the main `files`; once it succeeds, the main scripts run with the configured concurrency. Set `shareBrowserContext: true` so the successful setup attempt's browser context, including cookies and same-origin `localStorage`, is carried over. Every script receives an independent Page, so the setup Page's `sessionStorage`, DOM, URL, `window.name`, and other page-scoped state are not copied into main Pages. Every script in this shared batch must use a Puppeteer Web target; bridge mode is not supported. The shared Browser is launched or connected from the batch config once. Configure `cdpEndpoint`, `chromeArgs`, `acceptInsecureCerts`, and `downloadPath` in the batch config's global Web target. Defining any of these browser-level options in an individual setup or main script is rejected because it cannot be applied to the already-created shared Browser. ```yaml setup: ./scripts/login.yaml files: - ./scripts/search.yaml - ./scripts/report.yaml - ./scripts/settings.yaml shareBrowserContext: true concurrent: 3 retry: 2 ``` `retry` is the number of extra attempts, so `retry: 2` allows up to three attempts. Every setup retry gets a new BrowserContext and Page, preventing cookies, local storage, navigation state, and other browser-side effects from a failed attempt from leaking into the next one. After setup succeeds, its context is shared with the main scripts. A failed main script is retried in that same context so the successful setup state is preserved. If all setup attempts fail, the batch is aborted and the main scripts are reported as not executed. :::warning Page-scoped state is not shared `shareBrowserContext` follows browser-native storage boundaries: it does not copy or synchronize `sessionStorage` between Pages. If setup establishes authentication only in `sessionStorage`, the main scripts will not inherit that login. Prefer cookies, same-origin `localStorage`, or backend state for prerequisites that must be visible to multiple scripts. When scripts run concurrently, shared-state writes can race, so coordinate them explicitly. ::: `setup` also works with bridge mode, Android, iOS, HarmonyOS, Computer, and custom Interface targets. Omit `shareBrowserContext` for those targets. A retry creates a new script player and Agent, but the underlying browser profile, device, desktop session, or custom interface is not automatically reset. If a clean retry environment is required, make the setup script restore that environment explicitly. ## FAQ **How can I export cookies from Chrome as JSON?** Use this [Chrome extension](https://chromewebstore.google.com/detail/get-cookiestxt-locally/cclelndahbckbenkjhflpdbgdldlbecc) to export cookies. **How can I view dotenv debug logs?** Use the `--dotenv-debug` flag: ```bash midscene /path/to/yaml --dotenv-debug=true ```