Create and use test projects
This guide starts with project creation, then covers running an example, learning the available test operations, writing YAML cases, and inspecting results. If you already have a native Test project, start with Understand available test operations.
For the overall design, see Midscene Test overview. To implement business operations, see Develop custom Nodes. For runtime environments and execution settings, see Configure test projects.
Here, “legacy YAML” specifically means tasks/flow cases and YAML batch configuration executed by midscene; it does not include native cases/steps Test files. The legacy path receives compatibility maintenance and necessary fixes only, with no new capabilities. Existing projects can continue to run, while new projects and cases should use Midscene Test. Both paths share the execution kernel and new report page. See Migrate to Midscene Test for command boundaries and field mappings.
Create a project
1. Generate the project and install dependencies
Prepare Node.js ^20.19.0 || ^22.12.0 || >=24.0.0 and pnpm. For guided creation, run:
The command prompts for a project directory, platform, package manager, and whether to install dependencies. You can also specify these options directly. For example, create a Web test project with pnpm:
The available arguments are:
Each platform currently has one starter, selected with --platform. All starters use native cases/steps Test files. In a non-interactive environment such as CI, you must supply the directory and --platform; dependencies are installed unless --skip-install is set.
Common examples:
Follow the prompt to install dependencies. The command generates platform configuration and example cases; installation also generates the Node Spec. A Web project contains these main files:
The other platform starters define todo.setup in midscene.config.ts and do not include nodes/todo.ts, and their Spec filenames use the selected platform name, such as midscene-node-spec.ios.md. Creation and spec generation do not run tests, so this step does not require a model API Key, browser, or device. The generated README explains platform setup and how to run the example.
If installation was skipped or failed, enter the project directory and run pnpm install. To regenerate the Node Spec, run pnpm run nodes.
2. Configure the model and runtime environment
Copy .env.example to .env and follow Model configuration to set your model name, service URL, and API Key.
Verify model connectivity and Midscene compatibility before running tests:
The command reads .env from the current working directory and overrides existing shell environment variables. It does not require a browser or device. See Model debugging and observability for details.
For Web, also install Chromium:
For other platforms, change --platform when creating the project and complete the corresponding setup:
3. Run the generated example
All five starters use the public React TodoMVC page to demonstrate adding, deleting, and completing tasks. The page must be reachable from the browser or device used for the test.
Before each case, the Web starter runs todo.setup to open All view, clear page storage, reload, and verify an empty list through the DOM. todo.seed then creates three tasks. The first case checks the initial state; the second uses AI to delete and complete tasks, then checks the result. todo.captureState records the page state after each case. The Android, iOS, HarmonyOS, and desktop starters use AI to create three tasks, delete one, complete another, switch to the Completed filter, and check the result. Before changing tasks, those starters run todo.setup to open All view and wait for the input, then use aiAssert to check that the list is empty, including completed tasks, and no dialog covers the page. Existing tasks are left unchanged. If the check fails, select All and delete every task, or clear only TodoMVC website data; close blocking dialogs and rerun. For model or connection errors, resolve the reported cause first. The AI check can misjudge the page; inspect the report when it fails. All starters run serially and stop after the first failed case (test.bail: 1), while saving reports and releasing resources.
After the run finishes, view the report in midscene_run/report/ to inspect each case and step. Then edit cases/todo.yaml to write your own cases, using the generated Node Spec for available operations and their inputs.
Understand available test operations
Official operations and extensions
Midscene Test calls each capability you can invoke from YAML a Node.
Nodes can be called from case steps or from lifecycle hooks such as beforeEach and afterEach, using the same syntax.
All platforms provide these common capabilities: aiAct operates the interface from natural-language instructions, aiAssert checks an expected result, aiWaitFor waits for an observable condition, and wait pauses for a specified duration.
Alongside these shared capabilities, Midscene provides platform-specific operations. For example:
- Web:
gotoUrlopens a page,setCookiessets cookies, andsetViewportSizeadjusts the browser viewport. - Android:
launchstarts an app,backgoes back,homereturns to the home screen, andrunAdbShellexecutes an ADB shell command.
The project scaffold explicitly registers the official Nodes for the selected platform. A handwritten midscene.config.ts does not register any Nodes implicitly. Developers can also add custom operations, such as preparing a test order through a business API. See Register custom business Nodes.
Generate the current Project's Node Spec
The Nodes registered by different platforms, Execution Projects, and package versions can differ. Do not infer the current project's capabilities from another project's examples. midscene-test nodes generates Node Specs from the effective registry. This Node Spec is the source of truth for the current Project's Node names, input fields, and string shorthand. Both people and AI Agents should use it when writing YAML cases.
The scaffold generates the Spec after dependency installation. Regenerate it after changing agentClass, top-level nodes, projects[].nodes, or a Node schema, and after upgrading relevant Midscene packages:
You can also run the command directly and specify a test directory, configuration file, or Execution Project:
When projects is declared, midscene-test nodes generates one Spec per Execution Project by default, even when their Node definitions are identical. Files are named midscene-node-spec.<project-name>.md. Use --project <name> to generate only that Project’s Spec with the same filename. Without an explicit projects configuration, the filename remains midscene-node-spec.md. Files are written to the specified directory, or the current directory when omitted; run the command alongside package.json to keep the Specs there. Duplicate Project names or names that map to the same filename (including case differences) cause an error before any files are written.
The command loads project configuration and Node definitions only. It does not run setup or cases and does not call getAgent, so it does not need to start a browser, connect a device, or call a model. The Spec is generated output; do not edit it manually. Regenerate it after configuration changes.
The Spec has two sections: Available Nodes lists the Nodes you can use, and Node Details provides each Node's description, string shorthand, and input schema. Input schemas are converted from Zod inputSchema to standard JSON Schema. Both the Spec and terminal output list aiAct and aiAssert first, followed by the remaining Nodes in name order.
The Spec also lists case file patterns (Case files) and the configuration file path (Config file) to help case authors locate their files. Case files corresponds to each execution project's files.include and files.exclude. Both sets of paths are relative to the Spec's directory.
Write YAML test cases
A typical YAML file
A YAML file can contain multiple test cases and setup or cleanup steps. This example tests a store search: open the home page before each case, search for a product, and check the results. Replace the URL and product name with your application details.
The file has these parts:
beforeEach: optional steps that run before each case, such as opening a page or resetting state. See execution lifecycle for other hooks.cases: a list with at least one test case. Add another entry withnameandstepsto define another case.name: the case name, used to identify it in the results.steps: an ordered list with at least one step. Each step calls one Node, with its input after the colon.
This documentation calls the whole YAML file a Workflow Document, an individual test a Case, and one Node invocation a Step.
The example uses string shorthand: text after gotoUrl becomes url, while text after aiAct and aiAssert becomes prompt. To supply additional parameters, use the object form shown below. Check the project Node Spec for shorthand support.
Key Nodes: actions and assertions
Most cases use aiAct to describe actions and aiAssert to check their results:
Completing an action does not establish that the test passed. Use aiAssert to check the expected result explicitly. To provide a custom assertion failure message, expand the parameters:
For aiAct, optional options.effort selects planning effort: fast, balance, or deepThink. Omitting it retains the Agent's default behavior. When both effort and deepThink are provided, effort takes precedence. This is an action capability, not a test scheduling option:
Setting AI context
We recommend calling setAIContext in beforeAll so context is configured before the current document's cases run and applies to all subsequent relevant AI operations on the same Agent. You can also call it in beforeEach or a case's steps to set or clear context as needed; changes take effect from that point onward.
Its parameters match agent.setAIContext(target, context): target is default or an AI API name such as aiAct, aiTap, or aiAssert.
Set context: provide a context string to configure guidance for the selected API. Use target: default to configure the shared fallback for all AI APIs.
Clear context: omit context to pass undefined and remove the configuration for the selected target.
Clearing an API's context lets that API fall back to default; clearing default removes the shared fallback. To disable inherited context for an API, set its context to an empty string, "".
A call's options.context overrides the Agent-level value; values are not concatenated. Configuration belongs to the Agent instance: beforeAll and the cases must use the same Agent, and changes remain in effect until replaced or cleared.
Calling other Nodes
Platform operations and custom business Nodes use the same structure: put parameters beneath the Node name. These examples show calls with multiple parameters and with no parameters. Place the snippets inside a case's steps:
setViewportSize takes a parameter object; clearCookies takes no parameters, so use {}. Both Nodes are provided by Web projects. Consult your project's Node Spec for available mobile, desktop, and custom Nodes and their inputs.
For example, if your team registers an order.create Node, you can call it as follows. This is a business extension example that must be implemented and registered in your project before use:
Parameters can also contain nested objects or arrays. When passing a reference image to aiAct, the text and images form one prompt value, while options is a separate field:
Configure timeouts and error handling
Use $ for Step parameters controlled by Midscene Test. Midscene Test does not include these parameters in the Node's input.
The following fields are supported:
timeout: Step timeout in milliseconds.continue-on-error: when set totrue, allows subsequent Steps in the current phase to run after this Step fails. The default isfalse. Unfinished operations or resource cleanup problems still stop execution.
continue-on-error controls only whether execution continues. If any Step fails, the Case's final status is failed.
A timeout does not mean the underlying operation has stopped. Before continuing with subsequent Steps in the same phase, the framework waits for cancelled operations to finish, for up to 30 seconds by default. If they remain unfinished, later Steps do not start. This wait is recorded separately and does not consume the next Step's timeout budget. Once the operation finishes, the Agent can be used again, but the case must still verify or reset page and business state. See Timeouts and cancellation.
Use Project variables and environment variables
Midscene Test recursively resolves Node input before execution:
${name}reads a value from the current Execution Project'svariables. When the placeholder occupies the entire scalar, it preserves the original JSON type.${{ENV_NAME}}reads an environment variable. The result is always a string.- An object or array variable can be used as a complete value, but it cannot be embedded in a longer string.
- An undefined variable fails during collection. Variables are resolved only in Node input, not in
$.
Workflow YAML does not provide set, saveAs, or Step output expressions. Each Step runs independently and does not automatically receive previous Step results. To share required values explicitly, provide them through the Execution Project's context.
Filter Cases with tags
Framework maintainers configure tags.include and tags.exclude for each Execution Project. Exclusions always take precedence. When the include list is not empty, a Case is selected if it matches any included tag.
Define the execution lifecycle
This section covers the lifecycle of a single YAML file. For the full relationship between project setup, file hooks, cases, and cleanup, see Project, file, and case lifecycle.
Setup and cleanup steps
Lifecycle hooks prepare the environment, reset state, or clean up data around test cases. All four hooks are optional and appear at the same level as cases. Their steps use the same Node call syntax as case steps.
This Web example opens the store home page before each case and clears cookies afterward. data.prepare and data.cleanup are custom Nodes your project must implement and register to prepare and remove test products. gotoUrl, clearCookies, aiAct, and aiAssert are official Nodes registered by the Web scaffold. Replace the example URL and product names with your application details.
Execution order and retries
Without failures or retries, the file runs in this order:
When retries are configured, a failed case reruns beforeEach → steps → afterEach within the retry limit after cleanup completes and no operations remain running, before execution continues to subsequent cases. A retry does not rerun beforeAll; afterAll still runs once at the end of the file. The retry count is an upper limit: unfinished operations, pending cleanup, or failed cleanup stop retries early.
Handling failures
- If
beforeEachfails, the current case'sstepsare skipped, butafterEachstill runs. - If the case's
stepsfail,afterEachstill runs. - If
beforeAllfails, the file's cases are markednot-run, butafterAllstill runs. - Failures in
afterEachorafterAllare recorded as failures; cleanup errors are not ignored.
By default, a failed step stops the remaining steps in that phase. Set continue-on-error to continue with later steps in the same phase; this does not turn a failure into a success. Cleanup Nodes should handle partially completed setup.
A failed Case does not prevent the next Case from running by default. Use test.bail to stop scheduling new work after a failed-case threshold. Put file-wide prerequisites in beforeAll; if they fail, the file's Cases do not run. Keep dependent business steps in a single Case rather than relying on another Case's success.
Resource problems are an exception to these continuation rules: unfinished operations or pending or failed cleanup callbacks stop retries and subsequent Cases in the same file. The CLI also stops scheduling subsequent files in that Project because they share its context; independent Projects can continue. An ordinary afterEach Step failure remains a case failure and can be retried if operations have finished and registered cleanup has completed. Run infrastructure errors, such as result publication failures, cancel the entire run.
Project setup manages browser and Agent resources; see Create and clean up shared resources with setup. For resources created inside a Node, see Resource lifecycle and cleanup.
Run tests
Run YAML test cases from the project root:
Specify a test case directory or file
Generated projects use files.include to select .yaml and .yml files under cases/. Without a files configuration, Midscene Test recursively searches the test directory for YAML files, ignoring node_modules and .git.
Pass a specific directory or test case file as an argument to run only that target:
Filter execution targets and configuration files
If a project defines multiple platforms or environments, select specific Execution Projects or specify a custom configuration file on the command line:
To generate the Node Spec for a specific Execution Project, also use --project:
Keep unmigrated legacy YAML on the legacy command
Run top-level tasks/flow files and legacy YAML batch configuration with midscene:
midscene-test does not auto-detect or execute these files, and it does not accept legacy batch configuration or legacy CLI options. Convert a complete file to cases/steps before running it with midscene-test. Both kinds of files may coexist in separate directories while a repository is being migrated, but one command invocation must select only one format. See Migrate to Midscene Test for migration steps and field mappings.
Use --result-dir to write structured run results to a specific directory:
The CLI exits with code 1 when a test case fails, a document cannot be parsed, or an error occurs during collection.
View test results
After each run, the CLI prints the results and a Report: path. Open the HTML report to inspect Projects, Cases, retry attempts, screenshots, Step input/output, errors, and associated Agent traces.
The summary and result bar show final outcomes: Passed, Failed, and Not run. Passed includes Cases that recovered after earlier failed attempts. An outlined portion inside the green Passed segment represents those recovered Cases, with a passed after retry count in the legend. This is a subset of Passed and is never counted again as a separate outcome. Run metadata, Project counts, and File counts appear beside the summary.
The Test Report overview shows Project → File → Case. The left navigation groups Projects in the order Failed, Passed, and Not run. Each group shows its Project count; the Projects heading has no count. Each Project lists its Case total and nonzero final outcome counts. Long names and summaries are truncated; hover to read the full text. Recovered Projects, Files, and Cases keep their green Passed status. An amber circular warning beside affected Project and Case names explains passed after retry on hover or keyboard focus. Whole-file retry history remains in File attention and the Project warning without marking every always-passing Case as recovered.
Use the button beside the status filters to expand or collapse visible Files in the selected Project. Expansion for filtered-out Files and other Projects is preserved. Files with failures, scope errors, or recovered attempts start expanded. Empty and collection-failed Files remain visible. File errors appear beside their File, and Project setup or cleanup errors appear under the Project heading. Passing Cases do not override File or Project errors.
Compact shows Case names, final results, attempt counts, and durations. Detailed adds screenshot timelines and a short failure or not-run summary. Earlier failures are identified as earlier attempts, without changing the final result. Opening a passed Case shows its final successful attempt; select another attempt to inspect its history. Clicking an earlier-failure summary opens the corresponding failed Step, and clicking a screenshot opens its Step.
Search finds Projects, Files, Cases, and error details across the report. Only matching Projects remain in the navigation. The All / Failed / Passed / Not run checkboxes beside the selected Project heading filter that Project; multiple selections are combined with OR. Clear filters clears the search and restores All:
- Passed includes every finally successful Case, whether or not an earlier attempt failed. Warning icons and attempt history provide diagnostic context, not another status category.
- Failed includes final Case failures and keeps matching File or Project errors visible. A blocked, unexecuted Case remains Not run even when shown as context for a failed scope.
- Not run selects Cases that were never executed, including those blocked by setup or scheduling failures.
- A File cleanup failure can appear with all Cases passed and a 100% Case pass rate. Its error is still a File failure; it does not invent a failed Case.
- Collection failures without collected Cases remain visible without fabricating a Case count or pass rate.
New reports generated from legacy tasks / flow YAML use the same view: each task is a Case and its flow entries are Steps. Inline input is labeled Inline YAML. Host setup or cleanup errors belong to the File, without inventing YAML hooks. Legacy CLI batch and whole-file retry reports retain their source identities; this view does not combine independent reports into a single logical Project or infer Case retry history. A source containing only an Agent dump is labeled Source report / Agent report, without claiming a YAML file or task/flow navigation. YAML view is available only when a real source snapshot exists. Regenerate historical HTML reports to use the new layout.
In a multi-Case report, click the floating menu button on the left (View all cases) to open the Case directory. The panel opens below the button without dimming the report. It lists all Cases grouped by Project regardless of Overview filters, with each result and source file; the current Case is highlighted. Select another Case to navigate directly, including across Projects. Failed Cases open at their failed Step. Opening the directory focuses the current Case. Use Tab to move between controls and Escape to close it and return focus to the menu button. Click the menu button again or close the panel to keep the current Case. Returning to Overview restores the original filters, sort, expansion, and scroll position. Single-Case reports omit this button.
In a Case, the Execution header groups Visual / YAML, the Attempts dropdown (when multiple attempts exist), and Step status counts. YAML view is available when the report includes a source YAML snapshot. Visual view places the screenshot timeline above a shared Step list and the selected Step's details. Lifecycle groups are separated by dividers in the list. On wide screens, drag the divider to resize the Step list, use arrow keys after focusing it, or double-click to reset it. Failed Steps in the displayed attempt or File scope are selected and brought into view. Earlier failed attempts remain available through Attempts and explicit failure links.
The selected Step has Information and Events tabs, plus Record when recording or screenshot evidence is available:
- Record shows the associated Agent execution or a captured screenshot. Hover over timeline thumbnails to preview them; click one to select its owning Step.
- Information presents Input, Output, and Stable IDs as aligned rows. Configuration, planned input, and Information captured at appear when available. The capture time uses your browser's local time zone, includes milliseconds and a UTC offset, and is not necessarily the Step's end time. Hover over it for the original UTC timestamp and a reminder to check Events for later completion or cancellation.
- Events shows Time, Operation, Event, and Status in a table. If detailed operation events are unavailable, Step start and end times are shown when known. Steps with no events show an empty state.
The Operation column describes what is running. Hover over an operation or event label to see its original log name and an explanation. Operation hints also include the Operation ID for matching log entries.
These operations are not consecutive, non-overlapping stages. Run step includes its internal operations, so do not add their durations to the Step's total duration. For example, Runner's wait and custom Nodes that do not use the Agent adapter only have Run step events. An aiAct or aiAssert Step typically gets an Agent, then runs an Agent operation inside Run step. Planning, locating, and individual UI actions are shown in Record.
Execution still running and a pending badge mean the operation had not ended when the Runner returned its result. This can happen on timeout or cancellation and does not describe its state after the report's capture boundary. A Status of - means the event has no outcome, not that it failed.
Step errors appear above these tabs. Expand details to view the error type, code, and failed Node. Stacks share one scrollable panel: the original error stack appears first, followed by the wrapping errors and framework stack. Other diagnostics and related errors remain available below. Stacks are hidden while details are collapsed and are included when copying issue details. Sensitive values are redacted; diagnostics exceeding the report size or cause-depth limits are marked as truncated. Use View all … issues when several issues exist. Case and lifecycle cleanup errors remain above Execution. The header's theme button switches between light and dark themes.
The report preserves the test plan captured before execution, executed and unexecuted Steps, reasons Steps did not run, and reasons retries stopped. Time spent waiting for an earlier operation is recorded separately from Step execution time. The report is a snapshot at publication: cleanup pending means cleanup was unfinished at that point and resources remain quarantined. Cleanup that completes later does not update the published HTML.
Reports are saved to:
With external screenshot storage, the report is a directory containing index.html and screenshots/. Copy or upload the whole directory.
Set output.reportDir in midscene.config.ts to change the report directory.
Link to a report Step
Use the runner-step URL hash parameter to open a report at a specific Step. In addition to a concrete Step ID, reports accept portable selectors that do not depend on generated run IDs:
Without an explicit Case route, selectors search the complete report. When the URL includes runner-page=case, runner-project, and runner-case, the selector searches only that Case. If no Step matches, the report shows an informational message and does not open an unrelated Agent trace.
For example, this URL opens the last failed Step in a report:
Add runner-trace=page to open the Step's Agent trace as a full page when the Step has associated Agent details:
Test case design conventions
Midscene Test uses sequential execution and explicit state sharing to make each Node's inputs, execution context, and dependencies clear. YAML cases follow these design choices:
- Execute in declaration order: YAML describes a linear sequence of test steps, without DAG, branch, or loop syntax. For scenarios that require complex orchestration, we recommend building YAML script generation at a higher layer. That layer resolves the orchestration into an explicit sequence of steps for Midscene Test to execute.
- Each Node is stateless: each Node invocation relies on its declared inputs and the context explicitly provided by the project, without automatically inheriting outputs from previous Steps. YAML provides no syntax for referencing outputs across steps or cases; when business state needs to be shared, implement the corresponding business logic in custom Nodes that read and update the project
context.
For example, the following assertion incorrectly refers to "the icon from the previous step", relying on the previous instruction to identify its target:
Name the target and its expected state explicitly so the assertion expresses the complete check on its own:
The interface retains the effects of previous actions, but each Node's instruction should describe its target independently, without references such as "the previous step" or "that item".
Next steps
To add business Nodes or share runtime data, read Develop custom Nodes. For platform integration and multiple execution projects, read Configure test projects.

