#Run tests

#Basic usage

Run YAML test cases from the project root:

pnpm exec midscene-test

#Specify a test case directory or file

Projects generated from templates use files.include to match .yaml and .yml files under cases/ by default. If files is not declared in the configuration, Midscene Test recursively searches for all YAML files under the test root directory (automatically ignoring node_modules and .git).

When passing a directory or file path on the command line, be aware of how the test root changes and affects path resolution:

• Determining the test root: Without path arguments, the test root is the current working directory. With a directory argument, the test root becomes that directory. With a single file argument, the test root becomes the file's containing directory.
• Resolved relative to the test root: Default configuration file discovery (searches only within the test root, without traversing parent directories), the --config parameter path, and files.include / files.exclude patterns in the configuration.
• Resolved relative to the current working directory: The --result-dir parameter path.

Therefore, when midscene.config.ts lives in the project root and you only want to run a single file in the cases/ subdirectory, you need to explicitly pass --config (note that it resolves relative to the cases/ directory where the target file is located):

pnpm exec midscene-test ./cases/order.yaml --config ../midscene.config.ts

When targeting a single file directly, that file is not restricted by files.include / files.exclude rules, but tags filtering still applies. If you want to run only a specific subdirectory (such as cases/smoke), it is recommended to run the command from the project root and restrict the scope by setting files.include to cases/smoke/**/*.{yaml,yml} in the configuration, avoiding changes to the test root.

#Filter execution targets and configuration files

If a project defines multiple platforms or environments, you can use --project to run specific Execution Projects, or use --config to specify a custom configuration file:

# Run only the android-smoke and ios-regression Execution Projects
pnpm exec midscene-test --project android-smoke --project ios-regression

# Run tests with a specific configuration file
pnpm exec midscene-test --config ./config/midscene.config.ts

When generating the Node Spec for a specific Execution Project, you can also use --project:

pnpm exec midscene-test nodes --project android-smoke

#Output and exit codes

#Terminal output mode

By default, the terminal prints summarized execution results grouped by Case. To view detailed step diagnostics, use the verbose reporter. Run --help to view all supported CLI options:

# Show detailed step diagnostics
pnpm exec midscene-test --reporter verbose

# View all CLI options
pnpm exec midscene-test --help

#Result files and report paths

Each run generates two types of output artifacts. Their exact paths are printed at the end of the terminal output as Summary: and Report::

• Structured result files: By default written to .midscene/test-results/<runId>/ under the test root, containing machine-readable summary.json and human-readable result.md. You can customize the root results directory with --result-dir (each run will still create an isolated <runId> subdirectory underneath):

  pnpm exec midscene-test --result-dir ./test-results
  Note: The schema of summary.json may evolve across versions; external tools consuming this file should align with the actual output of the installed version.
• HTML visual report: Can be opened directly in a browser after tests finish. It provides a complete log of case outcomes, screenshots for each step, input/output arguments, error traces, and Agent visual replay to help diagnose issues quickly. The output directory is controlled by output.reportDir in the configuration file (defaults to midscene_run/report/), independent of the structured results directory.

#Exit codes

You can determine the overall outcome of a test run from the CLI exit code:

• 0: All selected cases and lifecycle hooks executed successfully.
• 1: An error occurred during execution (including case step failures, YAML parsing or collection errors, and project-level or file-level lifecycle errors). Even if all case steps pass, the exit code is still 1 if an exception occurs during the cleanup phase.

#Report archiving and URL deep links

#Report archiving

Reports default to a single HTML file: midscene_run/report/midscene-e2e-<runId>.html. When external screenshot mode is enabled, the report directory contains index.html and a screenshots/ directory; preserve the entire directory when archiving or uploading artifacts. See Run in GitHub Actions for CI artifact configuration.

#Link to a specific step (URL hash)

When sharing reports in CI notifications, chat bots, or PR comments, use the runner-step URL hash parameter to jump directly to a target Step. Beyond explicit Step IDs, reports support portable selectors:

Example: Jump directly to the last failed Step in a report:

https://example.com/report/index.html#runner-step=last-error

If the Step includes Agent visual replay, append runner-trace=page to open the Agent trace in full-screen view:

https://example.com/report/index.html#runner-step=last-error&runner-trace=page

To limit selection to a specific case, combine with runner-page=case, runner-project=<projectId>, and runner-case=<caseKey>. Use the report's internal projectId and the composite Case Key <projectId>:<documentId>:<caseId>. URL-encode each parameter value, for example with URLSearchParams. To obtain these values, open the target case in the report and copy its URL, then add runner-step and, if needed, runner-trace.

#Environment loading

Midscene Test reads model configurations and runtime parameters from system environment variables (process.env). You can provide environment variables based on your workflow:

#Local development with .env files

• Scaffolded projects (ready out of the box): Projects created via midscene-test create already load the .env file from the configuration directory in midscene.config.ts. Simply create a .env file in the project root to get started.
• Handwritten configuration: When writing a configuration file manually, import dotenv at the top of midscene.config.ts:

  import 'dotenv/config';

When loaded through dotenv, existing environment variables in your shell take precedence over values in the .env file by default.

#CI / CD environments

In continuous integration (such as GitHub Actions) or automated pipelines, you do not need a .env file. Simply inject API keys or credentials directly as pipeline environment variables.

#Model connectivity check (model verify)

When running midscene-test model verify to test model connectivity, the command automatically reads .env from the current working directory. To facilitate quick local checks, it prioritizes values in .env over matching shell environment variables. When troubleshooting, ensure the settings in .env match the model configuration used during test runs.

#Run in GitHub Actions

Midscene provides midscene-action to run Midscene Test in GitHub Actions.

• Automatically add test results to the workflow Summary.
• Save HTML reports and test data as artifacts for download.
• Optionally publish reports to Cloudflare Workers for online previews.

For setup instructions, configuration parameters, and complete examples, see midscene-action on GitHub Marketplace.