Automate with Scripts in YAML

In most cases, developers write automation just to perform some smoke tests, like checking the appearance of some content, or verifying that the key user path is accessible. Maintaining a large test project is unnecessary in this situation.

⁠Midscene offers a way to do this kind of automation with .yaml files, which helps you to focus on the script itself instead of the test infrastructure. Any team member can write an automation script without learning any API.

Here is an example of .yaml script, you may have already understood how it works by reading its content.

web:
  url: https://www.bing.com

tasks:
  - name: search weather
    flow:
      - ai: search for 'weather today'
      - sleep: 3000

  - name: check result
    flow:
      - aiAssert: the result shows the weather info
Demo Project

You can find the demo project with YAML scripts https://github.com/web-infra-dev/midscene-example/tree/main/yaml-scripts-demo

Setup AI model service

Set your model configs into the environment variables. You may refer to choose a model for more details.

# replace with your own
export OPENAI_API_KEY="sk-abcdefghijklmnopqrstuvwxyz"

or you can use a .env file locate at the same directory as you run the command to store the configuration, Midscene command line tool will automatically load it.

.env
OPENAI_API_KEY="sk-abcdefghijklmnopqrstuvwxyz"

Install Command Line Tool

Install @midscene/cli globally

npm i -g @midscene/cli
# or if you prefer a project-wide installation
npm i @midscene/cli --save-dev

Write a yaml file to bing-search.yaml to automate in web browser :

web:
  url: https://www.bing.com

tasks:
  - name: search weather
    flow:
      - ai: search for 'weather today'
      - sleep: 3000
      - aiAssert: the result shows the weather info

or to automate in Android device connected by adb :

android:
  # launch: https://www.bing.com
  deviceId: s4ey59

tasks:
  - name: search weather
    flow:
      - ai: open browser and navigate to bing.com
      - ai: search for 'weather today'
      - sleep: 3000
      - aiAssert: the result shows the weather info

Run this script

midscene ./bing-search.yaml
# or if you installed midscene inside the project
npx midscene ./bing-search.yaml

You should see that the output shows the progress of the running process and the report file.

Command line usage

Run single.yaml file

midscene /path/to/yaml

Run all.yaml files under a folder

midscene /dir/of/yaml/

# glob is also supported
midscene /dir/**/yaml/

YAML file schema

There are two parts in a .yaml file, the web/android and the tasks.

The web/android part defines the basic of a task. Use web parameter (also previously named as target) for web browser automation, and use android parameter for Android device automation. They are mutually exclusive.

Theweb part

web:
  # The URL to visit, required. If `serve` is provided, provide the path to the file to visit
  url: <url>

  # Serve the local path as a static server, optional
  serve: <root-directory>

  # The user agent to use, optional
  userAgent: <ua>

  # number, the viewport width, default is 1280, optional
  viewportWidth: <width>

  # number, the viewport height, default is 960, optional
  viewportHeight: <height>

  # number, the device scale factor (dpr), default is 1, optional
  deviceScaleFactor: <scale>

  # string, the path to the json format cookie file, optional
  cookie: <path-to-cookie-file>

  # object, the strategy to wait for network idle, optional
  waitForNetworkIdle:
    # number, the timeout in milliseconds, 2000ms for default, optional
    timeout: <ms>
    # boolean, continue on network idle error, true for default
    continueOnNetworkIdleError: <boolean>

  # string, the path to save the aiQuery/aiAssert result, optional
  output: <path-to-output-file>

  # string, the path to save the log content in JSON format, optional. If true, save to `unstableLogContent.json` file. If a string, save to the specified path. The structure of the log content may change in the future.
  unstableLogContent: <boolean | path-to-unstable-log-file>

  # boolean, if limit the popup to the current page, true for default in yaml script
  forceSameTabNavigation: <boolean>

  # string, the bridge mode to use, optional, default is false, can be 'newTabWithUrl' or 'currentTab'. More details see the following section
  bridgeMode: false | 'newTabWithUrl' | 'currentTab'

  # boolean, if close the new tabs after the bridge is disconnected, optional, default is false
  closeNewTabsAfterDisconnect: <boolean>

  # boolean, if allow insecure https certs, optional, default is false
  acceptInsecureCerts: <boolean>

  # string, the background knowledge to send to the AI model when calling aiAction, optional
  aiActionContext: <string>

Theandroid part

android:
  # The device id to use, optional, default is the first connected device
  deviceId: <device-id>

  # The url to launch, optional, default is the current page
  launch: <url>

Thetasks part

The tasks part is an array indicates the tasks to do. Remember to write a - before each item which means an array item.

The interfaces of the flow part are almost the same as the API, except for some parameter levels.

tasks:
  - name: <name>
    continueOnError: <boolean> # optional, default is false
    flow:
      # Auto Planning (.ai)
      # ----------------

      # perform an action, this is the shortcut for aiAction
      - ai: <prompt>
        cacheable: <boolean> # optional, whether cacheable when enabling [caching feature](./caching.mdx). True by default.

      # this is the same as ai
      - aiAction: <prompt>
        cacheable: <boolean> # optional, whether cacheable when enabling [caching feature](./caching.mdx). True by default.

      # Instant Action(.aiTap, .aiHover, .aiInput, .aiKeyboardPress, .aiScroll)
      # ----------------

      # tap an element located by prompt
      - aiTap: <prompt>
        deepThink: <boolean> # optional, whether to use deepThink to precisely locate the element. False by default.
        xpath: <xpath> # optional, the xpath of the element to operate. If provided, Midscene will first use this xpath to locate the element before using the cache and the AI model. Empty by default.
    - `cacheable?: boolean` - Whether cacheable when enabling [caching feature](./caching.mdx). True by default.
        cacheable: <boolean> # optional, whether cacheable when enabling [caching feature](./caching.mdx). True by default.

      # hover an element located by prompt
      - aiHover: <prompt>
        deepThink: <boolean> # optional, whether to use deepThink to precisely locate the element. False by default.
        xpath: <xpath> # optional, the xpath of the element to operate. If provided, Midscene will first use this xpath to locate the element before using the cache and the AI model. Empty by default.
    - `cacheable?: boolean` - Whether cacheable when enabling [caching feature](./caching.mdx). True by default.
        cacheable: <boolean> # optional, whether cacheable when enabling [caching feature](./caching.mdx). True by default.

      # input text into an element located by prompt
      - aiInput: <final text content of the input>
        locate: <prompt>
        deepThink: <boolean> # optional, whether to use deepThink to precisely locate the element. False by default.
        xpath: <xpath> # optional, the xpath of the element to operate. If provided, Midscene will first use this xpath to locate the element before using the cache and the AI model. Empty by default.
    - `cacheable?: boolean` - Whether cacheable when enabling [caching feature](./caching.mdx). True by default.
        cacheable: <boolean> # optional, whether cacheable when enabling [caching feature](./caching.mdx). True by default.

      # press a key (like Enter, Tab, Escape, etc.) on an element located by prompt
      - aiKeyboardPress: <key>
        locate: <prompt>
        deepThink: <boolean> # optional, whether to use deepThink to precisely locate the element. False by default.
        xpath: <xpath> # optional, the xpath of the element to operate. If provided, Midscene will first use this xpath to locate the element before using the cache and the AI model. Empty by default.
    - `cacheable?: boolean` - Whether cacheable when enabling [caching feature](./caching.mdx). True by default.
        cacheable: <boolean> # optional, whether cacheable when enabling [caching feature](./caching.mdx). True by default.

      # scroll globally or on an element located by prompt
      - aiScroll:
        direction: 'up' # or 'down' | 'left' | 'right'
        scrollType: 'once' # or 'untilTop' | 'untilBottom' | 'untilLeft' | 'untilRight'
        distance: <number> # optional, distance to scroll in px
        locate: <prompt> # optional, the element to scroll on
        deepThink: <boolean> # optional, whether to use deepThink to precisely locate the element. False by default.
        xpath: <xpath> # optional, the xpath of the element to operate. If provided, Midscene will first use this xpath to locate the element before using the cache and the AI model. Empty by default.
    - `cacheable?: boolean` - Whether cacheable when enabling [caching feature](./caching.mdx). True by default.
        cacheable: <boolean> # optional, whether cacheable when enabling [caching feature](./caching.mdx). True by default.

      # log the current screenshot with a description in the report file
      - logScreenshot: <title> # optional, the title of the screenshot, if not provided, the title will be 'untitled'
        content: <content> # optional, the screenshot description

      # Data Extraction
      # ----------------

      # perform a query, return a json object
      - aiQuery: <prompt> # remember to describe the format of the result in the prompt
        name: <name> # the name of the result, will be used as the key in the output json

      # More APIs
      # ----------------

      # wait for a condition to be met with a timeout (ms, optional, default 30000)
      - aiWaitFor: <prompt>
        timeout: <ms>

      # perform an assertion
      - aiAssert: <prompt>
        errorMessage: <error-message> # optional, the error message to print when the assertion fails

      # sleep for a number of milliseconds
      - sleep: <ms>

      # evaluate a javascript expression in web page context
      - javascript: <javascript>
        name: <name> # assign a name to the return value, will be used as the key in the output json, optional

  - name: <name>
    flow:
      # ...

More features

Use environment variables in.yaml file

You can use environment variables in .yaml file by ${variable-name}.

For example, if you have a .env file with the following content:

.env
topic=weather today

You can use the environment variable in the .yaml file like this:

#...
- ai: type ${topic} in input box
#...

Debug in headed mode

web scenario only

'headed mode' means the browser will be visible. The default behavior is to run in headless mode.

To turn on headed mode, you can use --headed option. Besides, if you want to keep the browser window open after the script finishes, you can use --keep-window option. --keep-window implies --headed.

When running in headed mode, it will consume more resources, so we recommend you to use it locally only when needed.

# run in headed mode
midscene /path/to/yaml --headed

# run in headed mode and keep the browser window open after the script finishes
midscene /path/to/yaml --keep-window

Use bridge mode

web scenario only By using bridge mode, you can utilize YAML scripts to automate the web browser on your desktop. This is particularly useful if you want to reuse cookies, plugins, and page states, or if you want to manually interact with automation scripts.

To use bridge mode, you should install the Chrome extension first, and use this configuration in the target section:

web:
  url: https://www.bing.com
+ bridgeMode: newTabWithUrl

See Bridge Mode by Chrome Extension for more details.

Run yaml script with javascript

You can also run a yaml script with javascript by using the runYaml method of the Midscene agent. Only the tasks part of the yaml script will be executed.

Config default behavior of dotenv

Midscene uses dotenv to load environment variables from .env file to set the environment variables.

Debug log

The debug log of dotenv will be printed by default. If you don't want to see these information, you can use --dotenv-debug option.

midscene /path/to/yaml --dotenv-debug=false

Use .env file to override global environment variables

By default, dotenv will NOT override the same name environment variable in the .env file if the global environment variable is already set.

If you want to do this, you can use --dotenv-override option.

midscene /path/to/yaml --dotenv-override=true

FAQ

How to get cookies in JSON format from Chrome?

You can use this chrome extension to export cookies in JSON format.

More

You may also be interested in Prompting Tips