Automate with Scripts in YAML
In most cases, developers write automation just to perform some smoke tests, like checking the appearance of some content, or verifying that the key user path is accessible. Maintaining a large test project is unnecessary in this situation.
Midscene offers a way to do this kind of automation with .yaml files, which helps you to focus on the script itself instead of the test infrastructure. Any team member can write an automation script without learning any API.
Here is an example of .yaml script, you may have already understood how it works by reading its content.
target:
url: https://www.bing.com
tasks:
- name: search weather
flow:
- ai: search for 'weather today'
- sleep: 3000
- name: check result
flow:
- aiAssert: the result shows the weather info
Preparation
Config the OpenAI API key in the environment variable
# replace with your own
export OPENAI_API_KEY="sk-abcdefghijklmnopqrstuvwxyz"
or you can use a .env file to store the configuration, Midscene command line tool will automatically load it when running yaml scripts.
.env
OPENAI_API_KEY="sk-abcdefghijklmnopqrstuvwxyz"
For more details about model and provider, see config model and provider
Start
Install @midscene/cli globally
npm i -g @midscene/cli
# or if you prefer a project-wide installation
npm i @midscene/cli --save-dev
Write a yaml file to bing-search.yaml
target:
url: https://www.bing.com
tasks:
- name: search weather
flow:
- ai: search for 'weather today'
- sleep: 3000
- aiAssert: the result shows the weather info
Run this script
midscene ./bing-search.yaml
# or if you installed midscene inside the project
npx midscene ./bing-search.yaml
You should see that the output shows the progress of the running process and the report file.
Command line usage
Run single .yaml file
Run all .yaml files under a folder
midscene /dir/of/yaml/
# glob is also supported
midscene /dir/**/yaml/
Debug in headed mode
'headed mode' means the browser will be visible. The default behavior is to run in headless mode.
To turn on headed mode, you can use --headed option. Besides, if you want to keep the browser window open after the script finishes, you can use --keep-window option. --keep-window implies --headed.
When running in headed mode, it will consume more resources, so we recommend you to use it locally only when needed.
# run in headed mode
midscene /path/to/yaml --headed
# run in headed mode and keep the browser window open after the script finishes
midscene /path/to/yaml --keep-window
YAML file schema
There are two parts in a .yaml file, the target and the tasks.
target part
The target part defines the basic of a task
target:
# The URL to visit, required. If `serve` is provided, provide the path to the file to visit
url: <url>
# Serve the local path as a static server, optional
serve: <root-directory>
# The user agent to use, optional
userAgent: <ua>
# number, the viewport width, default is 1280, optional
viewportWidth: <width>
# number, the viewport height, default is 960, optional
viewportHeight: <height>
# number, the device scale factor (dpr), default is 1, optional
deviceScaleFactor: <scale>
# string, the path to the json format cookie file, optional
cookie: <path-to-cookie-file>
# object, the strategy to wait for network idle, optional
waitForNetworkIdle:
# number, the timeout in milliseconds, 10000ms for default, optional
timeout: <ms>
# boolean, continue on network idle error, true for default
continueOnNetworkIdleError: <boolean>
# string, the path to save the aiQuery result, optional
output: <path-to-output-file>
# boolean, if limit the popup to the current page, true for default in yaml script
forceSameTabNavigation: <boolean>
# string, the bridge mode to use, optional, default is false, can be 'newTabWithUrl' or 'currentTab'. More details see the following section
bridgeMode: false | 'newTabWithUrl' | 'currentTab'
# boolean, if close the new tabs after the bridge is disconnected, optional, default is false
closeNewTabsAfterDisconnect: <boolean>
# boolean, if allow insecure https certs, optional, default is false
acceptInsecureCerts: <boolean>
# string, the background knowledge to send to the AI model when calling aiAction, optional
aiActionContext: <string>
tasks part
The tasks part is an array indicates the tasks to do. Remember to write a - before each item which means an array item.
The interfaces of the flow part are almost the same as the API, except for some parameter levels.
tasks:
- name: <name>
continueOnError: <boolean> # optional, default is false
flow:
# Auto Planning (.ai)
# ----------------
# perform an action, this is the shortcut for aiAction
- ai: <prompt>
# this is the same as ai
- aiAction: <prompt>
# Instant Action(.aiTap, .aiHover, .aiInput, .aiKeyboardPress, .aiScroll)
# ----------------
# tap an element located by prompt
- aiTap: <prompt>
deepThink: <boolean> # optional, whether to use deepThink to precisely locate the element
# hover an element located by prompt
- aiHover: <prompt>
deepThink: <boolean> # optional, whether to use deepThink to precisely locate the element
# input text into an element located by prompt
- aiInput: <final text content of the input>
locate: <prompt>
deepThink: <boolean> # optional, whether to use deepThink to precisely locate the element
# press a key (like Enter, Tab, Escape, etc.) on an element located by prompt
- aiKeyboardPress: <key>
locate: <prompt>
deepThink: <boolean> # optional, whether to use deepThink to precisely locate the element
# scroll globally or on an element located by prompt
- aiScroll:
direction: 'up' # or 'down' | 'left' | 'right'
scrollType: 'once' # or 'untilTop' | 'untilBottom' | 'untilLeft' | 'untilRight'
distance: <number> # optional, distance to scroll in px
locate: <prompt> # optional, the element to scroll on
deepThink: <boolean> # optional, whether to use deepThink to precisely locate the element
# Data Extraction
# ----------------
# perform a query, return a json object
- aiQuery: <prompt> # remember to describe the format of the result in the prompt
name: <name> # the name of the result, will be used as the key in the output json
# More APIs
# ----------------
# wait for a condition to be met with a timeout (ms, optional, default 30000)
- aiWaitFor: <prompt>
timeout: <ms>
# perform an assertion
- aiAssert: <prompt>
# sleep for a number of milliseconds
- sleep: <ms>
- name: <name>
flow:
# ...
More features
Use environment variables in .yaml file
You can use environment variables in .yaml file by ${variable-name}.
For example, if you have a .env file with the following content:
You can use the environment variable in the .yaml file like this:
#...
- ai: type ${topic} in input box
#...
Use bridge mode
By using bridge mode, you can utilize YAML scripts to automate the web browser on your desktop. This is particularly useful if you want to reuse cookies, plugins, and page states, or if you want to manually interact with automation scripts.
To use bridge mode, you should install the Chrome extension first, and use this configuration in the target section:
target:
url: https://www.bing.com
+ bridgeMode: newTabWithUrl
See Bridge Mode by Chrome Extension for more details.
Run yaml script with javascript
You can also run a yaml script with javascript by using the runYaml method of the Midscene agent. Only the tasks part of the yaml script will be executed.
FAQ
How to get cookies in JSON format from Chrome?
You can use this chrome extension to export cookies in JSON format.
More
You may also be interested in Prompting Tips
