Quick Experience by Chrome Extension

Midscene.js provides a Chrome extension. By using it, you can quickly experience the main features of Midscene on any webpage, without needing to set up a code project.

⁠The extension shares the same code as the npm @midscene/web packages, so you can think of it as a playground or a way to debug with Midscene.

Preparation

Prepare the config for the AI model you want to use. You can check the supported models in Choose a model

Install and config

Install Midscene extension from chrome web store: Midscene

Start the extension (may be folded by Chrome extension icon), setup the config by pasting the config in the K=V format:

OPENAI_API_KEY="sk-replace-by-your-own"
# ...all other configs here (if any)

Start experiencing

After the configuration, you can immediately experience Midscene. There are three main tabs in the extension:

  • Action: interact with the web page. This is also known as "Auto Planning". For example:
type Midscene in the search box click the login button
  • Query: extract JSON data from the web page
extract the user id from the page, return in \{ id: string \}
  • Assert: validate the page
the page title is "Midscene"
  • Tap: perform a single tap on the element where you want to click. This is also known as "Instant Action".
the login button

Enjoy !

For the different between "Auto Planning" and "Instant Action", please refer to the API document.

Want to write some code ?

After experiencing, you may want to write some code to integrate Midscene. There are multiple ways to do that. Please refer to the documents below:

FAQ

Extension fails to run and shows 'Cannot access a chrome-extension:// URL of different extension'

It's mainly due to conflicts with other extensions injecting <iframe /> or <script /> into the page. Try disabling the suspicious plugins and refresh.

To find the suspicious plugins:

  1. Open the Devtools of the page, find the <script> or <iframe> with a url like chrome-extension://{ID-of-the-suspicious-plugin}/....
  2. Copy the ID from the url, open chrome://extensions/ , use cmd+f to find the plugin with the same ID, disable it.
  3. Refresh the page, try again.