• English

      • Quick experience by Chrome extension

      Midscene.js provides a Chrome extension. By using it, you can quickly experience the main features of Midscene on any webpage, without needing to set up a code project.

      ⁠The extension shares the same code as the npm @midscene/web packages, so you can think of it as a playground or a way to debug with Midscene.

      Prompt : Sign up for Github, you need to pass the form validation, but don't actually click.

      View the full report of this task: report.html

      Install Chrome extension

      Install Midscene extension from chrome web store: Midscene

      Start the extension (may be folded by Chrome extension icon), you should see the sidebar named "Midscene" in the right side of the browser.

      Set up API keys for model

      Set the model configuration through environment variables. See Model strategy for guidance on choosing a model.

      export MIDSCENE_MODEL_BASE_URL="https://replace-with-your-model-service-url/v1"
export MIDSCENE_MODEL_API_KEY="replace-with-your-api-key"
export MIDSCENE_MODEL_NAME="replace-with-your-model-name"
export MIDSCENE_MODEL_FAMILY="replace-with-your-model-family"

      For all configuration options, see Model configuration.

      See more showcases: showcases

      Start experiencing

      After configuring the model, try these core tabs:

      • Act: interact with the interface through Auto Planning. This corresponds to aiAct. For example:
      Type “Midscene” in the search box, run the search, and open the first result
      Fill out the registration form and make sure every field passes validation
      • Query: extract structured data from the interface. This corresponds to aiQuery.

      Similar methods include aiBoolean(), aiNumber(), and aiString() for directly extracting booleans, numbers, and strings.

      Extract the user ID from the page and return JSON data in the { id: string } structure
      • Assert: verify a condition in the interface and throw an error when it is not met. This corresponds to aiAssert.
      There is a login button on the page, with a user agreement link below it
      • Tap: click an element through Instant Action. This corresponds to aiTap.
      Click the login button

      For the difference between Auto Planning and Instant Action, see the common API reference.

      FAQ

      Can I install the extension manually?

      If you cannot access the Chrome Web Store, you can download the installation package from the Releases page on GitHub and install it manually. However, this is not recommended as automatic updates will not be available.

      Extension fails to run and shows 'Cannot access a chrome-extension:// URL of different extension'

      It's mainly due to conflicts with other extensions injecting <iframe /> or <script /> into the page. Try disabling the suspicious plugins and refresh.

      To find the suspicious plugins:

      1. Open the Devtools of the page, find the <script> or <iframe> with a url like chrome-extension://{ID-of-the-suspicious-plugin}/....
      2. Copy the ID from the url, open chrome://extensions/ , use cmd+f to find the plugin with the same ID, disable it.
      3. Refresh the page, try again.

      Getting a 403 error when using Ollama model in Chrome extension

      You need to set the environment variable OLLAMA_ORIGINS="*" to allow the Chrome extension to access the Ollama model.