• English

      • Integrate with Playwright

      Playwright.js is an open-source automation library developed by Microsoft, mainly used for end-to-end testing and web scraping of web applications.

      There are two ways to integrate with Playwright:

      • Directly integrate and call the Midscene Agent via script, suitable for quick prototyping, data scraping, and automation scripts.
      • Integrate Midscene into Playwright test cases, suitable for UI testing scenarios.

      Set up API keys for model

      Set your model configs into the environment variables. You may refer to Model strategy for more details.

      export MIDSCENE_MODEL_BASE_URL="https://replace-with-your-model-service-url/v1"
export MIDSCENE_MODEL_API_KEY="replace-with-your-api-key"
export MIDSCENE_MODEL_NAME="replace-with-your-model-name"
export MIDSCENE_MODEL_FAMILY="replace-with-your-model-family"

      For more configuration details, please refer to Model strategy and Model configuration.

      Direct integration with Midscene agent

      Example Project

      You can find an example project of direct Playwright integration here: https://github.com/web-infra-dev/midscene-example/blob/main/playwright-demo

      Step 1: Install dependencies

      npm
      yarn
      pnpm
      bun
      deno
      npm install @midscene/web playwright @playwright/test tsx --save-dev

      Step 2: Write the script

      Save the following code as ./demo.ts:

      import { chromium } from 'playwright';
import { PlaywrightAgent } from '@midscene/web/playwright';
import 'dotenv/config'; // read environment variables from .env file

const sleep = (ms) => new Promise((r) => setTimeout(r, ms));

Promise.resolve(
  (async () => {
    const browser = await chromium.launch({
      headless: true, // 'true' means we can't see the browser window
      args: ['--no-sandbox', '--disable-setuid-sandbox'],
    });

    const page = await browser.newPage();
    await page.setViewportSize({
      width: 1280,
      height: 768,
    });
    await page.goto('https://www.ebay.com');
    await sleep(5000); // 👀 init Midscene agent
    const agent = new PlaywrightAgent(page);

    // 👀 type keywords, perform a search
    await agent.aiAct('type "Headphones" in search box, hit Enter');

    // 👀 wait for the loading
    await agent.aiWaitFor('there is at least one headphone item on page');
    // or you may use a plain sleep:
    // await sleep(5000);

    // 👀 understand the page content, find the items
    const items = await agent.aiQuery(
      '{itemTitle: string, price: Number}[], find item in list and corresponding price',
    );
    console.log('headphones in stock', items);

    const isMoreThan1000 = await agent.aiBoolean(
      'Is the price of the headphones more than 1000?',
    );
    console.log('isMoreThan1000', isMoreThan1000);

    const price = await agent.aiNumber(
      'What is the price of the first headphone?',
    );
    console.log('price', price);

    const name = await agent.aiString(
      'What is the name of the first headphone?',
    );
    console.log('name', name);

    const location = await agent.aiLocate(
      'What is the location of the first headphone?',
    );
    console.log('location', location);

    // 👀 assert by AI
    await agent.aiAssert('There is a category filter on the left');

    // 👀 click on the first item
    await agent.aiTap('the first item in the list');

    await browser.close();
  })(),
);

      For more Agent API details, please refer to API Reference.

      Step 3: Run the script

      Use tsx to run, and you will see the product information printed in the terminal:

      # run
npx tsx demo.ts

# The terminal should output something like:
#  [
#   {
#     itemTitle: 'JBL Tour Pro 2 - True wireless Noise Cancelling earbuds with Smart Charging Case',
#     price: 551.21
#   },
#   {
#     itemTitle: 'Soundcore Space One Wireless Headphones 40H ANC Playtime 2XStronger Voice',
#     price: 543.94
#   }
# ]

      Step 4: View the run report

      After the above command executes successfully, it will output: Midscene - report file updated: /path/to/report/some_id.html. Open this file in your browser to view the report.

      Integration in Playwright test cases

      Here we assume you already have a repository with Playwright integration.

      Example Project

      You can find an example project of Playwright test integration here: https://github.com/web-infra-dev/midscene-example/blob/main/playwright-testing-demo

      Step 1: Add dependencies and update configuration

      Add dependencies

      npm
      yarn
      pnpm
      bun
      deno
      npm install @midscene/web --save-dev

      Update playwright.config.ts

      export default defineConfig({
  testDir: './e2e',
+ timeout: 90 * 1000,
+ reporter: [["list"], ["@midscene/web/playwright-reporter", { type: "merged" }]], // type optional, default is "merged", means multiple test cases generate one report, optional value is "separate", means one report for each test case
});

      Reporter configuration options:

      • type: Report mode, can be merged (default) or separate. merged means multiple test cases generate one merged report, separate means each test case generates its own report.

      • outputFormat: Controls how the report is generated. 'single-html' (default) embeds all screenshots as base64 in a single HTML file. 'html-and-external-assets' saves screenshots as separate PNG files in a subdirectory, useful when report files are too large. Note: When using 'html-and-external-assets', reports must be served via HTTP server and cannot be opened directly using file:// protocol (because browser CORS restrictions block loading local images via relative paths from the file protocol). Navigate to the report directory and run one of the following commands:

        • Using Node.js: npx serve
        • Using Python: python -m http.server or python3 -m http.server

        Then access the report via http://localhost:3000 (or the port shown in the terminal).

      Step 2: Extend the test instance

      Save the following code as ./e2e/fixture.ts:

      import { test as base } from '@playwright/test';
import type { PlayWrightAiFixtureType } from '@midscene/web/playwright';
import { PlaywrightAiFixture } from '@midscene/web/playwright';

export const test = base.extend<PlayWrightAiFixtureType>(
  PlaywrightAiFixture({
    waitForNetworkIdleTimeout: 2000, // optional, the timeout for waiting for network idle between each action, default is 2000ms
  }),
);

      Step 3: Write test cases

      Review the full catalog of action, query, and utility methods in the Agent API reference. When you need lower-level control, you can use agentForPage to obtain the underlying PageAgent instance and call any API directly:

      test('case demo', async ({ agentForPage, page }) => {
  const agent = await agentForPage(page);

  await agent.recordToReport();
  const logContent = agent._unstableLogContent();
  console.log(logContent);
});

      Example code

      ./e2e/ebay-search.spec.ts
      import { expect } from '@playwright/test';
import { test } from './fixture';

test.beforeEach(async ({ page }) => {
  page.setViewportSize({ width: 400, height: 905 });
  await page.goto('https://www.ebay.com');
  await page.waitForLoadState('networkidle');
});

test('search headphone on ebay', async ({
  ai,
  aiQuery,
  aiAssert,
  aiInput,
  aiTap,
  aiScroll,
  aiWaitFor,
  aiRightClick,
  recordToReport,
}) => {
  // Use aiInput to enter search keyword
  await aiInput('Headphones', 'search box');

  // Use aiTap to click search button
  await aiTap('search button');

  // Wait for search results to load
  await aiWaitFor('search results list loaded', { timeoutMs: 5000 });

  // Use aiScroll to scroll to bottom
  await aiScroll(
    {
      scrollType: 'untilBottom',
    },
    'search results list',
  );

  // Use aiQuery to get product information
  const items = await aiQuery<Array<{ title: string; price: number }>>(
    'get product titles and prices from search results',
  );

  console.log('headphones in stock', items);
  expect(items?.length).toBeGreaterThan(0);

  // Use aiAssert to verify filter functionality
  await aiAssert('category filter exists on the left side');

  // Use recordToReport to capture the current state
  await recordToReport('Search Results', {
    content: 'Final search results for headphones',
  });
});

      For more Agent API details, please refer to API Reference.

      Step 4. Run test cases

      npx playwright test ./e2e/ebay-search.spec.ts

      Step 5. View test report

      After the command executes successfully, it will output: Midscene - report file updated: ./current_cwd/midscene_run/report/some_id.html. Open this file in your browser to view the report.

      Advanced

      About opening in a new tab

      Each Agent instance is bound to a single page. To make debugging easier, Midscene intercepts new tabs by default (for example, links with target="_blank") and opens them in the current page.

      If you want to restore opening in a new tab, set forceSameTabNavigation to false—but you’ll need to create a new Agent instance for each new tab.

      const mid = new PlaywrightAgent(page, {
  forceSameTabNavigation: false,
});

      Connect Midscene Agent to a Remote Playwright Browser

      Example Project

      You can find an example project of remote Playwright integration here: https://github.com/web-infra-dev/midscene-example/tree/main/remote-playwright-demo

      Connect to a remote Playwright browser when you already run browsers in your own infra or vendor grid. This keeps the browser close to the target environment, avoids repeated launches, and still lets Midscene drive it with the same AI APIs.

      Prerequisites

      npm
      yarn
      pnpm
      bun
      deno
      npm install playwright @playwright/test @midscene/web --save-dev

      Getting a CDP WebSocket URL

      You can get a CDP WebSocket URL from various sources, for example:

      • BrowserBase: Sign up at https://browserbase.com and get your CDP URL
      • Browserless: Use https://browserless.io or run your own instance
      • Local Chrome: Run Chrome with --remote-debugging-port=9222 and use ws://localhost:9222/devtools/browser/...
      • Docker: Run Chrome in a Docker container with debugging port exposed

      Code example

      import { chromium } from 'playwright';
import { PlaywrightAgent } from '@midscene/web/playwright';

// CDP WebSocket URL from your remote browser service
const cdpWsUrl = 'ws://your-remote-browser.com/devtools/browser/your-session-id';

// Connect and pick a page
const browser = await chromium.connectOverCDP(cdpWsUrl);
const context = browser.contexts()[0];
const page = context.pages()[0] || await context.newPage();

// Create Midscene agent (usage matches any Playwright agent)
const agent = new PlaywrightAgent(page);

// Use AI methods as usual
await agent.aiAct('navigate to https://example.com');
await agent.aiAct('click the login button');
const result = await agent.aiQuery('get page title: {title: string}');

// Cleanup
await agent.destroy();
await browser.close();

      Once connected, keep using PlaywrightAgent the same way you would with a locally launched browser.

      Provide custom actions

      Use the customActions option to extend the agent's action space with your own actions defined via defineAction. When provided, these actions will be appended to the built-in ones so the agent can call them during planning.

      import { getMidsceneLocationSchema, z } from '@midscene/core';
import { defineAction } from '@midscene/core/device';

const ContinuousClick = defineAction({
  name: 'continuousClick',
  description: 'Click the same target repeatedly',
  paramSchema: z.object({
    locate: getMidsceneLocationSchema(),
    count: z
      .number()
      .int()
      .positive()
      .describe('How many times to click'),
  }),
  async call(param) {
    const { locate, count } = param;
    console.log('click target center', locate.center);
    console.log('click count', count);
    // carry out your clicking logic using locate + count
  },
});

const agent = new PlaywrightAgent(page, {
  customActions: [ContinuousClick],
});

await agent.aiAct('click the red button five times');

      Check Integrate with any interface for more details about defining custom actions.

      FAQ

      The webpage continues to flash when running in headed mode

      In the local visualization interface, continuous flashing is usually caused by a mismatch between the viewport's deviceScaleFactor and the system/browser's pixel ratio (common on high-resolution or Retina screens).

      This flashing does not affect Midscene's screenshots or automation execution, but it does affect the local preview experience. To resolve this, set deviceScaleFactor to match your browser's window.devicePixelRatio, or use Puppeteer's auto-adaptation feature.

      // Playwright: Playwright does not support using 0 for auto-adaptation like Puppeteer
const page = await browser.newPage({
  deviceScaleFactor: 2, // Replace the number 2 with your window.devicePixelRatio
})

      If you are unsure of your browser's pixel ratio, you can press F12 on any page to open the console and type window.devicePixelRatio to check; or paste the following into the Chrome address bar and press Enter to see the value in a popup:

      data:text/html,<script>alert(`deviceScaleFactor of your browser: ${devicePixelRatio}`)</script>

      Customize the network timeout

      When doing interaction or navigation on web page, Midscene automatically waits for the network to be idle. It's a strategy to ensure the stability of the automation. Nothing would happen if the waiting process is timeout.

      The default timeout is configured as follows:

      1. If it's a page navigation, the default wait timeout is 5000ms (the waitForNavigationTimeout)
      2. If it's a click, input, etc., the default wait timeout is 2000ms (the waitForNetworkIdleTimeout)

      You can also customize or disable the timeout by options:

      • Use waitForNetworkIdleTimeout and waitForNavigationTimeout parameters in Agent.
      • Use waitForNetworkIdle parameter in Yaml or PlaywrightAiFixture.

      waiting for fonts to load or page.screenshot: Timeout ... exceeded when taking screenshots

      If you see an error like this in a Playwright-based environment:

      page.screenshot: Timeout 10000ms exceeded.
Call log:
- taking page screenshot
- waiting for fonts to load...

      This is usually not caused by Midscene itself. Playwright waits for fonts to finish loading before taking a screenshot. In some CI, container, or restricted network environments, font resources may load very slowly or never finish, which can eventually cause the screenshot to time out.

      You can work around it by setting this environment variable:

      export PW_TEST_SCREENSHOT_NO_FONTS_READY=1

      If you want to set it only for a single command, you can also write:

      PW_TEST_SCREENSHOT_NO_FONTS_READY=1 <your-command>

      For more background, see the Playwright issue: [BUG] Page.screenshot method hangs indefinitely.

      More