Integrate with Playwright

Playwright.js is an open-source automation library developed by Microsoft, primarily used for end-to-end testing and web scraping of web applications.

Here we assume you already have a repository with Playwright integration.

Example Project

You can find an example project of Playwright integration here: https://github.com/web-infra-dev/midscene-example/blob/main/playwright-demo

Prerequisites

Configure OpenAI API Key, or Custom Model and Provider

# Update with your own Key
export OPENAI_API_KEY="sk-abcdefghijklmnopqrstuvwxyz"

Step 1: Add Dependencies and Update Configuration

Add dependencies

npm
yarn
pnpm
bun
npm install @midscene/web --save-dev

Update playwright.config.ts

export default defineConfig({
  testDir: './e2e',
+ timeout: 90 * 1000,
+ reporter: [["list"], ["@midscene/web/playwright-report"]],
});

Step 2: Extend the test Instance

Save the following code as ./e2e/fixture.ts:

import { test as base } from '@playwright/test';
import type { PlayWrightAiFixtureType } from '@midscene/web/playwright';
import { PlaywrightAiFixture } from '@midscene/web/playwright';

export const test = base.extend<PlayWrightAiFixtureType>(PlaywrightAiFixture());

Step 3: Write Test Cases

Basic AI Operation APIs

  • ai or aiAction - General AI interaction
  • aiTap - Click operation
  • aiHover - Hover operation
  • aiInput - Input operation
  • aiKeyboardPress - Keyboard operation
  • aiScroll - Scroll operation

Query

  • aiQuery - AI Query

More APIs

  • aiAssert - AI Assertion
  • aiWaitFor - AI Wait

You can find more details in API Reference.

Example Code

./e2e/ebay-search.spec.ts
import { expect } from "@playwright/test";
import { test } from "./fixture";

test.beforeEach(async ({ page }) => {
  page.setViewportSize({ width: 400, height: 905 });
  await page.goto("https://www.ebay.com");
  await page.waitForLoadState("networkidle");
});

test("search headphone on ebay", async ({ 
  ai, 
  aiQuery, 
  aiAssert,
  aiInput,
  aiTap,
  aiScroll 
}) => {
  // Use aiInput to enter search keyword
  await aiInput('Headphones', 'search box');
  
  // Use aiTap to click search button
  await aiTap('search button');
  
  // Wait for search results to load
  await aiWaitFor('search results list loaded', { timeoutMs: 5000 });
  
  // Use aiScroll to scroll to bottom
  await aiScroll(
    { 
      direction: 'down',
      scrollType: 'untilBottom'
    },
    'search results list'
  );
  
  // Use aiQuery to get product information
  const items = await aiQuery<Array<{title: string, price: number}>>(
    'get product titles and prices from search results'
  );
  
  console.log("headphones in stock", items);
  expect(items?.length).toBeGreaterThan(0);
  
  // Use aiAssert to verify filter functionality
  await aiAssert("category filter exists on the left side");
});

For more Agent API details, please refer to API Reference.

Step 4. Run Test Cases

npx playwright test ./e2e/ebay-search.spec.ts

Step 5. View Test Report

After the command executes successfully, it will output: Midscene - report file updated: ./current_cwd/midscene_run/report/some_id.html. Open this file in your browser to view the report.

More