Bridge Mode by Chrome Extension#

The bridge mode in the Midscene Chrome extension is a tool that allows you to use local scripts to control the desktop version of Chrome. Your scripts can connect to either a new tab or the currently active tab.

Using the desktop version of Chrome allows you to reuse all cookies, plugins, page status, and everything else you want. You can work with automation scripts to complete your tasks. This mode is commonly referred to as 'man-in-the-loop' in the context of automation.

Setup AI model service#

Set your model configs into the environment variables. You may refer to choose a model for more details.

# replace with your own
export OPENAI_API_KEY="sk-abcdefghijklmnopqrstuvwxyz"

In bridge mode, the AI models configs should be set in the Node.js side instead of the browser side.

Step 1. Install Midscene extension from Chrome web store#

Install Midscene extension from Chrome web store

Step 2. install dependencies#

npm install @midscene/web tsx --save-dev

Step 3. write scripts#

Write and save the following code as ./demo-new-tab.ts.

import { AgentOverChromeBridge } from "@midscene/web/bridge-mode";

const sleep = (ms) => new Promise((r) => setTimeout(r, ms));
Promise.resolve(
  (async () => {
    const agent = new AgentOverChromeBridge();

    // This will connect to a new tab on your desktop Chrome
    // remember to start your chrome extension, click 'allow connection' button. Otherwise you will get an timeout error
    await agent.connectNewTabWithUrl("https://www.bing.com");

    // these are the same as normal Midscene agent
    await agent.ai('type "AI 101" and hit Enter');
    await sleep(3000);

    await agent.aiAssert("there are some search results");
    await agent.destroy();
  })()
);

Step 4. start the chrome extension#

Start the chrome extension and switch to 'Bridge Mode' tab. Click "Allow connection".

Step 5. run the script#

Run your scripts

tsx demo-new-tab.ts

After executing the script, you should see the status of the Chrome extension switched to 'connected', and a new tab has been opened. Now this tab is controlled by your scripts.

Constructor#

import { AgentOverChromeBridge } from "@midscene/web/bridge-mode";

const agent = new AgentOverChromeBridge();

Except the normal parameters in the agent constructor, AgentOverChromeBridge accepts one more parameter:

• closeNewTabsAfterDisconnect?: boolean: If true, the newly created tab will be closed when the bridge is destroyed. Default is false.

Methods#

Except the normal agent interface, AgentOverChromeBridge provides some other interfaces to control the desktop Chrome.

connectCurrentTab()#

Connect to the currently active tab.

• Type

function connectCurrentTab(options?: { 
  forceSameTabNavigation?: boolean 
}): Promise<void>;

• Parameters:

  • options?: object - Optional configuration object
    • forceSameTabNavigation?: boolean - If true (default), restricts pages from opening new tabs, forcing new pages to open in the current tab to prevent AI operation failures due to manual tab switching. This configuration usually doesn't need to be changed

• Returns:

  • Returns a Promise that resolves to void when connected successfully; throws an error if connection fails

• Example:

try {
  await agent.connectCurrentTab();
  console.log('Successfully connected to current tab');
} catch (err) {
  console.error('Connection failed:', err.message);
}

connectNewTabWithUrl()#

Create a new tab and connect to it immediately.

• Type

function connectNewTabWithUrl(
  url: string,
  options?: { 
    forceSameTabNavigation?: boolean 
  }
): Promise<void>;

• Parameters:

  • url: string - URL to open in the new tab
  • options?: object - Optional configuration object (same parameters as connectCurrentTab)

• Returns:

  • Returns a Promise that resolves to void when connected successfully; throws an error if connection fails

• Example:

// Open Bing and wait for connection
await agent.connectNewTabWithUrl(
  "https://www.bing.com",
  { forceSameTabNavigation: false }
);

destroy()#

Destroy the connection and release resources.

• Type

function destroy(closeNewTabsAfterDisconnect?: boolean): Promise<void>;

• Parameters:

  • closeNewTabsAfterDisconnect?: boolean - If true, the newly created tab will be closed when the bridge is destroyed. Default is false. The will override the closeNewTabsAfterDisconnect parameter in the constructor.

• Returns:

  • Returns a Promise that resolves to void when destruction completes

• Example:

// Destroy connection after completing operations
await agent.ai('Perform final operation...');
await agent.destroy();

Use bridge mode in yaml-script#

Yaml scripts is a way for developers to write automation scripts in yaml format, which is easy to read and write comparing to javascript.

To use bridge mode in yaml script, set the bridgeMode property in the target section. If you want to use the current tab, set it to currentTab, otherwise set it to newTabWithUrl.

Set closeNewTabsAfterDisconnect to true if you want to close the newly created tabs when the bridge is destroyed. This is optional and the default value is false.

For example, the following script will open a new tab by Chrome extension bridge:

target:
  url: https://www.bing.com
+ bridgeMode: newTabWithUrl
+ closeNewTabsAfterDisconnect: true
tasks:

Run the script:

midscene ./bing.yaml

Remember to start the chrome extension and click 'Allow connection' button after the script is running.

Unsupported options#

In bridge mode, these options will be ignored (they will follow your desktop browser's settings):

• userAgent
• viewportWidth
• viewportHeight
• viewportScale
• waitForNetworkIdle
• cookie

FAQ#

• Where should I config the OPENAI_API_KEY, in the browser or in the terminal?

When using bridge mode, you should config the OPENAI_API_KEY in the terminal.