• English

      • Use a model API key

      Connect Midscene using an API key from a model provider. This guide covers choosing a model, configuring it, and verifying the connection. If you already have a ChatGPT subscription, you can also follow Use a ChatGPT subscription.

      The model configuration shown on this page consists of environment variables. Provide it according to how you use Midscene:

      • In Playground, paste the configuration text from this page directly into the settings page.
      • With the SDK or CLI, load the configuration as described in Set environment variables.

      For complete parameter definitions, see Model configuration reference.

      Supported models

      Midscene supports these multimodal models for operating user interfaces: Doubao Seed, Qwen, DeepSeek, Google Gemini, OpenAI GPT, Moonshot Kimi, Xiaomi MiMo, Zhipu GLM-V, and Jev (experimental), as well as the other compatible models AutoGLM and UI-TARS described below.

      Each configuration requires a Base URL, API key, model name, and MIDSCENE_MODEL_FAMILY. The model family (MIDSCENE_MODEL_FAMILY) determines how Midscene adapts to the selected model.

      Doubao Seed Series

      Model
      MIDSCENE_MODEL_NAME      		MIDSCENE_MODEL_FAMILYNotes
      2.x series
      doubao-seed-2-1-pro-260915 doubao-seed-2-1-turbo-260628      		doubao-seeddoubao-seed-2-1-turbo-260628 offers fast localization and strong localization quality. Recommended.
      1.x series
      Doubao-Seed-1.6-Vision Doubao-Seed-1.8      		doubao-seedThe 1.x series is an older generation of Doubao models and is no longer competitive overall. We recommend using the 2.x series instead. For compatibility with existing configurations, MIDSCENE_MODEL_FAMILY="doubao-vision" remains supported. New configurations should use doubao-seed.

      Environment variable configuration example, using doubao-seed-2-1-turbo-260628:

      🤖 Use as default model
      🎯 Use as Locate model
      🧠 Use as Planning model
      🔎 Use as Insight model
      MIDSCENE_MODEL_BASE_URL="https://ark.cn-beijing.volces.com/api/v3" # Volcano Engine endpoint
MIDSCENE_MODEL_API_KEY="......"
MIDSCENE_MODEL_NAME="doubao-seed-2-1-turbo-260628"
MIDSCENE_MODEL_FAMILY="doubao-seed"

      If your Volcano Engine account has Fast Tier quota enabled, add the extra request body below to use it. This usually improves model response speed by about 30%-50%.

      MIDSCENE_MODEL_EXTRA_BODY_JSON={"service_tier":"fast"}

      Qwen Series

      Model
      MIDSCENE_MODEL_NAME      		MIDSCENE_MODEL_FAMILYNotes
      Qwen3.x series
      qwen3.7-plus qwen3.5-plus qwen3.6-plus      		qwen3Based on localization evaluation results, the recommended order is Qwen3.7 > Qwen3.5 > Qwen3.6. The previous qwen3.5 and qwen3.6 families remain compatible.
      Qwen3-VL series
      qwen3-vl-plus      		qwen3-vlAs an older model generation, it is not recommended. Use the Qwen3.x series instead.
      Qwen2.5-VL series
      qwen-vl-max-latest      		qwen2.5-vlAs an older model generation, it is not recommended. Use the Qwen3.x series instead.

      Environment variable configuration example, using qwen3.7-plus:

      🤖 Use as default model
      🎯 Use as Locate model
      🧠 Use as Planning model
      🔎 Use as Insight model
      MIDSCENE_MODEL_BASE_URL="https://dashscope.aliyuncs.com/compatible-mode/v1" # Alibaba Cloud endpoint
MIDSCENE_MODEL_API_KEY="......"
MIDSCENE_MODEL_NAME="qwen3.7-plus"
MIDSCENE_MODEL_FAMILY="qwen3" # If you use another Qwen version, replace this with the corresponding model family

      DeepSeek Series

      Midscene supports DeepSeek starting from v1.12.0.

      Model
      MIDSCENE_MODEL_NAME      		MIDSCENE_MODEL_FAMILYNotes
      DeepSeek V4 series
      deepseek-flash (V4.1 Flash) deepseek-v4-flash-vision-exp (V4 Flash Vision)      		deepseekDeepSeek-V4-Pro does not support visual input and cannot be used with Midscene.

      Environment variable configuration example, using deepseek-flash:

      🤖 Use as default model
      🎯 Use as Locate model
      🧠 Use as Planning model
      🔎 Use as Insight model
      MIDSCENE_MODEL_BASE_URL="https://api.deepseek.com" # DeepSeek API endpoint
MIDSCENE_MODEL_API_KEY="......"
MIDSCENE_MODEL_NAME="deepseek-flash"
MIDSCENE_MODEL_FAMILY="deepseek"

      Google Gemini Series

      Model
      MIDSCENE_MODEL_NAME      		MIDSCENE_MODEL_FAMILYNotes
      Gemini 3.x series
      gemini-3.5-flash gemini-3-flash-preview      		geminigemini-3.5-flash currently performs best for localization in our private evaluation set.

      Environment variable configuration example, using gemini-3.5-flash:

      🤖 Use as default model
      🎯 Use as Locate model
      🧠 Use as Planning model
      🔎 Use as Insight model
      MIDSCENE_MODEL_BASE_URL="https://generativelanguage.googleapis.com/v1beta/openai/" # Google Gemini API endpoint
MIDSCENE_MODEL_API_KEY="......"
MIDSCENE_MODEL_NAME="gemini-3.5-flash"
MIDSCENE_MODEL_FAMILY="gemini"

      OpenAI GPT Series

      Info

      To connect using your ChatGPT subscription, see Use a ChatGPT subscription.

      • Common model provider: OpenAI
      Model
      MIDSCENE_MODEL_NAME      		MIDSCENE_MODEL_FAMILYNotes
      GPT-6 series
      gpt-6-astra gpt-6-sol gpt-6-luna      		gpt-6GPT-6 Sol delivers excellent visual localization accuracy at just one-fifth the price of GPT-6 Astra. GPT-6 Luna is very inexpensive while also offering good visual localization capabilities.
      GPT-5 series
      gpt-5.4 gpt-5.5 gpt-5.6-sol gpt-5.6-terra gpt-5.6-luna      		gpt-5Models before GPT-5.4 do not support visual localization and can only be used as Planning or Insight models. In practical localization tests, GPT-5.5 and GPT-5.6 perform noticeably better than GPT-5.4; we recommend using GPT-5.5 or GPT-5.6 first.

      Use an API Key

      Environment variable configuration example, using gpt-6-sol:

      ?
      🤖 Use as default model
      🎯 Use as Locate model
      🧠 Use as Planning model
      🔎 Use as Insight model
      MIDSCENE_MODEL_BASE_URL="https://api.openai.com/v1" # OpenAI API endpoint; or your compatible endpoint
MIDSCENE_MODEL_API_KEY="......"
MIDSCENE_MODEL_NAME="gpt-6-sol"
MIDSCENE_MODEL_FAMILY="gpt-6"

      Enable Fast mode (service_tier="fast")

      Both GPT-5.6 Sol and GPT-6 Astra support Fast mode. Add the service_tier request body parameter to enable it. For example, with GPT-5.6 Sol:

      MIDSCENE_MODEL_NAME="gpt-5.6-sol"
MIDSCENE_MODEL_FAMILY="gpt-5"
MIDSCENE_MODEL_EXTRA_BODY_JSON='{"service_tier":"fast"}'

      According to OpenAI, Fast mode can deliver up to 2–2.5× the speed at twice the Standard rate. Refer to the official documentation for current speed, pricing, and usage restrictions.

      To reuse a Codex CLI session, see Use Codex App Server.

      When using GPT, note the following:

      • GPT-5.4 or later is required for UI localization. Midscene does not support earlier versions.
      • According to OpenAI, GPT-5 may still struggle with non-Latin text and with text that is too small in the image. See the Images and Vision guide.
      • In the computer use guide, OpenAI says they observe good performance around 1440x900 and 1600x900. We recommend adjusting screenshot sizes accordingly. In Midscene, you can control screenshot compression with screenshotShrinkFactor in the agent options. For browser automation, you can also control the page size and scale through the browser viewport.
      • Azure OpenAI's Chat Completions API may not handle "detail": "original" correctly, causing click-coordinate offsets. If you encounter this issue, you can switch to the Responses API. See Clicks are offset when using Azure OpenAI.

      Moonshot Kimi Series

      Model
      MIDSCENE_MODEL_NAME      		MIDSCENE_MODEL_FAMILYNotes
      K3 series
      kimi-k3      		kimi3According to the Kimi documentation, K3 always has reasoning enabled and cannot be disabled. Its reasoning effort defaults to max.
      K2.x series
      kimi-k2.5 kimi-k2.6      		kimi—

      Environment variable configuration example, using kimi-k3:

      🤖 Use as default model
      🎯 Use as Locate model
      🧠 Use as Planning model
      🔎 Use as Insight model
      MIDSCENE_MODEL_BASE_URL="https://api.moonshot.cn/v1" # Moonshot AI API endpoint
MIDSCENE_MODEL_API_KEY="......"
MIDSCENE_MODEL_NAME="kimi-k3"
MIDSCENE_MODEL_FAMILY="kimi3" # For kimi-k2.6, use "kimi"

      Xiaomi MiMo Series

      Model
      MIDSCENE_MODEL_NAME      		MIDSCENE_MODEL_FAMILYNotes
      V2.x series
      mimo-v2.6-pro mimo-v2.6-flash mimo-v2.6-pro-ultraspeed mimo-v2.5      		xiaomi-mimoAll 2.6 models support multimodal input; for 2.5, only the Omni series supports multimodal input.

      Environment variable configuration example, using mimo-v2.6-pro:

      🤖 Use as default model
      🎯 Use as Locate model
      🧠 Use as Planning model
      🔎 Use as Insight model
      MIDSCENE_MODEL_BASE_URL="https://api.xiaomimimo.com/v1" # Xiaomi MiMo API endpoint
MIDSCENE_MODEL_API_KEY="......"
MIDSCENE_MODEL_NAME="mimo-v2.6-pro"
MIDSCENE_MODEL_FAMILY="xiaomi-mimo"

      Zhipu GLM-V Series

      Model
      MIDSCENE_MODEL_NAME      		MIDSCENE_MODEL_FAMILYNotes
      GLM-5V series
      glm-5v-turbo      		glm-v—
      GLM-4.6 series
      glm-4.6v      		glm-vglm-4.6v is open-source.

      Environment variable configuration example, using glm-5v-turbo:

      🤖 Use as default model
      🎯 Use as Locate model
      🧠 Use as Planning model
      🔎 Use as Insight model
      MIDSCENE_MODEL_BASE_URL="https://open.bigmodel.cn/api/paas/v4" # BigModel API endpoint; use https://api.z.ai/api/paas/v4 for Z.AI
MIDSCENE_MODEL_API_KEY="......"
MIDSCENE_MODEL_NAME="glm-5v-turbo"
MIDSCENE_MODEL_FAMILY="glm-v"

      Learn more about the open-source GLM-4.6V model

      Jev (experimental)

      Jev support is experimental. Results are heavily influenced by the Jev model.

      Jev can be used for Planning, Locate, or Insight. Because Jev cannot generate input content or other open-ended text, you must also configure a general-purpose default model to handle tasks Jev cannot perform. Jev cannot serve as the default model. See Multiple models for default model configuration.

      The following example uses jev-1.14.0 through TypeSafe's official service. Select the role you want Jev to perform:

      🎯 Use as Locate model
      🧠 Use as Planning model
      🔎 Use as Insight model
      MIDSCENE_LOCATE_MODEL_BASE_URL="https://api.typesafe.ai/v1" # TypeSafe official API endpoint
MIDSCENE_LOCATE_MODEL_API_KEY="......"
MIDSCENE_LOCATE_MODEL_NAME="jev-1.14.0"
MIDSCENE_LOCATE_MODEL_FAMILY="jev"

      When the corresponding model role is configured to use Jev, tasks are handled as follows:

      • Locate: Jev performs element localization.
      • Insight: Jev evaluates aiBoolean, aiAssert, and aiWaitFor. The general-purpose default model handles aiQuery, aiString, aiNumber, and aiAsk.
      • Planning: In aiAct, Jev selects actions such as tapping, scrolling, and waiting, and determines whether the task is complete. The general-purpose default model handles the current planning step when it requires input content generation or other parameters Jev cannot produce. If Locate is configured to use another model, the default model also generates the concrete target descriptions required for tap and scroll steps.

      Model selection is based on the capabilities required by each task. Failed Jev requests do not automatically switch to the default model.

      Jev uses TypeSafe's System One protocol, not an OpenAI protocol. Some OpenAI model settings do not apply, such as temperature, native reasoning, response format, and streaming output.

      Image input limits

      Jev requires the longest edge of an input image to be no more than 1568 pixels. Midscene therefore scales images that exceed this limit while preserving their aspect ratio, then maps localization results back to the original image coordinates. Edges shorter than 280 pixels are padded. This image processing behavior may change in future versions.

      Set environment variables

      Midscene reads model configuration from environment variables. Choose the method that matches how you use Midscene, or keep using your project's existing environment-management approach.

      Set environment variables in the current shell

      The following example works in Bash and Zsh. These variables are available only to the current shell session and its child processes.

      # Replace every value with the configuration for your selected model provider
export MIDSCENE_MODEL_BASE_URL="https://replace-with-your-model-service-url/v1"
export MIDSCENE_MODEL_API_KEY="replace-with-your-api-key"
export MIDSCENE_MODEL_NAME="replace-with-your-model-name"
export MIDSCENE_MODEL_FAMILY="replace-with-the-family-for-your-model"

      Load .env with the CLI

      Create a .env file in the directory where you run the midscene command. @midscene/cli loads this file automatically.

      # Replace every value with the configuration for your selected model provider
MIDSCENE_MODEL_BASE_URL="https://replace-with-your-model-service-url/v1"
MIDSCENE_MODEL_API_KEY="replace-with-your-api-key"
MIDSCENE_MODEL_NAME="replace-with-your-model-name"
MIDSCENE_MODEL_FAMILY="replace-with-the-family-for-your-model"

      Do not add export at the beginning of each line. When running YAML tasks, existing variables in the current shell take precedence over values with the same name in .env. To let .env override them, use --dotenv-override.

      midscene model verify is an exception. This command uses values from .env to override variables with the same name in the current shell.

      Load .env with dotenv for the JavaScript SDK

      The Midscene JavaScript SDK reads model configuration directly from the Node.js process environment (process.env). If the shell, container, or deployment platform already provides these variables, you do not need dotenv. Use dotenv only when the configuration is stored in a .env file and needs to be loaded into process.env.

      npm
      yarn
      pnpm
      bun
      deno
      npm install dotenv

      Create a .env file in the directory where you run the script:

      # Replace every value with the configuration for your selected model provider
MIDSCENE_MODEL_BASE_URL="https://replace-with-your-model-service-url/v1"
MIDSCENE_MODEL_API_KEY="replace-with-your-api-key"
MIDSCENE_MODEL_NAME="replace-with-your-model-name"
MIDSCENE_MODEL_FAMILY="replace-with-the-family-for-your-model"

      Import dotenv before creating a Midscene Agent:

      import 'dotenv/config';

      If a variable with the same name already exists in the current shell, dotenv keeps the existing value by default. The Midscene demo project uses this loading method as well.

      Verify the setup

      After setting the environment variables, run one of the following commands:

      # Use the CLI installed in the current project
npx midscene model verify

# Or use the latest CLI version
npx @midscene/cli@latest model verify

      For failed checks and connectivity problems, see Model debugging and observability.

      Optional: configure multiple models

      Multi-model setup is optional. For most use cases, a Default model is enough for UI localization and actions. Add a Locate, Planning, or Insight model to control localization, planning, or page understanding independently. Each role without a dedicated model uses the Default model.

      For guidance on when to combine models, see Model strategy.

      A Default model is still required when combining multiple models. It handles auxiliary tasks such as judging whether an element description is order-sensitive when generating caches, as well as any role without a dedicated model. Removing a dedicated model configuration returns that role to the Default model.

      The following example uses Qwen 3.8 Flash as the Default model, Gemini 3.5 Flash for localization, GPT-6 Astra for planning, and GPT-6.1 Sol for extraction, assertions, and page understanding. Use the actual model IDs provided by your service provider.

      # Default model
export MIDSCENE_MODEL_BASE_URL="https://..."
export MIDSCENE_MODEL_API_KEY="..."
export MIDSCENE_MODEL_NAME="qwen3.8-flash"
export MIDSCENE_MODEL_FAMILY="qwen3"

# Locate model
export MIDSCENE_LOCATE_MODEL_BASE_URL="https://..."
export MIDSCENE_LOCATE_MODEL_API_KEY="..."
export MIDSCENE_LOCATE_MODEL_NAME="gemini-3.5-flash"
export MIDSCENE_LOCATE_MODEL_FAMILY="gemini"

# Planning model
export MIDSCENE_PLANNING_MODEL_BASE_URL="https://..."
export MIDSCENE_PLANNING_MODEL_API_KEY="..."
export MIDSCENE_PLANNING_MODEL_NAME="gpt-6-astra"
export MIDSCENE_PLANNING_MODEL_FAMILY="gpt-6"

# Insight model
export MIDSCENE_INSIGHT_MODEL_BASE_URL="https://..."
export MIDSCENE_INSIGHT_MODEL_API_KEY="..."
export MIDSCENE_INSIGHT_MODEL_NAME="gpt-6.1-sol"
export MIDSCENE_INSIGHT_MODEL_FAMILY="gpt-6"

      Other compatible models

      The following smaller models are also compatible with Midscene and designed for automation. They require less deployment hardware but may struggle with complex tasks or large screenshots. Evaluate them against your tasks and deployment constraints before choosing one.

      Zhipu AutoGLM Series

      Zhipu AutoGLM is an open-source mobile UI automation model (9B parameters) from Zhipu AI.

      After obtaining an API key from Z.AI (Global) or BigModel (CN), configure:

      MIDSCENE_MODEL_BASE_URL="https://api.z.ai/api/paas/v4" # Or https://open.bigmodel.cn/api/paas/v4
MIDSCENE_MODEL_API_KEY="......"
MIDSCENE_MODEL_NAME="autoglm-phone"
MIDSCENE_MODEL_FAMILY="auto-glm" # Or "auto-glm-multilingual"

      About MIDSCENE_MODEL_FAMILY Configuration

      AutoGLM provides two model versions, distinguished by MIDSCENE_MODEL_FAMILY:

      • auto-glm - Corresponds to AutoGLM-Phone-9B, optimized for Chinese mobile applications
      • auto-glm-multilingual - Corresponds to AutoGLM-Phone-9B-Multilingual, supports English and other languages

      Choose the appropriate version based on your application language.

      Info

      AutoGLM is best suited for mobile interaction. APIs such as aiAssert and aiQuery require page understanding. When using these APIs, configure a separate Insight model with the MIDSCENE_INSIGHT_MODEL_... environment variables. See Model strategy for details.

      Learn more about Zhipu AutoGLM

      UI-TARS Series

      Use the deployed doubao-1.5-ui-tars on Volcano Engine:

      MIDSCENE_MODEL_BASE_URL="https://ark.cn-beijing.volces.com/api/v3"
MIDSCENE_MODEL_API_KEY="...."
MIDSCENE_MODEL_NAME="ep-2025..." # Inference endpoint ID or model name from Volcano Engine
MIDSCENE_MODEL_FAMILY="vlm-ui-tars-doubao-1.5"

      About MIDSCENE_MODEL_FAMILY

      This variable selects the UI-TARS version. Supported values:

      • vlm-ui-tars – for the 1.0 release
      • vlm-ui-tars-doubao – for the 1.5 release deployed on Volcano Engine (equivalent to vlm-ui-tars-doubao-1.5)
      • vlm-ui-tars-doubao-1.5 – for the 1.5 release deployed on Volcano Engine
      Info

      The legacy configurations MIDSCENE_USE_VLM_UI_TARS=DOUBAO or MIDSCENE_USE_VLM_UI_TARS=1.5 are still supported but deprecated. Please migrate to MIDSCENE_MODEL_FAMILY.

      Migration mapping:

      • MIDSCENE_USE_VLM_UI_TARS=1.0 → MIDSCENE_MODEL_FAMILY="vlm-ui-tars"
      • MIDSCENE_USE_VLM_UI_TARS=1.5 → MIDSCENE_MODEL_FAMILY="vlm-ui-tars-doubao-1.5"
      • MIDSCENE_USE_VLM_UI_TARS=DOUBAO → MIDSCENE_MODEL_FAMILY="vlm-ui-tars-doubao"

      Next steps