Midscene AppControlBench Benchmark Report
This is Midscene's benchmark report for AppControlBench. In this evaluation, each model ran the same set of 60 tasks, with the following results:
| Model | Completion | Time / run | Cost / run | Task outcomes |
|---|---|---|---|---|
| Doubao Seed 2.1 Turbo | 98% | 52s | $0.0261/¥0.1772 | |
| Qwen3.7 Plus | 97% | 54s | $0.0097/¥0.0657 | |
| DeepSeek V4 Flash Vision Exp | 90% | 1m 26s | $0.0045/¥0.0308 |
Note: Model costs are calculated using OpenRouter pricing, with CNY converted at ¥6.8 per $1.
The results show that the Midscene + GUI vision approach is highly competitive in both cost and pass rate.
About AppControlBench
AppControlBench evaluates how well an Agent can operate real iOS apps in isolated Simulator runs. The final screenshot is graded against a task-specific solved-screen description. Each run covers 30 Bluesky tasks and 30 Element iOS tasks.
Run configuration
Case Intent Calibration
For this evaluation, the action-route description of 1 case was calibrated against the actual UI of the pinned Element iOS version. The calibration only removes a stale or unavailable UI-entry description; it does not change the final target state or judge rules of the case.
Report files
Doubao Seed 2.1 Turbo · Pass@1 96.67% · 58 PASS · 1 PARTIAL · 1 FAIL
| # | Task | Status | Model Cost | Report |
|---|---|---|---|---|
| 1 | bsky-01 | PASS | $0.0107/¥0.0728 | report |
| 2 | bsky-02 | PASS | $0.0107/¥0.0728 | report |
| 3 | bsky-03 | PASS | $0.0217/¥0.1478 | report |
| 4 | bsky-04 | PASS | $0.0163/¥0.1111 | report |
| 5 | bsky-05 | PASS | $0.0108/¥0.0733 | report |
| 6 | bsky-06 | PASS | $0.0109/¥0.0743 | report |
| 7 | bsky-07 | PASS | $0.0109/¥0.0739 | report |
| 8 | bsky-08 | PASS | $0.0339/¥0.2306 | report |
| 9 | bsky-09 | PASS | $0.0220/¥0.1495 | report |
| 10 | bsky-10 | PASS | $0.0164/¥0.1117 | report |
| 11 | bsky-11 | PASS | $0.0107/¥0.0725 | report |
| 12 | bsky-12 | PASS | $0.0107/¥0.0728 | report |
| 13 | bsky-13 | PASS | $0.0107/¥0.0729 | report |
| 14 | bsky-14 | PASS | $0.0277/¥0.1885 | report |
| 15 | bsky-15 | PASS | $0.0278/¥0.1888 | report |
| 16 | bsky-16 | PASS | $0.0608/¥0.4134 | report |
| 17 | bsky-17 | PASS | $0.0162/¥0.1104 | report |
| 18 | bsky-18 | PASS | $0.0106/¥0.0724 | report |
| 19 | bsky-19 | PASS | $0.0219/¥0.1488 | report |
| 20 | bsky-20 | PASS | $0.0219/¥0.1490 | report |
| 21 | bsky-21 | PASS | $0.0162/¥0.1104 | report |
| 22 | bsky-22 | PASS | $0.0106/¥0.0723 | report |
| 23 | bsky-23 | PASS | $0.0162/¥0.1099 | report |
| 24 | bsky-24 | PASS | $0.0220/¥0.1498 | report |
| 25 | bsky-25 | FAIL | $0.0217/¥0.1476 | report |
| 26 | bsky-26 | PASS | $0.0396/¥0.2690 | report |
| 27 | bsky-27 | PASS | $0.0461/¥0.3136 | report |
| 28 | bsky-28 | PASS | $0.0218/¥0.1484 | report |
| 29 | bsky-29 | PASS | $0.0218/¥0.1483 | report |
| 30 | bsky-30 | PASS | $0.0280/¥0.1906 | report |
| 31 | element-01 | PASS | $0.0106/¥0.0720 | report |
| 32 | element-02 | PASS | $0.0107/¥0.0728 | report |
| 33 | element-03 | PASS | $0.0274/¥0.1864 | report |
| 34 | element-04 | PASS | $0.0161/¥0.1095 | report |
| 35 | element-05 | PASS | $0.0162/¥0.1103 | report |
| 36 | element-06 | PARTIAL | $0.2547/¥1.7319 | report |
| 37 | element-07 | PASS | $0.0163/¥0.1106 | report |
| 38 | element-08 | PASS | $0.0277/¥0.1881 | report |
| 39 | element-09 | PASS | $0.0110/¥0.0747 | report |
| 40 | element-10 | PASS | $0.0468/¥0.3181 | report |
| 41 | element-11 | PASS | $0.0164/¥0.1117 | report |
| 42 | element-12 | PASS | $0.0163/¥0.1110 | report |
| 43 | element-13 | PASS | $0.0280/¥0.1907 | report |
| 44 | element-14 | PASS | $0.0164/¥0.1116 | report |
| 45 | element-15 | PASS | $0.0220/¥0.1497 | report |
| 46 | element-16 | PASS | $0.0222/¥0.1512 | report |
| 47 | element-17 | PASS | $0.0218/¥0.1485 | report |
| 48 | element-18 | PASS | $0.0222/¥0.1512 | report |
| 49 | element-19 | PASS | $0.0471/¥0.3200 | report |
| 50 | element-20 | PASS | $0.0107/¥0.0724 | report |
| 51 | element-21 | PASS | $0.0295/¥0.2005 | report |
| 52 | element-22 | PASS | $0.0401/¥0.2725 | report |
| 53 | element-23 | PASS | $0.0220/¥0.1493 | report |
| 54 | element-24 | PASS | $0.0339/¥0.2302 | report |
| 55 | element-25 | PASS | $0.0163/¥0.1110 | report |
| 56 | element-26 | PASS | $0.0107/¥0.0726 | report |
| 57 | element-27 | PASS | $0.0405/¥0.2754 | report |
| 58 | element-28 | PASS | $0.0274/¥0.1861 | report |
| 59 | element-29 | PASS | $0.0275/¥0.1872 | report |
| 60 | element-30 | PASS | $0.0273/¥0.1858 | report |
Qwen3.7 Plus · Pass@1 93.33% · 56 PASS · 4 PARTIAL · 0 FAIL
| # | Task | Status | Model Cost | Report |
|---|---|---|---|---|
| 1 | bsky-01 | PASS | $0.0073/¥0.0497 | report |
| 2 | bsky-02 | PASS | $0.0074/¥0.0501 | report |
| 3 | bsky-03 | PASS | $0.0108/¥0.0736 | report |
| 4 | bsky-04 | PASS | $0.0092/¥0.0624 | report |
| 5 | bsky-05 | PASS | $0.0053/¥0.0362 | report |
| 6 | bsky-06 | PASS | $0.0075/¥0.0507 | report |
| 7 | bsky-07 | PASS | $0.0075/¥0.0507 | report |
| 8 | bsky-08 | PASS | $0.0170/¥0.1155 | report |
| 9 | bsky-09 | PASS | $0.0110/¥0.0746 | report |
| 10 | bsky-10 | PASS | $0.0111/¥0.0756 | report |
| 11 | bsky-11 | PASS | $0.0053/¥0.0360 | report |
| 12 | bsky-12 | PASS | $0.0053/¥0.0357 | report |
| 13 | bsky-13 | PASS | $0.0054/¥0.0365 | report |
| 14 | bsky-14 | PASS | $0.0086/¥0.0585 | report |
| 15 | bsky-15 | PASS | $0.0107/¥0.0724 | report |
| 16 | bsky-16 | PARTIAL | $0.0196/¥0.1332 | report |
| 17 | bsky-17 | PASS | $0.0050/¥0.0342 | report |
| 18 | bsky-18 | PASS | $0.0033/¥0.0224 | report |
| 19 | bsky-19 | PASS | $0.0067/¥0.0457 | report |
| 20 | bsky-20 | PASS | $0.0068/¥0.0460 | report |
| 21 | bsky-21 | PASS | $0.0049/¥0.0335 | report |
| 22 | bsky-22 | PASS | $0.0032/¥0.0219 | report |
| 23 | bsky-23 | PASS | $0.0050/¥0.0340 | report |
| 24 | bsky-24 | PASS | $0.0069/¥0.0469 | report |
| 25 | bsky-25 | PARTIAL | $0.0068/¥0.0463 | report |
| 26 | bsky-26 | PASS | $0.0125/¥0.0849 | report |
| 27 | bsky-27 | PASS | $0.0145/¥0.0988 | report |
| 28 | bsky-28 | PASS | $0.0068/¥0.0461 | report |
| 29 | bsky-29 | PASS | $0.0068/¥0.0462 | report |
| 30 | bsky-30 | PASS | $0.0171/¥0.1160 | report |
| 31 | element-01 | PASS | $0.0052/¥0.0356 | report |
| 32 | element-02 | PASS | $0.0032/¥0.0214 | report |
| 33 | element-03 | PASS | $0.0087/¥0.0593 | report |
| 34 | element-04 | PASS | $0.0048/¥0.0326 | report |
| 35 | element-05 | PASS | $0.0107/¥0.0727 | report |
| 36 | element-06 | PARTIAL | $0.0640/¥0.4355 | report |
| 37 | element-07 | PASS | $0.0088/¥0.0600 | report |
| 38 | element-08 | PASS | $0.0086/¥0.0582 | report |
| 39 | element-09 | PASS | $0.0032/¥0.0217 | report |
| 40 | element-10 | PASS | $0.0234/¥0.1593 | report |
| 41 | element-11 | PASS | $0.0050/¥0.0341 | report |
| 42 | element-12 | PASS | $0.0050/¥0.0338 | report |
| 43 | element-13 | PASS | $0.0070/¥0.0474 | report |
| 44 | element-14 | PASS | $0.0051/¥0.0345 | report |
| 45 | element-15 | PASS | $0.0068/¥0.0461 | report |
| 46 | element-16 | PARTIAL | $0.0346/¥0.2352 | report |
| 47 | element-17 | PASS | $0.0206/¥0.1403 | report |
| 48 | element-18 | PASS | $0.0112/¥0.0761 | report |
| 49 | element-19 | PASS | $0.0108/¥0.0738 | report |
| 50 | element-20 | PASS | $0.0032/¥0.0219 | report |
| 51 | element-21 | PASS | $0.0033/¥0.0222 | report |
| 52 | element-22 | PASS | $0.0128/¥0.0872 | report |
| 53 | element-23 | PASS | $0.0069/¥0.0466 | report |
| 54 | element-24 | PASS | $0.0109/¥0.0739 | report |
| 55 | element-25 | PASS | $0.0050/¥0.0341 | report |
| 56 | element-26 | PASS | $0.0033/¥0.0224 | report |
| 57 | element-27 | PASS | $0.0051/¥0.0344 | report |
| 58 | element-28 | PASS | $0.0106/¥0.0723 | report |
| 59 | element-29 | PASS | $0.0087/¥0.0588 | report |
| 60 | element-30 | PASS | $0.0087/¥0.0590 | report |
DeepSeek V4 Flash Vision Exp · Pass@1 86.67% · 52 PASS · 4 PARTIAL · 4 FAIL
| # | Task | Status | Model Cost | Report |
|---|---|---|---|---|
| 1 | bsky-01 | PASS | $0.0005/¥0.0035 | report |
| 2 | bsky-02 | PASS | $0.0005/¥0.0035 | report |
| 3 | bsky-03 | PASS | $0.0015/¥0.0100 | report |
| 4 | bsky-04 | PASS | $0.0009/¥0.0062 | report |
| 5 | bsky-05 | PASS | $0.0005/¥0.0037 | report |
| 6 | bsky-06 | PASS | $0.0021/¥0.0144 | report |
| 7 | bsky-07 | PASS | $0.0010/¥0.0066 | report |
| 8 | bsky-08 | PASS | $0.0019/¥0.0128 | report |
| 9 | bsky-09 | FAIL | $0.0155/¥0.1055 | report |
| 10 | bsky-10 | PASS | $0.0018/¥0.0121 | report |
| 11 | bsky-11 | PASS | $0.0011/¥0.0077 | report |
| 12 | bsky-12 | PASS | $0.0011/¥0.0073 | report |
| 13 | bsky-13 | PASS | $0.0012/¥0.0082 | report |
| 14 | bsky-14 | PASS | $0.0029/¥0.0200 | report |
| 15 | bsky-15 | PASS | $0.0012/¥0.0081 | report |
| 16 | bsky-16 | PARTIAL | $0.0160/¥0.1085 | report |
| 17 | bsky-17 | PASS | $0.0017/¥0.0114 | report |
| 18 | bsky-18 | PASS | $0.0011/¥0.0078 | report |
| 19 | bsky-19 | PASS | $0.0025/¥0.0168 | report |
| 20 | bsky-20 | PASS | $0.0022/¥0.0152 | report |
| 21 | bsky-21 | PASS | $0.0023/¥0.0156 | report |
| 22 | bsky-22 | PASS | $0.0011/¥0.0074 | report |
| 23 | bsky-23 | PASS | $0.0017/¥0.0119 | report |
| 24 | bsky-24 | PARTIAL | $0.0072/¥0.0493 | report |
| 25 | bsky-25 | PASS | $0.0029/¥0.0199 | report |
| 26 | bsky-26 | PASS | $0.0025/¥0.0171 | report |
| 27 | bsky-27 | PASS | $0.0034/¥0.0231 | report |
| 28 | bsky-28 | PASS | $0.0023/¥0.0157 | report |
| 29 | bsky-29 | PASS | $0.0023/¥0.0159 | report |
| 30 | bsky-30 | FAIL | $0.0290/¥0.1970 | report |
| 31 | element-01 | PASS | $0.0010/¥0.0069 | report |
| 32 | element-02 | PASS | $0.0010/¥0.0071 | report |
| 33 | element-03 | PASS | $0.0030/¥0.0201 | report |
| 34 | element-04 | PASS | $0.0038/¥0.0256 | report |
| 35 | element-05 | PASS | $0.0018/¥0.0122 | report |
| 36 | element-06 | FAIL | $0.0232/¥0.1575 | report |
| 37 | element-07 | PASS | $0.0019/¥0.0127 | report |
| 38 | element-08 | PASS | $0.0028/¥0.0193 | report |
| 39 | element-09 | PASS | $0.0011/¥0.0077 | report |
| 40 | element-10 | PASS | $0.0024/¥0.0161 | report |
| 41 | element-11 | PASS | $0.0019/¥0.0131 | report |
| 42 | element-12 | PASS | $0.0019/¥0.0130 | report |
| 43 | element-13 | PASS | $0.0026/¥0.0176 | report |
| 44 | element-14 | PASS | $0.0020/¥0.0139 | report |
| 45 | element-15 | PASS | $0.0010/¥0.0067 | report |
| 46 | element-16 | PARTIAL | $0.0549/¥0.3734 | report |
| 47 | element-17 | PASS | $0.0026/¥0.0178 | report |
| 48 | element-18 | PASS | $0.0027/¥0.0182 | report |
| 49 | element-19 | PARTIAL | $0.0106/¥0.0723 | report |
| 50 | element-20 | PASS | $0.0011/¥0.0075 | report |
| 51 | element-21 | FAIL | $0.0236/¥0.1604 | report |
| 52 | element-22 | PASS | $0.0024/¥0.0160 | report |
| 53 | element-23 | PASS | $0.0014/¥0.0094 | report |
| 54 | element-24 | PASS | $0.0020/¥0.0138 | report |
| 55 | element-25 | PASS | $0.0009/¥0.0059 | report |
| 56 | element-26 | PASS | $0.0006/¥0.0039 | report |
| 57 | element-27 | PASS | $0.0010/¥0.0065 | report |
| 58 | element-28 | PASS | $0.0016/¥0.0107 | report |
| 59 | element-29 | PASS | $0.0016/¥0.0107 | report |
| 60 | element-30 | PASS | $0.0016/¥0.0107 | report |