USENIX Security ’26 Accepted
COGNITION: From Evaluation to Defense against Multimodal LLM CAPTCHA Solvers
Recognition-oriented CAPTCHAs are already fragile against multimodal LLMs; stronger defenses must shift toward precise grounding, ordered interaction, and stateful consistency.
- Venue
- 35th USENIX Security Symposium
- Status
- Accepted
- Artifact
- Artifact Evaluated — Available
- Year
- 2026
Overview
Evaluation first. Defense second.
COGNITION maps the practical boundary of automated visual CAPTCHA solving and turns that evidence into concrete redesign guidance. The study pairs broad evaluation with a defense case study that reduces state-of-the-art solver success from over 95% to 0%.
Abstract
This paper studies how multimodal large language models (MLLMs) undermine the security guarantees of visual CAPTCHA. We identify the attack surface where an adversary can cheaply automate CAPTCHA solving using off-the-shelf models. We evaluate 7 representative MLLMs across 18 CaptchaWorld task types and 3 supplemental external categories, totaling 21 visual CAPTCHA task types, measuring single-shot accuracy, success under limited retries, end-to-end latency, and per-solve cost. We further validate our findings through a supplemental external dataset and an adaptive-attacker setting with session memory, while also analyzing the impact of task-specific prompt engineering and few-shot demonstrations on solver effectiveness. We reveal that MLLMs can reliably solve recognition-oriented and low-interaction CAPTCHA tasks at human-like cost and latency, whereas tasks requiring fine-grained localization, multi-step spatial reasoning, or cross-frame consistency remain significantly harder for current models.
At a glance
The evaluation in four numbers.
- 7
- multimodal modelsRepresentative proprietary and open MLLMs
- 21
- task types18 CaptchaWorld tasks plus 3 external categories
- 458
- instancesMeasured across accuracy, retries, latency, and cost
- 95→0%
- defense resultSolver success after structural Select_Animal redesign
Key findings
Where current solvers succeed—and where they break.
Finding 01
Recognition-heavy CAPTCHAs are already fragile.
Path finding, animal selection, and image recognition tasks can be solved reliably by current MLLMs within practical retry and time budgets.Finding 02
Grounding precision creates the real hardness gap.
Tasks requiring fine-grained localization, ordering, counting, or cross-frame consistency remain substantially harder for current models.Finding 03
Structural redesign can sharply reduce success.
Adding fine-grained localization and implicit counting to Select_Animal reduced state-of-the-art MLLM success from over 95% to 0%.Defense takeaway
Design around the capabilities models still lack.
The evaluation reveals a practical design principle: move beyond simple recognition and require precise spatial grounding, ordering, counting, and cross-frame consistency. These properties preserve usability while raising the cost and failure rate of automated solvers.
Research poster
The paper, mapped as one visual argument.
The full research poster connects the research question, evaluation framework, task-hardness taxonomy, defense redesign, and artifact.
Resources
Read, cite, and inspect the work.
BibTeX citation
@misc{wang2026cognitionevaluationdefensemultimodal,
title={COGNITION: From Evaluation to Defense against Multimodal LLM CAPTCHA Solvers},
author={Junyu Wang and Changjia Zhu and Yuanbo Zhou and Lingyao Li and Xu He and Mingkui Wei and Junjie Xiong},
year={2026},
eprint={2512.02318},
archivePrefix={arXiv},
primaryClass={cs.CR},
url={https://arxiv.org/abs/2512.02318},
}