Overall Resolution
31/50
62% resolution rate
Django
18/25
72% resolution rate
Sphinx
13/25
52% resolution rate
Avg Cost / Instance
$0.287
Total $14.37 · flex tier
Avg Wall Time
181s
per instance (primary run)
Avg Cache Hit Rate
79%
prompt token cache hits (flex)
Instances Evaluated
50
LP-optimised · proxies full 500-instance set
QueryEngine tool calls across all 50 instances
QueryEngine tools only · mean calls per instance across all 50 runs
| Instance | Repo | Status | Cost | Cache | Tokens | LLM Calls |
|---|---|---|---|---|---|---|
| django__django-11790 | django | $0.226 | 77% | 230,022 | 16 | |
| django__django-11815 | django | $0.268 | 82% | 248,860 | 17 | |
| django__django-11848 | django | $0.117 | 74% | 117,811 | 11 | |
| django__django-11880 | django | $0.119 | 70% | 150,168 | 12 | |
| django__django-11885 | django | $0.775 | 81% | 794,208 | 34 | |
| django__django-11951 | django | $0.255 | 81% | 213,089 | 16 | |
| django__django-11964 | django | $0.364 | 77% | 245,592 | 16 | |
| django__django-11999 | django | $0.294 | 83% | 146,535 | 13 | |
| django__django-12039 | django | $0.271 | 81% | 251,168 | 18 | |
| django__django-12050 | django | $0.114 | 80% | 131,330 | 12 | |
| django__django-12143 | django | $0.240 | 79% | 201,022 | 16 | |
| django__django-12155 | django | $0.149 | 77% | 222,650 | 15 | |
| django__django-12193 | django | $0.355 | 80% | 155,559 | 14 | |
| django__django-12209 | django | $0.245 | 82% | 554,136 | 30 | |
| django__django-12262 | django | $0.307 | 77% | 225,197 | 15 | |
| django__django-12273 | django | $0.232 | 83% | 441,940 | 26 | |
| django__django-12276 | django | $0.293 | 77% | 216,967 | 15 | |
| django__django-12304 | django | $0.218 | 76% | 213,638 | 16 | |
| django__django-12308 | django | $0.160 | 78% | 254,508 | 19 | |
| django__django-12325 | django | $0.273 | 78% | 124,698 | 11 | |
| django__django-12406 | django | $0.603 | 85% | 536,443 | 27 | |
| django__django-12708 | django | $0.254 | 72% | 484,988 | 27 | |
| django__django-12713 | django | $0.213 | 72% | 180,965 | 13 | |
| django__django-12774 | django | $0.151 | 80% | 193,749 | 15 | |
| django__django-9296 | django | $0.123 | 65% | 169,510 | 13 | |
| sphinx-doc__sphinx-10323 | sphinx | $0.189 | 73% | 141,632 | 11 | |
| sphinx-doc__sphinx-10435 | sphinx | $0.197 | 74% | 212,339 | 16 | |
| sphinx-doc__sphinx-10466 | sphinx | $0.200 | 74% | 148,978 | 11 | |
| sphinx-doc__sphinx-10673 | sphinx | $0.522 | 100% | 1,171,642 | 37 | |
| sphinx-doc__sphinx-11510 | sphinx | $0.610 | 100% | 1,483,790 | 49 | |
| sphinx-doc__sphinx-7590 | sphinx | $0.615 | 100% | 1,113,090 | 47 | |
| sphinx-doc__sphinx-7748 | sphinx | $0.619 | 100% | 1,725,262 | 59 | |
| sphinx-doc__sphinx-7757 | sphinx | $0.201 | 70% | 176,228 | 13 | |
| sphinx-doc__sphinx-7985 | sphinx | $0.273 | 78% | 411,984 | 22 | |
| sphinx-doc__sphinx-8035 | sphinx | $0.314 | 81% | 499,291 | 26 | |
| sphinx-doc__sphinx-8056 | sphinx | $0.308 | 74% | 400,943 | 21 | |
| sphinx-doc__sphinx-8265 | sphinx | $0.395 | 71% | 537,258 | 21 | |
| sphinx-doc__sphinx-8269 | sphinx | $0.135 | 76% | 142,363 | 12 | |
| sphinx-doc__sphinx-8475 | sphinx | $0.189 | 75% | 172,744 | 14 | |
| sphinx-doc__sphinx-8548 | sphinx | $0.444 | 100% | 917,250 | 39 | |
| sphinx-doc__sphinx-8551 | sphinx | $0.232 | 65% | 197,190 | 13 | |
| sphinx-doc__sphinx-8638 | sphinx | $0.224 | 75% | 212,935 | 14 | |
| sphinx-doc__sphinx-8721 | sphinx | $0.230 | 74% | 179,147 | 12 | |
| sphinx-doc__sphinx-9229 | sphinx | $0.322 | 77% | 411,176 | 23 | |
| sphinx-doc__sphinx-9230 | sphinx | $0.347 | 70% | 437,377 | 20 | |
| sphinx-doc__sphinx-9281 | sphinx | $0.175 | 78% | 181,783 | 14 | |
| sphinx-doc__sphinx-9320 | sphinx | $0.194 | 79% | 258,184 | 19 | |
| sphinx-doc__sphinx-9367 | sphinx | $0.133 | 74% | 127,280 | 12 | |
| sphinx-doc__sphinx-9461 | sphinx | $0.452 | 99% | 730,643 | 35 | |
| sphinx-doc__sphinx-9698 | sphinx | $0.231 | 70% | 238,292 | 15 |
SWE-bench Verified Mini · GPT-5.4 (OpenAI flex tier)