MaxxBench – "Nice DeepSWE score. But is it true?"
.
. , , . , , , .
. . , .
. 89 , .
: ...
, . . . . . .
, . 2 .
. , .
, . . ; . ..
. , .
, ; .
/ ___.
/ ___.
. .
, . , . .
, 17 (3.5%). .
...
5.5 3.8 74%. , 76.4% 53.4%.
23- . ; .
-6 5 , 71.9% 63.5% .
12 , , 20.9 48.3 . 1.5 74% , 23.0 .
. 5.5 76.4%, 74.2% , -6 -5.6 2.2 3.6 , . 74% , 4.1 3.8 18.6 20.5 .
.
, . , .
64 , 5.3 64, 253. , : “ 64; >2^53 .” 253 + 1, . , , (“ ? ; ”) (“ / , ”).
. , 5 : “ … .” , . 5 “ ,” — “ 50/50 , 0.55 0.80” — , . , 5, 5.3, 5.3 , .
. 5.5 100% , 16.7% , 30.8% 5 40.3% 5.3.
40.3%(31 77 )
36.7%(29 79 )
30.8%(20 65 )
27.5%(22 80 )
18.2%(10 55 )
16.7%(7 42 )
14.6%(14 96 )
12.0%(6 50 )
10.8%(7 65 )
8.6%(11 128 )
1.2%(1 83 )
0%(0 83 )
0%(0 109 )
, ’ .
. 5 , :
“ … … .”
1,000,100 1,000,900 . 0 [0, 1], .
, :
“ ”
. .
__ “0”. .
“ __ "0" . … "0","","" "" . . .”
, , , .
“ ” :
, .() .
, :
, ( := ()): .() .
. , , =<> ___=<>.
___ . .
13 , . 13 , , .
14
.
11
, .
. . 14
.
. . 70
.
. . 13
.
34
, .
. 105
.
4.1 . _ _. , , , :
; . 130 ↗
. . , 20 178 .
. , “ , .” , “ .”
4.1 , 100 ↗, , . , , , .
, . , : .
, . , . . , , . 33 , . ..
`.()`! " ".
, 107 ↗, 108 ↗
, .
, 110 ↗ , 114 ↗
/.
/.
35 / 68
469 / 469
↗
.
, : , . .
, . , , , .
, ( [1,2]) {} , 39’ :
(39 §7.3.2)
, 89 . 65.8% 64.5%, 1.3 , .
. , .
. , .
, , . , , .
. , , , , .
:
( ; )