9:41

Serving GLM-5.3 at 607.8 tok/s across two Nvidia DGX stations with vLLM

; . -8 , .

-5.3 4 , . () , 753 , 40 . , . 464 , 252 3 300 . , .

, .

-5.3 , , . (=2) 300 . , . , .

Two stations serve one model: each GPU holds and computes part of the model over the selected RDMA connection. Their memories remain distinct.
Figure 1. Two Stations serve one model: each GPU holds and computes part of the model over the selected RDMA connection. Their memories remain distinct

  • 300 , , .
  • . . , , .
  • 464 , .
  • -5.3 , . .
  • , / : , , . 5 .

.

1.

, , . . , . “ .”

: ( 0) , ( 1) , .

, :

2 , 80 81. , 81, :

  • , 2 . 5_1 1 ==> 81 (), 5_1.
  • 4 , - . 81 192.168.101.1/24, 192.168.101.1.

; 4 .

, ~/53. . :

<...> , , .

, 81, . ="1":

_ , 2 .

, , , :

. 400000/ , 9000, , . 300. , .

2 , . , ( ), , :

2.

. :

. 64 .

, . 464 , 1 /, 10 /:

, . :

; , .

, ( ), :

_ . _ : 4 , .

3.

: , ~/53., . : -- 98304, , , -- 0.93, 93% . , .

, .

, ~/53.:

(), : , , . , .

8 16- , . . ; , , :

, :

; . , , , . .

, (-- ):

, + ; .

. , : .

, . ( ). , . , -- .

4.

, (127.0.0.1 ):

_ . , , . _=, :

__ . , ::_0 ::_1, . 218 , , .

, :

:

_: . _ __. -5.3 , , . (--), .

5.

, , . , , :

.

. 80 , , . , /: , . , , :

, . :

, . _:

1, 8, 16, 32. , 3× , 10× , . .

() 8,192- ; . .

, -- , -- .

, . 1 8 , 2, .

, , . .

: 2. ~/53. 3 , -- -- 16 32 . , , ( -- 60 "$", "$"). , ~/53. . , , .

, -- -- 3 , . 2.

2 . .

Overall output throughput and mean per-user throughput at configured concurrency 1, 8, 16, and 32, with prefix caching enabled. C1: 107.4 and 115.3 tok/s; C8: 386.0 and 53.0; C16: 607.8 and 42.4; C32: 385.0 and 13.0.
Figure 2. Overall output throughput versus mean per-user throughput on SPEED-Bench Coding, with prefix caching enabled. C1, C8, C16, and C32 denote configured concurrency

2, , .

607.8 / 16. 385.0 / 32, 453.5 . , 115.3 / 1 13.0 / 32.

, , :

= =, ; , , . , "$" , , 4 .

  • : ~/53. ; .
  • : , / , , .
  • : "$" ; , ( 07).
  • : -- , .

753- , . : , , .

, . , .