9:41

Show HN: TrnRT – An open-source LLM inference engine for AWS Trainium

TrnRT: the open LLM inference engine for AWS Trainium, by RunAnywhere
CI
OpenSSF Scorecard
License: Apache 2.0
Python 3.13
AWS Trainium2
Neuron SDK 2.32

· · · · · .

. 2.48, ( ), . -5.3- ; .

, : -5.3- 2 .

3.57M tokens per minute on one trn2.48xlarge (AgentX, batch 12). 24 ms batch-8 decode step, down from 32.2 ms with NKI megakernels. $0.040 per million tokens at the spot price, $0.167 on a capacity block.

, 2.48 , 86_64 . ? /..

1. ( 2.32 ( 2.27), 3.13)

2. ( 86_64 , ; )

3. ( 2.48, $/ $/ )

4.

, , 4, 8 12 : · · .

Hardware: AWS Trainium2 (trn2.48xlarge, 16 chips, 96 GiB HBM each, Neuron SDK 2.32), supported. Models: GLM-5.3-Flash (320B MoE, 18B active, FP8, 1M-token context), supported. More open models: coming soon.

-5.3- . , , , /. .

. 2.48 1 32 , :

( ) . : $8.60/, -2, 3 2026. $35.76/, 12 $0.167 . , , /..

. (/_.):

  • : 32 , 157 -12 .
  • : 384 384 .
  • : .

Clients reach a session-affine router on port 8000, which sends each session to one of two engines. Each engine is tensor-parallel over 32 logical NeuronCores on 8 chips. Offline, five steps build the compile cache the engines load.

; , . : , , . /. .

/..

/. , . ( ), . , . .

, . -5.3-, . -2.0 : __..

2.0.

RunAnywhere

.

, , , ., . . . , .