AWS opens TorchNeuron as a public beta, a native PyTorch path onto Trainium
The PyTorch blog, for a launch at the PyTorch Conference on Tuesday: torch-neuronx is an Apache 2.0 backend that registers Trainium2 and Trainium3 as a neuron device, with eager mode and torch.compile. Hugging Face says Transformers, Diffusers and TRL call it directly, and that 89 percent of the top 100 text-generation architectures run, against 46 percent on the older XLA stack. Deepgram’s own benchmark puts Trainium at about 60 percent more audio seconds per dollar than its GPU baseline.
Feature3 minIn-depth
Eli Navarro3 min read
- AWS
- Trainium
- PyTorch
- Hugging Face
- Chips

Photo: PyTorch FoundationSource page
AWS on Tuesday put TorchNeuron into public beta: an open-source backend that lets PyTorch treat a Trainium chip as an ordinary device. The PyTorch blog, published for the PyTorch Conference in North America, says the code is Apache 2.0 at github.com/aws-neuron/torch-neuronx, and that it installs with pip install torch-neuronx from the AWS Neuron index. It runs on Trainium2 and Trainium3. The change a researcher is supposed to make, in the post’s example, is model.to("neuron"). Jana van Greunen, director of PyTorch engineering at Meta, is quoted saying native support “expands that choice” so teams can use eager execution and the distributed APIs they already know.
PyTorch has, for several years, let a chip vendor register hardware that is not an Nvidia GPU. The slot is called PrivateUse1. TorchNeuron uses it, plus the torch.accelerator APIs, so torch.nn, tensors, autocast, torch.distributed and the profiler are supposed to see neuron the way they see any other device. Eager mode runs each operation as the Python hits it, which is the mode for debugging: prints and breakpoints still work, and there is no compile step first. torch.compile(model, backend="neuron") captures a graph, lowers it through Torch-MLIR into StableHLO, and hands the Neuron compiler a file it can run. The post also describes an experimental “adaptive eager” mode that tries to fuse operations without giving up the interactive loop. Custom kernels go through the Neuron Kernel Interface, NKI, and a shared library of those kernels, NKI-lib, includes FlashAttention and mixture-of-experts blocks.
The old path was a compiler
Until this release, Hugging Face’s route onto Trainium and Inferentia was Optimum Neuron, which sits on a torch-neuronx package built with PyTorch XLA. Régis Pierrard, in a Hugging Face post dated 19 October, describes what that required: lazy tensors instead of eager execution, static shapes only, and a fresh compilation whenever the graph broke. Optimum Neuron hid some of that behind its own model classes and trainers. TorchNeuron, he writes, makes neuron a device inside PyTorch, and Hugging Face now recommends calling Transformers, Diffusers, Accelerate, PEFT and TRL directly rather than going through Optimum Neuron. On inference, he says, 89 percent of the top 100 most-used text-generation architectures work out of the box, against 46 percent with Optimum Neuron v0.4.5. A model loads with device_map="neuron".
The same post is plain about what still does not work. The Neuron backend cannot yet compile Dynamo’s automatic dynamic shapes, so the training example turns that off and pads batches to a multiple of 128 tokens. Asynchronous weight loading in Transformers is not supported, and the workaround is an environment flag. Generation examples use a static cache, because a changing shape forces a recompile. The PyTorch blog’s own account of the first release lists further gaps: sparse tensors, forward-mode autograd, arbitrary attention masks in native multi-head attention, and dynamic and symbolic shapes. Causal masks and no mask are supported. BF16, FP32 and FP16 work. FP8, through TorchAO, is in a private beta and is not in this release. vLLM Neuron and vLLM Omni Neuron are plugins on the same compile path, for language models and for image and video diffusion.
Two customers, and a sweep without a pass rate
Deepgram’s chief technology officer, Adam Sypniewski, says that in the benchmarks his company ran after a port with AWS, Trainium processed about 60 percent more input audio seconds per dollar than Deepgram’s existing GPU baselines. That is Deepgram’s comparison, on its models. Arjun Desai, a cofounder of Cartesia, says the company ran its standard inference pipeline for the Sonic and Ink voice models “with minimal changes” to the PyTorch, and wrote NKI kernels only for the layers that dominate latency. Jeff Boudier of Hugging Face says that until now a Trainium deployment meant Neuron-specific software, and that TorchNeuron makes Trainium a native backend for Transformers and Diffusers.
AWS says it has been running a sweep of 100,000 Hugging Face models, chosen to cover 99.9 percent of real-world use by download count, and recording whether each one loads, runs a forward and backward pass, and compiles, in both eager and compiled mode. The graph in the post is dated September 2026. The prose does not print a pass rate. It says the remaining gaps are being closed in later releases. Distributed training uses the ordinary PyTorch names: DDP, FSDP2, tensor parallelism, pipeline parallelism, DTensor and DeviceMesh. Collectives stay on Trainium’s own cores inside a machine and on Elastic Fabric Adapter between machines. The beta is the install and the device string. The post does not publish a training-throughput table against an Nvidia box.


