Google open-sources TorchTPU for PyTorch on TPUs, and DeepSeek open-sources DeepInfer
TorchTPU, Tuesday at the PyTorch Conference: a Google and Meta backend that runs PyTorch on a TPU with model.to("tpu"), in eager mode and through torch.compile. The PyTorch post says about 450 operators are in, and that the 5,000 most-downloaded Hugging Face models were checked. DeepInfer, the same day, is Apache 2.0 and runs DeepSeek-V4.1-Flash on Nvidia SM100 and Huawei Ascend 950. The README says it is not yet a full serving stack.
Feature4 minIn-depth
Eli Navarro4 min read
- PyTorch
- TPU
- DeepSeek
- Inference

Photo: PyTorchSource page
Google and DeepSeek both opened, on Tuesday, software they use to run models on chips that are not a generic Nvidia box. Google put TorchTPU on GitHub, a PyTorch backend for TPUs co-developed with Meta’s PyTorch team and announced at the PyTorch Conference. DeepSeek put DeepInfer on GitHub under the Apache 2.0 licence. DeepInfer, the company said on X, is the framework behind its production service. For now the README supports one model, DeepSeek-V4.1-Flash, on two kinds of hardware: Nvidia SM100 GPUs and Huawei Ascend 950 processors.
A device named tpu
The change TorchTPU asks for, in the PyTorch post, is model.to("tpu") in place of model.to("cuda"). Eager mode runs each operation as the Python reaches it, which is the mode for prints and debugging. torch.compile, through Torch Dynamo, is the path for speed. Google’s older PyTorch/XLA route compiled graphs and, the PyTorch post says, did not offer PyTorch’s ordinary eager execution, asked for source changes to mark what could be compiled, and lagged new distributed features. TorchTPU’s authors say they kept JAX out of the runtime. They also say a one-line device change will run a model and will not, by itself, hit peak speed, because shapes and sharding written for a GPU can be a poor fit for a TPU.
The PyTorch post’s inventory, as of Tuesday, is about 450 ATen operators, including forward and backward for the core operators the major architectures need, and a check that the 5,000 most-downloaded Hugging Face models run out of the box. The list it names includes Qwen, DeepSeek, Gemma, Llama and Mistral, plus vision and speech models. Custom kernels can be written in Pallas or in Helion. Google’s cloud post says Transformers v5 deploys to a TPU with device_map="tpu". A TPU device, the PyTorch post says, can be used by only one user-space process, so the stack uses one process per chip. Reinforcement-learning jobs that need to share a chip are not supported as shipped. The compiler still wants shapes known when it compiles. The post says they are adding bounded dynamism, where the maximum size is known ahead and the actual size arrives at runtime. Matrix multiplies run in bfloat16. The default precision mode, called medium, accumulates in 32-bit floats.
Serving and training sit on that device. Google says a vLLM backend, built with Inferact, and an SGLang backend, built with RadixArk, keep each project’s scheduler and API server and add TPU kernels underneath. Both support prefix caching, speculative decoding and a split between prefill and decode. On training, Google says TorchTitan recipes have been run on as many as 8,192 Ironwood TPUs, and that one path reached up to 52 percent model-flops utilization on Qwen 3 8B. That utilization figure is Google’s, on that model. A customer named in the same post, Zenteiq, says prefix-aware routing on TPU v6e lifted its synthetic-data job from 1.47 billion to 5.97 billion tokens a day per 1,000 chips. That is Zenteiq’s workload.
TorchTPU is a massive step forward — allowing us to co-develop hardware flexibly with the core machine learning systems team while supporting major LLM, Ads, and Recommendation systems workloads on key production use cases.
— Jana van Greunen, director of PyTorch engineering at Meta, in Google’s post, Oct 20, 2026
DeepInfer leaves out the service around the model
DeepInfer’s README is explicit about the hole. A complete inference service still needs a KV-cache system, a split between prefill and decode, and load balancing across data-parallel replicas. On the split, the README says most of the required engine work is already in the code. What a developer gets today is offline inference: kernels, including FlashMLA, DeepGEMM and DeepEP, parallelism for attention and for mixture-of-experts experts, speculative decoding that drafts tokens and then checks them, an asynchronous scheduler, and structured output against a regex, a JSON schema or a grammar. Batch-invariant runs and deterministic sampling are marked NVIDIA only.
The Next Web, reading the same release, reported that DeepSeek first built the stack mainly on vLLM and also drew on SGLang and Nvidia’s TensorRT-LLM, and that the vLLM project thanked the company and said it would take ideas back into its own V4.1-Flash support. DeepSeek’s published benchmarks, The Next Web wrote, were measured on four SM100 GPUs for prompts and 64 for generating output. They are the company’s figures. Bloomberg’s Saritha Rai reported that no one outside DeepSeek has independently tested the gains, The Next Web said. DeepSeek also shipped deepseek-recipe, a library in Rust and Python that is supposed to match how the official API turns text into tokens and back. The Next Web said the GitHub repository had passed 8,000 stars by Tuesday afternoon, and that the release came a day after civilian federal agencies were ordered to remove DeepSeek products. The code is the production stack DeepSeek is willing to publish. The README’s own list of what a full service still needs is the part a team would be renting, or building, on top.


