Member of Technical Staff, Inference Systems
Job Description
Job DescriptionAbout the Role\n
Join a small AI systems team building a high-performance inference platform from the ground up. You will shape the runtime architecture and improve how large language models are served across GPUs and nodes.
\nWhat You'll Do\n
- Build a Rust inference runtime, owning batching, scheduling, request routing, and the serving stack.
- Design KV cache management, prefix caching, and other optimizations to reduce latency and cost per token.
- Scale inference across multiple GPUs and nodes.
- Profile and benchmark the inference pipeline, then deliver performance improvements.
- Collaborate with the team on core architectural decisions.
\nWhat We're Looking For\n
- Two to ten years of backend or distributed systems engineering experience.
- Hands-on experience building, operating, or optimizing LLM inference or serving systems, beyond using hosted APIs.
- Strong understanding of transformer inference, including attention, KV cache, batching, scheduling, and performance bottlenecks.
- Experience with production inference engines such as vLLM, SGLang, or TensorRT-LLM, and performance-critical systems where latency, throughput, and cost matter.
- Strong systems programming skills in Rust, C++, Go, or systems-level Python and PyTorch, with the ability to become productive in Rust quickly.
- Experience with multi-GPU or multi-node serving, CUDA or Triton, or techniques such as prefix caching, speculative decoding, and prefill/decode disaggregation is valuable.
- Strong computer science and systems fundamentals.
\nCompensation & Benefits\n
Base salary: $230,000 to $350,000 annually. Visa sponsorship is available.
\nLocation\n
On-site in Palo Alto, California, United States.
