Back to list

Vacancy description

Performance Engineer, Inference in Bangalore, भारत

sarvam

  • HybridMix of office and remote
  • On-siteWork from the office
  • Full-timeStandard weekly hours

128k

Bangalore, भारत

Performance Engineer, Inference

Part of Sarvam's Performance Engineering team. We are hiring two specialized performance roles - Inference (this posting) and Kernels (companion posting). They are a vertical stack: the kernels team authors the µs-level GPU code, and the inference team integrates it into a running serving stack and owns the system-level numbers. If your depth genuinely spans both, apply to either and tell us - but most candidates are strongest in one, and we hire for that depth.

Location: [Bengaluru / Chennai / Hybrid / On-site] · Team: Performance Engineering · Level: Senior

About the team

Sarvam serves multiple model families - small and large LLMs, Mixture-of-Experts, Indic ASR & TTS, streaming models and multimodal models - across a multi-node, multi-tenant fleet of Hoppers and Blackwells. The Performance Engineering team owns the numbers the rest of the company plans against: how fast we serve, how much it costs, and how much we get out of every GPU. This team works at the intersection of the serving runtime, the kernel layer, and the SRE org that keeps the fleet alive.

About the role

You will own Sarvam's production serving path for large distributed models end to end. You should be source-level fluent in at least one of SGLang, vLLM, NVIDIA Dynamo, or TensorRT-LLM - able to read and modify it where stock behavior does not fit our workloads - and you should operate and extend a distributed-serving stack at depth: disaggregated prefill-decode across nodes, distributed KV/cache transfer, and the routing and scheduling that span them. You will integrate artifacts from the model and kernel teams into a running multi-node, multi-tenant stack, and you will build and train your own speculators - draft models, distillation from the target, acceptance-rate tuning against the live serving distribution - rather than only wiring in stock implementations. You will produce, and defend, the latency and throughput numbers the company plans against, and you will spend significant time in cross-team work with architecture co-design, the kernels team, the model team, and SRE.

Your scoreboard: TTFT (p50 / p95 / p99), TPOT, throughput, GPU utilization, and cost per million tokens.

What we're looking for

  • 5+ years in ML systems, with 2+ years on inference serving at production scale. Your record shows concrete outcomes - tokens per day, throughput wins, p99 reductions - rather than "deployed a model."

  • Experience serving 100B+ parameter models in production across multi-node tensor, pipeline, or expert parallelism.

  • Source-level fluency in one of SGLang, vLLM, Dynamo, or TensorRT-LLM - you have modified the scheduler, the KV allocator, or the disaggregation path - and reading-level familiarity with the other three.

  • Distributed serving at operating-and-extending depth: a disaggregated prefill-decode stack, distributed KV/cache transfer (Mooncake or equivalent), and cross-node routing and scheduling. You have run one of these in production and modified it where it didn't fit.

  • Speculative decoding as a build-and-train competency: you have trained your own draft models or speculators (EAGLE / DFlash or otherwise), distilled them from a target model, measured and tuned acceptance rate against a real serving distribution, and composed speculation with the rest of the stack - not only integrated a stock implementation.

  • Deep understanding of KV cache internals: block tables, copy-on-write, prefix sharing, and fragmentation.

  • Working command of TP / PP / EP, NCCL primitives, and how they interact with the scheduler.

  • Multi-tenant serving: model co-location and MIG / MPS isolation.

  • Profiling fluency with Nsight Systems, framework tracing, and py-spy / perf.

  • C++ and CUDA at a read-and-modify level.

  • On-call ownership of an inference SLO.

Strong pluses

  • Upstream contributions to SGLang, vLLM, Dynamo, llm-d, TensorRT-LLM, or LMDeploy on non-trivial code paths.

  • Direct production experience with Dynamo or llm-d at scale.

  • Having operated a forked runtime in production.

  • Published or shipped speculator work - a draft model or speculative-decoding technique you trained and measured.

  • MoE serving at scale, long-context (128K+), multi-model serving, or Indic and multilingual workloads.

Similar vacancies

More like this role

  • Full-time
  • IT
... technology advances, client-side AI (local execution) will play a key role in shaping digital experiences. The LocalAI team is seeking a Senior Systems Software Engineer to build efficient on-device AI software for RTX and DGX-class systems. This role focuses on high-performance local inference, low latency, efficient memory ...

Pune, भारत

  • Remote
  • Full-time

400 / month

... multiple projects, and comfortable with frequent context switching and prioritization. About Together AI Together AI, the AI Native Cloud, is purpose-built for AI engineers. AI application developers get high-performance inference that scales reliably, fine-tuning and reinforcement learning for creating frontier-level specialized ...

भारत

$1.33 USD

... and optimize batch and streaming pipelines for large‑scale datasets - Create and manage feature pipelines and curated datasets for AI/ML model training and inference - Collaborate with data scientists and ML engineers to enable scalable ML workflows - Support MLOps pipelines, including data versioning, feature stores, and ...

Bangalore, भारत

  • On-site
  • Full-time
  • Contract

Rs 33.24 - 40.62 lakhs p.a.

... production AI/ML systems. - Deep expertise in GenAI application development: LLM orchestration, RAG, agentic frameworks (LangChain, LlamaIndex, DSPy), prompt engineering, and evaluation pipelines. - Strong foundations in ML fundamentals — model training, fine-tuning, inference optimization, quantization, and performance benchmarking. ...

Bangalore, भारत

Coupang Internal

Staff Software Engineer

... Engineering teams to enable seamless deployment and operation of AI workloads. - Lead architecture and design reviews, raise engineering standards, and mentor senior engineers across multiple teams. - Optimize system performance, latency, throughput, and infrastructure efficiency for large-scale inference workloads. - Define observability ...

Bangalore, भारत

Coupang Internal

Staff Software Engineer

  • Remote
  • Hybrid
  • On-site
  • Contract
  • IT
  • Backend
... Engineering teams to enable seamless deployment and operation of AI workloads. - Lead architecture and design reviews, raise engineering standards, and mentor senior engineers across multiple teams. - Optimize system performance, latency, throughput, and infrastructure efficiency for large-scale inference workloads. - Define observability ...

Bangalore, भारत

  • Full-time
  • AI / ML
... startups. OpenAI recently announced a multi-year partnership with Cerebras, to deploy 750 megawatts of scale, transforming key workloads with ultra high-speed inference. About the role We are looking for an ML Engineer who can take a model from its reference implementation to efficient execution on AI accelerator hardware. ...

Bangalore, भारत

CRESCENDO GLOBAL LEADERSHIP HIRING INDIA PRIVATE LIMITED

Senior Backend & MLOps Engineer - Python/Kubeflow

... low-latency inference, and model drift challenges.- Opportunity to work with a modern, cloud-native MLOps stack including Kubeflow, KServe, vLLM, Kubernetes, and vector databases.- High-impact engineering role driving automation, reliability, and performance in a fast-growing AI/ML platform environment. (ref:hirist.tech) []

Pune, भारत

  • Remote
  • Full-time
  • Contract
  • Data Science
  • Research

$150 USD - $200 USD / hour

... scientist or applied scientist . - Experience with training fine-tuning or optimizing models in PyTorch or JAX . - Deep knowledge in at least one area: LLMs GPU inference and performance Reinforcement learning (RL) or Scientific ML . - Ability to explain why a problem is hard for a frontier model. Preferred - Experience writing ...

Bangalore, भारत

... feedback to improve datasets, monitor drift and retrain models; track experiments and validate changes in shadow deployments.- Collaborate with backend and SDK engineers on reliable, low-latency inference and partner-model integration; support liveness or deepfake evaluation where relevant as the roadmap https://jobeax.com/link/EpAVfYYppfcla46L ...

Delhi, भारत