Staff Backend Engineer - K8 (Envoy) in Bangalore, भारत

Responsibilities

• Architect and build an AI Inference Gateway platform serving large-scale machine learning and generative AI workloads.
• Design request routing, model endpoint abstraction, traffic shaping, load balancing, failover, caching, and policy enforcement mechanisms.
• Drive the technical vision and roadmap for scalable, secure, and reliable AI inference infrastructure across cloud and on-premises environments.
• Develop infrastructure components in Go, Java, or Python with emphasis on performance, resiliency, and operational excellence.
• Design multi-tenant capabilities for authentication, authorization, quota management, cost attribution, rate limiting, and governance.
• Partner with ML Platform, Model Serving, Data, and Product Engineering teams to deploy and operate AI workloads.
• Lead architecture and design reviews, raise engineering standards, and mentor senior engineers across multiple teams.
• Optimize latency, throughput, resource utilization, and infrastructure efficiency for inference workloads.
• Define observability standards using metrics, tracing, logging, and SLO-based operations.
• Investigate production issues, conduct root-cause analysis, and implement long-term architectural solutions.

Requirements

• 8+ years of professional software development experience.
• 5+ years of experience designing and operating large-scale distributed systems in production.
• Strong hands-on programming expertise in one or more of Go, Java, or Python.
• Experience building highly available, mission-critical platform or infrastructure services.
• Experience designing API platforms, service gateways, service mesh, or large-scale networking infrastructure.
• Deep understanding of microservices architecture, distributed systems design, and cloud-native technologies.
• Production experience with Kubernetes and containerized workloads.
• Experience operating services on AWS, Azure, or GCP.
• Strong understanding of observability, reliability engineering, capacity planning, and production operations.
• Preferred experience with AI/ML inference platforms, LLM gateways, model-serving infrastructure, or GPU-accelerated workloads.
• Preferred expertise in Kubernetes ecosystem technologies, platform networking, Gateway API, ingress controllers, and service mesh technologies.
• Preferred experience with vLLM, Triton Inference Server, TensorRT-LLM, Ray Serve, KServe, SGLang, or similar inference-serving technologies.
• Preferred understanding of high-performance networking, gRPC, streaming protocols, API gateways, and service proxy architectures.
• Preferred experience designing traffic management systems involving routing, retries, circuit breaking, rate limiting, and request prioritization.
• Preferred familiarity with GenAI and LLM ecosystems, model deployment, prompt routing, RAG systems, model observability, and AI governance.
• Preferred experience with Kafka, Cassandra, Redis, MongoDB, or similar distributed data systems.
• Preferred understanding of concurrency, synchronization, asynchronous programming, and non-blocking I/O.

Benefits

• Hybrid, onsite, or remote work models are available; the hybrid model requires at least 3 days in the office per week with flexibility to work from home 2 days per week, depending on role requirements.
• Some businesses may require additional office time based on the nature of the work.