Architect and build an AI Inference Gateway platform serving large-scale machine learning and generative AI workloads.
Design request routing, model endpoint abstraction, traffic shaping, load balancing, failover, caching, and policy enforcement mechanisms.
Drive the technical vision and roadmap for scalable, secure, and reliable AI inference infrastructure across cloud and on-premises environments.
Develop infrastructure components in Go, Java, or Python with emphasis on performance, resiliency, and operational excellence.
Design multi-tenant capabilities for authentication, authorization, quota management, cost attribution, rate limiting, and governance.
Partner with ML Platform, Model Serving, Data, and Product Engineering teams to deploy and operate AI workloads.
Lead architecture and design reviews, raise engineering standards, and mentor senior engineers across multiple teams.
Optimize latency, throughput, resource utilization, and infrastructure efficiency for inference workloads.
Define observability standards using metrics, tracing, logging, and SLO-based operations.
Investigate production issues, conduct root-cause analysis, and implement long-term architectural solutions.
Requirements
8+ years of professional software development experience.
5+ years of experience designing and operating large-scale distributed systems in production.
Strong hands-on programming expertise in one or more of Go, Java, or Python.
Experience building highly available, mission-critical platform or infrastructure services.
Experience designing API platforms, service gateways, service mesh, or large-scale networking infrastructure.
Deep understanding of microservices architecture, distributed systems design, and cloud-native technologies.
Production experience with Kubernetes and containerized workloads.
Experience operating services on AWS, Azure, or GCP.
Strong understanding of observability, reliability engineering, capacity planning, and production operations.
Preferred experience with AI/ML inference platforms, LLM gateways, model-serving infrastructure, or GPU-accelerated workloads.
Preferred expertise in Kubernetes ecosystem technologies, platform networking, Gateway API, ingress controllers, and service mesh technologies.
Preferred experience with vLLM, Triton Inference Server, TensorRT-LLM, Ray Serve, KServe, SGLang, or similar inference-serving technologies.
Preferred understanding of high-performance networking, gRPC, streaming protocols, API gateways, and service proxy architectures.
Preferred experience designing traffic management systems involving routing, retries, circuit breaking, rate limiting, and request prioritization.
Preferred familiarity with GenAI and LLM ecosystems, model deployment, prompt routing, RAG systems, model observability, and AI governance.
Preferred experience with Kafka, Cassandra, Redis, MongoDB, or similar distributed data systems.
Preferred understanding of concurrency, synchronization, asynchronous programming, and non-blocking I/O.
Benefits
Hybrid, onsite, or remote work models are available; the hybrid model requires at least 3 days in the office per week with flexibility to work from home 2 days per week, depending on role requirements.
Some businesses may require additional office time based on the nature of the work.