2,349 monitoring and evaluation jobs in India.

... use/function calling and enterprise AI integrations. - Experience with MCP (Model Context Protocol) or similar AI integration approaches. - Experience with model evaluation, observability, and AI application monitoring. - Knowledge of fine-tuning, model adaptation, embeddings, and NLP techniques. - Experience integrating AI solutions ...

... maintaining acceptable output quality.- Establish systematic evaluation and regression testing for model quality, retrieval relevance, groundedness, hallucinations, and application reliability.- Implement robust production practices, including logging, monitoring, retries, timeouts, caching, versioning, and failure recovery.- ...

... Skills: Multi-Agent System Design & Orchestration Full-stack AI Integration (Frontend to Backend) Advanced Prompt Engineering LLM Observability, Evaluation & Monitoring (LLMOps) Data Engineering Skills: Proficient in Pyspark coding and concepts. Example: Lazy evaluation, window functions, spark architecture and performance ...

... registries, lineage management, and rollback procedures.- Experience with champion-challenger frameworks, backtesting, and model performance evaluation.- Strong understanding of model governance, data quality, and bias/fairness monitoring.- Experience leading engineering teams and communicating effectively with business and technical ...

... across agent development, LLM infrastructure, and model evaluation. - End-to-end agent lifecycle: requirements through architecture, build, evaluation, deployment, and production monitoring. - RAG and knowledge infrastructure: document ingestion pipelines, chunking strategies, embedding, vector search, and knowledge graph grounding. ...

... RMF), ISO 27001, FFIEC, SOC Reporting.Good-to-Have Skills : - Experience in AI-enabled vendor risk evaluation.- Exposure to cybersecurity threat intelligence and external risk monitoring.- Knowledge of vendor remediation and risk governance processes.- Experience working with enterprise risk management and compliance https://jobeax.com/link/qGnMiVoH8iNOW5pe ...

... - Lead research and implementation for long-context handling and cascading-error challenges in multi-step agent pipelines - Drive context engineering strategy and experimentation frameworks across the team Evaluation and Production Monitoring - Define structured rubrics for evaluating AI outputs on nuanced, ambiguous research ...

... Infrastructure and MLOps: Build and improve the infrastructure required to operate models reliably, including model versioning, CI/CD pipelines, observability, and production monitoring. - Experimentation and Evaluation: Design and run experiments, including A/B tests, to evaluate model performance and determine whether ...

... Experience integrating commercial and open-source LLMs through APIs or inference frameworks.- Strong understanding of REST APIs, microservices, distributed systems, and application architecture.- Experience with Docker, Kubernetes, CI/CD, and cloud platforms such as AWS, Azure, or GCP.- Knowledge of AI evaluation, monitoring, ...

... institutional capacity and systems change. 2. Systems and scaling science: Contribute to research in scaling science, aligning outcome delivery with tangible and high-value monitoring and learning. 3. Delivery pathways: Embed innovations in public delivery systems (policy, extension, seed systems) and in market and enterprise ...

... similar GCP https://jobeax.com/link/pb3hYSrIwv9hVTPR to Have : - Google Agent Development Kit (ADK).- Gemini Enterprise / Agent Engine.- RAGAS, Evals, or custom evaluation pipelines.- Model monitoring and observability.- Multi-agent architecture.- Human-in-the-loop workflows.- Knowledge graph or enterprise search https://jobeax.com/link/SskUR30IxJmT4jLY ...

... ML models and AI services - including fine-tuning, prompt engineering, and model selection.2. Build and maintain robust data pipelines for model training, evaluation, and monitoring using tools such as Airflow, dbt, or Spark.3. Define and enforce AI quality standards - including automated evals, red-teaming, and latency/accuracy ...

... optimization, and evaluation- Work with LLMs (open & closed), fine tuning (LoRA/PEFT), prompt engineering, and alignment- Optimize inference for latency, cost, and scale (quantization, batching, GPUs)- Build MLOps/LLMOps pipelines: CI/CD, model versioning, monitoring, and drift detection- Deploy AI systems on cloud native ...

... latency, and operational efficiency in production environments.- Build evaluation and observability into every agentic release, using test datasets, automated evaluation, tracing, and cost and latency monitoring to prove quality before and after deployment.- Evaluate emerging AI, machine learning, and agentic framework technologies ...

... development and deployment of production-grade NLP and ML models.- Work with large language models (LLMs), RoBERTa, GPT APIs, and hybrid architectures.- Guide and review the work of junior scientists and interns.- Develop pipelines for training, evaluation, and continuous model monitoring.- Collaborate with cross-functional ...

... answer-writing evaluation, personalised study aids and voice/vision-enabled learning; and internal automation that accelerates content creation, data pipelines and business operations. You will own the full stack of an AI feature: data, retrieval, models, serving, evaluation and monitoring. Key Responsibilities - LLM & RAG ...

... improve the quality of customer interactions for an international electricity and gas utilities process. The role involves monitoring calls, conducting quality evaluations and calibrations, identifying process and compliance gaps, performing root-cause analysis, preparing quality reports, and supporting coaching and continuous ...

... inspections and monitoring across production stages, ensuring customer specifications are met or exceeded. The role involves pre-production coordination, fabric evaluation, inline and final inspections, quality reporting, and troubleshooting support. Responsibilities - Conduct pre-production meetings, fabric evaluations, and ...

... observability, evaluation, and auditability.- Implement human approval, deterministic validation, and rollback controls for consequential engineering actions.- Design evaluation frameworks using representative Intel Design Quality and Reliability use cases.- Establish automated testing, CI/CD, secure configuration, monitoring, documentation, ...

... deployment, and monitoring.- Experience with Azure OpenAI, Amazon Bedrock, or Vertex AI is an advantage.6. Evaluation & Responsible AI :- Knowledge of LLM evaluation, hallucination reduction, and response quality assessment.- Understanding of AI guardrails, data privacy, access control, and prompt-injection mitigation.- ...