151 monitoring and evaluation jobs in Pune, India.

... vendor scoring, risk prediction, and control monitoring. The Senior Consultant provides technical depth to assessment delivery, contributes to methodology design, and supports junior analysts in executing high-quality risk evaluations across diverse industry sectors. Key Responsibilities - Lead and independently execute Third ...

... International Voice Customer Service experience, including 1–2 years in a dedicated Quality Analyst / Quality Monitoring role. - Hands-on experience in call monitoring, quality evaluation and calibration. - Strong understanding of quality scoring parameters, critical/fatal errors and compliance standards. - Experience evaluating ...

... practices, including source control, code review, automated testing, and technical documentation.- Implement MLOps and GenAIOps practices, including CI/CD, model and prompt versioning, experiment tracking, automated deployment, rollback, and lifecycle management.- Establish production monitoring and evaluation to track model ...

... engineering, training, deployment, and inference.- Experience building or integrating REST APIs and working with backend services.- Familiarity with structured and unstructured datasets.- Experience with cloud platforms, preferably GCP.- Understanding of model monitoring, evaluation, versioning, and performance https://jobeax.com/link/WrPpyfU0ZsWMfPbm ...

... maintaining acceptable output quality.- Establish systematic evaluation and regression testing for model quality, retrieval relevance, groundedness, hallucinations, and application reliability.- Implement robust production practices, including logging, monitoring, retries, timeouts, caching, versioning, and failure recovery.- ...

... registries, lineage management, and rollback procedures.- Experience with champion-challenger frameworks, backtesting, and model performance evaluation.- Strong understanding of model governance, data quality, and bias/fairness monitoring.- Experience leading engineering teams and communicating effectively with business and technical ...

... retry logic, and failure handling. - Prototype to Production Delivery: Take agentic systems from initial concept to scalable production, managing deployment, monitoring, cost, latency, and operational support against live enterprise client data. - Evaluation Infrastructure: Build and maintain rigorous evaluation suites, task ...

... Familiarity with API Gateway CloudWatch IAM Secrets Manager S3 queues databases and related AWS services • Understanding of cloud security permissions monitoring and production support AI agents and orchestration • Design build and deploy AI agents and AI-enabled workflows within the approved architecture • Use Amazon and ...

... programming, Git- Deployment : Cloud platforms, Docker, CI/CD, MLOps, LLMOps, monitoringIdeal Candidate Profile :A hands-on AI professional with strong problem-solving and software engineering skills, capable of leading technical discussions, working directly with clients, and delivering secure, reliable, and scalable AI solutions ...

... training, evaluation, and performance optimization.- Deploy machine learning models into production environments using scalable APIs and cloud infrastructure.- Build and maintain end-to-end ML pipelines, including training, inference, monitoring, and model versioning.- Work closely with product, engineering, and business teams ...

... with hands-on experience in PyTorch and Hugging Face.- Deep experience in NLP : information extraction, text classification and structuring unstructured text.- Hands-on experience applying LLMs, including prompting, fine-tuning (full and parameter-efficient methods such as LoRA) and evaluation.- Strong statistics and experiment ...

... extraction, and reasoning. - Develop agentic AI solutions using LLMs, LangChain, LangGraph, and related frameworks. - Apply prompt engineering, orchestration, evaluation, monitoring, text processing, embeddings, and vector-based retrieval practices. - Integrate AI services with MSCI data science platforms and APIs and collaborate ...

... & Optimization : - Test AI agents and workflows for accuracy, reliability, latency, scalability, and cost efficiency.- Implement AI observability, evaluation, and monitoring practices.- Ensure data privacy, security, permissions, and responsible AI usage.- Contribute to standards for AI agent lifecycle management and https://jobeax.com/link/KsGA28k3kWQGcNzc ...

... inference server such as vLLM, TGI or Ollama, on GPU or CPU.- Solid grounding in text classification, named entity recognition and summarisation, including evaluation metrics and calibration.- Experience exposing models as APIs (FastAPI or similar) and packaging them with Docker.- Understanding of PII handling and data protection ...

... processes and generate intelligent insights.- Build and ship production-ready software using AI as a core part of the engineering lifecycle.- Apply prompt engineering and AI evaluation frameworks to develop and validate solutions.- Maintain solution quality and compliance while ensuring scalability and responsible AI use.- Align ...

... GitHub, portfolio, or projects you’ve built/shipped. That’s what we’ll look at first. What helps Each one moves you up the list. • Conversational interfaces, and multi-turn state across channels • LLM evaluation and monitoring — Ragas, LangSmith, Langfuse, or your own harness • Working knowledge of Generative UI. • AdTech, ...

... retry logic, and failure handling. - Prototype to Production Delivery: Take agentic systems from initial concept to scalable production, managing deployment, monitoring, cost, latency, and operational support against live enterprise client data. - Evaluation Infrastructure: Build and maintain rigorous evaluation suites, task ...

... or optimization.- Establish baselines and evaluation criteria; interpret results, investigate failure modes, check for data leakage and communicate limitations and confidence clearly.- Partner with data engineers and analysts on data profiling, feature engineering, data pipelines, dataset quality, reproducibility and reliable ...

... cost, and measurable business value. - Contribute to standards for agent permissions, observability, evaluation, and lifecycle management. AI & LLM Technologies Candidates should have hands-on experience with or strong practical knowledge of: - Claude / Anthropic and/or other leading foundation models - Prompt and context ...

... Engineers, supporting due diligence and contributing to successful project financing. - Collaborate with O&M Teams to implement advanced diagnostic techniques (EL and FTR) for monitoring and maintaining module health. - Develop and Standardize SOPs and Checklists, promoting operational excellence and consistency across processes. ...