Back to list

Vacancy description

Quality Assurance Engineer in Pune, भारत

Pune, भारत

Responsibilities

  • Design, execute, and maintain test strategies and automation frameworks for AI-powered agentic workflows.
  • Develop specialized testing approaches for LLM applications, including response accuracy, relevance, grounding, hallucination detection, and edge-case behavior.
  • Evaluate autonomous and semi-autonomous agents for planning, tool selection and sequencing, agent-to-agent interaction, and error or timeout recovery.
  • Validate RAG document ingestion, chunking, metadata quality, embedding accuracy, and citation or source attribution.
  • Conduct AI safety and security testing for prompt injection, data leakage, unauthorized tool execution, and privilege escalation vulnerabilities.
  • Establish AI evaluation metrics and quality gates covering accuracy, groundedness, hallucination rate, tool-selection accuracy, cost, and latency.
  • Build and maintain automated test suites for APIs, UI, integration, RAG, and agent trajectories to support continuous evaluation.
  • Analyze production AI interactions and failed agent journeys, create regression tests from incidents, and support decision auditability.

Requirements

  • Bachelor’s degree or above in Computer Science, Engineering, Information Technology, or a related discipline.
  • At least 5 years of experience in software quality assurance and testing, including test planning, functional, integration, regression, and end-to-end testing.
  • 1–2 years of hands-on experience testing GenAI, LLM, or agentic AI applications.
  • Strong experience building test automation frameworks and test utilities rather than relying solely on manual tools.
  • Strong programming ability in at least one of Java, Kotlin, or Python.
  • Understanding of generative AI technologies including LLMs, prompt engineering, RAG, embeddings, vector databases, AI agents, agentic workflows, and tool or function calling.
  • Hands-on experience with LLM and agent evaluation tools such as LangSmith, Langfuse, Ragas, or equivalent frameworks.
  • Nice-to-have experience with API and data integrations, JSON, SQL, databases, Kafka, CI/CD quality gates, distributed systems, AI observability, and advanced AI testing.
  • Financial services experience and knowledge of regulatory reporting, trade lifecycles, transaction reporting, reconciliations, and regulatory controls are beneficial.
  • Experience with AI or LLM orchestration frameworks such as LangChain, LangGraph, LlamaIndex, Vertex AI, OpenAI, or similar technologies is beneficial.
  • Ability to use AI tools productively while applying critical judgment for responsible and ethical use of data and AI outputs.

Benefits

  • Best-in-class leave policy.
  • Gender-neutral parental leave.
  • 100% reimbursement under the childcare assistance benefit.
  • Sponsorship for industry-relevant certifications and education.
  • Employee Assistance Program for employees and family members.
  • Comprehensive hospitalization insurance for employees and dependents.
  • Accident and term life insurance.
  • Complimentary health screening for employees aged 35 and above.
  • Training and development, coaching and expert support, continuous learning, and flexible benefits.
  • Location: Pune, India.
Loading similar vacancies…