Vacancy description
Quality Assurance Engineer in Pune, भारत
Pune, भारत
Responsibilities
- Design, execute, and maintain test strategies and automation frameworks for AI-powered agentic workflows.
- Develop specialized testing approaches for LLM applications, including response accuracy, relevance, grounding, hallucination detection, and edge-case behavior.
- Evaluate autonomous and semi-autonomous agents for planning, tool selection and sequencing, agent-to-agent interaction, and error or timeout recovery.
- Validate RAG document ingestion, chunking, metadata quality, embedding accuracy, and citation or source attribution.
- Conduct AI safety and security testing for prompt injection, data leakage, unauthorized tool execution, and privilege escalation vulnerabilities.
- Establish AI evaluation metrics and quality gates covering accuracy, groundedness, hallucination rate, tool-selection accuracy, cost, and latency.
- Build and maintain automated test suites for APIs, UI, integration, RAG, and agent trajectories to support continuous evaluation.
- Analyze production AI interactions and failed agent journeys, create regression tests from incidents, and support decision auditability.
Requirements
- Bachelor’s degree or above in Computer Science, Engineering, Information Technology, or a related discipline.
- At least 5 years of experience in software quality assurance and testing, including test planning, functional, integration, regression, and end-to-end testing.
- 1–2 years of hands-on experience testing GenAI, LLM, or agentic AI applications.
- Strong experience building test automation frameworks and test utilities rather than relying solely on manual tools.
- Strong programming ability in at least one of Java, Kotlin, or Python.
- Understanding of generative AI technologies including LLMs, prompt engineering, RAG, embeddings, vector databases, AI agents, agentic workflows, and tool or function calling.
- Hands-on experience with LLM and agent evaluation tools such as LangSmith, Langfuse, Ragas, or equivalent frameworks.
- Nice-to-have experience with API and data integrations, JSON, SQL, databases, Kafka, CI/CD quality gates, distributed systems, AI observability, and advanced AI testing.
- Financial services experience and knowledge of regulatory reporting, trade lifecycles, transaction reporting, reconciliations, and regulatory controls are beneficial.
- Experience with AI or LLM orchestration frameworks such as LangChain, LangGraph, LlamaIndex, Vertex AI, OpenAI, or similar technologies is beneficial.
- Ability to use AI tools productively while applying critical judgment for responsible and ethical use of data and AI outputs.
Benefits
- Best-in-class leave policy.
- Gender-neutral parental leave.
- 100% reimbursement under the childcare assistance benefit.
- Sponsorship for industry-relevant certifications and education.
- Employee Assistance Program for employees and family members.
- Comprehensive hospitalization insurance for employees and dependents.
- Accident and term life insurance.
- Complimentary health screening for employees aged 35 and above.
- Training and development, coaching and expert support, continuous learning, and flexible benefits.
- Location: Pune, India.
Loading similar vacancies…