Senior QA Engineer in Pune, भारत

Responsibilities

• Design and maintain end-to-end business scenarios covering agent routing, tool and function calls, orchestration, handoffs, and multi-turn conversations.
• Evaluate agent responses for correctness, relevance, groundedness, consistency, tone, and completeness using human and automated methods.
• Build and maintain evaluation datasets, golden sets, edge cases, adversarial prompts, and automated regression suites.
• Identify cross-agent, orchestration, context, integration, tool-call, looping, and runaway-behavior issues.
• Test for hallucinations, prompt injection, jailbreaks, data leakage, and unsafe or out-of-scope responses.
• Detect and quantify regressions caused by model, prompt, retrieval, or knowledge changes.
• Triage issues with Engineering and Product, determine root causes, re-test fixes, and validate the end-user experience.
• Document test strategy, evaluation results, and quality trends for global stakeholders.

Requirements

• Bachelor's degree in software engineering, computer science, or a related field.
• 3-5 years of QA experience, including at least 1-2 years testing LLM-based applications, AI agents, or agentic workflows.
• Hands-on experience evaluating non-deterministic outputs by defining pass/fail criteria, rubrics, and acceptance thresholds.
• Experience building or maintaining AI evaluation or regression suites, including LLM-as-judge methods, golden datasets, or scoring rubrics.
• Understanding of tool and function calling, multi-agent orchestration, planning, memory and context, and RAG.
• Experience with test automation tools such as Selenium, Playwright, or TestComplete, and with API testing.

Benefits

• Health and well-being programs supporting work-life balance.
• Collaborative culture with growth and advancement opportunities.
• Virtual-first work arrangement with primary work performed from home.
• Occasional travel to a physical office may be required for collaboration, design, strategy, and alignment.
• Compensation packages are based on experience and desired skill set.