Responsibilities

Design and maintain end-to-end business scenarios covering agent routing, tool and function calls, orchestration, handoffs, and multi-turn conversations.
Evaluate agent responses for correctness, relevance, groundedness, consistency, tone, and completeness using human and automated methods.
Build and maintain evaluation datasets, golden sets, edge cases, adversarial prompts, and automated regression suites.
Identify cross-agent, orchestration, context, integration, tool-call, looping, and runaway-behavior issues.
Test for hallucinations, prompt injection, jailbreaks, data leakage, and unsafe or out-of-scope responses.
Detect and quantify regressions caused by model, prompt, retrieval, or knowledge changes.
Triage issues with Engineering and Product, determine root causes, re-test fixes, and validate the end-user experience.
Document test strategy, evaluation results, and quality trends for global stakeholders.

Requirements

Bachelor's degree in software engineering, computer science, or a related field.
3-5 years of QA experience, including at least 1-2 years testing LLM-based applications, AI agents, or agentic workflows.
Hands-on experience evaluating non-deterministic outputs by defining pass/fail criteria, rubrics, and acceptance thresholds.
Experience building or maintaining AI evaluation or regression suites, including LLM-as-judge methods, golden datasets, or scoring rubrics.
Understanding of tool and function calling, multi-agent orchestration, planning, memory and context, and RAG.
Experience with test automation tools such as Selenium, Playwright, or TestComplete, and with API testing.

Health and well-being programs supporting work-life balance.
Collaborative culture with growth and advancement opportunities.
Virtual-first work arrangement with primary work performed from home.
Occasional travel to a physical office may be required for collaboration, design, strategy, and alignment.
Compensation packages are based on experience and desired skill set.