Design, execute, and maintain test strategies and automation frameworks for AI-powered agentic workflows.
Develop specialized testing approaches for LLM applications, including response accuracy, relevance, grounding, hallucination detection, and edge-case behavior.
Evaluate autonomous and semi-autonomous agents for planning, tool selection and sequencing, agent-to-agent interaction, and error or timeout recovery.
Validate RAG document ingestion, chunking, metadata quality, embedding accuracy, and citation or source attribution.
Conduct AI safety and security testing for prompt injection, data leakage, unauthorized tool execution, and privilege escalation vulnerabilities.
Establish AI evaluation metrics and quality gates covering accuracy, groundedness, hallucination rate, tool-selection accuracy, cost, and latency.
Build and maintain automated test suites for APIs, UI, integration, RAG, and agent trajectories to support continuous evaluation.
Analyze production AI interactions and failed agent journeys, create regression tests from incidents, and support decision auditability.
Requirements
Bachelor’s degree or above in Computer Science, Engineering, Information Technology, or a related discipline.
At least 5 years of experience in software quality assurance and testing, including test planning, functional, integration, regression, and end-to-end testing.
1–2 years of hands-on experience testing GenAI, LLM, or agentic AI applications.
Strong experience building test automation frameworks and test utilities rather than relying solely on manual tools.
Strong programming ability in at least one of Java, Kotlin, or Python.
Understanding of generative AI technologies including LLMs, prompt engineering, RAG, embeddings, vector databases, AI agents, agentic workflows, and tool or function calling.
Hands-on experience with LLM and agent evaluation tools such as LangSmith, Langfuse, Ragas, or equivalent frameworks.
Nice-to-have experience with API and data integrations, JSON, SQL, databases, Kafka, CI/CD quality gates, distributed systems, AI observability, and advanced AI testing.
Financial services experience and knowledge of regulatory reporting, trade lifecycles, transaction reporting, reconciliations, and regulatory controls are beneficial.
Experience with AI or LLM orchestration frameworks such as LangChain, LangGraph, LlamaIndex, Vertex AI, OpenAI, or similar technologies is beneficial.
Ability to use AI tools productively while applying critical judgment for responsible and ethical use of data and AI outputs.
Benefits
Best-in-class leave policy.
Gender-neutral parental leave.
100% reimbursement under the childcare assistance benefit.
Sponsorship for industry-relevant certifications and education.
Employee Assistance Program for employees and family members.
Comprehensive hospitalization insurance for employees and dependents.
Accident and term life insurance.
Complimentary health screening for employees aged 35 and above.
Training and development, coaching and expert support, continuous learning, and flexible benefits.
Quality Engineer – TPI Planning and Painting Department: Quality Role Summary The Quality Engineer – TPI Planning and Painting is responsible for leading quality assurance and quality control activities related to t hird-party inspection planning, inspection coordination, inspection readiness, and release management , as ...
... test failures, diagnose defects, determine root causes, and collaborate with stakeholders to resolve issues. - Review system requirements and track and report quality assurance metrics. - Support functional, non-functional, integration, performance, security, and database testing. - Use AI tools across test design, scripting, ...
Are you ready to make a significant impact in the world of technology and innovation? Join our team as a Futures Quality Assurance Engineer and become an integral part of our forward-thinking business unit. You'll engage in strategic projects that are essential for shaping the future of our operations and innovation. This ...
Responsibilities - Develop and execute automated and manual test plans, test scripts, and functional test cases. - Design, develop, enhance, maintain, and extend QA automation frameworks and test script libraries.
Mid-level quality assurance role at Dyad Tech in Ahmedabad. Needs 3+ years. What you'll do - Analyse business and technical requirements to define effective test scenarios and cases - Perform functional, regression, integration, system, API, and end-to-end testing - Design, develop, execute, and maintain robust automated ...
... standards, and best practices - Report on test coverage, quality metrics, and release readiness What You’ll Bring - Bachelor’s or advanced degree in mechanical engineering, physics, or a related discipline - Proven Quality Assurance experience, with a strong emphasis on complex software products - Strong practical experience ...
Are you ready to make a significant impact in the world of technology and innovation? Join our team as a Futures Quality Assurance Engineer and become an integral part of our forward-thinking business unit. You'll engage in strategic projects that are essential for shaping the future of our operations and innovation. This ...
... experience in manual testing, API testing, SQL, and defect tracking while working in a fast-paced Agile environment. You will play a key role in ensuring the quality, reliability, and performance of our applications by identifying bugs, improving testing processes, and collaborating closely with cross-functional https://jobeax.com/link/ZlR3JZToBtLJKlA9 ...
Mid-level quality assurance role at Dyad Tech in Ahmedabad. Needs 2–4 years. What you'll do - Analyze business and technical requirements to create clear, traceable test scenarios and test cases - Perform functional, regression, integration, system, UI, API, and end-to-end testing - Execute manual test cycles with a focus ...
Mid-level QA and testing role at Cloud Counselage in Hyderabad. E.. E. or BCA - Employment type: Full-time - Languages: English [{"name": "full_time", "text": "Full time", "categoryName": "Type of employment"}]