Back to list

Vacancy description

Adversarial Evaluator, AI Search Red Teaming Evaluator in भारत

Truelancer AI

  • RemoteWork from anywhere
  • ContractFixed-term or freelance

भारत

Contract · Remote · 5-week pilot · 15 openings

About the role

Truelancer AI is building a specialist red team for an AI search safety pilot with a global enterprise client. You’ll work as an individual contributor. You’ll own specific attack surfaces, design and run adversarial scenarios against an AI search and RAG system, and produce evidence that holds up under QA review.

We value complete evidence and defensible coverage more than prompt volume.

What you’ll do

  • Plan attacks: form risk hypotheses and design original, bounded adversarial scenarios.

  • Run attacks: execute multi-turn, adaptive prompt attacks, including context escalation, on authorised systems.

  • Evaluate outputs for these failure types:

    • grounding errors

    • source contradictions

    • stale information

    • unverifiable citations

    • trust-boundary violations, such as indirect prompt injection

  • Record evidence that can be reproduced:

    • exact prompt sequences

    • session and version context

    • timestamps

  • Submit finding cards to QA, and adjust your judgements as rubrics and guidelines are updated.

  • Classify each outcome as a candidate finding, a quality issue or inconclusive. Know when to stop and escalate an unsafe or uncertain case.

  • Stay within scope: keep to client-approved boundaries and privacy policies at all times.

  • Keep current on client taxonomies, system changes, policy revisions and new risk definitions.

  • Work with the team: the Red-Team Lead, Project Manager and QA analysts.

Who you are

  • Experience: 3 or more years in one or more of the following:

    • AI red teaming, LLM evaluation or GenAI testing and QA

    • Trust & Safety, content safety or content policy operations

    • Fraud, abuse or risk investigations

    • Responsible AI, AI governance or AI safety

  • Methods: hands-on experience with LLM and model evaluation, RLHF or human-feedback work, and prompt engineering.

  • Adversarial skill: working knowledge of prompt injection, jailbreak testing and adversarial techniques.

  • Mindset: sceptical and analytical. You can rule out false positives and check citations against sources.

  • Communication: excellent written and spoken US English, suited to structured reporting.

  • Detail: very high attention to detail and ownership of what you deliver.

  • Education: graduate or undergraduate degree.

How you’ll be measured

  • Defensible coverage of your assigned surfaces, and how reliably your findings reproduce

  • QA calibration and adjudication scores, with a target of at least 85% agreement

  • Compliance with client boundary and privacy policies

  • Quality of your finding cards and how well you take on calibration updates

Guidelines

Truelancer AI verifies each project before posting. Apply on Truelancer AI; never pay to get work.

Similar vacancies

More like this role

  • AI / ML
... and Automation) Join the AI Security practice to independently secure AI/GenAI systems and lead adversarial testing initiatives. The role focuses on protecting AI components across lifecycle and implementing robust controls in enterprise environments. Your Key Responsibilities - Lead AI red teaming execution including prompt ...

Bangalore, भारत

Société Générale

Lead AI Security Engineer- IDP

  • Full-time
  • Cybersecurity
... Experience- - Experience with AI security engineering, AI red teaming, adversarial testing, model robustness evaluation, secure RAG design, agent security, or AI platform security in regulated environments. - Experience with enterprise AI platforms such as Azure OpenAI, Microsoft AI Foundry, Azure AI Services, Databricks, ...

Bangalore, भारत

  • Remote
  • Contract

₹2850 INR - ₹3800 INR / hour

HR contract role at Mercor. Remote; pays ₹2,850–3,800 an hour. What you'll do - Conduct daily search queries and rate the results to improve AI model outputs - Complete assigned queries each day to ensure timely evaluation - Provide short notes and annotations to enhance training data quality Job details - Location: Remote, ...

Bombay, भारत

  • Remote
  • Full-time
  • Contract

$18 USD - $22 USD / hour

... detail. - Sound judgment around sensitive and dual-use information. Preferred - Based in India or the wider South Asia region. - Experience reviewing grading or red-teaming written or technical content. - Background in trust and safety content moderation policy evaluation or adversarial testing. Start Date - Immediate Application ...

Bombay, भारत

  • Remote
  • Contract
... as required by project protocols. - Record findings and submit evaluation datasets on AI training platforms in line with project standards and milestones. - Maintain a high standard of attention to detail and rigor across all evaluation tasks. Preferred Qualifications - Experience with AI products, AI models, or AI agent ...

India, भारत

  • Full-time
  • Contract
... API keys, tokens, and autonomous agent credentials, across AI and agentic workflows. - Work with AI, detection engineering, and data teams to implement red teaming, adversarial testing, model risk scoring, and anomaly detection for AI systems. - Help define data requirements, feedback loops, and operational guardrails ...

भारत

  • Remote
  • Contract

$30 USD - $40 USD / hour

About the job Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors include Benchmark , General Catalyst , Peter Thiel , Adam D'Angelo , Larry Summers , and Jack Dorsey . Position: Search Generalist Type: Contract Compensation: $30–$40/hour Location: ...

Bombay, भारत

  • On-site
  • Contract
  • AI / ML
  • Consulting
... platforms. - Conduct AI red teaming assessments (manual and automated) to identify safety, security, misuse, and adversarial risks. - Design and implement Responsible AI frameworks for bias detection, fairness evaluation, explainability, and robustness. - Build and maintain automated AI validation, monitoring, and alerting pipelines ...

Bombay, भारत

  • On-site
  • Contract
  • AI / ML
  • Consulting
... platforms. - Conduct AI red teaming assessments (manual and automated) to identify safety, security, misuse, and adversarial risks. - Design and implement Responsible AI frameworks for bias detection, fairness evaluation, explainability, and robustness. - Build and maintain automated AI validation, monitoring, and alerting pipelines ...

Bombay, भारत

  • Remote
  • Contract
... products, models or agents - Preferred: precise, consistent instruction-following and strong attention to detail - Preferred: a regular, everyday AI user - Preferred: comfort with repetitive, high-volume tasks under strict timeframes - Preferred: past work in digital products, software testing or AI research Suits people who ...

भारत