Vacancy description
FreshTalent
India
AI Quality Analyst - English in work from home, India is listed on Jobeax. Browse 30,000+ vacancies available.
Remote | Contractor | 3-Month Engagement
4–40 hours per week | 4 hours of overlap with PST required
As an AI Quality Analyst – English , you will evaluate a new personalization feature for an advanced AI assistant. You will assess how effectively the model uses information from previous conversations, email, search activity, and video activity to make responses more relevant, natural, and helpful.
This role combines creative prompt design, analytical evaluation, attention to detail, and strong written communication . You will create prompts based on your own personal experiences and context, then evaluate how accurately and naturally the AI incorporates that information into its responses.
You will assess dimensions such as Grounding, Integration, and Helpfulness , while identifying incorrect personalization, unsupported inferences, hallucinations, and unnatural use of personal information.
Design and execute multi-turn conversational prompts, typically involving 1–5 turns.
Create prompts that require the AI to appropriately use personal information and experiences.
Approach evaluations creatively to thoroughly test the model’s personalization capabilities.
Develop prompts based on realistic personal context and intended outcomes.
Evaluate model responses based on the intent established in the starting prompt.
Determine whether personalization was applied appropriately and meaningfully.
Identify incorrect personalization, poor inferences, forced connections, and other quality issues.
Assess whether responses are natural, relevant, useful, and easy to use.
Analyze responses for grounding issues.
Verify that claims about you are supported by available evidence.
Identify flawed inferences, unsupported claims, and hallucinations.
Evaluate whether personal information is incorporated accurately rather than simply referenced unnecessarily.
Assess how naturally personal information is integrated into responses.
Identify robotic behavior, excessive explanation, or unnecessary "overnarrating."
Evaluate whether personalization improves the overall quality of the response.
Rigorously compare two model responses side-by-side.
Stack-rank responses based on overall helpfulness, usability, naturalness, and enjoyment.
Identify subtle differences between responses that may affect the user experience.
Provide clear reasoning for your rankings.
Write concise, structured, and defensible rationales for model comparisons.
Explicitly reference relevant turn numbers when explaining issues or positive aspects.
Provide constructive feedback and detailed annotations.
Extract and verify available debugging information to confirm that chat summaries and relevant data sources were properly utilized.
Maintain strict data hygiene throughout evaluation activities.
Delete evaluation conversations as required to prevent them from affecting future chat history.
Strong English reading and writing skills, with a high degree of comprehension.
Ability to evaluate nuanced and ambiguous AI responses.
Strong analytical thinking and judgment.
Experience designing creative, multi-turn prompts based on personal context.
Ability to identify incorrect personalization, poor inferences, and forced connections.
Exceptional attention to detail when comparing model responses.
Ability to write clear, concise, and structured evaluation rationales.
Ability to provide constructive feedback and detailed annotations.
Strong communication and collaboration skills.
Ability to work independently in a remote environment.
Willingness to use your primary personal Google account, rather than a testing account, and enable the required personal data sources for genuine evaluation.
Desktop or laptop with a reliable internet connection.
Full-time availability within your local time zone is required, as the team operates across a global 24-hour schedule.
BS/BA degree or equivalent experience in a relevant field such as:
Policy
Law
Ethics
Linguistics
Journalism
Computer Science
A related analytical field
Experience in data annotation, AI quality evaluation, content moderation, or a related role is strongly preferred.
Domain-Specific Languages
Engagement Type: Contractor
Engagement Length: 3 months
Availability: At least 4 hours per day, up to 40 hours per week
Schedule Requirement: 4 hours of overlap with PST
Work Arrangement: Remote
Success in this role means consistently producing accurate, evidence-based, and well-documented AI evaluations . You will demonstrate strong judgment when assessing personalization quality, distinguish genuine contextual relevance from forced or incorrect personalization, and clearly explain why one model response performs better than another.
Your evaluations should be precise, defensible, and grounded in specific evidence from the conversations being assessed.
Shortlisted candidates will receive a Job Interest Form .
Following profile review, an assessment will be shared.
The assessment must be completed within 24 hours .
Candidates selected based on assessment results will be contacted regarding pre-onboarding requirements .
Similar vacancies
Business Analyst - Vijayawada 4+ years of total experience (Mandatory) 2–4 years of consulting experience in HCM – Cornerstone OnDemand (Mandatory) Client: Phenom Cloud Vijayawada (Work From Office) Shift: US Shift | 7:00 PM – 4:00 AM Working Days: Monday to Friday About the Client – Phenom Cloud At Phenom Cloud, innovation ...
India, Vijayawada
... should we account for sudden variability at the beginning) A postman is assigned to deliver mail and can only deliver to two neighborhoods in one day, with certain rules (e.g. Note: No other prior specialized domain experience is needed. English Proficiency: High level of reading and writing comprehension in English. Analytical ...
India, Delhi
claidroid
... strength lies in combining deep domain understanding, robust governance, rigorous quality assurance, and strong business alignment to deliver scalable, high-quality technology solutions that create measurable enterprise value. Role: Business Analyst We are looking for experienced Business Analysts to join our growing team ...
India, Pune
... should we account for sudden variability at the beginning) A postman is assigned to deliver mail and can only deliver to two neighborhoods in one day, with certain rules (e.g. Note: No other prior specialized domain experience is needed. English Proficiency: High level of reading and writing comprehension in English. Analytical ...
India, Rāniganj
... the executed Test cases [ Pass, Fail ] Any new requirements to be suggested, as identified GAP. Skills Must have Overall 7+ years of experience as a Business Analyst in the Settlement and Custody domains. Experience as a Business Analyst in the IT industry in the Finance domain. Knowledge of Capital Market activities, Financial ...
India, Bengaluru
... the executed Test cases [ Pass, Fail ] Any new requirements to be suggested, as identified GAP. Skills Must have Overall 7+ years of experience as a Business Analyst in the Settlement and Custody domains. Experience as a Business Analyst in the IT industry in the Finance domain. Knowledge of Capital Market activities, Financial ...
India, Pune
Accenture
... Marketing field Roles and Responsibilities: Have good understanding of End Market trade operation. Support and ensuring key trade data being updated/validated /maintained as per business need in system/ repository Upload and maintain data in the TM&D platform/repository to ensure accuracy and timeliness. Ensure smooth operation ...
India, Mumbai
Recruitment Room
$145600 USD - $228800 USD
... expertise in valuation, quantitative analysis, and investment recommendations to evaluate AI-generated financial and investment analysis. What You'll Do - Evaluate AI-generated valuation models, investment recommendations, industry analyses, and other analytical artifacts for quality and accuracy. - Compare AI-generated responses ...
India
Design, execute, and maintain test cases for web applications to ensure high-quality software releases. • Proficient in various testing types, including regression, ad hoc, sanity, smoke, usability, and integration testing. - Perform API testing with Postman, and Perform comprehensive database testing to assess data integrity ...
India, Noida
Mindrift - Data annotation
$8 USD / hour
... accepted. Compensation Paid per accepted task. Your rate depends on the qualification tier you reach and how efficiently you complete tasks — up to the equivalent of $8/hr . Because payment is per task, a faster pace raises your effective hourly rate. Keep in mind, quality standards must be maintained, regardless of speed.
India, Hyderabad
Mindrift - Data annotation
$8 USD / hour
... accepted. Compensation Paid per accepted task. Your rate depends on the qualification tier you reach and how efficiently you complete tasks — up to the equivalent of $8/hr . Because payment is per task, a faster pace raises your effective hourly rate. Keep in mind, quality standards must be maintained, regardless of speed.
India, Mumbai
Mindrift - Data annotation
$8 USD / hour
... accepted. Compensation Paid per accepted task. Your rate depends on the qualification tier you reach and how efficiently you complete tasks — up to the equivalent of $8/hr . Because payment is per task, a faster pace raises your effective hourly rate. Keep in mind, quality standards must be maintained, regardless of speed.
India, Delhi
Mindrift - Data annotation
$8 USD / hour
... accepted. Compensation Paid per accepted task. Your rate depends on the qualification tier you reach and how efficiently you complete tasks — up to the equivalent of $8/hr . Because payment is per task, a faster pace raises your effective hourly rate. Keep in mind, quality standards must be maintained, regardless of speed.
India, Kolkata
Anyone AI
$150 USD / hour
... in a highly technical or analytical finance role, such as: Quant researcher Quant trader Quantitative analyst Risk modeler Financial engineer Researcher or analyst at a hedge fund Researcher or analyst at an investment bank Strong ability to write clearly and precisely in fluent English. Ability to explain complex Quantitative ...
India, Delhi
Patch Infotech Private Limited
... nature of machine learning models. By implementing advanced testing strategies, you will directly influence the quality of our product releases, ensuring that our AI solutions deliver consistent, high-value outcomes for our global enterprise https://jobeax.com/link/OBSYxKmspgkYWOcn Responsibilities : - Design and maintain scalable ...
India, India
awign expert
... technical specifications. Experience with Whiz AI or product-based environments is preferred. A strong product mindset is mandatory. Candidates who have worked for Saas and UI/UX products Business Analyst who has used AI/ML OR LLMs, OR agentic AI. Super good communication Single round of interview – direct with Client
India, Pune
polycab india limited
... continuous improvement initiatives. Key Responsibilities: Conduct electrical and high-voltage testing of cables. Ensure compliance with product specifications and quality standards. Perform Product, Process, 5S, Safety, IMS, and Internal audits. Monitor product quality and investigate quality deviations. Prepare quality reports ...
India, Hālol
Synthires
$80 USD - $120 USD / hour
... detailed written feedback and professional evaluations of AI-generated work. Help establish high-quality evaluation standards for finance and investment-focused AI systems. Contribute domain expertise to improve financial reasoning and analytical capabilities within AI models. Required Qualifications - 4+ years of experience ...
India, Mumbai
FreshTalent
... skills in English. - Strong analytical, critical-thinking, and problem-solving abilities. - Ability to evaluate complex financial information objectively and explain conclusions clearly. Ideal Candidate This opportunity is well suited to Finance Analysts, Investment Bankers, FP&A Professionals, Accountants, Investment Professionals, ...
India