AI Benchmarking Lead, Performance Benchmarking Evaluation

Job ID: | ADCI HYD 13 SEZ - H84

Join our mission-critical team supporting Seller Assistant, Amazon's Gen-AI powered copilot that helps sellers navigate Amazon's complex ecosystem and grow their businesses. As a Quality Assurance Specialist, you'll play a pivotal role in ensuring the reliability and accuracy of AI model evaluations as we scale from 61% to 90%+ active seller coverage worldwide.

About Seller Assistant

Seller Assistant is a conversational AI copilot that understands the full context of a seller's business. It intelligently orchestrates backend tools to deliver actionable, drilled-down responses and can independently complete complex tasks on behalf of sellers with their permission.

Our Scale and Impact

• Expanded to 2.44MM sellers (45x growth vs. Dec 2024)
• Currently serving 61% of active sellers worldwide across 9 international stores (CN2XX, IN, UK, DE, JP, BR, MX, AE, SA)
• Supporting four languages: English, Chinese, German, and Japanese
• 2026 Goal: Scale to 90%+ active sellers WW with 5 new store launches (France, Italy, Spain, Canada, Australia)

As a Quality Assurance Specialist/AI Benchmarking Lead, you will benchmark Seller Assistant AI models for relevancy, correctness, and completeness. The role focuses on evaluating audits, improving reliability, and ensuring quality standards.

Your primary responsibilities include:

1. Evaluate audits performed by the core auditing team to increase confidence in evaluation metrics.
2. Improve audit reliability and consistency through systematic measurement of auditor accuracy.
3. Conduct targeted calibration to ensure quality standards across the auditing function.
4. Enforce quality standards by quality-checking audits and providing actionable feedback to team members.
5. Drive continuous improvement in audit processes and methodologies.

Key Responsibilities

• Conduct quality checks on audits performed by the core auditing team.
• Identify rubric gaps and evaluation ambiguities that lead to inconsistent audit outcomes.
• Surface high-confidence product issues earlier by validating and categorizing model failures.
• Serve as point of contact for annotation tasks across ML data process areas, ensuring quality execution and delivery.
• Understand dependencies across ML data workflows and articulate customer impact effectively.
• Modify existing annotation methods and update SOPs.
• Document SOP changes, secure approval, share knowledge with the team, and audit adoption and execution.
• Test new SOPs and tools, providing feedback on quality and improvement recommendations to support onboarding.
• Structure data collection, analyze results and share inputs for SOP changes.
• Collate, track, and report progress on key metrics agreed to with respective stakeholders (e.g., Program managers, Applied Scientist) specific to your functional area.
• Identify operational issues related to process and tooling and recommend suggestions to improve key project metrics such as productivity and quality.

Basic Qualifications

• Bachelor's degree or equivalent.
• Experience in natural language data labeling, data annotation, linguistic annotation or other forms of data markup.
• Technical Skills: Proficiency in MS Excel; basic understanding of SQL and Python.
• Experience with Microsoft Office products and applications.
• Strong verbal and written communication skills in English.
• Knowledge about SOA and processes that deal with sellers.

Preferred Qualifications

• 1 to 3 years of equivalent experience.
• Performed annotation related tasks across ML data process areas.
• Strong knowledge of process documentation, analysis knowledge.
• Technical proficiency in SQL querying and Python programming for data analysis.
• Strong analytical and problem-solving skills.
• Ability to work independently and as part of a team.

Our inclusive culture empowers Amazonians to deliver the best results for our customers. If you have a disability and need a workplace accommodation or adjustment during the application and hiring process, including support for the interview or onboarding process, please visit for more information. If the country/region you're applying in isn't listed, please contact your Recruiting Partner.

Posted: April 24, 2026 (Updated 17 days ago)

Amazon is an equal opportunity employer and does not discriminate on the basis of protected veteran status, disability, or other legally protected status.