AI Evaluation Guidelines & Rubric Specialist

Your work at IXO

Write the instructions that allow different human reviewers to judge the same AI output consistently. You will turn incomplete or ambiguous specifications into practical guidance, identify where a rubric fails and refine the document until a reviewer can apply it with a defensible explanation.

Responsibilities

• Translate program requirements into explicit rating criteria, decision rules and examples.
• Find contradictions, vague terms and missing cases in draft guidelines.
• Test how the instructions handle difficult or borderline outputs, then revise the guidance to reduce unnecessary escalation.
• Explain domain-specific distinctions without assuming every rater has the same specialist background.
• Reconcile feedback from subject experts and maintain consistent rules across related guideline sets.
• Produce documented changes that show how a revision improves clarity and agreement.

Qualifications

• You need at least three years of professional experience in linguistics, instructional design, technical writing or a closely related field, including direct work on human-rater guidelines or rubrics in generative AI or RLHF.
• Bring concrete examples of resolving ambiguity or contradiction in written specifications, a record of increasing professional responsibility and the ability to explain nuanced instructions clearly. You should have demonstrated work across multiple subject areas and be available for at least 35 weekday hours per week for the intended work pattern.

Deliverables and terms

Deliverables include usable guideline sets, rating rubrics, edge-case examples and revision rationales. The opportunity is for US-based contributors. IXO will agree the assignment scope and schedule before work starts.

Pay

Pay is $50 $70/hr (USD).