Evaluation Data Specialist

About the role

The work includes rubric design, careful review, factual validation, consistent judgment, error analysis, and feedback on evaluation quality.

What you'll do

• Create evaluation examples and rubrics that express clear, testable criteria.
• Review outputs consistently and document the evidence behind each decision.
• Identify ambiguous cases, recurring errors, and gaps in evaluation instructions.
• Recommend changes that improve evaluation accuracy, consistency, and usefulness.

What we're looking for

• Experience in model evaluation, data annotation, quality review, research support, or a relevant specialist field.
• Strong analytical reasoning and exceptional attention to factual accuracy.
• The ability to make consistent, defensible decisions under incomplete or ambiguous instructions.
• Clear writing that separates observations, evidence, and recommendations.

Preferred experience

• Experience with evaluation datasets, annotation workflows, SQL, Python, or quality measurement.

Equal opportunity

Entlegnant, the company behind Leiolai, is an equal opportunity employer. We consider qualified applicants without regard to race, color, religion, sex, gender identity or expression, sexual orientation, national origin, ancestry, age, disability, veteran status, genetic information, or any other status protected by law.

Applicants must be at least 18. If you need an accommodation during the application process, email talent@leiolai.com.