Datasets and Annotation Engineer

Level of qualifications required : Graduate degree or equivalent

Fonction : Temporary scientific engineer

Context

Following the priorities established in May 2024 by the Seoul Declaration for Safe, Innovative and Inclusive AI, to which France is a signatory, the French government decided to create INESIA, an institute whose mission is to bring together , without creating a new legal entity, national stakeholders involved in AI evaluation and safety , in particular:

• the French Cybersecurity Agency (ANSSI),
• the National Laboratory of Metrology and Testing (LNE),
• the Digital Regulation Expertise Center (PEReN),
• and the French National Institute for Research in Digital Science and Technology (Inria).

Within this framework, Inria primarily contributes to activities related to systemic risk analysis in the field of national security, as well as the evaluation of the performance and reliability of AI models.

This work is strategically coordinated with Inria’s AI Evaluation research program and materializes through the design and development of an AI evaluation platform, particularly focused on systems based on Large Language Models (LLMs).

The platform aims to provide an integrated, secure, and robust environment supporting the program’s research projects, while enabling the development of evaluation applications such as benchmarking campaigns and red teaming exercises. It relies on open-source tools from the AI ecosystem as well as internally developed components.

You will join a team operating in a fast-paced, iterative development environment: the platform will evolve progressively through regular operational deliverables. We are looking for individuals capable of proposing solutions, making technical trade-offs, and transforming technical requirements into operational systems.

As a Datasets and Annotation Engineer, you will play a central role in ensuring the methodological quality of evaluations by guaranteeing the relevance, representativeness, and documentation of the datasets used.

This position offers the opportunity to contribute to a strategic and ambitious project at the heart of current challenges related to AI safety, transparency, and governance, spanning technical, scientific, and societal dimensions.

Assignment

Design, structure, and document evaluation datasets used by the platform while ensuring their quality, traceability, and compliance with ethical and regulatory requirements.

Main activities

• Build datasets dedicated to evaluation purposes (benchmarks, red teaming scenarios, etc.)
• Define annotation protocols
• Conduct annotation campaigns

Required Skills

• Experience in dataset construction and management
• Understanding of the challenges and specificities of data used for LLMs
• Proficiency in Python for data processing and manipulation
• Awareness of ethical and regulatory issues related to data
• Ability to write technical documentation

Preferred Skills

• Experience in projects related to AI evaluation or AI safety

Additional Appreciated Skills

• Experience in academic research
• Technical English proficiency, both written and spoken
• Awareness of AI trustworthiness and safety challenges

We encourage you to apply even if you do not meet every requirement — we value candidates who are eager to learn and grow new skills.

Benefits package

• Partial reimbursement of public transport costs
• Leave: 7 weeks of annual leave + 10 extra days off due to RTT (statutory reduction in working hours) + possibility of exceptional leave (sick children, moving home, etc.)
• Possibility of teleworking (after 6 months of employment) and flexible organization of working hours
• Professional equipment available (videoconferencing, loan of computer equipment, etc.)
• Social, cultural and sports events and activities