Data Engineer - Foundational

About Harmattan AI

Harmattan AI is a next-generation defense prime building autonomous and scalable defense systems. Following the close of a $200M Series B, valuing the company at $1.4 billion, we are expanding our teams and capabilities to deliver mission‑critical systems to allied forces.

About the Role

As a Data Engineer on the Foundational team, you will serve as the "plumber" for deep learning. Based in Paris, you will manage terabytes—and eventually petabytes—of raw, unstructured, and noisy video data (EO and IR). Your mission is to ensure our ML engineers spend their time designing architectures, not waiting for data loaders or wrangling corrupted files.

Responsibilities

• Build ETL/ELT pipelines to extract, decode, and store raw Electro‑Optical (EO) and Infrared (IR) video from field logs into highly optimised formats such as WebDataset, TFRecords, or Parquet.
• Develop algorithms to synchronise EO and IR frames temporally and spatially, providing paired inputs for model training.
• Architect storage‑to‑GPU pipelines to maintain >90% GPU utilisation on multi‑node training clusters without I/O bottlenecks.
• Write and optimise distributed data processing jobs using Apache Spark, Ray, or Apache Beam to process thousands of hours of tactical video logs.
• Implement automated quality checks to filter corrupted or blank frames and maintain 100% reproducible training runs through robust versioning and lineage tracking.
• Evaluate and implement advanced storage solutions (e.g., MinIO, S3 tiering) to manage growing datasets while optimising for cost and latency.

Candidate Requirements

• Educational background: BS or MS in Computer Science, Software Engineering, or Distributed Systems (highly preferred). Deep knowledge of operating systems, networking, and parallel computing is essential.
• 5–6+ years of experience building and maintaining terabyte‑scale pipelines for unstructured data such as video, images, or point clouds.
• Proven track record of maximising multi‑node GPU utilisation and optimising data loaders for frameworks like PyTorch or JAX.
• Strong command of distributed computing tools (Spark, Ray, Beam) and ML data versioning tools (DVC, Apache Iceberg, or Pachyderm).
• Systems‑thinker who thrives in a fast‑paced startup environment and views messy data as an engineering problem to be solved via automation.
• Commitment to 100% dedication to Harmattan AI’s mission of providing a defensive edge to allied nations through ethical, high‑impact technology.

We look forward to hearing how you can help shape the future of autonomous defense systems at Harmattan AI.