Data Engineer

We're looking for a Senior Data Engineer to design, build, and optimise robust data pipelines that process massive datasets in both batch and real-time. Working at the intersection of software engineering and data science, you'll ensure our data architecture is scalable, reliable, and follows industry best practices - building systems that power B2B data products at global scale.

• Contract type: Permanent/Contract

Responsibilities

• Design and implement complex data processing pipelines using Apache Spark
• Build scalable, distributed systems for high-throughput data streams and large-scale batch processing
• Design and implement real-time pipelines with strong data quality and validation controls
• Manage and provision cloud infrastructure using Terraform
• Implement CI/CD workflows using GitHub Actions for automated testing and deployment
• Uphold rigorous engineering standards — unit/integration testing, code reviews, and clean documentation
• Translate business requirements into technical specifications alongside cross-functional stakeholders

Qualifications

• Degree in Computer Science, Engineering, Mathematics, or related field preferred
• 5+ years in data engineering and the software development lifecycle
• 4+ years hands‑on with AWS cloud services
• 4+ years building production data applications across relational and columnar data stores
• Deep expertise in Apache Spark — Core, SQL, and Structured Streaming
• Strong proficiency in Scala or Java for JVM-based production development
• Advanced SQL for transformation, analysis, and performance tuning
• Hands‑on experience with Terraform and GitHub Actions
• Experience with Apache Flink for low-latency stream processing
• Proficiency in Python for automation and scripting
• Familiarity with Kafka, event‑driven architectures, and streaming data
• Experience with workflow engines — Airflow, Luigi, or Azure Data Factory
• Familiarity with Lakehouse architectures (Delta Lake, Iceberg) or NoSQL databases