Banner BG.png

Senior Data Engineer

Reports to: 
Location: United Kingdom
Head of AI

About the Job

We are looking for a high calibre individual who is passionate about technology, designing and building software solutions and delivering applications with high utility.

You will be instrumental in driving the design and implementation of the eXate platform to accommodate higher throughput, uptime and an extensible set of features, ensuring that the platform is robust, well tested and functional.

To comply with data privacy and protecting data is hard. We have solved this problem by aggregating and simplifying the use of different privacy enhancing technologies and need to get the word out to potential clients.

We are looking for a real computer scientist with an obsession with solving data challenges. If you would love to work for a start-up, loves a challenge, and has a creative mindset alongside exceptional problem solving skills this could be a potential fit.

Major Accountabilities

Collaborate with the CIO on application Architecture and Design of our ETL (Extract, Transform, Load) and other aspects of Data Pipelines. Our stack is built on top of the well known Spark Ecosystem (e.g Scala, Python, etc)

Periodically evaluate architectural landscape for efficiencies in our Data Pipelines and define current state, target state architecture and transition plans, road maps to achieve desired architectural state

Conducts/leads and implements proof of concepts to prove new technologies in support of architecture vision and guiding principles (e.g. Flink)

Assist in the ideation and execution of architectural principles, guidelines and technology standards that can be leveraged across the team and organization. Specially around ETL & Data Pipelines

Promotes consistency between all applications leveraging enterprise automation capabilities

Provide architectural consultation, support, mentoring, and guidance to project teams, e.g. architects, data scientist, developers, etc.

Collaborate with the DevOps Lead on technical features

Define and manage work items using Agile methodologies (Kanban, Azure boards, etc)

Leads Data Engineering efforts (e.g. Scala Spark, PySpark, etc)

Key Requirements

Extensive data engineering development experience (e.g. ETL), using well known stacks (e.g. Scala Spark, NiFi)

Experience in Technical Leadership positions (or looking to gain experience)

Background – software engineering

The ability to write technical documentation

Solid understanding of virtualization and/or cloud computing technologies (e.g. docker, kubernetes)

Experience in designing software solutions and enjoys UML and the odd sequence diagram

Experience operating within an Agile environment

Ability to work independently and with minimum supervision

Strong project development management skills, with the ability to successfully manage and prioritise numerous time pressured analytical projects/work tasks simultaneously

Able to pivot quickly and make rapid decisions based on changing needs in a fast-paced environment

Works constructively with teams and acts with high integrity

Passionate team player with an inquisitive, creative mindset and ability to think outside the box

Knowledge & Experience

Experienced with Spark, Delta Lake, and Scala to work with Petabytes of data (to work with Batch and Streaming flows)

Knowledge of a wide variety of open source technologies including but not limited to; NiFi, Kubernetes, Docker, Hive, Oozie, YARN, Zookeeper, PostgreSQL, RabbitMQ, Elasticsearch

A strong understanding of AWS/Azure and/or technology as a service (Iaas, SaaS, PaaS)

Strong verbal and written communications skills are a must, as well as the ability to work effectively across internal and external organizations and virtual teams

Appreciation of building high volume, low latency systems for the API flow

Core Dev skills (SOLID principles, IOC, 12-factor app, CI-CD, GIT)

Messaging, Microservice Architecture, Caching (Redis), Containerisation, Performance, and Load testing, REST API’s

Knowledge of HTML, JavaScript frameworks (preferably Angular 2+), Typescript

Appreciation of Python and C# .NET Core or Java

Appreciation of global data privacy requirements and cryptography

Experience in System Testing and experience of automated testing – e.g. unit tests, integration tests, mocking/stubbing

Relevant industry and other professional qualifications

Tertiary qualifications (degree level)

We are an inclusive employer and welcome applicants from all backgrounds.

We pride ourselves on our commitment to Equality and Diversity and are committed to removing barriers throughout our hiring process.

If you have any special requirements or require reasonable adjustments to help you access career opportunities at eXate, please do let us know at