Data Engineer

NewDay Ltd; City of London; Hybrid; GBP 60,000 - 80,000; Full time

Job summary

A leading data solutions provider in London is seeking a Data Engineer to monitor daily data pipelines and enhance data solutions. The successful candidate will have experience with ETL processes, Big Data integration technologies, and AWS services. This role offers a hybrid working environment with a focus on mentoring junior engineers and improving technical documentation.

Qualifications

• Experience with BAU processes, ETL, and data refresh.
• Experience in Big Data integration (Spark, Kafka).
• Proficient in Python.
• Hands-on experience with AWS services.

Responsibilities

• Monitor daily BAU data pipelines and ensure data solutions are refreshed.
• Work within an Agile framework on development and CI/CD.
• Create and maintain technical documentation and RCA reports.
• Implement monitoring dashboards and alert systems.

Skills

Experience with data solution BAU processes (ETL, table refresh)

Experience in Big Data data integration technologies (Spark, Kafka)

Experience in programming language such as Python

Experience using AWS (Athena, Glue, EMR, Step functions, Cloud watch)

Analytical and problem-solving skills

Experience of CI/CD

Experience in handling IT incidents

Tools

Snowflake

DBT

Grafana

Job description

Data Engineer page is loaded## Data Engineerlocations: Londontime type: Full timeposted on: Posted Todayjob requisition id: JR0197**What you will deliver*** Monitoring daily BAU data pipelines and ensure our data solution is refreshed up to date every day* Working with the team within an Agile framework using agile methodology tooling that controls our development and CI/CD release processes* Get up to speed with existing data infrastructure, tools, and processes.* Monitoring and responding to IT incidents.* Build familiarity with key pipelines/processes and support documentation and enhance documentation.* Begin proactive monitoring and suggest quick wins for stability and efficiency.* Working on incident RCA along with other engineers within the team.* Create and maintain technical documentation, RCA reports, and knowledge base articles.* Implement monitoring dashboards and alerts for key jobs and data quality checks.* Mentor junior engineers or act as a technical go-to for pipeline and cloud topics.**Your Skills and Experience****Essential*** Experience with data solution BAU processes (ETL, table refresh etc.)* Experience in Big Data data integration technologies such as Spark, Kafka* Experience in programming language such as Python.* Experience using AWS (Athena, Glue, EMR, Step functions, Cloud watch), DBT and Snowflake.* Analytical and problem-solving skills, applied to data solution* Experience of CI/CD* Experience in handling IT incidents.* Experience with IT incident management process.* Basic understanding of event streaming.* Basic knowledge on Scala(nice to have).**Desirable*** Experience of ETL technologies* Previous exposure to own data solution BAU monitoring and enhancement* Exposure to grafana#LI-Hybrid#LI-TM1**We work with Textio to make our job design and hiring inclusive.**Permanent