Databricks Senior Data Engineer | Data Pipelines & DataOps

AnaVation LLC; Greater London; On-site; GBP 70,000 - 90,000; Full time

Job summary

A prominent tech company in Greater London is seeking a Senior Data Engineer to lead the design and development of robust data pipelines. The successful candidate will integrate data from diverse sources and collaborate closely with business and analytics teams to ensure high-quality deliverables. Key skills required include strong programming in Python and SQL, experience with Databricks, and a solid understanding of DataOps practices. This role is crucial for maintaining the quality and governance of ingested data.

Qualifications

• Proven experience in building and managing large-scale data pipelines in Databricks.
• Strong programming skills in Python and SQL for data processing.
• Deep understanding of ETL/ELT frameworks and distributed data processing.
• Hands-on experience with modern DataOps practices like CI/CD.

Responsibilities

• Develop and maintain data pipelines.
• Integrate data from various sources.
• Collaborate with business and analytics teams.
• Ensure quality, reliability, security and governance of ingested data.

Skills

Data pipeline development

Python programming

SQL proficiency

ETL/ELT frameworks

DataOps practices

Troubleshooting performance issues

Version control with Git

Tools

Databricks

AWS

Azure

GCP

Job description

A prominent tech company in Greater London is seeking a Senior Data Engineer to lead the design and development of robust data pipelines. The successful candidate will integrate data from diverse sources and collaborate closely with business and analytics teams to ensure high-quality deliverables. Key skills required include strong programming in Python and SQL, experience with Databricks, and a solid understanding of DataOps practices. This role is crucial for maintaining the quality and governance of ingested data.