Data Engineer

Thoughtswinsystems; Greater London; On-site; GBP 70,000 - 90,000; Full time

Job summary

A leading technology firm in the UK is seeking a Senior Data Engineer to create and manage datasets that inform strategic decision-making. You will design scalable data pipelines, collaborate with product managers and engineers, and ensure data integrity across complex environments. This role is ideal for those with extensive experience in data architecture and optimization technologies. Join a dynamic team focused on innovative solutions.

Qualifications

• 5+ years of work experience with ETL, Data Modeling, and Data Architecture.
• Experience with the AWS or MS Azure technology stacks.
• Experience operating very large data warehouses or data lakes.

Responsibilities

• Interface with project managers to understand requirements.
• Design and develop scalable data pipelines.
• Adopt best practices in data processing and analysis.

Skills

ETL

Data Modeling

Data Architecture

SQL

AWS

Azure

Apache Spark

Data processing

Education

Bachelor’s degree in Computer Science or related field

Tools

Redshift

RDS

S3

EMR

Azure BLOB storage

Azure Synapse

Job description

Looking for Senior Data Engineers who are experienced, self-driven, analytical, and strategic.In this role, you will work with clients across large and complex data lake/warehouse environments.You are passionate about working with disparate datasets in order to bring data together to answerbusiness questions. You should have deep expertise in the creation and management of datasets andthe proven ability to translate the data into meaningful insights through collaboration with productmanagers, data engineers, business intelligence developers, operations managers and client leaders.In this role, your skills and expertise will be integral in strategic decision-making, answering complexquestions by taking ownership of the end-to-end development of data engineering solutions.

Responsibilities

• Interface with project managers, business customers, data architects and data modelers to understand requirements and implement solutions.
• Design, develop, and operate highly-scalable, high-performance, low-cost, and accurate data pipelines in distributed data processing platforms providing ad hoc access to large datasets and computing power.
• Explore and learn the latest big data technologies. Evaluate and make decisions around the use of new or existing software products to design data architectures.
• Recognize and adopt best practices in data processing, reporting, and analysis: data integrity, test design, analysis, validation, and documentation.

Qualifications

• Bachelor’s degree in Computer Science or related technical field, or equivalent work experience.
• 5+ years of work experience with ETL, Data Modeling, and Data Architecture.
• 5+ years of experience with SQL and large data sets, data modeling, ETL development, and data warehousing, or similar skills.
• 2+ years experience with the AWS or MS Azure technology stacks. AWS should include Redshift, RDS, S3, EMR or similar solutions built around Hive/Spark etc. MS Azure should include ADF, Azure BLOB storage, Azure Synapse etc.
• Excellent in ETL optimization, designing, coding, and tuning big data processes using Apache Spark or similar technologies.
• Experience operating very large data warehouses or data lakes.
• Experience with building data pipelines and applications to stream and process datasets at low latencies.
• Demonstrate efficiency in handling data – tracking data lineage, ensuring data quality, and improving discoverability of data.
• Knowledge of distributed systems and data architecture – design and implement batch and stream data processing pipelines, knows how to optimize the distribution, partitioning, and MPP of high-level data structures.
• Experience with full software development life cycle, including coding standards, code reviews, source control management, build processes, and testing.

Join Our Team

We are looking for passionate and innovative individuals ready to make a meaningful impact. Our collaborative and inclusive work environment values your ideas and fosters professional growth. Be part of a dynamic team that celebrates success and strives for excellence in every endeavor. Shape the future with us – explore our diverse range of roles and embark on a rewarding journey.

Name

Job Post

Phone

Email

Message

Upload CV

Ready to Empower Your Data to Decision Journey

Let us know if you have any queries for us or want to learn more about our services. Our team will be happy to assist and will respond at our soonest opportunity. Please complete this form and include your questions in the message box.

Name

Email

Company Name

Message

Explore how ThoughtsWin can assist you in maximizing the value from your data. Schedule a consultation with a specialist today.

ThoughtsWin

ThoughtsWin leads the way in transformative technology solutions. Our proficiency in Data & Cloud Engineering, AI & Analytics, Modernization & Migration to Cloud, and Strategy & Governance powers business innovation.

• British Columbia: Unit 204, 9900 King George Blvd, Surrey, BC V3T 0K9
• USA
• Texas: Suite 220, 8990 Kirby Drive, Houston 77054
• India
• Rajasthan: Plot No. B-2/23, Sector 2, Chitrakoot Scheme, Gandhi Path, Vaishali Nagar, Jaipur