Lead Data Engineer (Azure/Databricks)

IntelliAM AI Ltd; Sheffield; Hybrid; GBP 80,000 - 100,000; Full time

Job summary

An intelligent asset management company in Sheffield is seeking a Data Engineering Technical Lead to own the data engineering domain, architect new data systems, and mentor a talented team. This full-time role requires expertise in building data platforms on Azure, particularly in Databricks. The ideal candidate should be capable of balancing academic perfection with business value and have experience in data pipeline architecture. Benefits include 5 weeks of paid leave, tax-efficient stock options, and opportunities for professional development.

Benefits

5 weeks paid annual leave

Tax efficient stock options

Workplace pension with employer contributions

Salary Sacrifice EV Scheme

Training and professional development opportunities

Qualifications

• Proven experience building data platforms on Azure.
• Strong opinions on schema design and data standardization.
• Ability to explain architectural decisions to business stakeholders.

Responsibilities

• Own the redesign and rebuild of the data transformation layer in Databricks.
• Support the implementation of the 'Unified Name Space' across the data estate.
• Establish a stable data serving layer for Grafana and Analytics.

Skills

Advanced Data Engineering

Data Modelling Proficiency

Code Quality

Communication

Job description

Location: Sheffield (2 daysperweek/ Hybrid)

Reporting to: VP of Engineering

The Team: 2 Data Scientists, 1 Data Engineer

IntelliAM isan intelligent asset management companyspecialisingin industrial IoT data. We ingest high-frequency factory and sensor data, apply machine learning techniques, and present high value actionable insights to analysts and customers.

We are building a cutting-edge UNS aligned Industrial IoT data platform with a highly scalable architecture to handle our next phase of growth. We require adelivery-focused technical leaderto own the data engineering domain, re-architect and build our pipelines from the ground up, and mentor a small but talented team.

The Mission

You will not be maintaining a legacy system; you will be architecting the new one.You will take ownership of the data flows within the platform, designing a robust architecture that supports both real-time operational views and deep historical analysis.

The Tech Stack

Primary Objectives (First 6–12 Months)

• Re-platforming: Own the redesign and rebuild of the data transformation layer (post-event bus) in Databricks. Move from ad-hoc scripts to software engineering standards (CI/CD, testing, modular code).
• Modelling Implementation: Support the implementation of the “Unified Name Space” (UNS) across the data estate, to include schema and pathstandardisation; machine hierarchies and semantic relationships; and methodologies for validation and processing of data against contracts.
• Enablement: Establisha stable data serving layer for Grafana and Analytics (via Databricks/MLflow), unblocking the Data Science team.
• Mentorship: Upskill the existing team in good practices (dbt, git workflows, SQLoptimisation, data modelling).

Experience & Background

• Pragmatic Architect: Capable of making trade-offs between “academic perfection” and “business value”. They must align architecture to the 80/20 rule.
• Technical Authority: Comfortablediscussing and constructively challengingarchitectural decisions with the Head of Software and VP of Engineering. Must be able to justify changes to the core architecture and challenge existing thinking.
• Mentorvs.Manager: Willing to sit and pair-program with junior engineers. They should measure their success by the team’s outcomes, not simply their own. A full “Team Manager” is notrequired;thisisn’ta pastoral-focused hire.
• Business Translator: Able to explain to non-technical stakeholders why changes arerequired, and what their impacts may be.Similarly,the abilityto provide a bridge between technical approaches and limitations, and business requirements.
• Industrial / IoT Context: Highly desirable. Youshould be comfortable withthe chaotic nature of sensor data, time-series continuity, and the physical reality of the machines we are modelling.
• Alternatively, experience in high-volume data systems that are subject to similar challenges (duplicated or missing data, spiky load, variable schemas,etc.)would bebeneficial.

Skills

• Advanced Data Engineering: Proven experience building data platformsonAzure. You know Databricks inside out – not just how to write a notebook, but how to architect a Lakehouse, manage clusters, andoptimisecosts.
• Data Modelling Proficiency: You understand that data engineering is more than moving JSON blobs. You have strong opinions on schema design, datastandardisation, and ideallyhave workedwith graph data or hierarchical models.
• Code Quality: You treat data pipelines as software. You use git, CI/CD, and automated testing. You can teach these practices to a junior engineer.
• Communication: You can explain to a business stakeholder why architectural decisionsmatter fortheir bottom line, and you can debate architectural trade-offs with Software Engineers.

Why Join?

• Autonomy: You have the support of leadership to make architectural decisions. Youaren’tfighting for permission to fix the tech debt; you are being hired specifically to fix it.
• Impact: Your work directly enables the Data Science team to stop cleaning data and start building better predictive models.
• Modern Stack: We are moving toward a cutting-edgeIIoTstack (UNS + Digital Twins), offering you the chance to work on industry-leading patterns.

Whatyou’llget

• 5 weeks paid annual leave, plus UK bank holidays
• Tax efficient stock options (subject to scheme rules and eligibility)
• Workplacepensionwith employer contributions, in line with UK auto-enrolment legislation(subject toeligibility andscheme terms)
• Salary Sacrifice EV Scheme (subject to eligibility)
• Training and professional development opportunities

This is a full-time role.Due to the collaborative and mentoring nature of this role, regular in-person attendance is required. The team collaborates in person from our Sheffield officetwiceweeklyfor planning,workshopsand team development,providingvaluable opportunities for mentoring,feedbackand connection. Werecognisethe importance of flexibility and are open to discussing reasonable flexible working arrangements in line with business needs.

Join the Team!

If your backgrounddoesn’texactly align with the job description,but youpossesstransferable skills or experience that could be a strong match, we encourage you to highlight this in a cover letter.We’recommitted to personal growth as our company evolves, so ifyou’reexcited to be part of that journey,we’dlove to hear from you.

We are an equal opportunities employer and are committed to creating an inclusive workplace.We welcome applications from all suitably qualified candidates regardless of age, disability, gender reassignment, marriage or civil partnership, pregnancy or maternity, race, religion or belief, sex, or sexual orientation.

Applicants must have the right to work in the UK.

We will process your personal data as part of the recruitment processin accordance withapplicable data protection laws. Please refer to our Recruitment Privacy Notice for further information on how we collect, use, and store candidate information.