
Senior Data Engineer
AI
Senior Data Engineer required for a permanent hybrid role in London within the insurance sector. The position demands 10+ years of data engineering experience, strong Azure Databricks, PySpark and SQL skills, and proven delivery of enterprise data platforms and business-facing data products.
What you'll do
- Design, develop, and maintain data products within an Azure Databricks lakehouse environment
- Build and optimise Silver and Gold layer transformations using Medallion Architecture principles
- Contribute to data modelling and architecture discussions
What you need
Must-have
SQLPySparkAzure DatabricksDelta LakeData lakehouse architecturesMedallion ArchitectureData modellingPower BI
Nice-to-have
Databricks Unity CatalogAILLMsRetrieval-Augmented Generation (RAG)Databricks GenieDimensional modellingStar SchemasSlowly Changing Dimensions (SCDs)
Soft skills
Stakeholder managementCommunicationAbility to explain technical concepts to non-technical audiences
Data EngineeringInsuranceAzure DatabricksPySparkPower BISeniorHybridLondon