Cognitive Group | Part of the Focus Cloud Group logo

Data Ingestion Engineer

AI
Cognitive Group | Part of the Focus Cloud GroupGreater LondonHybridContract2 weeks ago

A contract Data Ingestion Engineer is needed for a leading AI and robotics business in the automotive space, based in London. The role involves building and running batch and streaming pipelines that bring large, complex datasets into the data platform for analytics, data science and machine learning teams. Strong Python, PySpark and SQL skills with large-scale data experience are essential.

What you'll do

  • Design, build and maintain batch and streaming ingestion pipelines using Python, PySpark and SQL
  • Ingest structured, semi-structured and sensor or time-series data from files, APIs, databases and device feeds
  • Process and transform datasets from gigabytes to multiple terabytes efficiently and cost-effectively

What you need

Must-have

PythonPySparkSQLData modellingData qualitySchema evolutionAWSAzure

Nice-to-have

Automotive dataIMU dataTime-series dataComputer visionRobotics datasetsLidarSensor fusionFinancial services

Soft skills

Understanding of data domainsCollaboration with data scientists and analystsDocumentationTroubleshooting
Data EngineeringData IngestionPySparkContractAutomotiveAIRoboticsLondonInside IR35