
Data Ingestion Engineer
AI
A contract Data Ingestion Engineer is needed for a leading AI and robotics business in the automotive space, based in London. The role involves building and running batch and streaming pipelines that bring large, complex datasets into the data platform for analytics, data science and machine learning teams. Strong Python, PySpark and SQL skills with large-scale data experience are essential.
What you'll do
- Design, build and maintain batch and streaming ingestion pipelines using Python, PySpark and SQL
- Ingest structured, semi-structured and sensor or time-series data from files, APIs, databases and device feeds
- Process and transform datasets from gigabytes to multiple terabytes efficiently and cost-effectively
What you need
Must-have
PythonPySparkSQLData modellingData qualitySchema evolutionAWSAzure
Nice-to-have
Automotive dataIMU dataTime-series dataComputer visionRobotics datasetsLidarSensor fusionFinancial services
Soft skills
Understanding of data domainsCollaboration with data scientists and analystsDocumentationTroubleshooting
Data EngineeringData IngestionPySparkContractAutomotiveAIRoboticsLondonInside IR35