Staff Software Engineer (Agentic Search) - Crawler

NebiusAmsterdamOn-siteFull-time3 weeks ago

Nebius is seeking a Staff Software Engineer to build web-scale crawling and content acquisition infrastructure for an agent-native search platform. The role requires 8+ years of backend or distributed systems experience, strong Go or C++ expertise, and deep knowledge of large-scale distributed systems and web protocols. This is an on-site full-time position based in Amsterdam.

What you'll do

Design, implement, and operate web-scale crawling systems for acquiring content from the internet
Build ingestion workflows for internal and external data sources, including crawlers, structured feeds, and partner integrations
Develop crawl scheduling, prioritisation, recrawl policies, and freshness strategies
Build systems for URL discovery, deduplication, content extraction, and crawl orchestration
Ensure reliable operation of crawling infrastructure under high-throughput conditions
Define observability and quality metrics for crawl coverage, freshness, throughput, and content quality
Monitor resource usage, bandwidth consumption, and infrastructure cost
Collaborate with indexing and ML teams to ensure acquired content meets retrieval and ranking requirements
Enable safe experimentation with crawling strategies and content acquisition policies

What you need

Must-have

GoC++Distributed SystemsHTTPDNSTLSWeb CrawlingScraping

Nice-to-have

Apache KafkaApache PulsarNATSRabbitMQApache SparkApache FlinkApache BeamMapReduce

Soft skills

Collaboration

Software EngineeringDistributed SystemsWeb CrawlingSearch EngineAICloud InfrastructureGoC++Staff LevelAmsterdam

/What you'll do

/What you need

Must-have

Nice-to-have

Soft skills

What you'll do

What you need