Member of Technical Staff - Backend Software Engineer, Search Crawler

About the role

A company is looking for an experienced Backend Software Engineer to join its Crawler team. You will design, develop and operate systems that ingest, process and manage web-scale data to support next-generation search technologies. The role requires expertise across both backend and frontend parts of the data acquisition stack.

Key responsibilities

• Lead projects that build large-scale web crawlers, ingestion pipelines and data processing systems.
• Build, maintain and optimise core backend and frontend components, including storage, retrieval and UI dashboards for data management.
• Work with the Search and Infrastructure teams to ensure reliable, high-quality ingestion and processing of billions of web pages.
• Architect and implement full-stack features and scalable distributed systems handling high-load and real-time data operations.
• Iterate quickly, experiment with new approaches and improve system performance, usability and reliability.

Required profile

• At least 5 years of software development experience.
• Strong knowledge of data structures and algorithms in Python, C++, Rust or Go.
• Experience building, deploying and optimising high-load, distributed and hardware-adjacent services.
• Deep understanding of cloud infrastructure, with hands-on experience in Kubernetes and AWS.
• Experience with large-scale web crawlers is highly desirable.