Member of Technical Staff - Search Crawler Analyst

About the role

Perplexity's crawling and storage system is complex, with several stages (URL discovery, crawling, parsing, indexing) and many opportunities for improvement. This role sits at the intersection of data analysis and engineering: you will design metrics, build data pipelines and improve the quality of the search and answer systems.

Key responsibilities

• Find and diagnose quality issues in the crawling pipeline
• Train small models that optimize particular aspects of the pipeline, such as parsing quality
• Build datasets for model training, including LLM-as-a-judge labeling pipelines
• Improve page selection algorithms for indexing
• Design and analyze experiments to validate improvements

Required profile

• 4+ years of experience as a data analyst, ML engineer or in a related role
• Experience designing metrics from scratch
• Experience training ML models that shipped to production with measurable metric improvements
• Nice to have: direct experience working on web crawling or indexing pipelines

Required skills

• Strong coding skills, able to write production-grade code at the level of a mid-level backend engineer