Web Access Engineer (Crawler) - Member of Technical Staff

About the role

Perplexity is hiring a Web Access Engineer for its Crawler team to build reliable fetching and rendering systems for AI-powered search. The engineer works on publicly available web content within site permissions and approved access policies, developing the infrastructure that feeds content to downstream indexing and retrieval systems. The focus is on diagnosing network errors, page-loading issues and incomplete rendering, and turning findings into production fixes.

Key responsibilities

• Own fetching and rendering improvements end to end, from investigation through implementation and production rollout, partnering with crawler engineers and SREs.
• Investigate fetch and rendering failures across DNS, TLS, HTTP, caching, redirects and client-side rendering, and classify root causes.
• Improve page loading, completion detection and content capture so that relevant content is preserved and dynamic pages are handled reliably.
• Ensure the crawler respects site-imposed access restrictions and rate limits, identifies itself clearly and retries only transient technical errors.
• Build tools to reproduce failures, compare crawler output against expected page content and turn findings into regression tests.
• Define metrics for fetching reliability, content completeness and error classification, and use them to prioritize improvements.

Required profile

• 5+ years of software engineering experience in web infrastructure, browsers, CDNs or crawling.
• Strong programming skills in at least one of Go, Python, Rust or C++.
• Ability to work independently when the path forward is not fully defined.
• Nice to have: experience with CDN, caching or web performance systems; experience operating crawlers, headless browsers or distributed content-acquisition systems at scale; contributions to a browser engine or JavaScript runtime.

Required skills

• HTTP/1.1, HTTP/2 and HTTP/3, including status semantics, caching, conditional requests, content encoding and redirects; strong knowledge of TLS, DNS, cookies and sessions.
• CDN caching and content delivery, HTTP error handling, retry backoff and server-defined rate limits.
• Page lifecycle, network idle, service workers, client-side rendering and shadow DOM; debugging JavaScript.
• Fluency with Chrome DevTools, CDP, HAR files, curl and packet captures.