What Is PerplexityBot? The Complete Technical Reference

PerplexityBot is the crawler Perplexity uses to discover and index web pages so those pages can be retrieved and cited when Perplexity answers a question. It is one of two distinct Perplexity user agents, and confusing the two is the most common mistake site owners make when configuring access.

This is a technical reference: what each agent does, how to allow or block them correctly, how to verify a request is genuine rather than spoofed, and why the allow-or-block decision matters more for Perplexity than for most engines.

PerplexityBot vs Perplexity-User

Perplexity operates two agents with different jobs, and they should be reasoned about separately.

PerplexityBot is the indexing crawler. It browses the web in the background, following links and building an index, the same broad pattern a traditional search engine crawler follows. It is not tied to any individual user's question. If you block it, your pages are absent from the corpus Perplexity can draw on, which means you cannot be cited even when a user asks a question your page answers perfectly.

Perplexity-User is the user-triggered fetcher. When a person asks something that requires reading a specific page right now, this agent fetches that page in real time. It is acting on behalf of a human who has effectively requested your content.

The practical consequence: these two agents warrant different decisions. Blocking the indexer removes you from consideration entirely. Blocking the user-triggered fetcher breaks a live retrieval for someone who is, in effect, already trying to read your page.

Configuring access in robots.txt

Both agents are configured through standard robots.txt directives. To allow PerplexityBot explicitly:

User-agent: PerplexityBot

Allow: /

To block it:

User-agent: PerplexityBot

Disallow: /

A few things routinely go wrong here.

A blanket disallow catches everything. If your robots.txt contains User-agent: * with a broad Disallow: / and you have not added an explicit allow for the AI crawlers, you are blocking them without having decided to. This is the most common cause of accidental AI invisibility, and it is worth checking rather than assuming. The robots.txt checker will show you exactly which AI crawlers your current file permits.

Specific rules beat general ones. A specific User-agent: PerplexityBot block overrides the wildcard block for that agent, so you can allow AI crawlers while restricting others.

robots.txt is not access control. It is a request that well-behaved crawlers honour. It does not stop anything that chooses to ignore it. If you need enforcement rather than a request, that is a server-level or firewall-level decision, not a robots.txt one.

Verifying a genuine PerplexityBot request

The user-agent string is a header the client sets. Anyone can send it. Treating a user-agent string as proof of identity is a mistake, and it is regularly exploited by scrapers that want the access real crawlers get.

Verify properly using forward-confirmed reverse DNS, the same method used to verify Googlebot:

1. Take the IP address of the request.
2. Do a reverse DNS lookup on it. It should resolve to a hostname on a Perplexity-controlled domain.
3. Do a forward DNS lookup on that hostname. It should resolve back to the original IP.

If any step fails, the request is not what it claims to be. You can also check the IP against the address ranges Perplexity publishes for its crawlers, which is faster but requires keeping the ranges current.

This matters practically: if you are making decisions based on crawler traffic, whether that is serving different content, allowing higher rate limits, or measuring AI crawler activity, unverified user-agent strings will give you inflated and misleading numbers.

Why the Perplexity decision matters more than most

Most engines fall into one of two categories, and Perplexity sits firmly in the more actionable one.

Retrieval-based engines fetch live sources at answer time. Perplexity is the clearest example. What this means for you is that content it can reach can influence answers within days. You publish, it gets crawled, it becomes eligible for citation.

Training-data-based answering depends on the model's retraining cadence, which you cannot influence and which operates on a much longer timescale.

That difference makes Perplexity the fastest feedback loop available for AI visibility work. It is where you find out whether a change worked, well before slower engines reflect it. Blocking PerplexityBot forfeits that feedback loop entirely, which is why the default recommendation for most brands is to allow it.

The practical tactics for earning those citations are covered in how to get cited by Perplexity and the broader Perplexity SEO guide.

What PerplexityBot needs to see

Allowing the crawler is necessary but not sufficient. Two technical issues cause pages to be crawled and still never cited.

JavaScript-dependent content. If your page renders its substance client-side, a crawler that does not execute JavaScript sees an empty shell. Server-render anything you want cited. This is one of the most common gaps between "we are not blocking anything" and "we are still not being cited," and it is covered in detail in preparing your website for AI crawlers.

Unextractable structure. A retrieval engine needs a clean, self-contained claim it can quote. Pages that bury the answer in a long narrative are harder to cite than pages that state it plainly and then support it. Content optimization for LLMs covers the structural patterns that make extraction reliable.

Other AI crawlers worth configuring deliberately

PerplexityBot is one of several agents you should have an explicit position on rather than a default. The others most sites care about are the crawlers operated by OpenAI, Anthropic and Google for their respective assistants, each with its own documented user-agent strings and its own indexing versus user-triggered distinction.

The useful discipline is to decide once, deliberately, for each: do I want this engine able to cite me? For most brands trying to be discovered, the answer is yes across the board, and the risk is not over-permission but accidental blocking that nobody notices for months.

Once access is correct, the question becomes whether the citations are actually happening. That is measurable rather than a matter of inference: an AI visibility tracker shows which engines are naming and citing you over time, and the free AI Visibility Scorecard gives you a baseline across the major assistants in about a minute.