Search Architecture

How to Optimize Your Website for Claude: The Three Ways Claude Finds You

How to Optimize Your Website for Claude: The Three Ways Claude Finds You

In This Article

    Someone asks Claude to recommend a vendor in your category, Claude answers without searching anything, and you are either in its memory or you are not.

    That is the first thing to understand, and most guides skip it. Claude does not search the web for every question, so being found by Claude means being present in three different places: its training memory, its search results and the pages that users point it to. Each one has a different lever, a different speed and a different owner inside your company. Optimizing a website for Claude means working out which of the three you are failing in, and not treating Claude as a search engine with a different logo.

    What is in This Series

    This article is part of cluster How to Measure Your Presence in AI Search Engines, the series hub containing the: How to Optimize Your Website for ChatGPT, How to Measure Your Presence in ChatGPT, How to Optimize Your Website for Perplexity, How to Measure Your Presence in Perplexit, How to Optimize Your Website for Claude, How to Measure Your Presence in Claude.

    Also read AI Search Visibility Share of Voice, for turning mentions into a competitive number

    What “Optimizing for Claude” Means

    Optimizing for Claude is the work of making your brand known to the model before anyone asks (training memory), retrievable when Claude does search (live search), and readable when a user hands Claude one of your URLs (fetch). Claude is Anthropic’s AI assistant, available in the Claude app and through an API that developers build on, and that second fact matters later in this article.

    I use “GEO” (Generative Engine Optimization, the practice of making content retrievable and citable by AI answer engines) as the wider frame. Claude is the engine where GEO is least like classic SEO, and the reason is the first sentence of the next section.

    Three Ways Claude Can Know About You

    Anthropic’s documentation for its web search tool says Claude decides whether to search based on the prompt. It searches for information that is current, changing or outside its training data, and it answers directly for stable knowledge. That single design choice produces the three-way split below. The table is my working model, built from Anthropic’s documented behavior. The “my read” column is my judgment.

    PathWhat triggers itYour leverSpeedMy read
    Training memoryClaude answers directly, no searchBeing consistently present on the open web before training, and ClaudeBot accessSlow. Tied to model versionsHardest to influence, easy to ignore
    Live searchClaude judges the question to need current informationCrawl access, indexable pages, quotable answersFasterWhere page-level work pays off
    User-directed fetchA user gives Claude your URLPages that render without JavaScript and answer cleanlyImmediateCheap to fix, often forgotten

    If Claude answers from memory, no page you publish today can change that answer today. That is uncomfortable for anyone who bought an optimization budget expecting a dial. For stable category knowledge (“what are the main approaches to X”), the memory path is the likely one, and a competitor that was widely and consistently described before the last training run has a head start that a content calendar cannot erase. How training data gets assembled and how long changes take to appear: Not enough available data. I will not guess a timeline.

    Step 1: Control the Three Crawlers (This Part Is Documented)

    Anthropic’s crawler documentation names three bots, and the split is the most useful thing on that page.

    User agentAnthropic’s stated purposeWhat blocking it does
    ClaudeBotCollects web content that may be used to train Anthropic’s modelsSignals that future content should be excluded from training datasets
    Claude-UserAccesses sites when a user’s question calls for itStops retrieval for user queries, which may reduce visibility in user-directed search
    Claude-SearchBotBrowses the web to improve the relevance and accuracy of search resultsStops indexing for search, which may reduce visibility and accuracy in search results

    Anthropic says its bots honor robots.txt, and that crawl-delay is supported as a non-standard extension. It also says it does not bypass CAPTCHAs or other anti-circumvention measures. One practical warning on the same page: blocking by IP address may not work reliably as an opt-out, because it can stop the bots from reading your robots.txt in the first place. Anthropic publishes its crawler IP list for verification.

    There is also a newer item. Per the same page, as of October 30, 2026, Anthropic recognizes a machine-readable “no AI training” signal based on an IETF draft, and a robots.txt rule that names ClaudeBot takes precedence over that general signal for the affected paths. I have not tested how this works in practice, so check the page before relying on it.

    An example of a split, to adapt and not paste:

    User-agent: Claude-SearchBot
    Allow: /
    
    User-agent: Claude-User
    Allow: /
    
    User-agent: ClaudeBot
    Disallow: /
    

    Visibility and training permission are separate decisions, and they have different owners. Allowing Claude-SearchBot and Claude-User is a visibility call. Allowing ClaudeBot is an intellectual property call that should go through legal, and I covered that side in Everything You Need to Know About AI Intellectual Property. But be honest about the tradeoff. If you block ClaudeBot, you are choosing to stay out of future training data, which is the memory path above. That may be the right call. Just make it knowingly, because the visibility cost is real and nobody will send you an invoice for it.

    A separate documented item: Anthropic-hosted browsers, which act on a user’s behalf, identify with a standard browser user agent plus the token Claude-Agent, and Anthropic lists their IP addresses so you can verify requests. Anyone can send the same user agent, so match the IP list, not just the string. Anthropic says crawler preferences are set independently from this traffic. For a structured check of your own crawl and indexation setup, start with my Indexation and Crawl Diagnostic.

    Step 2: Write for a Short Citation and a Plain Fetch

    Two documented details change how I would write for Claude.

    First, citations. In the web search tool, citations are always on, and each one carries the URL, the title and a cited_text field of up to 150 characters. My inference, not a documented ranking rule: the unit a user sees quoted is short. So every important claim on a page should survive as one self-contained sentence of that length, with its subject named, not “this approach improves it”. If a claim only makes sense three paragraphs later, it will not travel.

    Second, fetch. Anthropic’s web fetch documentation says Claude can only fetch URLs that already appeared in the conversation, that results are cached and may not reflect the latest version of a page, and that the tool does not render JavaScript-dependent pages. So the practical rules are plain:

    • Keep key facts in the HTML text, not behind scripts or tabs that need JavaScript to appear.
    • Expect stale copies. A fix to a page may not be what Claude reads next hour.
    • Make the URL worth pasting. Fetch only happens when someone supplies the link, so this is about pages your sales team, your documentation and your partners already share.

    The search tool also returns a page_age field (when the site was last updated). Whether it influences which results are chosen is not documented. But a visible, honest update date costs nothing. I would never fake one, because a faked date helps nobody and does not survive a closer look.

    Want to see how AI systems describe your brand right now, before you touch a page? Run the AI Visibility Inspector (my diagnostic tool for checking how AI systems see a brand). A baseline first, or you will credit your edits for noise.

    The writing side of all this is in How to Write a Blog Post for AI SEO and Citations, the structure side in AI Content Structure for Enterprise Visibility and the engineering side in AI Citation Engineering.

    Step 3: Work on the Memory Path Honestly

    This is the slow path, and what I can say about it is limited. Anthropic does not document what makes a brand well represented in training data, so I will not claim a formula. What I can say is what follows from how the three paths work.

    • Say the same true thing everywhere. One name, one description, one category label across your site, profiles and the sources that describe you. An engine that has seen five versions of you will blur them. I wrote about this failure in Entity Ambiguity Resolution and about the framework behind it in the Entity Engineering Framework.
    • Be described by sources you do not control. Independent mentions are copied, quoted and summarized in places a model may have seen. That is the logic behind Authority Engineering, and I treat it as a hypothesis, not a measured effect on Claude.
    • Do not leave ClaudeBot blocked by accident. A blanket “block all AI bots” rule, copied from a template, quietly sets your memory-path strategy for you.

    Which Claude? The Search Backend Depends on the Deployment

    Anthropic’s web search tool documentation does not name the search provider and does not describe how results are ranked or selected. That is a documented gap, not an oversight on my part. And “Claude” is not one deployment. Anthropic’s Claude Desktop documentation for third-party deployments says search runs through the inference provider you configure, or through a built-in option where the organization picks a backend: Brave, Tavily, Exa or its own server. That documentation covers enterprise setups, not the consumer app, so it does not tell us what the consumer Claude app uses. It does show that there is no single “Claude index” to optimize against.

    Treat any article that tells you exactly which search engine feeds Claude, and how to rank in it, as a guess with a confident tone. Including mine, the day I stop saying “not documented”. If you want to confirm what your buyers’ Claude actually retrieves, you test it, which is what the measurement article covers.

    What Is Documented vs What Is Guesswork

    TacticEvidence levelMy read
    Allow Claude-SearchBot and Claude-UserDocumented by AnthropicDo first. Binary
    Decide ClaudeBot separatelyDocumented by AnthropicA legal call, made knowingly
    Do not block by IPDocumented warningUse robots.txt
    Keep key facts in HTML, not JavaScriptDocumented limit of the fetch toolCheap. Do it
    Short, self-contained claimsInference from the 150-character cited text fieldCheap. Do it
    Visible update datepage_age returned, influence not documentedDo it, honestly
    Consistent entity descriptionMy reasoning, not documentedDo it, slowly
    Ranking in “Claude’s search engine”Provider not named in the documentation I readNot enough available data

    What This Is NOT

    This is not a ranking guide, because Anthropic documents no ranking and names no search provider for its search tool. It is not a promise that publishing more will change what Claude says, since memory-based answers do not move with your latest page. It does not replace SEO, though I cannot tell you how much of Claude’s search depends on classic search indexes (Not enough available data). And it is not finished when you publish. You change something, then you check whether the answers changed, over repeated runs, which is exactly what How to Measure Your Presence in Claude covers.

    Estimated Gain After Implementation

    Here is what I can say honestly. Fixing a crawler block is the only change with a clear before and after, because a blocked Claude-SearchBot or Claude-User means you are asking Claude to find you with one hand tied. Writing and fetch fixes are quick to make and quick to test. The memory path is the long game, and anyone promising a percentage on it is guessing.

    The Uncomfortable Part

    The temptation is to spend the whole Claude budget on the path you can see, the pages, and neglect the one that may decide many answers, the memory. I cannot prove the proportion, and I would be inventing it if I tried. But the logic is documented: Claude answers stable questions without searching. So if your category is described in stable terms, your pages may never be consulted at all. The uncomfortable work is to be consistently and accurately described everywhere, for a long time, before anyone asks.

    If you want to know which of the three paths is failing for you (training memory, live search or fetch), that is the kind of work I do in Enterprise Search Advisory, and you can talk to me here.

    FAQ

    No. According to Anthropic’s documentation, Claude decides per prompt. It searches for current, changing or out-of-training information and answers directly for stable knowledge. That is why I split Claude’s visibility into memory, search and fetch.

    Anthropic names ClaudeBot (training), Claude-User (user-initiated retrieval) and Claude-SearchBot (search indexing). Allowing the last two is a visibility decision. ClaudeBot is a separate call that I would route through legal.

    Anthropic says blocking ClaudeBot signals that future content should be excluded from training datasets. In my reading, that reduces the memory path over time. Search and fetch are controlled by the other two bots.

    Anthropic’s documentation warns that IP blocking may not work reliably as an opt-out, because it can stop the bots from reading robots.txt. Use robots.txt, and use the published IP list only to verify that a request really comes from Anthropic.

    Per Anthropic’s web fetch documentation, no. The tool does not render JavaScript-dependent pages, and results are cached, so a fix may take a while to show.

    In the web search tool, the cited text field holds up to 150 characters. I conclude from that that important claims should stand alone as short sentences. That conclusion is my inference, not a ranking rule.

    Anthropic’s web search tool documentation does not name the provider. In third-party enterprise deployments, the organization can choose the backend. Anything more specific for the consumer app: Not enough available data.

    No. Not enough available data exists for an honest range. Promise a baseline, a fixed set of changes and repeated measurement.

    Measure repeated runs of a fixed prompt set before and after, and separate memory-based answers from search-based ones. I explain how in How to Measure Your Presence in Claude.

    What To Do Next?

    Start with the AI Visibility Inspector for a first read. When you need Claude tracked over time against named competitors, NovaX (my AI Visibility Intelligence Platform) covers it. And if you want the memory, search and fetch gaps scoped for your market, contact me.

    This article was researched and drafted with the assistance of AI tools and reviewed and edited by author prior to publication.

    Share in 𝕏 ✉
    Ivica Srncevic
    Author

    Ivica Srncevic is an independent AI strategist, researcher, framework author, and international speaker focused on AI sovereignty, knowledge infrastructure, governance, AI retrieval, and the evolving relationship between organizations and intelligent systems. His work examines what AI systems can see, retrieve, infer, and reconstruct from organizational information, and how organizations can retain greater control over their data, knowledge, and AI infrastructure. In 2026, he spoke at the AIFOD Geneva Summit at UN Geneva on what nations must own and what they can safely share, with a particular focus on data ownership, control, and sovereign AI infrastructure.

    Articles: 259