AI crawlers read internal links as a discovery path and a relationship signal. This guide covers raw-HTML crawlability, anchor text, cluster wiring, Shopify product and collection linking, schema, and a six-stage implementation order.
49 min read
Firon Marketing is a Generative Engine Optimization consultancy that engineers how AI assistants read, interpret and recommend commerce brands. This article covers internal linking as a technical component of our Identity Architecture and content architecture services, and is written for technical marketers, developers and heads of growth at DTC, Shopify Plus and subscription brands. Internal linking in the era of generative retrieval is not a link equity exercise. It is a graph construction problem. The links between your pages are the primary mechanism by which an AI crawler infers that a set of documents belongs together, determines which document is canonical for a subject, and locates the adjacent context it needs to answer a compound question. Sites that link well are traversable. Sites that link badly are read as isolated pages with no aggregate authority.
Traditional internal linking theory assumed a crawler that discovered URLs and distributed a ranking signal along the graph. Links were valued for how much authority they passed and where they concentrated it. That model still operates inside classical search, but it describes only a fraction of what internal links do for generative systems.
An AI retrieval pipeline uses internal links in three distinct ways, and each imposes a different design requirement.
First, discovery and freshness. Crawlers operated by AI search providers traverse the link graph to find and revisit content, and most of them read only the HTML your server returns. Vercel and MERJ's analysis of more than 500 million GPTBot fetches found no evidence of JavaScript execution, and the same held for ClaudeBot and PerplexityBot; Googlebot, which renders pages, is the notable exception. Navigation built by a JavaScript framework, faceted filters that assemble URLs client-side and related-post modules injected after page load are therefore discovered inconsistently or not at all. If a link is absent from the server-delivered HTML, treat it as absent.
Crawl accessibility is broken more often than teams assume. Cloudflare's July 2025 analysis of crawler behavior across its network found that only about 37 percent of the top 10,000 domains it serves had a robots.txt file at all, which means most sites are not expressing crawl preferences deliberately in either direction. The same analysis put June 2025 crawl-to-referral ratios at roughly 1,700 to 1 for OpenAI and 73,000 to 1 for Anthropic. These systems read far more than they send back, so the value to a brand lies in being read accurately rather than in referral traffic, and a page the crawler cannot reach forfeits that value entirely.
Crawler access can also change without anyone on your team touching it. On September 15, 2026, Cloudflare replaced its single Block AI Bots switch with separate Search, Training and Agent controls and made blocking agents on ad-carrying pages its recommended setting for ad-supported sites, so review all three settings deliberately rather than assuming last year's configuration still applies.
Google spells out its own dependency. Its May 2026 guidance on optimizing for generative AI features says a page must be indexed and eligible to appear with a snippet before it can surface in AI Overviews or AI Mode, adds that the site must be included in generative AI features through Search Console, and tells owners to keep content crawlable because those features draw on publicly accessible pages. Internal links are how a crawler reaches a page in the first place. Everything downstream of discovery, including every schema property and every carefully written passage, is contingent on that first step succeeding.
Second, relationship inference. When a model needs to determine whether two documents represent one body of expertise, the presence and character of the links between them is direct evidence. Anchor text carries the semantic payload. A link labeled "read more" asserts nothing. A link labeled "how to build topical authority that AI models recognize" asserts the destination's subject and the relationship between the two pages simultaneously.
Third, context expansion during agentic retrieval. Agentic systems increasingly follow links from a retrieved page to gather supporting context before answering a compound question. Google's guidance notes that browser agents gather what they need by inspecting the DOM and the accessibility tree, the layer in which a descriptive anchor tells the agent where a link leads and a generic one tells it nothing. When an assistant is asked to compare two approaches and your page addresses one, a well-labeled link to your treatment of the other keeps the entire answer sourced from your domain. A missing link hands the second half of the answer to a competitor.
Internal linking failures are invisible from the front end and obvious to a crawler, and they surface as gaps in what AI engines say about you. Before a 30-minute walkthrough call, Firon builds a free AI Perception Report: ChatGPT, Claude, Gemini and Perplexity tested on your brand from clean, logged-out sessions with no account history, an AI leaderboard of the brands recommended in your category, a scored read of what each model gets right or wrong about you, and the structural reason for the gap, whether schema, entity signals, crawlability or content structure. You keep the report and its three prioritized fixes whether or not you become a client. Find out which structural gaps are costing you AI recommendations.
Anchor text has more influence on how a model reads a link than any other variable, and it is the one most commonly wasted. Three rules govern it.
Anchor text must name the destination's subject in natural language. Not the keyword alone, and not a fragment. "Internal linking" is a keyword. "Internal linking strategy for LLM-readable sites" is a subject. The second form gives a model both the topic and the scope of the destination document, which is what it needs to decide whether following the link is worthwhile.
Anchor text must be varied but consistent in meaning. Repeating one exact-match phrase across four hundred links reads as manipulation to classical systems and adds no information to semantic ones, because identical strings produce identical embeddings. Varying the surface form while holding the meaning constant, so that "wiring internal links for AI crawlers" and "internal linking strategy for LLM-readable sites" both point to the same node, strengthens the semantic association rather than diluting it.
Anchor text must sit inside prose that supports it. A link surrounded by a sentence explaining why the destination is relevant provides the surrounding context that gets embedded alongside the anchor. Links dropped into a bare list at the end of an article are stripped by many extraction pipelines along with other boilerplate, and even when preserved they carry no contextual signal.
The failure mode worth naming is the navigation-only link. Sites that link between related articles exclusively through sidebar widgets, related-post carousels and footer blocks are relying on the parts of the page most likely to be removed during content extraction, and where those modules are injected by client-side script, on parts most AI crawlers never see at all. Links in body prose survive extraction; links in template chrome mostly do not.
Placement determines survival. Content extraction routines that feed retrieval systems typically discard headers, footers, sidebars, cookie banners, share widgets and anything else that repeats across templates. What remains is the main content block. Every internal link that matters to your topical graph therefore belongs inside that block.
Within the main content, position matters most for the third-party retrieval systems that split pages into passages before ranking them. Google says its own systems do not need content broken into small pieces and can surface the relevant part of a long page, so article length should follow the subject, and the effort belongs in deciding where the most important link sits. A link in the first third of the article, embedded in the passage that establishes the problem, sits in the passage most likely to be retrieved for a definitional prompt. A link at the end of the final section sits in a passage that is retrieved less often. Neither is wrong, but the first link in a document should point at the node you most want associated with the page's subject.
Link density needs a ceiling. A page with forty body links dilutes every one of them and produces passages in which the link text overwhelms the substantive content. Firon's working standard for a cluster article is between three and eight body links: one upward link to the canonical definition node, two to four lateral links to sibling articles, and one or two links to service or evidence pages where they earn their place. That is enough to establish the graph without turning the prose into a directory.
The wiring extends the node structure Firon uses when structuring a topic cluster so LLMs read it as one authoritative entity, and three link classes do the structural work.
Upward links run from every resolution article to the canonical definition node, using anchor text containing the canonical subject term. These establish which page is the authority for the subject and consolidate the definitional signal on one URL rather than spreading it across a dozen partial definitions.
Lateral links run between sibling resolution articles, using anchor text that states the sibling's specific question. These are the links that create the co-occurrence pattern from which a model infers that the documents constitute a coherent body of work. A cluster in which every article links only up and never sideways reads as a hub and spoke, not a subject.
Downward links run from the definition node to each resolution article, placed in the body prose of the section that summarizes that subtopic. The long-form pillar page usually breaks here, and the 5,000-word pillar engineered for a ranking algorithm is the clearest example: it delegates to its cluster through a footer index rather than through prose, and the index is stripped before the model ever sees it.
Orphan detection should be a standing check rather than an annual audit. Any page not reachable from server-delivered HTML within three clicks of the homepage is a candidate for either integration into a cluster or consolidation into a stronger sibling. Retaining an orphan on the theory that it might rank is an expensive habit in an environment where authority is aggregated across a graph.
Commerce sites run a second graph beside the editorial one, and it is the graph AI shopping answers draw on. When an assistant is asked which product suits a specific need, the pages that answer are product and collection pages, supported by buying guides that explain the category. On most Shopify Plus stores these three page types are poorly connected: guides point at products through a single button, collections reach products only through the product grid, and products link sideways only through a recommendations block.
That recommendations block is the weakest link in the store. Shopify's theme documentation states that product recommendations load asynchronously with JavaScript, because the recommendations object is empty when the section first renders. For GPTBot, ClaudeBot and PerplexityBot, which read only the initial HTML, the "You might also like" section on a product page usually contains nothing. Carry the relationships you care about in server-rendered Liquid instead: a product reference list metafield for complementary and alternative products, output in the product template with descriptive anchors, gives every crawler the same lateral links a shopper sees.
The three link classes translate directly. The collection page is the canonical node for a category, so buying guides and product pages link up to it with anchors that name the category and use case rather than the collection's merchandising title: "waterproof trail running shoes for wide feet" tells a model far more than "Shop the Trail Edit". Lateral links run between products that answer adjacent questions, such as the alternative at a different price point, the accessory that completes the purchase, or the version built for a different use. Downward links run from the collection and the buying guide into specific products, placed in prose that explains who each product suits, which is exactly the job a collection page's introductory copy is usually too thin to do.
Two platform details decide whether this graph consolidates or fragments. Some themes link products through collection-scoped paths such as /collections/trail/products/shoe while the canonical tag points at /products/shoe, and linking the canonical URL directly keeps every internal link pointing at the address you want cited. Collection filtering or pagination that depends on client-side script can also leave much of a large catalogue reachable only through the sitemap, which is the least reliable discovery path for crawlers that do not render.
Prose links let a model infer relationships. Structured data lets you state them for the systems that parse it, and the gap between those two audiences has become clearer over the past year. Google's May 2026 guidance says structured data is not required for its generative AI features and that no special schema.org markup exists for them, while still recommending it for rich result eligibility. Independent testing points the same way for LLM retrieval: searchVIU's October 2025 tests across five AI systems found that ChatGPT, Claude, Gemini, Perplexity and Google AI Mode did not return facts that existed only in JSON-LD during direct page fetches. The practical conclusion is that body links carry the relationship signal and schema corroborates it. Schema remains worth maintaining, rendered server-side and consistent with the visible page, but it cannot rescue a graph that the prose does not already express.
The pattern is straightforward. The canonical node declares hasPart references to each resolution article. Each resolution article reciprocates with isPartOf, and can also use mentions to name entities it discusses and citation to reference the specific sources it draws on. Every node references the same Organization and Person identifiers so that authority aggregates rather than fragments. Where you want to list related pages explicitly, note that schema.org defines relatedLink on WebPage rather than on Article, so a @graph that separates the page from the article it carries keeps the markup valid.
Sample markup (illustrative only; placeholder values):
{
"@context": "https://schema.org",
"@graph": [
{
"@type": "WebPage",
"@id": "https://example.com/geo/internal-linking#webpage",
"url": "https://example.com/geo/internal-linking",
"mainEntity": { "@id": "https://example.com/geo/internal-linking#article" },
"breadcrumb": { "@id": "https://example.com/geo/internal-linking#breadcrumb" },
"relatedLink": [
"https://example.com/geo/topic-clusters",
"https://example.com/geo/topical-authority"
]
},
{
"@type": "Article",
"@id": "https://example.com/geo/internal-linking#article",
"headline": "Internal Linking Strategy for LLM-Readable Sites",
"isPartOf": {
"@type": "CollectionPage",
"@id": "https://example.com/geo#pillar",
"name": "Content Architecture for GEO"
},
"mentions": [
{ "@type": "Thing", "name": "Generative Engine Optimization" },
{ "@type": "Thing", "name": "Structured data" }
],
"author": { "@id": "https://example.com/team/jane-doe#person" },
"publisher": { "@id": "https://example.com/#organization" }
},
{
"@type": "BreadcrumbList",
"@id": "https://example.com/geo/internal-linking#breadcrumb",
"itemListElement": [
{
"@type": "ListItem",
"position": 1,
"name": "GEO",
"item": "https://example.com/geo"
},
{
"@type": "ListItem",
"position": 2,
"name": "Internal Linking Strategy for LLM-Readable Sites"
}
]
}
]
}
This markup is an illustrative sample, not deployable code. Every example.com URL, the jane-doe Person identifier and the Content Architecture for GEO pillar name are placeholders: replace them with your own canonical URLs and with the Organization and Person @id values already defined elsewhere on your site, so that every page references one identity rather than creating new ones. Validate the result with the Schema Markup Validator and Google's Rich Results Test before release, and output it server-side so crawlers that do not execute JavaScript receive it in the initial HTML.
Alongside this, ensure the site's primary navigation is expressed in server-delivered HTML with descriptive anchors, and that breadcrumbs appear on every content page both as visible links and as BreadcrumbList markup. The visible trail gives any crawler the document's hierarchical position in the layer it reliably reads, and the markup states the same hierarchy for Google's systems, which is considerably more reliable than inferring it from URL path segments.
Firon's Three-Check Protocol evaluates Clarity, Credibility and Reputation. Internal linking is primarily a Clarity instrument. It answers the machine's first question, which is what this document is about and what else on this domain relates to it, without requiring the model to infer relationships from keyword overlap alone.
Clarity is the check that must pass first. Credibility work, meaning specificity, sourcing and named expertise, produces no aggregate benefit if a model cannot tell which of your pages belong together. Reputation work, meaning third-party citation, concentrates on whichever URL external sources happen to point at, and a poorly wired graph fails to distribute that signal to the rest of the cluster. That ordering is why Firon Marketing lands Clarity work first inside its sequenced Generative Engine Optimization and Agentic Commerce Protocol program: every later investment depends on it.
The order matters more than the thoroughness of any single step, because each stage depends on the one before it. The work runs in six stages.
Start with access. Confirm that robots.txt and CDN bot settings admit the search and agent crawlers you want, including OAI-SearchBot, Claude-SearchBot and PerplexityBot alongside Googlebot, and that the site is included in generative AI features in Search Console. No linking work matters until a crawler receives a successful response.
Second, map the graph as a crawler sees it. Crawl the site with JavaScript disabled, compare the result with the sitemap and CMS inventory, and record orphans, pages deeper than three clicks from the homepage, and links that exist only after rendering. On a Shopify store, inspect the product recommendations block and any app-injected sections specifically.
Third, fix upward links. Give every cluster and every product category one canonical node, and point each supporting page at it from body copy. Consolidating the definitional signal first makes every later change easier for a model to interpret.
Fourth, add lateral links between siblings, beginning with pages that are already cited or ranking, since those are the pages a retrieval system is most likely to fetch and follow onward from.
Fifth, rewrite anchors across existing links, replacing generic and merchandising labels with phrases that name the destination's subject, and move structurally important links out of widgets and into prose. Schema updates belong at this stage, as corroboration for links that now exist in the body.
Last, measure. Fix a prompt set for each engine before any change ships so you have a baseline, track which URLs are cited, and use the Generative AI performance report in Search Console for Google's surfaces. The signal to watch is whether a cited page starts pulling its linked siblings into later answers. Results lag the changes, because each engine recrawls and refreshes its retrieval index on its own schedule. If cited pages still fail to lift their neighbors after the changed pages have been recrawled, examine the content of those neighbors before touching the links again.
Firon's working standard for a cluster article is between three and eight links inside the body content: one upward link to the canonical definition node, two to four lateral links to sibling articles, and one or two links to service or evidence pages where genuinely relevant. The constraint is dilution. A page carrying forty body links produces passages where link text crowds out substantive content, which weakens the extractability of every passage. Fewer, better-labeled links inside prose consistently outperform a dense directory of them.
Rarely. Content extraction routines that feed retrieval systems typically discard headers, footers, sidebars and related-post widgets as boilerplate, because those elements repeat across every page of a template. Related-post modules injected by client-side JavaScript fail twice, since most AI crawlers never execute the script that creates them. These links may still assist classical crawl discovery, so there is no need to delete them, but any link that carries structural meaning for your topical graph must also appear in the server-delivered body prose.
Descriptive natural-language phrases that name the destination's subject and scope, rather than bare keywords or generic instructions. An anchor reading "how to build topical authority that AI models recognize" tells a system both what the destination covers and how it relates to the current document. Vary the surface wording across links to the same destination while holding the meaning constant, since identical strings add no further semantic information. Avoid "click here", "read more" and "learn more" entirely, as they transmit nothing extractable.
Indirectly but materially. Internal links determine whether an AI crawler can discover a page at all, whether a model can infer that your documents form one coherent body of expertise, and whether an agentic system following links from a retrieved page finds supporting context on your domain rather than a competitor's. These act as prerequisites for being a viable retrieval candidate rather than as classical ranking signals, and they decide whether authority aggregates across your content or is judged page by page.
Run a crawl that reads only the raw HTML your server returns, without executing JavaScript, because GPTBot, ClaudeBot and PerplexityBot behave the same way. Compare the discovered URL set against your sitemap and CMS inventory. Anything present in the inventory but absent from the raw HTML crawl is functionally orphaned for those systems. Apply the same test to pages buried more than three clicks from the homepage, then either integrate each orphan into a relevant cluster through body links or consolidate it into a stronger sibling with a redirect.
Yes, if they are set without review. Since September 15, 2026, Cloudflare manages AI crawler access through separate Search, Training and Agent controls rather than a single Block AI Bots switch. Choosing Block for Training now also stops mixed-use crawlers such as Googlebot, Applebot and Bingbot, while Disallow AI Training refuses training and keeps search crawling open. Cloudflare's recommended setting for ad-supported sites also blocks agents on ad-carrying pages. Check each control deliberately, because a crawler that receives a block response never sees your internal links.
Disclaimer: "Firon Marketing is a strategic consultancy. All technical implementations should be reviewed by your engineering team to ensure compatibility with your specific tech stack."
A content graph that a machine cannot traverse does not exist for that machine, however much it cost to produce, and the consequence shows up in what the engines say about you. Book a 30-minute walkthrough and Firon will have your AI Perception Report built before the call: where you sit on the AI leaderboard for your category, the answer each engine gives about your brand today set against the one it should give, and three prioritized fixes named page by page. The report is yours to keep either way. See what AI engines are telling buyers about your brand.
