SEO, robots.txt & llms.txt

What Nimbu does for search engines and AI agents automatically, and how a theme can add robots.txt rules, language annotations and llms.txt content

Nimbu handles most technical SEO for you: canonical and hreflang links, /sitemap.xml, /robots.txt and /llms.txt. A theme only has to output {{ content_for_header }} in its layouts. This page explains what is generated and where a theme or the site settings can change it.

Most settings below live in the Search Engines section of Settings → General in the Nimbu admin.

Canonical and hreflang links

{{ content_for_header }} adds to every HTML page:

<link rel="canonical"> pointing at the current page on the primary domain, in the current locale.
<link rel="alternate" hreflang="..."> for every locale the page is available in. Hidden locales, and locales where the page is not available, are skipped.
The same canonical and alternate links as HTTP Link response headers.

Filter and pagination parameters (?vendor=ara, ?page=2) stay in the canonical URL, because each of those pages can rank on its own. Tracking parameters are removed: utm_*, gclid, fbclid, msclkid, _gl, mc_cid, autologin and similar.

Don't add canonical or hreflang tags in your theme. Nimbu already outputs them, and a second set conflicts with the first. If a layout does not output {{ content_for_header }}, you get none of them.

Two settings adjust the language annotations:

Search language overrides: a language tag per locale, for example nl-BE for nl when the Dutch site only targets Belgium. Leave blank to use the locale code.
Fallback language (x-default): this locale also gets hreflang="x-default", the version search engines show when no language matches. Visitors are not redirected.

See Multilingual for the matching <html lang> setup.

Pages that are not indexed

Page	What Nimbu sends
Pages set to Hidden from Search Bots	`<meta name="robots" content="none">` and `X-Robots-Tag: none`
The themed 404 page	`<meta name="robots" content="noindex">` and `X-Robots-Tag: noindex`, no canonical or `hreflang` links
Pages opened with a preview link (`?preview=...`)	`X-Robots-Tag: noindex`

robots.txt

/robots.txt is built from up to three sources, merged in this order:

Nimbu's rules: Disallow: /admin/, hidden locales, pages hidden from search bots, a Crawl-delay group for bingbot and for SEO crawlers, and the sitemap.
The robots.txt field in the Search Engines settings.
templates/robots.txt.liquid in the live theme, rendered without a layout.

Later sources can only add rules. A theme or the admin field cannot remove Nimbu's rules, so /admin/, hidden locales and crawler throttling always stay in place. A site without its own rules gets Nimbu's file unchanged.

Adding rules from the theme

Create templates/robots.txt.liquid. It is plain text, and Liquid works as in any template:

User-agent: *
Disallow: /checkout
Disallow: /cart
Disallow: /*/checkout

User-agent: GPTBot
Disallow: /members/

How rules are merged

Start every block with a User-agent: line. Rules before the first User-agent: are ignored, as crawlers ignore them.
Only field: value lines are kept. Other lines are dropped, including the Liquid error: ... text a broken template prints. If the template fails to render altogether, it is skipped and robots.txt is served without it. /robots.txt never returns an error because of your template (an error there would read as "disallow everything" to Google).
To open something Nimbu blocks, add a more specific Allow:. Crawlers use the longest matching rule, so Allow: /admin/public-page wins over Disallow: /admin/.
User-agent: * rules also reach named crawlers. A crawler that has its own group ignores * (RFC 9309), and Nimbu names bingbot and the SEO crawlers. So Nimbu copies your * rules into those groups. The exception is a source that names that crawler itself: User-agent: * / Disallow: / next to User-agent: Googlebot / Disallow: means "everyone except Googlebot", and that source's * rules stay out of the Googlebot group.
Each source contributes its most specific matching group. A theme's Googlebot-Image group also receives the admin field's Googlebot rules, unless the admin field has its own Googlebot-Image group.
Crawl-delay can only slow crawlers down. Nimbu uses 5 seconds for bingbot and 10 seconds for SEO crawlers (AhrefsBot, SemrushBot, MJ12bot, DataForSeoBot, dotbot and others). A Crawl-delay under User-agent: * applies to them too, but only when it is higher: Crawl-delay: 30 raises them to 30 seconds, Crawl-delay: 3 keeps 5 and 10.
Sitemap: lines from all sources are combined, without duplicates, at the end of the file.

nimbu server renders your local templates/robots.txt.liquid, so you can check the merged result before pushing the theme.

sitemap.xml

Nimbu generates /sitemap.xml from published pages, the entries (or customer profiles) shown by channel template pages, blog articles, and, when the webshop is enabled, collections and products. Pages that are customer-only or hidden from search bots are left out, and every URL is listed once per available locale. There is no Liquid tag or template for the sitemap.

llms.txt

/llms.txt gives AI agents a Markdown index of the site:

The site name as title, and the homepage's SEO description as introduction. When the homepage has none, the default locale's file uses the site's meta description; other locales get no introduction.
Pages: up to 100 published pages available in the locale, with their SEO description. Channel template pages, customer-only pages and pages hidden from search bots are left out.
Blog: the 25 most recent published articles in the locale, from public blogs.

Each locale has its own file at /<locale>/llms.txt. The default locale lives at /llms.txt; /<default locale>/llms.txt redirects there with a 301.

Settings:

Setting	Effect
Serve llms.txt	On by default. When off, `/llms.txt` is not served and is handled like any unknown URL.
llms.txt override	Your own content, served as-is for every locale instead of the generated file. Leave blank to generate it.
Markdown for AI agents	On by default. See below.

Markdown for AI agents

When Markdown for AI agents is on, a GET request for a page or blog article with Accept: text/markdown gets a Markdown version of the rendered HTML instead of the HTML. Browsers never ask for this, so visitors are not affected. The generated llms.txt then mentions that pages can be fetched as Markdown.

curl -H "Accept: text/markdown" https://www.example.com/about

Only successful (200) responses are converted.

Layouts - Where {{ content_for_header }} goes
Multilingual - Locales and language switchers
Templates - Themed 404 page

On this page