Controlling the Discoverability Range of Posts on My Hugo Blog

Oct 5, 2026

Not every piece of writing on my personal site is meant to be indexed by search engines or archived in permanent web caches.

While public essays and documentation should remain easily discoverable via search engines, informal musings, work-in-progress drafts, or semi-private notes shared only with specific peers require tighter boundaries. These posts still need to be accessible via direct link, but they should not be crawled, ranked, or cached for the broader web.

To establish this level of fine-grained control, I implemented a selective indexing mechanism directly into my Hugo site architecture.

1. Defining Custom Front Matter Directives

The first step was establishing a clean way to flag specific content files for exclusion. In the front matter of any post that should remain unindexed, I introduced a robots parameter.

Front Matter Configuration

---
title: "Unlisted Technical Notes"
date: 2026-10-18
robots: "noindex, noarchive"
---

(This can also be represented in TOML using robots = "noindex, noarchive").

Directive Purposes

noindex: Explicitly instructs search crawlers (Googlebot, Bingbot, etc.) not to include the page in search indices.
noarchive: Prevents search engines from retaining or displaying a cached snapshot of the page.

2. Adapting the Hugo Head Layout

To make Hugo react to this parameter during the static build step, the <head> template required conditional logic.

In the head template (layouts/partials/head.html), the standard robots meta tag was replaced with a dynamic check:

{{ if .Params.robots }}
  <meta name="robots" content="{{ .Params.robots }}">
{{ else }}
  <meta name="robots" content="index, follow">
{{ end }}

During build generation:

When Hugo encounters a post with the robots parameter defined, it outputs <meta name="robots" content="noindex, noarchive">.
For standard articles where the parameter is omitted, it falls back to <meta name="robots" content="index, follow">.

If using an external theme, placing this file in the project’s root layouts/partials/head.html overrides the theme’s version safely without risking changes being wiped during theme updates.

3. Excluding Posts from the Sitemap

By default, Hugo compiles a global sitemap.xml file that lists every published page on the site.

Even with a noindex tag present on the page itself, leaving unlisted URLs inside sitemap.xml can trigger contradictions in search engine search consoles (such as warnings for Submitted URL marked ’noindex’).

Hugo’s built-in sitemap template supports an exclusion flag. Adding sitemap.disable: true to the front matter cleanly removes the post from the generated XML:

---
title: "Unlisted Technical Notes"
robots: "noindex, noarchive"
sitemap:
  disable: true
---

When Hugo builds the site, its internal sitemap generator checks this property and skips the entry altogether.

4. Verification

Before deploying changes, verification involves two local checks via hugo server:

Header validation: Inspecting the page source (Ctrl+U / Cmd+Option+U) of the unlisted post to confirm the presence of <meta name="robots" content="noindex, noarchive">.
Sitemap validation: Opening localhost:1313/sitemap.xml and verifying that the excluded post path is absent from the <urlset>.