Add Search to a Hugo site with Lunr.js and Node.js

Hugo “The world’s fastest framework for building websites” is a great option for JAMStack (JavaScript, APIs, prebuild Markup) sites.

Lunr.js is “A bit like Solr, but much smaller and not as bright”, it’s a pure JavaScript implementation of a Solr-like search engine.

One of the only things it doesn’t provide out of the box is Search. It does give you a few options to integrate at “Search for your Hugo Website”.

None were plug and play so I wrote my own simple solution using Lunr.js, a small Node.js script and a few lines of client-side HTML/JavaScript code.

This is a great example of a benefit of Node.js: it’s a breeze to integrate a pure JavaScript library and pre-compute the search index.

You can see Search in action at codewithhugo.com/search/?q=lunrjs.

Find the full gist at gist.github.com/HugoDF

Load up all markdown content with frontmatter

We only want to index a single directory, the “content/posts” directory, our loadPostsWithFrontMatter function will accept the full path of the posts directory.

First thing this function does it read the directory contents to get all the file names. It then reads each file and parses the frontmatter and markdown. It flattens the content and the frontmatter data into a single object. It also truncates the content to 3000 characters to avoid generating a huge (2MB+) index file.

const fs = require('fs').promises;
const {promisify} = require('util');
const frontMatterParser = require('parser-front-matter');
const parse = promisify(frontMatterParser.parse.bind(frontMatterParser));

async function loadPostsWithFrontMatter(postsDirectoryPath) {
  const postNames = await fs.readdir(postsDirectoryPath);
  const posts = await Promise.all(
    postNames.map(async fileName => {
      const fileContent = await fs.readFile(
        `${postsDirectoryPath}/${fileName}`,
        'utf8'
      );
      const {content, data} = await parse(fileContent);
      return {
        content: content.slice(0, 3000),
        ...data
      };
    })
  );
  return posts;
}

Creating the Lunr.js index

Given a list of posts, we want to use the title as a reference (more on that later), and index the title, content and tags fields.

const lunrjs = require('lunr');

function makeIndex(posts) {
  return lunrjs(function() {
    this.ref('title');
    this.field('title');
    this.field('content');
    this.field('tags');
    posts.forEach(p => {
      this.add(p);
    });
  });
}

Putting it all together

The following script needs to have the previously defined JavaScript functions in scope to work, and be at the root of the Hugo project in order to read all the posts into the search index.

See the full file at gist.github.com/HugoDF/aac2e529f79cf90d2050d7183571684b.

This function actually just logs the stringified index out. To get it into a file, we could add await fs.writeFile('./path/to/index.json', JSON.stringify(index), 'utf8') or we can redirect the output of a file (which is a bit more flexible).

async function run() {
  const posts = await loadPostsWithFrontMatter(`${__dirname}/content/post`);
  const index = makeIndex(posts);
  console.log(JSON.stringify(index));
}

run()
  .then(() => process.exit(0))
  .catch(error => {
    console.error(error.stack);
    process.exit(1);
  });

Generating the index file

I personally created a static/gen folder that includes a .gitkeep file. Added the empty folder to git and then ignored it, then my Lunr.js search index generation command is:

node ./build-lunrjs-index.js > static/gen/search-index.json

You can also just stuff the search index into the root of your static folder:

node ./build-lunrjs-index.js > static/search-index.json

Or even put it in public directly:

node ./build-lunrjs-index.js > public/search-index.json

In each of these cases, be weary of trying to redirect output to a directory that doesn’t exist (especially in your continuous integration pipeline).

Consuming the Lunr.js index client-side

To consume the Lunr.js index, we just have to load it and call lunr.Index.load, as illustrated below:

fetch('/gen/search-index.json').then(function (res) {
  return res.json();
}).then(function (data) {
  const index = lunr.Index.load(data);
  const matches = index.search(searchString);
});

A more fully featured integration might be as follows.

We want a search box (form) with a submit button and a clear link. When the page loads, we first check what the q param contains by trying to parse it as a URLSearchParams.

If it’s empty, display an information message.

If there is a search query, we load up the search index using fetch, load into into memory using lunr.Index.load and search against it. What we’ve also done before this point is generate a post title -> search result mapping using Hugo slices and a bit of JavaScript to marshal it.

Using the title -> result mapping, we display relevant search results.

<form method="get" action="">
  <input id="search" name="q" type="text" />
  <button type="submit" class="button">Search</button>
  <a href="/search">Clear</a>
</form>
<div id="#app"></div>

<script src="https://unpkg.com/lunr/lunr.js"></script>
<!-- Generate a list of posts so we can display them -->
{{ $p := slice }}
{{ range (where .Site.RegularPages "Section" "==" "post") }}
  {{ $post := dict "link" .RelPermalink "title" .Title "content" (substr .Plain 0 200) -}}
  {{ $p = $p | append $post -}}
{{ end }}
<script>
const posts = JSON.parse(
  {{ $p | jsonify }}
);

const query = new URLSearchParams(window.location.search);
const searchString = query.get('q');
document.querySelector('#search').value = searchString;
const $target = document.querySelector('#app');

// Our index uses title as a reference
const postsByTitle = posts.reduce((acc, curr) => {
  acc[curr.title] = curr;
  return acc;
}, {});

fetch('/gen/search-index.json').then(function (res) {
  return res.json();
}).then(function (data) {
  const index = lunr.Index.load(data);
  const matches = index.search(searchString);
  const matchPosts = [];
  matches.forEach((m) => {
    matchPosts.push(postsByTitle[m.ref]);
  });

  if (matchPosts.length > 0) {
    $target.innerHTML = matchPosts.map(p => {
      return `<div>
        <h3><a href="${p.link}">${p.title}</a></h3>
        <p>${p.content}...</p>
      </div>`;
    }).join('');
  } else {
    $target.innerHTML = `<div>No search results found</div>`;
  }
});

You can see Search in action at codewithhugo.com/search/?q=lunrjs.

See the full gist at gist.github.com/HugoDF

N.