Building a Search Index on every Netlify Deploy

Stork is compatible with any website, but works especially well with static sites built with a static site generator and hosted on static hosting services like Netlify. In this guide, you'll learn:

  1. How to dynamically generate a Stork configuration file as part of your site's build process,
  2. How to build an index using the Stork binary during your site's build process, on the Netlify task runner, and
  3. How to load that index and build a search page on your site.

I'll be building a Stork interface on top of the Eleventy Base Blog example.

!

Note:

You can check out the final repository on Github to see what we're going to build. The repository's most recent commit represents all the work we're going to do in this guide.

Step 1: Generate the Stork Configuration File

A Stork configuration file is required to tell Stork how to index your content. Stork configuration files are, in essence, a listing of each document that should be indexed. We could write this file by hand, but we'll use Eleventy to build this configuration file for us.

Every static site generator lets you build a page that lists out all your "posts" or "pages" or "articles." You might be using that functionality to build a "post archive" page, or perhaps an RSS feed. We can use that same functionality to build a Stork configuration file.

I'll create a new Nunjucks file that will act as a template for my Stork configuration file. In it, I'll loop through collections.posts and generate a new TOML entry under [[input.files]] for each post with the post's path on disk, the URL it will ultimately have once it's published, and the post's title.

I'm also configuring the output file to be located in the _site directory: the directory where Eleventy writes the final site.

stork-posts.njk
---
permalink: "stork-posts.toml"
eleventyExcludeFromCollections: true
---
[input]
url_prefix = "{{metadata.url}}"
{%- for post in collections.posts %}
{% set absolutePostUrl %}{{ post.url | url }}{% endset %}
[[input.files]]
path = "{{post.inputPath}}"
url = "{{absolutePostUrl}}"
title = "{{post.data.title | safe}}"
{%- endfor %}

Now, when Eleventy builds my site, it will create a file at _site/stork-posts.toml with the full Stork configuration file. (We can run npm run eleventy build and examine the contents of the _site directory to confirm this!)

Next, we'll configure Netlify to use that configuration file to generate the Stork index after it builds the site.

Step 2: Building the Index

By default, when Netlify builds my site, it runs npx eleventy, a Terminal command which will build the site. In this step, we want to change the Netlify build process so that it:

  1. Runs the Eleventy site generation to build my static pages and my Stork configuration file, then
  2. Runs stork build on the generated configuration file to build an index file.

Stork doesn't come preinstalled on Netlify's build machines, so in between steps 1 and 2, we have to install the Stork binary as well.

I'll add the following to a new file called build.sh in the root of my project:

build.sh
#!/usr/bin/env bash
wget https://files.stork-search.net/releases/latest/stork-ubuntu-16-04
chmod +x stork-ubuntu-16-04
ELEVENTY_ENV=production npx @11ty/eleventy --config=eleventy.js
./stork-ubuntu-16-04 build --input _site/stork-posts.toml --output _site/stork-posts.st

From top to bottom, this script:

  1. Downloads the Stork command line program from Github, built for Ubuntu 16.04, the Ubuntu version that Netlify's build system uses,
  2. Makes that downloaded program executable,
  3. Runs Eleventy and builds the site, and
  4. Uses the Stork binary to generate a search index.

Netlify lets users configure the build command to run when publishing a new version of the site (again, by default, it's running npx eleventy) by editing a file named netlify.toml in the root of the repository. I'll change the file so it has the following contents:

netlify.toml
[build]
publish = "_site"
command = "chmod +x build.sh && ./build.sh"

After I change the Netlify configuration and re-deployed my site, I'll check to make sure Netlify is generating the search index properly. At this point, your index should be available on Netlify's CDN; you should be able to visit https://<your-site>/stork-posts.st to download the index file

Step 3: Building the Search page

Now that Netlify is automatically building the search index, I can build a new page on my site that will display the Stork interface. I'll add the following page to my site:

search.njk
---
layout: layouts/home.njk
permalink: /search/
eleventyNavigation:
key: Search
order: 4
---
<h1>Search</h1>
<link rel="stylesheet" href="https://files.stork-search.net/basic.css" />
<div class="stork-wrapper">
<input data-stork="posts" class="stork-input" />
<div data-stork="posts-output" class="stork-output"></div>
</div>
<script src="https://files.stork-search.net/stork.js"></script>
<script>stork.register('posts', '/stork-posts.st');</script>

Eleventy will use the default page layout and publish this page to /search when it builds the site. Visit https://stork-netlify-example.netlify.app/search to see the final result in action!

Extra credit, for you to explore

  • On my personal site, I have multiple "collections" of content on my site, including a portfolio of projects I've worked on. What would you add to also index the portfolio project descriptions as well?
  • What would you change to get the search interface embedded in my site's header or footer, on every page of the site?
  • This setup runs the Stork indexer on the Markdown files in the site's repository. How would you change it so that it instead indexes the HTML files generated by Eleventy?

© 2019–2021

Stork is built and shepherded by James Little, who's really excited that you're checking it out. If you have any questions or comments, feel free to tweet at him or open an issue on Github.

This site is open source. Please file a bug or open a PR if you see something confusing or incorrect. PRs are always welcome!

Logo by Bruno Monts. Please contact James Little before using the Stork logo. Thanks to Bruno and the fission.codes team for making this logo happen.