Building a Search Index on every Netlify Deploy

Stork is compatible with any website, but works especially well with static sites built with a static site generator and hosted on static hosting services like Netlify. In this guide, you'll learn:

1. How to dynamically generate a Stork configuration file as part of your site's build process,
2. How to build an index using the Stork binary during your site's build process, on the Netlify task runner, and
3. How to load that index and build a search page on your site.

I'll be building a Stork interface on top of the Eleventy Base Blog example.

Step 1: Generate the Stork Configuration File

A Stork configuration file is required to tell Stork how to index your content. Stork configuration files are, in essence, a listing of each document that should be indexed. We could write this file by hand, but we'll use Eleventy to build this configuration file for us.

Every static site generator lets you build a page that lists out all your "posts" or "pages" or "articles." You might be using that functionality to build a "post archive" page, or perhaps an RSS feed. We can use that same functionality to build a Stork configuration file.

I'll create a new Nunjucks file that will act as a template for my Stork configuration file. In it, I'll loop through collections.posts and generate a new TOML entry under [[input.files]] for each post with the post's path on disk, the URL it will ultimately have once it's published, and the post's title.

I'm also configuring the output file to be located in the _site directory: the directory where Eleventy writes the final site.

---
permalink: "stork-posts.toml"
eleventyExcludeFromCollections: true
---
 
[input]
url_prefix = "{{metadata.url}}"
 
{%- for post in collections.posts %}
{% set absolutePostUrl %}{{ post.url | url }}{% endset %}
 
[[input.files]]
path = "{{post.inputPath}}"
url = "{{absolutePostUrl}}"
title = "{{post.data.title | safe}}"
 
{%- endfor %}

Now, when Eleventy builds my site, it will create a file at _site/stork-posts.toml with the full Stork configuration file. (We can run npm run eleventy build and examine the contents of the _site directory to confirm this!)

Next, we'll configure Netlify to use that configuration file to generate the Stork index after it builds the site.

Step 2: Building the Index

By default, when Netlify builds my site, it runs npx eleventy, a Terminal command which will build the site. In this step, we want to change the Netlify build process so that it:

1. Runs the Eleventy site generation to build my static pages and my Stork configuration file, then
2. Runs stork build on the generated configuration file to build an index file.

Stork doesn't come preinstalled on Netlify's build machines, so in between steps 1 and 2, we have to install the Stork binary as well.

I'll add the following to a new file called build.sh in the root of my project:

From top to bottom, this script:

1. Downloads the Stork command line program from Github, built for Ubuntu 16.04, the Ubuntu version that Netlify's build system uses,
2. Makes that downloaded program executable,
3. Runs Eleventy and builds the site, and
4. Uses the Stork binary to generate a search index.

Netlify lets users configure the build command to run when publishing a new version of the site (again, by default, it's running npx eleventy) by editing a file named netlify.toml in the root of the repository. I'll change the file so it has the following contents:

[build]
publish = "_site"
command = "chmod +x build.sh && ./build.sh"

After I change the Netlify configuration and re-deployed my site, I'll check to make sure Netlify is generating the search index properly. At this point, your index should be available on Netlify's CDN; you should be able to visit https://<your-site>/stork-posts.st to download the index file

Step 3: Building the Search page

Now that Netlify is automatically building the search index, I can build a new page on my site that will display the Stork interface. I'll add the following page to my site:

Eleventy will use the default page layout and publish this page to /search when it builds the site. Visit https://stork-netlify-example.netlify.app/search to see the final result in action!

Extra credit, for you to explore

• On my personal site, I have multiple "collections" of content on my site, including a portfolio of projects I've worked on. What would you add to also index the portfolio project descriptions as well?
• What would you change to get the search interface embedded in my site's header or footer, on every page of the site?
• This setup runs the Stork indexer on the Markdown files in the site's repository. How would you change it so that it instead indexes the HTML files generated by Eleventy?