I use a system written in Pollen and Racket ⊕ This post assumes some familiarity with Pollen and SQL. to generate this blog. It renders the HTML file for most individual pages in less than half a second per page —not blazing, certainly, but serviceable. However, when rendering pages that combine multiple posts, my code has been interminably slow—40 seconds to render the home page, for example, and 50 seconds to render the RSS feed.
By creating my own cache system using a SQLite database, I was able to drop those times to less than a second each.
My old approach, which used only Pollen's functions for retrieving docs and metas, had some pretty glaring inefficiencies For example, in the code for my RSS feed, I was fetching the docs and metas from every document in the entire pagetree (that is, every post I had ever written) into a list of structs, even though I would only use the five most recent posts in the feed itself.—so glaring, in fact, that I probably could have cut my render times for these "problem pages" down to 3–6 seconds just by addressing them. But I thought I could get even further by making my own cache system.
Pollen's cache system is optimized for fetching individual docs and metas one by one, not for grabbing and dynamically sorting a whole bunch of them at once. Also, Pollen's cache doesn't store the rendered HTML of each page. There are good reasons for both of these things; Pollen is optimized for making books, not blogs, and the content of a book doesn't often change. But Pollen was also designed to be infinitely flexible, which allows us to extend it with whatever facilities we need.
In the template code for individual posts (i.e. at the same time the post is being rendered to HTML) save the post's title, date and a copy of the rendered HTML in the database.
In the "combination pages" (such as index.html and feed.xml), query the SQLite database for the HTML and spit that out.
This still isn't nearly as fast as some static site generators Hugo has been benchmarked building a complete 5,000 post site in less than six seconds total.. But it's plenty good enough for my needs.
I made a new file, util-db.rkt ⊕ You can browse the source code for this blog on its public repository. to hold all the database functionality and to provide functions that save and retrieve posts. My pollen.rkt re-provides all these functions, which ensures that they are available for use in my templates and .pp files.
The expression init-db ensures the database file exists, and runs some CREATE TABLE IF NOT EXISTS queries to ensure the tables are set up correctly. The save-post expression saves the post's metas and rendered HTML into the database.
Some more notes on the code above: The split-body-comments and post-header functions come from another module I wrote, util-template.rkt. The first separates any comments (that is, any ◊comment tags in the Pollen source) from the body of the post, which lets me save just the body HTML in the database. The second provides an X-expression for the post's header, which includes or omits various things depending on the post's metas.
Internally, util-db.rkt has some very basic functions that generate and execute the SQL queries I need.
The schema and queries are designed to be idempotent, meaning I can safely run them over and over again and end up with same set of records every time. So, no matter what state things are in, I don't have to worry about ending up with duplicate records or other out-of-whacknesses when I render the site.
The database is also designed to need as few queries as possible, both when saving and when fetching data: for example, using a single query to either create something if it doesn't exist, or to replace it if it does exist. Also, where applicable, creating multiple records in a single query (as in the last example above).
Finally, in the interests of keeping things "simple", I have tried to keep the database disposable: that is, no part of the site's content has its origin or permanent residence in the database itself. I can delete it at any point and quickly rebuild it. That way my use of it remains limited to my original plan for it: a speed enhancement, and nothing more.
Since home page, RSS feed, and other aggregation pages are now getting all their content from the SQLite database, it's important that the database be up to date before those pages are rendered.
If any of the "core files" have changed, force a re-render of all individual posts.
If any posts or core files have been changed or added, force a re-render of the "aggregation pages", such as the home page, the Topics page, and the RSS feed.
This way, the pages that update the cache database are rendered before the pages that draw from the database, and everything is hunky-dory.
This blog makes use of "topics", which are basically like tags. A post can have zero or more topics, and a topic can have or more posts.
"SELECT `topic`, p.pagenode, p.title FROM `posts-topics` t INNER JOIN `posts` p ON t.pagenode = p.pagenode ORDER BY `topic` ASC"
The SQL query uses INNER JOIN to link the posts in the posts-topics table with their titles in the posts table, resulting in a list of rows with three columns each: topic, pagenode (the relative path to the HTML file), and title. The topic-list function then groups these rows by the topic name and returns the resulting list.
One last consideration: removing a post is no longer as simple as deleting its Pollen source document. I have to remove it from the SQLite database as well, otherwise it will continue to appear on the home page, RSS feed, etc.
There are a few ways I could do this. The simplest would be to delete the database file and rebuild the site from scratch. Or I could open the database in a program like DB Browser for SQLite and delete the rows manually. Or I could write a script to automate this. I don't often need to delete posts, so I'll probably put off writing any scripts for now.