Note: My RSS feed URL has changed to https://feather.onl/feed/blog. While I have left a redirect on the old one I can't guarantee it will be around forever so it's probably a good idea to update it~

Goodbye Static Site

Welcome to newly non-static site! My old site was generated in a big bash script primarily using pandoc. This worked okay, but it there was a couple of things I always meant to do that ended up being so annoying I didn't bother, most obviously, adding tags. I don't doubt that I could have done it, but that script was already at "this is way too much to be doing in bash" levels. Also, I don't like bash at the best of times to be honest, fine for small things, probably shouldn't have been generating a website with it.

Go

I've made small programming projects for random things for myself over the years. I always used python and I don't think I ever got "good" at it. Go caught my attention a few months ago, it looked it had a nice syntax and I was interested in learning something else because I just felt like I wanted a fresh start on programming and a language I can compile to a single binary sounds a lot nicer than running python scripts. A lot of the things I self host are also written in Go so who knows, maybe it would open up contributions if I ever felt that I got good enough.

Syntax aside, Go has a web server and templating system built-in to the standard library which helps immensely. I was learning it on and off over a couple of months, going through the Go tour, reading random sites and watching a few videos.

I'd recommend anyone looking to learn to give The Little Go Book a look. It hasn't been updated in a while, but it's mostly still relevant, and free. Let's Go is also very nice, though it is paid. I quite liked it as it explained why things are done a certain way a lot, which is weirdly lacking in a lot of places.

goldfinch

So I started on what would become goldfinch, which is now running this website. I've actually been using a modified version of my original bash script for another website which is updated more frequently, and the process just sort of started annoying me to be honest, on top of not actually doing everything I wanted it to.

I planned out, mostly, what I wanted this project to do:

• Generate HTML from files and put into a template

  • Parsing YAML frontmatter for config data

  • Support both markdown and mycomarkup

• Create index pages that list all pages within a directory when it's URL is visited

• Allow pages to be marked as drafts, unlisted or private

  • Drafts are ignored completely

  • Unlisted pages simply don't show up in the indexes

  • Private pages require a key

• Tags

  • Listed on pages, clicking one shows other pages with the same tag

  • A page that lists all tags used and how pages are using it

• Templates

  • Basic templating that puts page content into a layout

  • Custom template used for page depending on root directory, eg, a blog template for anything in the /blog folder

• Multiple RSS feeds

Early on I was just loading the files to a slice of structs and returning pages from variables. Bad idea looking back, but I fairly quickly moved on to loading everything into an SQLite database. Sorting through data with variables gets messy quickly and SQL greatly helps with that too since SQL queries are nicer when it comes to both reading and writing. I also assume that they're more efficient.

I took my time with it, both due still learning Go and life stuff but I slowly built up all the features I wanted, going back occasionally to redo or restructure things but eventually I got to where it is now, and I'm pretty happy with it. The private pages are something I'm still thinking about. I went over how I wanted to do them many times, most importantly I needed to key generated to always be the same, obviously. I ultimately just settled on setting a key in the YAML and combining it with a "base key" before base64 encoding it. It's one I'll probably keep mulling over.

Adding mycomarkup was also high on my list of things I wanted. I've been using mycorrhiza for my note taking and knowledge base for a couple of months and I really love the mycomarkup language. Mycomarkup is actually a Go project as well so adding support for it was quite easy since I could just import it :3 I'm fairly sure there's some specific CSS styling I'll need to add later but for most things it's working perfectly well. So goldfinch can use both regular markdown and mycomarkup at the same time, this post is being written in mycomarkup while all the older ones in markdown!

Another thing I initially wanted was for it to automatically regenerate the database if any pages were changed. This is another thing I ended up settling on a different solution for. I thought over things like checksumming every page and periodically checking them all but even after doing that it would mean doing a bit more than updating the content field of pages depending on exactly what changes were made. I ended up just making a /rebuild route that scraped the whole database and generates it from scratch. My website, nor the other two I'm using goldfinch for, update very frequently so this isn't much a problem anyway and it's still much easier than manually generating and syncing HTML files, even if it was scripted.

Goldfinch supports building multiple RSS feeds based on a directory, which I'll probably make use of on another site. It also builds feeds that contain the entire post content, another thing I always meant to make the old generator do. Love full content RSS~

When adding the files to the database I decided to do the conversion beforehand and save the actual HTML output to the database so it wouldn't be constantly converting every time a page was visited, seemed like it made the most sense given that's the bulk of generation. The template output is still built on demand but I felt like pre-converting the main content would be more efficient.

I was curious how it would do with a large website so I did an unscientific test. I generated a lorem ipsum with markdown from this very nice site and made 100 files, then copied them into 50 folders. So, 5000 files in total but since they're split up into folders, that's also 50 extra entries for index page routes, plus some random other files that were there from regular testing. I ran goldfinch and timed it, forty seconds to generate 5084 pages, not bad! Of course in reality I don't ever expect to get anywhere near that and did another test with just ten folders, 1000 files, ish. It took four seconds. I don't think I'm going to be running into any problems regenerating the database any time soon.

The above test was done on my own PC, I did the same 5000 file test on my Hetzner VPS, the lowest tier ARM server, which is currently called CAX11. It has two vCPUs and 2GB of RAM. It finished in 25 seconds. Honestly surprised by that.

I'm going to keep working on it, I think there's definitely a lot of improvements to be made but I think it's done feature-wise, and I had fun with it so that's a plus :D

You can find the source in my fossil repo (a topic of another blog post perhaps) or the git mirror on codeberg if you prefer.

Links

• The Little Go Book

• Let's Go

• mycomarkup

• mycorrhiza

• Markdown lorem ipsum generator

• goldfinch

  • fossil repo

  • git repo