Deploying A Jekyll Static Site with Circle CI

May 31, 2018

One of the primary steps in making each iteration of this site happen is deploying its generated HTML files to my Digital Ocean server. Since these are just static files and there isn’t a CMS backing them up (in a traditional sense), there needs to be an automatic process that takes care of it after I make a change. If I had to manually push or build the site every time I added something, I’d:

  1. never do it
  2. go back to a CMS

This is where Circle CI enters the picture. Used mainly for software development, Circle CI allows those who are more inclined in the software development realm to build, test, and deploy code. If it can run on a Linux command line, Circle CI can run it.

For the grand starting price of zero and the promise of keeping code open source, I’m offered up to four concurrent builds and 25 hours of build time. As the operator of a static site powered by Jekyll, this is way more than I’ll need, but I’m glad it’s there.

Goals

  • Break down how I make this site happen with Circle CI.
  • We’ll take a look at my Circle CI config file and go over my workflow a bit.

It used to be way more complicated before I wrote this post but as I was thinking about what to write, I realized I was doing way more work than I needed to in the build process.

If you’d like to follow along, this entire site is available to browse through it in GitHub repo form and the Circle CI config file is here.

Table of Contents

The Circle CI File

Starting off first, let’s take a look at the defaults I have set:

defaults: &defaults
  docker:
    - image: circleci/ruby:2.5.1-node-browsers
  working_directory: ~/repo

What you’re looking at here are values that I’ll always need, no matter how many steps, jobs, etc. I’ll end up with. Since I have only one job, this is more of my “do not touch” section in that these values will never change and only new ones will be added. (You can find a more in-depth explanation of the purpose of defaults here.)

version: 2
jobs:
  the_only_job:
    <<: *defaults

Now we’re entering job territory. This is where I specify the actual tasks I need Circle CI to run. I only have one job, now–the_only_job–but if I had more, they’d be broken down like this:

version: 2
jobs:
  the_only_job:
    <<: *defaults
  except_its_not:
    <<: *defaults
  a_third_job:
    <<: *defaults

Each job would call upon the defaults because Circle CI treats each job as a separate build and would need its own container. In multi-job scenarios, having a set of defaults to share across all jobs is truly a no-brainer.

Deployment

Inside our job, we have a set of steps:.

Note: This is a list of tasks that should be performed by the container. Everything in this section is in the context of:

jobs:
  the_only_job:
    <<: *defaults
    steps:

So never mind the lack of full indentation. It saves me from repeating lines a dozen times.

Pre-game Tasks

- add_ssh_keys:
    fingerprints:
      - "69:fe:2c:df:c8:34:c5:e6:3f:6e:18:64:43:97:58:02"

The very first thing I have the container do is add an SSH key using the add_ssh_keys step. I’ve provided Circle CI with a key to the production server as a specific deploy-only user. This adds the key to the container so it can connect to the server later without me needing to provide hardcoded credentials. Doing so would be a massive security breach as my Circle CI builds are open to the public.

- checkout

Once that’s good to go, I have Circle CI checkout the latest code from the master branch of johlym/johnathan.org. Simple enough.

- attach_workspace:
    at: ~/repo

The third step is attach_workspace. This was more relevant when I had multiple jobs but the idea here is that we’re creating a persistent and consistent location within the job container to do all our task work. In this case, I need to make it clear that we’ll be doing all our work in ~/repo from here on out.

Cache Handling, Part 1

- restore_cache:
    keys:
      - v1-bundle-{{ checksum "Gemfile.lock" }}-{{ checksum "package.json" }}

This part is important if there’s even a stretch goal of having a speedy build process. The restore_cache step looks for a cache file that we’ve already built (something we’ll do at the end) to save time with things like bundle and npm. Without this, we could spend a few minutes just installing Rubygems and Node modules. bleh.

The cache file uses MD5 hashes of the Gemfile.lock and package.json files combined. If those files never change, the MD5s won’t either, so this cache will remain valid. If I were to update a gem, for example, the cache would be invalid and Rubygems and Node modules would be installed.

One potential spot for improvement here is to break this out into two separate caches, but Circle CI doesn’t handle that well, so this’ll be fine.

Installations

- run: 
    name: Install Rubygems if necessary
    command: |
      bundle install --path vendor/bundle --jobs 4 --retry 3
- run: 
    name: Install Node modules if necessary 
    command: |
      cd ~/repo && npm install
- run: 
    name: Install Rsync
    command: |
      sudo apt install rsync

This part is pretty straight forward. I need to make sure all the required Rubygems, Node modules, and Rsync are installed. In this case, I’m making sure bundle puts everything in vendor/bundle (remember, this is relative to ~/repo since we declared that to be our workspace earlier) when it installs. The Node modules I don’t need to worry so much about. The package.json would be located in the ~/repo directory since that’s where the code was checked out to so we hop in there and get to it. It’ll plop its node_modules file at ~/repo/node_modules as a result. This is totally acceptable. Lastly, we install rsync via apt. Nothing special.

Site building

- run: 
    name: Build site
    command: |
      bundle exec jekyll build --profile --verbose --destination /home/circleci/repo/_site

For those who’ve worked with Jekyll before, this command shouldn’t come as a surprise. We’re asking Jekyll to build out the site and place it at ~/repo/_site. The --profile and --verbose flags are for CI output, only, in case there’s an error or my curiosity gets the better of me.

Site tweaking

- run:
    name: Install and run Gulp
    command: |
      cd ~/repo && npx gulp

When considering how I wanted to handle minification of HTML and JavaScript, I considered the jekyll-assets plugin, but decided against it because of the amount of overhead and work that would be required to implement it in my already moderately-sized site. This is where I decided to bring in Gulp, instead. I have a simple Gulpfile that’s set up to use a couple Gulp modules to minify all the HTML and local JavaScript. Over the 400-something pages I have, this saves me about 20% on the site size overall. Not too shabby.

You’ll notice we need to use npx here. For some reason, I was never able to get Gulp to run on its own… it would look for the gulp binary in strange places I could not control. npx allows me to run gulp wherever, so long as it can find the corresponding node_modules folder for reference. Brilliant, eh? Portable Gulp.

Server Push

- run: 
    name: Deploy to prod server if triggered via master branch change
    command: |
      if [ $CIRCLE_BRANCH = 'master' ]; then rsync -e "ssh -o StrictHostKeyChecking=no" -va --delete ~/repo/_site [email protected]:/var/www/johnathan.org/static; fi

This is pretty straight forward, as well, though it can look complicated to the untrained eye. What we’re doing is here is first checking if the branch this build is based off of is the master branch. We’ll find that value in the CIRCLE_BRANCH ENV variable. If it is not, we’ll skip this, but if it is, we’ll run rsync to push the contents of ~/repo/site over to the production Digital Ocean server. I’m using the IP here because of Cloudflare, though I have a TODO item to use a hostname instead.

Post-Deployment

For all intents and purposes, the deployment is done, but because of Cloudflare, we have one additional step to make sure everyone’s seeing the freshest code.

Cloudflare

(we’re still in the jobs context)

- run: 
    name: Bust Cloudflare cache if triggered via master branch change
    command: |
      if [ $CIRCLE_BRANCH = 'master' ]; then 
        curl -X POST "https://api.cloudflare.com/client/v4/zones/$CLOUDFLARE_ZONE_ID/purge_cache" \
        -H "X-Auth-Email: $CLOUDFLARE_API_EMAIL" \
        -H "X-Auth-Key: $CLOUDFLARE_API_KEY" \
        -H "Content-Type: application/json" \
        --data '{"purge_everything":true}'; 

Using the Cloudflare API, we’re submitting a POST request to dump the entire cache for the johnathan.org DNS zone. I’ve provided Circle CI with the necessary information as ENV variables and am calling upon them here. This keeps them safe and the job step functional.

I wouldn’t recommend this for high-volume sites, but because I have Cloudflare caching just about everything combined with the fact that I maybe do this a couple times a week, this feels like the right level of effort and precision.

Cache Handling, Part 2

- save_cache:
    key: v1-bundle-{{ checksum "Gemfile.lock" }}-{{ checksum "package.json" }}
    paths:
      - ~/repo/vendor/bundle
      - ~/repo/node_modules

Earlier, we called upon the generated cache. Here is where we create it if necessary. This’ll do the same check step before acting. If the cache file already exists with the same MD5s, we’ll skip creating it, but if it’s missing, we’ll build it out, making sure to capture everything from the ~/repo/vendor/bundle and ~/repo/node_modules folders we referenced with Rubygems and NPM.

Workflow Management

workflows:
  version: 2
  build_site:
    jobs:
      - the_only_job

I used to have multiple jobs running in a breakout-combine pattern, and this is leftover from that. Although I only have one job, now, I didn’t want to re-craft it to not use Workflows, so I just operate with a one-job Workflow instead. XD

Wrap Up

That about does it for my overview. This process is turning out to work very well for me and I’m glad I took the time to both develop it and explain it for posterity. Over time, it’ll morph, I’m sure, but right now this feels like a really good bass to work off of.

Thanks for taking the time to read this. Cheers!

more about:circle ci / jekyll / static sites

The Static Site Book

Wandering into uncharted territory in hopes of scratching a long-running itch.
May 29, 2018

Over the last few weeks, I’ve spent a not-insignificant amount of time thinking about how I wanted to convey all the information and tips I have to share about building a static site and making it good, making it something you want to maintain, and even going so far as to sprinkle a bit of enjoyment on top.

The best way I can think of to do this in a format that makes the most sense to me and how I communicate, is via text and images in a book. Over the coming weeks, I’ll take a look at what it’ll take to self-publish a book and how I can best offer it to my visitors.

Here’s what I know I want to include for sure:

  • An overview of the concept of static sites
  • Jekyll 101
  • Reviewing common tools
  • More advanced topics like SEO, caching, and asset management
  • A few example sites and how to build them from start to finish using what we’ve just learned.

As far as a release date, I can only imagine it’ll be somewhere between the end of 2018 and all of 2019. I’ve never done this before so it’ll likely take a lot longer than more seasoned writers.

The working title is simply: The Static Site Book.

If there’s anything in particular you’d like to see, please let me know. I’d love to have the community’s help in making this happen.

A Week of Cloudflare Argo

There's fast... and then there's Argo.
May 21, 2018

I recently made a pretty heavy shift over to Cloudflare. The majority of assets and HTML on this site load from Cloudflare’s servers now, instead of my own. The only time this differs is when I push changes up to its GitHub repo. The last step in the build process after deploying to my server is hitting the API and requesting a complete cache dump. I could be more programmatic about it but updates aren’t frequent enough to warrant a more careful approach.

On its own, Cloudflare’s caching and CDN is great, but there’s another piece that makes it even better. Argo is Cloudflare’s smart routing feature that makes real-time choices about the best path to take from the Cloudflare POP (point of presence) to the origin (my server). This can dramatically reduce requests that do have to make their way to my San Francisco-located server.

It’s $5 a month plus $.10 per GB. Since most of the content on this site is text and a smattering of images, I’m not concerned with the price and $5 is fun money, basically. I saved that moving from DNSimple to Cloudflare for DNS, too, so there’s that.

Looking at some of the stats, now…

As reported by Updown.io (the service that powers status.johnathan.org and the uptime percentage at the bottom of the page), the majority of my requests come in in no time at all, with the obvious winner being Los Angeles (closest to San Francisco, my DigitalOcean location). The outlier is France, though I’m not too concerned with it. It seems to be fluctuating.

(Update May 24, 2018: I noticed response times from Updown.io had dropped to sub 100ms averages thanks to France falling in line with the rest of the countries. The only outlier at this point is Sydney’s lookup taking 3-9x longer) than the other locations.)

Moving over to Pingdom

It’s pretty obvious when Argo was enabled. I’m not sure what happened with the spike half-way between the 16th and 17th, but a response time being cut in half is amazing, even after doing zero work on the server, itself.

Lastly, the Cloudflare stats…

We see a clear difference in their metrics of response time improvements. Including the entire TLS handshake process, these sub 200ms response times in most cases is wonderful. At scale, Argo would have the potential to be mind-numbingly fast.

I’ll continue to use Argo for the foreseeable future. Cloudflare is free and works great for hobbyists like myself. Argo is $5 + $0.10/GB and I’d consider that peanuts. If it’s of any help, I’ve sent maybe 50MB through it since I signed up a week ago (remember, mostly text and a few images).

more about:Argo / Cloudflare / Networking

Hi friend. What's your email address?

...or follow the RSS feed.