Automated deployment of Hugo blogs with Bitbucket Pipelines & rsync

· by Steve · Read in about 8 min · (1495 Words)
development web

I’ve waxed lyrical before about how much I like Hugo for blogging; the ability to just use a static site with no need to worry about security patches, database connections etc, but still with the convenience of a simple blogging platform, is very attractive.

However, it does mean you can’t easily write or tweak content from simpler environments like your phone if you notice a typo, since you need a full Hugo build environment to change content.

Also if you switch between Windows and Mac like I do, the line endings in the generated content will be different on each platform, meaning that if you use rsync to upload the changes, you’ll end up uploading more than you need to because the checksums won’t match simply due to those line ending changes. It doesn’t make that much difference, but it’s not ideal.

Enter Bitbucket Pipelines

Bitbucket Pipelines is a Continuous Integration tool which automatically runs scripts against your git repository when you push new commits. The typical use for these tools is running automated tests, and deploying build results.

So, we can make Bitbucket Pipelines build our Hugo blog for us whenever it receives a new commit, and deploy the result to our server. Then we can edit our blog in much simpler environments, such as using a mobile Git client like Working Copy on iOS or just editing files directly on Bitbucket from any device using the web-based file editor. Very useful to fix those typos you only notice when checking on your phone!

Enabling Pipelines on your repo

The first step is to simply click on “Pipelines” in the navigation bar on your repo in Bitbucket, then click “Enable Pipelines”.

The next thing you need is a new file in your repo root called bitbucket-pipelines.yml. This file contains all the instructions required to run your build steps on receiving a new commit. Before we can do that though, we need to think about how Pipelines is going to get the software it needs.

A brief Docker aside

Like many CI tools, Pipelines runs Docker containers, which are like lightweight virtual machines which run a standard OS image which can be configured to include just the software you need.

We will need hugo, rsync and ssh to run our build. I’ve already created a Docker image containing these things so you don’t have to; to use it you simply put this at the top of your bitbucket-pipelines.yml file:

image: sjstreeting/pipelines-hugo

Per-branch build steps

If I’m working on a phone and therefore don’t have access to a full Hugo build/test environment, I’m going to want to be able to preview my changes on a test site.

In order to do this, I set up my Pipelines configuration so that only the master branch deploys to the final live blog location. All other branches will deploy to a test location instead, and will include draft and future dated posts.

To do that, the bitbucket-pipelines.yml file looks like this:

image: sjstreeting/pipelines-hugo

pipelines:
  # Master branch deployment to live - no drafts
  branches:
    master:
      - step:
          script:
            - hugo
            - rsync -clvrz public/ $DEPLOY_URL
  # All other branches include drafts and deploy to test URL only
  default:
    - step:
        script:
          - hugo -D -F -b "$TEST_BASEURL"
          - rsync -clvrz -e public/ $TEST_DEPLOY_URL

Notice that there’s a specific pipeline for the master branch, which performs a default hugo build, and a pipeline for all other branches which includes drafts (-D), future posts (-F), and also changes the baseUrl setting using -b "$TEST_BASEURL" so that links will work when I upload it to a non-live URL.

Also notice how each rsync uploads to $DEPLOY_URL from the master branch and to $TEST_DEPLOY_URL from every other branch.

Also of note is the -c option to rsync; because the Hugo blog is going to be completely regenerated every time, we can’t use the date/time to upload only new content. Instead -c causes rsync to use a checksum to determine what to upload.

Environment variables

You’ve obviously noticed the variables in the script like $TEST_BASEURL and $DEPLOY_URL and will want to know where they get their values from.

They are references to environment variables, which you set per repository in Settings > Pipelines > Environment Variables:

Environment variables

It’s just useful not to have to hardcode URLs in your repo, especially when you’re initially setting it up and want to send all the results somewhere else for testing.

At this point you’re almost ready to commit & push your bitbucket-pipelines.yml, but we need to sort out one more thing; SSH access.

Configuring SSH

You’re going to need your Pipelines script to be able to connect to your server using an SSH key, so it doesn’t have to log in. There are several steps to this:

Generate a new key

It’s a good idea to use a new SSH key for your deployment, that way you can turn it on/off independently of any other SSH login.

  1. Go to Settings > Pipelines > SSH Keys
  2. Click Generate Keys
  3. Once generated, copy the public key
  4. On your server, paste it into the ~/.ssh/authorized_keys file of the user which you want the script to use to upload. Ideally this is a non-admin user but it depends on how flexible your hosting is.

Tell Bitbucket the fingerprint of your server

SSH clients will only connect to servers they recognise; unless you tell Bitbucket the fingerprint of your SSH server, it will reject the connection when the pipeline attempts it.

The easy way to do this, if your SSH server is running on the standard port:

  1. Go to Settings > Pipelines > SSH Keys
  2. Under “Known Hosts”, enter yourserver.com or whatever server you used for $DEPLOY_URL
  3. Click “Fetch”

This will grab the fingerprint of your server and authorise Bitbucket to talk to it over SSH.

However, if your SSH server is running on non-standard port this won’t work. See the next section for how to resolve this.

SSH servers on non-standard ports

If your SSH server is running on a port other than the standard port 22 (this is often a good idea since it avoids being a target of numerous brute-force attacks), you’ll have to make a few extra changes to your bitbucket-pipelines.yml file.

There are 2 things you need to do:

  1. Manually insert the key fingerprint in the pipeline script since the “Known Hosts” fetching tool in Bitbucket won’t find it
  2. Modify the rsync call so it knows to use a different port

Here’s an example file where yourserver.com is running SSH on port 8022:

image: sjstreeting/pipelines-hugo

pipelines:
  # Master branch deployment to live - no drafts
  branches:
    master:
      - step:
          script:
            - hugo
            - ssh-keyscan -p 8022 -t rsa yourserver.com >> ~/.ssh/known_hosts
            - rsync -clvrz -e "ssh -p 8022" public/ $DEPLOY_URL
  # All other branches include drafts and deploy to test URL only
  default:
    - step:
        script:
          - hugo -D -F -b "$TEST_BASEURL"
          - ssh-keyscan -p 8022 -t rsa yourserver.com >> ~/.ssh/known_hosts
          - rsync -clvrz -e "ssh -p 8022" public/ $TEST_DEPLOY_URL

The ssh-keyscan call reads the fingerprint and inserts it during the pipeline run, which is exactly what the more friendly Bitbucket admin version will do when using standard ports; the difference is we always do it when we run a pipeline, which is a bit less secure, since you never get to check the fingerprint ahead of time. Then again, I know few people who are diligent about this anyway 😉 If you’re paranoid, replace the hostname in the ssh-keyscan call with an IP instead, that way if DNS ever does get hijacked the fingerprint will fail to be the same as the one coming from the raw IP.

We’ve also modified the rsync call to include the option -e "ssh -p 8022" so that it knows to use SSH over a custom port.

Ready to go

Now you just need to commit & push your bitbucket-pipelines.yml. I recommend doing this from a non-master branch to begin with so you go through your test route in the first instance.

It’s also worth setting your environment variables so that $DEPLOY_URL is initially a non-live location, so you can prove everything works to your satisfaction even on master before using it for real. Once it looks good, you can change the environment variable and either push a new commit or re-run the latest pipeline to make it deploy to live.

If all goes well, you’ll see something like this on the main page of your Bitbucket repo:

Pipelines Success

You can always click this or on Pipelines in the repo sidebar to see the status of all your Pipelines that have run in the past.

That’s it! I hope this blog helps you bridge the gap between the simplicity of Hugo and the convenience of an automatically built website. Best of luck!

Big thanks to Karel Bemelmans whose earlier post heavily informed this one. I’ve updated & expanded on it because I wanted a general purpose rsync deployment over SSH instead of AWS.