The case for self-hosting VCS
For game development, I like hosting my own source code repositories. The reason? They get big really fast. We’re hardly making AAA assets but even so, things adds up very quickly unless you’re doing something low-fi.
Big files mean increased storage costs, and slower network transfer speeds if you use remote hosted solutions. If you self-host, storage is much cheaper to add compared to, for example, $5/month per 50GB on GitHub, and you can locate your server closer to your work machines to speed up data transfers. As a small team we host ours on our local network, since that’s measured in Gb/s rather than Mb/s.
What I had before
Up until recently, my solution was a plain Git repo with fileshare-based LFS store. This worked great for quite a while, but this year we switched to using Unreal Engine 4, one of the consequences of which is having to manage a lot more binary files, and a subsequent increase in the risk of creating accidental unmergeable changes by people editing something in parallel.
That meant that we really needed locking support - the ability to exclusively reserve edit access to a certain file. This is something older version control systems used to do for everything, before merging became more reliable, but binary files remain inherently unmergeable. Unreal has built-in support for locking / unlocking files during editing, so ideally we wanted to support that workflow.
My plain Git setup couldn’t support the locking API, so I needed something else. I wasn’t keen on installing what I perceived to be “heavyweight” server software like GitLab; it seemed overkill and I didn’t want to have to maintain something complicated.
Evaluating the Options
Option 1: Perforce
Perforce is the default option in Unreal and a bunch of the automation tools are written with it in mind. I’ve only used Perforce very briefly, and I didn’t like it very much; to me it felt quite old fashioned. Also, I don’t like tying my project repositories to a commercial license, even if it’s free up to a certain number of users. I figure they can change the conditions any time they want, and I just prefer my repos to be in an open format. Perforce is almost certainly the way to go if you’re a big professional studio, but that’s not us.
(The same principle applies to PlasticSCM, which potentially solves the ‘old fashioned’ part but not the ‘you need an active license to proprietary software to access your data’ part)
Option 2: Subversion
Unreal also supports Subversion, and locking works fine with it. I spent about a month going down this route and almost used it for full production.
But holy cow, Subversion feels SO antiquated these days, I’d forgotten in the last 11/12 years just how much DVCSs had improved things. There’s how slow it is, and how even simple things like ignoring files is incredibly clunky (properties attached to folders that have to pre-exist, really?)
I wrote a bunch of scripts to deal with its nonsense and really didn’t like it. I was this close to using it in production and feeling pretty grumpy about that. Then…
Option 3: Gitea
Then I discovered that Gitea, a lightweight Go-based server for Git supported the Git LFS locking API. I had no idea locking had been adopted so well - when I was working on Git LFS it was in its nascent stage. I guess I lost track of time and now it seems it’s quite well implemented.
I like Go a lot, I like how it promotes simplicity and removes the need to run separate web servers with lots of moving parts, language plugins, app containers, all that clutter - within the scale I care about anyway. And it’s not the sprawling rats nest of Node.js (sorry Node fans).
I tried it out and guess what: it works really well. It’s about as simple as it can get beyond plain Git hosting while still supporting all the things you expect from a personal GitHub. I was impressed.
Note: Unreal Git Locking Support
The standard Git plugin in UE4 doesn’t support LFS locking. Luckily, there’s a Git LFS 2 plugin in active development which does. I’ll talk about that later.
I’m going to quickly describe my setup here; this won’t be a step-by-step, because honestly it’s not that complicated (that’s why I like it!) and the official documentation for both “ends” covers almost everything.
Instead, I’ll refer you to the major moving parts, and point out the important details and caveats I personally found along the way which should help you avoid some bumps in the road.
Install with Docker
The easiest way to get up and running quickly is to use Docker.
docker-compose to configure everything - this makes it
really easy to create test containers if you want and to version control your
The Gitea documentation describes a number of Docker setups, I personally started with the “Basics” setup which uses SQLite3 as the database, which is a great way to just play with it. Later on I switched to the MySQL setup for use by both of us. My docker-compose.yml looks basically exactly the same as the docs linked above, but with different passwords 😉 It’s really quite ridiculously simple.
Best of all there’s only 2 things running on the server; the gitea binary (which is both the web and app server), and when you’re fully deployed, the mysql binaries - when testing with SQLite there’s not even that. I love the simplicity, I really hate complex web / app server setups.
Data and configuration are exposed from the Docker container so they’re persistent, and we’ll cover backups later in this post.
The Install Wizard and Admin User
On first access you’ll be redirected to the install wizard, which is one page and very straight forward.
After that you’re in, and the first user you register will be the admin. Everything works pretty much just like GitHub, so you know how everything else works. It’s really that simple!
Installing the Git LFS 2 Plugin
Right now you need to include the Git LFS 2 plugin inside your UE4 project, under Plugins/UE4GitPlugin.
Personally I do this by cloning as a submodule, because I do work on it sometimes (I’ve submitted a few PRs) but you can just download the ZIP and put it there.
After installing the plugin and restarting UE4, you use the “Connect To Source Control” option in the toolbar and select “Git LFS 2”, as shown in the plugin docs.
You configure it much like the normal Git plugin, except right now you need to also tell it your server-side username, because that’s how it determines which locks are yours right now (I will be submitting a PR for that later to make it unnecessary, but go with it).
My custom Git LFS 2 plugin
I’ve been submitting PRs to the LFS 2 plugin but not all of them have been merged yet. If you’d like to use the version I use right now, see my fork on GitHub.
Important DefaultEngine.ini change
Because UE4 is tuned to Perforce style access patterns, saving will be very slow at first using the LFS locking plugin. You need to add this to your DefaultEngine.ini to fix that:
[SystemSettingsEditor] r.Editor.SkipSourceControlCheckForEditablePackages = 1
Set up Git LFS Lockable flags
We now need to tell Git LFS which files in our project are lockable. This will cause them to be made read-only on checkout, which will help prevent accidental edits to files we haven’t locked yet. This is also what will trigger UE to prompt for “Check Out” (what old man VCS’s used to call “lock” 😉) when you try to edit a file you don’t have a lock on yet.
The git-lfs locking documentation covers this process in full detail, but for example the main ones we’ll want to do for UE4 are:
git lfs track "*.uasset" --lockable git lfs track "*.umap" --lockable
You’ll need to commit/push your
.gitattributes file after making these changes,
and everyone should check out fresh copies of the repo to make sure all the
attributes are set correctly (although see my helper scripts later for another
Ready to go!
You’re done on both ends now; inside the UE4 editor when you try to save an asset, you’ll get a prompt asking you to “Check Out” (lock) the file. Files that you have locked are shown with a red check mark:
If someone else has locked a file, the checkmark is blue (and you can see details by hovering your mouse pointer), and you won’t be allowed to save the file yourself by default. Overriding this is a bad idea because it’s likely to create unmergeable changes.
You generally want to release your lock on files when you push your changes.
If you push inside UE4 this happens automatically. The “Submit To Source Control” option both commits and pushes and unlocks files. The “Push” command from inside UE4 also unlocks, should the push fail and you need to repeat after pulling.
I’ve also created some command line tools for easy lock state management outside of UE4.
My fork of the Git LFS 2 plugin has an option to ONLY commit on “Submit” rather than pushing too. I find this useful because I prefer to push manually. I submitted it as a PR but it’s not merged at time of writing. Pushing in UE4 unlocks, and so does my push-and-unlock script in the command line tools.
Extra Useful Information
While locking can prevent accidental parallel changes that aren’t mergeable, there are caveats. Due to Git’s inherently parallel nature, 2 people (or even 1 person) could still create a conflict without ever having the files locked at the same time - they could simply have locked the file at different times, but when their local versions were at different states.
For example, I lock a binary file for 10 minutes at version X, then commit, push and release my lock, creating version Y. Anyone checking the file after will show it as available to be locked. But unless that other person pulls my changes before locking it, they’ll still have version X. If they lock and make a change based on that, they’ll still create an umergeable conflict, even though we never had the file locked at the same time.
For this reason, the UE4 plugin only allows you to take out new locks if you’re at the “HEAD” revision of the upstream remote branch that you’re tracking. If there are newer commits which update that file, you can’t lock it, forcing you to pull the other person’s changes first and resolving the problem. It looks like this:
This has 2 unfortunate side effects though:
- If you create a new branch that you haven’t pushed, there is no upstream branch yet and you can’t lock anything.
- If you have multiple branches, it only checks the upstream of your current branch, not any other branches which you might want to merge. So you could still end up with conflicts that way.
The way I deal with this is that we only make .uasset / .umap changes on the main git branch. I only use feature branches for C++, or things I know are completely owned by me. This makes sense anyway, since running multiple branches making parallel changes to binary files is a really bad idea. But it does require a little extra thought if you’re used to ‘branchy’ workflows.
There are 2 ways you can view all the current file locks on your repository.
- On the command line:
git lfs locks
- In Gitea:
- Repo > Settings > LFS > Locks
Both allow you to force unlocking if you want, but be careful about doing this. You could cause someone to lose their changes, and read-only attributes can get out of sync if you unlock from somewhere other than the working copy the file was locked in.
Command Line Tools
Git clients have mostly not caught up with Git LFS locking yet, and while unlocking files when you Submit/Push in UE4 is easy, sometimes you might push from elsewhere, either because you have to merge some other changes, or you’re committing non-UE files as well.
There can sometimes be other issues, like you accidentally clicked “Make Writeable” instead of “Check Out” when you edited a file in UE, or the attributes got out of sync some other way.
I’ve created a few command line tools to help address these cases:
- Push and unlock
- Unlock files you don’t have changes for
- Fix read-only attributes based on lock state
You can find all these scripts in my GitScripts repo on GitHub.
Gitea Data Backups
The Gitea docs are a bit vague about backups. There’s a
gitea dump command
which works well for database and config but it doesn’t include Git LFS data.
It makes sense because LFS data is both large and immutable (each file is named by its hash),
so there’s no point zipping snapshots of it, but it’s important to realise.
I use a combination of
gitea dump and rsync for backups - the former backs up
the DB and Git repos to zips, the latter syncs LFS data. The backup data goes to both a second
server for local recovery and to Backblaze for offsite backups (encrypted).
Obviously I scripted the recovery process. Given a snapshot, my restore scripts can re-build the entire Docker container from scratch. I used it to move my original install to a completely separate server, so that was a good test of it.
My scripts are fairly specific to my own needs and so not perfectly re-usable, but since they deal with the nuts and bolts pretty well I’ve published them in case they help someone else.
What Server Do I Need?
The great thing is all this runs on pretty much any server you want; anywhere Docker runs really (and you could run it without Docker, it’s just more faff)
I run Gitea on a small Synology NAS; the only concession is that it’s one of the “+” editions (DS218+) so that it has an Intel processor; that just makes Docker easier to install. You can build Docker for other CPUs though.
I have 2 of these little boxes; they each run different things in my office and act as hot backups for each other, in case something non-RAID related fails, like a PSU or mainboard. In my experience even the smallest DS2xx+’s are plenty powerful enough to run this with spare headroom for a small team.
I hope this post has been useful and / or interesting. And yes, I know I haven’t done part 2 of the UE4 conversion post yet; I’ve parked that for now and will probably do it more as a retrospective on our current project; I think that will be more useful than more of the same as part 1 (the themes are mostly the same)