The case for self-hosting VCS
For game development, I like hosting my own source code repositories. The reason? They get big really fast. We’re hardly making AAA assets but even so, things adds up very quickly unless you’re doing something low-fi.
Big files mean increased storage costs, and slower network transfer speeds if you use remote hosted solutions. If you self-host, storage is much cheaper to add compared to, for example, $5/month per 50GB on GitHub, and you can locate your server closer to your work machines to speed up data transfers. As a small team we host ours on our local network, since that’s measured in Gb/s rather than Mb/s.
What I had before
Up until recently, my solution was a plain Git repo with fileshare-based LFS store. This worked great for quite a while, but this year we switched to using Unreal Engine, one of the consequences of which is having to manage a lot more binary files, and a subsequent increase in the risk of creating accidental unmergeable changes by people editing something in parallel.
That meant that we really needed locking support - the ability to exclusively reserve edit access to a certain file. This is something older version control systems used to do for everything, before merging became more reliable, but binary files remain inherently unmergeable. Unreal has built-in support for locking / unlocking files during editing, so ideally we wanted to support that workflow.
My plain Git setup couldn’t support the locking API, so I needed something else. I wasn’t keen on installing what I perceived to be “heavyweight” server software like GitLab; it seemed overkill and I didn’t want to have to maintain something complicated.
Evaluating the Options
Option 1: Perforce
Perforce is the default option in Unreal and a bunch of the automation tools are written with it in mind. I’ve only used Perforce very briefly, and I didn’t like it very much; to me it felt quite old fashioned. Also, I don’t like tying my project repositories to a commercial license, even if it’s free up to a certain number of users. I figure they can change the conditions any time they want, and I just prefer my repos to be in an open format. Perforce is almost certainly the way to go if you’re a big professional studio, but that’s not us.
(The same principle applies to PlasticSCM, which potentially solves the ‘old fashioned’ part but not the ‘you need an active license to proprietary software to access your data’ part)
Option 2: Subversion
Unreal also supports Subversion, and locking works fine with it. I spent about a month going down this route and almost used it for full production.
But holy cow, Subversion feels SO antiquated these days, I’d forgotten in the last 11/12 years just how much DVCSs had improved things. There’s how slow it is, and how even simple things like ignoring files is incredibly clunky (properties attached to folders that have to pre-exist, really?)
I wrote a bunch of scripts to deal with its nonsense and really didn’t like it. I was this close to using it in production and feeling pretty grumpy about that. Then…
Option 3: Gitea
Then I discovered that Gitea, a lightweight Go-based server for Git supported the Git LFS locking API. I had no idea locking had been adopted so well - when I was working on Git LFS it was in its nascent stage. I guess I lost track of time and now it seems it’s quite well implemented.
I like Go a lot, I like how it promotes simplicity and removes the need to run separate web servers with lots of moving parts, language plugins, app containers, all that clutter - within the scale I care about anyway. And it’s not the sprawling rats nest of Node.js (sorry Node fans).
I tried it out and guess what: it works really well. It’s about as simple as it can get beyond plain Git hosting while still supporting all the things you expect from a personal GitHub. I was impressed.
Note: Unreal Git Locking Support
The standard Git plugin in UE4 doesn’t support LFS locking. Luckily, there’s a Git LFS 2 plugin in active development which does. I’ll talk about that later.
The Setup
I’m going to quickly describe my setup here; this won’t be a step-by-step, because honestly it’s not that complicated (that’s why I like it!) and the official documentation for both “ends” covers almost everything.
Instead, I’ll refer you to the major moving parts, and point out the important details and caveats I personally found along the way which should help you avoid some bumps in the road.
Gitea Install
Install with Docker
The easiest way to get up and running quickly is to use Docker.
Specifically, use docker-compose to configure everything - this makes it
really easy to create test containers if you want and to version control your
setup configuration.
The Gitea documentation describes a number of Docker setups, I personally started with the “Basics” setup which uses SQLite3 as the database, which is a great way to just play with it. Later on I switched to the MySQL setup for use by both of us. My docker-compose.yml looks basically exactly the same as the docs linked above, but with different passwords 😉 It’s really quite ridiculously simple.
Best of all there’s only 2 things running on the server; the gitea binary (which is both the web and app server), and when you’re fully deployed, the mysql binaries - when testing with SQLite there’s not even that. I love the simplicity, I really hate complex web / app server setups.
Data and configuration are exposed from the Docker container so they’re persistent, and we’ll cover backups later in this post.
The Install Wizard and Admin User
On first access you’ll be redirected to the install wizard, which is one page and very straight forward.
After that you’re in, and the first user you register will be the admin. Everything works pretty much just like GitHub, so you know how everything else works. It’s really that simple!
Unreal Engine Setup
Updated September 2022: I’ve changed my recommendation for the UE Git plugin to the “ProjectBorealis” fork, which is more up to date and UE5-compatible.
Installing the Git LFS 2 Plugin
Right now you need to include the Git LFS 2 plugin inside your Unreal Engine project, under Plugins/UEGitPlugin.
Personally I do this by cloning as a submodule, but you can just download the ZIP and put it there if you prefer.
After installing the plugin and restarting the editor, you use the “Connect To Source Control” option in the toolbar and select “Git LFS 2”, as shown in the plugin docs.
Important DefaultEngine.ini change
Because Unreal is tuned to Perforce style access patterns, saving will be very slow at first using the LFS locking plugin. You need to add this to your DefaultEngine.ini to fix that:
[SystemSettingsEditor]
r.Editor.SkipSourceControlCheckForEditablePackages = 1
Set up Git LFS Lockable flags
We now need to tell Git LFS which files in our project are lockable. This will cause them to be made read-only on checkout, which will help prevent accidental edits to files we haven’t locked yet. This is also what will trigger UE to prompt for “Check Out” (what old man VCS’s used to call “lock” 😉) when you try to edit a file you don’t have a lock on yet.
The git-lfs locking documentation covers this process in full detail, but for example the main ones we’ll want to do for Unreal are:
git lfs track "*.uasset" --lockable
git lfs track "*.umap" --lockable
You’ll need to commit/push your .gitattributes file after making these changes,
and everyone should check out fresh copies of the repo to make sure all the
attributes are set correctly (although see my helper scripts later for another
way).
Ready to go!
Locking files
You’re done on both ends now; inside the Unreal editor when you try to save an asset, you’ll get a prompt asking you to “Check Out” (lock) the file. Files that you have locked are shown with a red check mark:
Locked File
If someone else has locked a file, the checkmark is blue (and you can see details by hovering your mouse pointer), and you won’t be allowed to save the file yourself by default. Overriding this is a bad idea because it’s likely to create unmergeable changes.
Unlocking files
You generally want to release your lock on files when you push your changes.
If you push inside Unreal this happens automatically. The “Submit To Source Control” option both commits and pushes and unlocks files. The “Push” command from inside Unreal also unlocks, should the push fail and you need to repeat after pulling.
I’ve also created some command line tools for easy lock state management outside of Unreal. I use these most of the time rather than the editor versions.
Extra Useful Information
Lock Limitations
While locking can prevent accidental parallel changes that aren’t mergeable, there are caveats. Due to Git’s inherently parallel nature, 2 people (or even 1 person) could still create a conflict without ever having the files locked at the same time - they could simply have locked the file at different times, but when their local versions were at different states.
For example, I lock a binary file for 10 minutes at version X, then commit, push and release my lock, creating version Y. Anyone checking the file after will show it as available to be locked. But unless that other person pulls my changes before locking it, they’ll still have version X. If they lock and make a change based on that, they’ll still create an umergeable conflict, even though we never had the file locked at the same time.
For this reason, the Unreal plugin only allows you to take out new locks if you’re at the “HEAD” revision of the upstream remote branch that you’re tracking. If there are newer commits which update that file, you can’t lock it, forcing you to pull the other person’s changes first and resolving the problem. It looks like this:
Lock error when you need to pull
Status Branches
By default, the plugin only checks the upstream of your current branch. You’re unlikely to have a lot of active branches in UE development because merging binary files isn’t possible, but if you do use some, for example maintaining separate release branches, there’s a way to handle this. Epic calls them “Status Branches”, but it’s really a fairly traditional approach of maintaining a chain of branches, with more stable ones merging into less stable ones, for example:
- A bugfix branch for the public release
- A “Next release” branch where things stablise for the next major release
- A “Development” branch where everything else for beyond that goes
You merge only downwards in this list. If you ever need to move something backwards, you cherry-pick it.
If you maintain branches like this then you need to tell the plugin how to check all of them for outdated versions before you can lock, and not just the current branch. See the Git Plugin Status Branch Documentation for details.
At our team size we only make .uasset / .umap changes on the main git branch. I only use feature branches for C++, or things I know are completely owned by me, it just keeps things simple. I use a far less “branchy” model of development than I do for non-Unreal projects.
Examining locks
There are 2 ways you can view all the current file locks on your repository.
- On the command line:
git lfs locks
- In Gitea:
- Repo > Settings > LFS > Locks
Both allow you to force unlocking if you want, but be careful about doing this. You could cause someone to lose their changes, and read-only attributes can get out of sync if you unlock from somewhere other than the working copy the file was locked in.
Command Line Tools
Git clients have mostly not caught up with Git LFS locking yet, and while unlocking files when you Submit/Push in Unreal is easy, sometimes you might push from elsewhere, either because you have to merge some other changes, or you’re committing non-UE files as well.
There can sometimes be other issues, like you accidentally clicked “Make Writeable” instead of “Check Out” when you edited a file in UE, or the attributes got out of sync some other way.
I’ve created a few command line tools to help address these cases:
- Push and unlock
- Unlock files you don’t have changes for
- Fix read-only attributes based on lock state
You can find all these scripts in my GitScripts repo on GitHub.
Gitea Data Backups
The Gitea docs are a bit vague about backups. There’s a gitea dump command
which works well for database and config but it doesn’t include Git LFS data.
It makes sense because LFS data is both large and immutable (each file is named by its hash),
so there’s no point zipping snapshots of it, but it’s important to realise.
I use a combination of gitea dump and rsync for backups - the former backs up
the DB and Git repos to zips, the latter syncs LFS data. The backup data goes to both a second
server for local recovery and to Backblaze for offsite backups (encrypted).
Obviously I scripted the recovery process. Given a snapshot, my restore scripts can re-build the entire Docker container from scratch. I used it to move my original install to a completely separate server, so that was a good test of it.
My scripts are fairly specific to my own needs and so not perfectly re-usable, but since they deal with the nuts and bolts pretty well I’ve published them in case they help someone else.
What Server Do I Need?
The great thing is all this runs on pretty much any server you want; anywhere Docker runs really (and you could run it without Docker, it’s just more faff)
I run Gitea on a small Synology NAS; the only concession is that it’s one of the “+” editions (DS218+) so that it has an Intel processor; that just makes Docker easier to install. You can build Docker for other CPUs though.
I have 2 of these little boxes; they each run different things in my office and act as hot backups for each other, in case something non-RAID related fails, like a PSU or mainboard. In my experience even the smallest DS2xx+’s are plenty powerful enough to run this with spare headroom for a small team.
Fin
I hope this post has been useful and / or interesting!