When I was evaluating Mercurial, I touched briefly on Mercurial Queues (MQ), but at the time felt that they were probably a little more complicated than I needed. After all, one of the things I liked about Mercurial was that it all felt rather straightforward and intuitive, while MQ seemed downright confusing. However, I’ve just discovered a really powerful use case that has meant I’m now committed to using MQ for my every day workflow. But I’ll come back to that.

First, some explanation of why I revisited this. I’ve been working with Mercurial in a live situation now for a few weeks now, and my experience has been uniformly good. Particularly the main thing that used to be a total pain in my ass before – merges – has become infinitely less frustrating, and I even managed to use it to take a client’s highly modified version of Ogre which was forked from quite an old base, and bring it up to date with later versions via 3-way merges in a fraction of the time it would have taken me with Subversion’s relatively brute-force, roundtrip-generating merge system.

However, one thing that has come up is that some other team members have found the ‘uncommitted changes’ error message annoying – that is, there are some things that Mercurial won’t let you do if you have uncommitted changes; mainly merge and rebase, since both of these work on committing (or committed) changesets. Some of this confusion has been caused by me telling people to always use ‘hg pull –rebase’, to avoid unnecessary merge points when parallel development has happened, but the knock-on effect is that using the ‘–rebase’ option makes Mercurial stop if you haven’t committed your current work.

Resolving This Without Mercurial Queues

1. Finish what you’re working on and commit it before pulling new changes (with –rebase). In fact, most of the time this is just fine, unless for some reason you’re dependent on someone else’s work to finish yours. Some people got into the habit of pulling SVN changes every morning regardless of whether they’re in the middle of something or not, but actually you don’t need to, and in fact doing so can introduce unexpected and unwanted variables into your existing work. But, there are some times when you genuinely want to pull other people’s changes before you’ve finished, so..
2. Drop the ‘pull with rebase’ and use a regular ‘pull with update’ instead, ie “hg pull -u”. This is compatible with outstanding uncommitted changes – however if you’ve also got unpushed changesets of your own locally, bear in mind you’ll end up with multiple heads (branches) after you pull, and you’ll need to remember to call “hg rebase” directly (instead of as part of your pull) before you push next time to unify them.
3. Use “hg shelve” (extension) or simply ‘hg diff > somepatchfile’  to stash your uncommitted changes, then pull –rebase, then unshelve  (or import / patch) your changes in again. However, if you’re considering this option I urge you to try to use MQ instead.

Mercurial Queues – juggle many unfinished changes at once

The important thing to understand about MQ, which I didn’t grasp at first, is that it’s basically just designed to automate the process of using ‘hg diff’ to generate named patch files which you stash away somewhere as an unfinished change. I used to do this all the time in CVS and SVN when working on multiple things that weren’t quite ready to be committed yet – dump my current changes to a patch file, put it on a server or USB stick (so I can apply it on other machines & back it up), revert my changes then start work on something else, pulling the patch file back again when I want to re-start it. You can still use this manual approach with Mercurial of course, but this is also exactly what MQ does, it just has a bunch of explicit commands for it, and keeps track of which patches you have applied already in your local copy.

For example, ‘hg qnew mypatch’ creates a new patch and imports your current uncommitted changes to the patch (after which your ‘hg status’ shows as unchanged since hg is MQ-aware). It’s actually created a patch file of the changes in .hg/patches/mypatch. If you subsequently make changes, hg status shows the changes since you did the hg qnew, and you can update the patch to the latest set of changes by using ‘hg qrefresh’ – note how this just updates that patch file, it doesn’t create any changesets or anything.

If you want to go back to the status of the repo without this patch applied, just call ‘hg qpop’, or ‘hg qpush mypatch’ to bring it back. The reason for the ‘push’ and ‘pop’ concept is that each patch (which you create with qnew, you can call it at any time to spawn a new patch with just your current alterations) is considered to be in an ordered stack, with each one depending on the one underneath (and applying the patch at the top of the stack implies applying all those beneath it too). You can navigate around the stack whenever you like, qrefreshing individual patches (remember, they’re not changesets so they’re mutable – this is how you get around the fact that Mercurial’s history is not changeable), but you’re always in a stack. This is the one thing I’m not so fond of about MQ, I think patches should be in a list of stacks to allow you to have unrelated (and potentially breaking) sequences of patches tracked differently in parallel. The sharp-minded among you will note that git’s equivalent – which is to use local branches – effectively allows this by nature of the branch structure. If you’re really in need of this behaviour, you could try the LocalBranch extension or use local clones instead of MQ.

However, there’s something else I particularly like about MQ over the local branch approach, and that’s what happens when you want to synchronise your work-in-progress changes with other machines (or other people).

Repositories within repositories

When using local branches (in Git or via LocalBranch), you’re always working with regular committed changesets. They’re not pushed to others by default, but if you want to synchronise them you have ways to force them to push (or pull) – for example when you work across multiple machines. You might do this direct, or if the machines are not on at the same time you may do it via an intermediate server or USB stick. The main problems I have with this approach when related to work-in-progress changes are:

• Inherently these changesets are not finished, and you’re not going to publish them as-is elsewhere. In Git you can collapse them together later of course with the finished state, but it means that even locally you’re synchronising unstable state, which is fragile if you screw up
• If you need an intermediary for the sync (server or USB stick), it has to be a full repository clone even though you only need your WIP changes.

Of course, you could generate patch files yourself in both cases to remove both of these issues. But that’s why MQ is ideal, because it already does that for you, without you having to track external patch files. And best of all, it has a simple way to synchronise and version control just those mutable patches with others, without ever raising the issue of history modification: it simply creates a Mercurial repository within the .hg/patches directory! This repository only contains your MQ patches, and can be safely published, synchronised without ever looking like a proper ‘history’ of the main repository. Best of all, most of the regular hg commands take a ‘–mq’ option to perform actions on this nested patch repository instead of the main repository.

Putting it all together

Since Mercurial 1.5 you haven’t strictly needed to call ‘hg qinit’ to start using Mercurial Queues, but in order to create a nested repository for sync, you still want to call it, like this:

hg qinit -c

That will create a nested repository in .hg/patches which maintains its own history and can be synchronised with others, independently of the main repository. You might want to take the opportunity to edit .hg/patches/.hg/hgrc and provide a default remote path if like me you want to specify a server you will use to keep all your WIP patches for this repository.
So let’s say you’ve been doing some work on a change, but it’s not ready yet and you want to put a stake in the ground where you are without committing. Simple, you just do:

hg qnew awesome-feature-1 -m"I'm working on an awesome feature"

That says “create me a new patch in .hg/patches/awesome-feature-1 with my current changes, and give it an initial description” – you can change this description later whenever you want when you update the patch with qrefresh, see below. At this point hg considers that you have no local uncommitted changes anymore so you can happily do rebases and merges without the ‘outstanding uncommitted changes’ message. But, there’s more, you may want to do the following so you can sync the current state of this patch with other machines:

hg commit --mq -m"My awesome feature WIP #1"

Woah, woah, I said you didn’t want to commit, right, because this is a work in progress? Notice though, that I called commit with the ‘–mq’ option, which means ‘commit changes to the MQ patches repository’, and not ‘commit working copy changes’. If you look now, you’ll see that you have a patch file called .hg/patches/awesome-feature-1, which was automatically added to the nested patch repository when you called ‘qnew’, and which was committed to that repository in its current state (remember, this patch repository is completely separate from the main repo).

If I wanted to push the current state of my WIP patches to somewhere else, I can do it now (I’m assuming that I’ve already edited .hg/patches/.hg/hgrc to include a default remote path:

hg push --mq

That pushes the commits I’ve made to the patch queue (not the main repo) to that remote location. So for example I use a small area on a local server to track this, but there’s no reason I couldn’t push this somewhere public too if I wanted – again all explicitly separate from my actual repo, which is perfect. All the other main commands take a –mq option too, so you can use “hg outgoing –mq” etc too.

At this point, I could carry on working on my same patch, and every time I wanted to update the patch file in .hg/patches with my local changes I would call:

hg qrefresh

You can use the ‘-e’ command if you want to update the commit message that will eventually be used. And again, if I wanted to commit the changes to that patch, I’d call ‘hg commit –mq’ (and potentially ‘hg push –mq’). If at some point I wanted to quickly work on a different change, I could call ‘hg qnew’ again to create a new patch file based on the current diff, and deal with that separately. To navigate around the patch queue to identify which patch I’m currently working on, I’d use qpop and qpush.

Finally, at some point I’m going to want to turn these patches into real commits. Assuming that my commit message is already correct in the patch (if not, use ‘hg qrefresh -e’ to edit it), and that your current position on the patch stack is the position you want to create a proper commit for, all you need to do is:

hg qfinish --applied

Which will move the patches you currently have applied into the real Mercurial repository history, ready to be officially pushed with everyday work. Alternatively if you want to just pick a revision or range of revisions, you can name them instead of using –applied.

Conclusion

I didn’t really see the advantage of using MQ over just keeping my own patch files generated from my working copy via ‘hg diff’, particularly since MQ makes you work in a stack-based structure instead of via independent patches, until I learned how to use the nested patch repository. Manually handling patch files is fine, but once you get more than a couple of them on the go it can get tricky to manage. Having an automated approach to this with the nested repository works really very well, so I’m now using it as my primary method of managing work-in-progress.

Git users will no doubt think the local branch approach is better, and in at least one way (not having to keep patches in a single stack), it is – however I personally like how MQ mirrors my normal approach to unfinished changes as ‘volatile patches’ rather than treating unfinished changes as regular changesets and relying on history modification to fix them up later, and as someone who regularly moves between machines, MQ’s ability to sync patches separately from the main repository is very valuable. As is ever the case with Mercurial and Git, the perfect solution would probably be some combination of the two.