Mercurial queues just clicked

When I was evaluating Mercurial, I touched briefly on Mercurial Queues (MQ), but at the time felt that they were probably a little more complicated than I needed. After all, one of the things I liked about Mercurial was that it all felt rather straightforward and intuitive, while MQ seemed downright confusing. However, I’ve just discovered a really powerful use case that has meant I’m now committed to using MQ for my every day workflow. But I’ll come back to that.

First, some explanation of why I revisited this. I’ve been working with Mercurial in a live situation now for a few weeks now, and my experience has been uniformly good. Particularly the main thing that used to be a total pain in my ass before – merges – has become infinitely less frustrating, and I even managed to use it to take a client’s highly modified version of Ogre which was forked from quite an old base, and bring it up to date with later versions via 3-way merges in a fraction of the time it would have taken me with Subversion’s relatively brute-force, roundtrip-generating merge system.

However, one thing that has come up is that some other team members have found the ‘uncommitted changes’ error message annoying – that is, there are some things that Mercurial won’t let you do if you have uncommitted changes; mainly merge and rebase, since both of these work on committing (or committed) changesets. Some of this confusion has been caused by me telling people to always use ‘hg pull –rebase’, to avoid unnecessary merge points when parallel development has happened, but the knock-on effect is that using the ‘–rebase’ option makes Mercurial stop if you haven’t committed your current work.

Resolving This Without Mercurial Queues

  1. Finish what you’re working on and commit it before pulling new changes (with –rebase). In fact, most of the time this is just fine, unless for some reason you’re dependent on someone else’s work to finish yours. Some people got into the habit of pulling SVN changes every morning regardless of whether they’re in the middle of something or not, but actually you don’t need to, and in fact doing so can introduce unexpected and unwanted variables into your existing work. But, there are some times when you genuinely want to pull other people’s changes before you’ve finished, so..
  2. Drop the ‘pull with rebase’ and use a regular ‘pull with update’ instead, ie “hg pull -u”. This is compatible with outstanding uncommitted changes – however if you’ve also got unpushed changesets of your own locally, bear in mind you’ll end up with multiple heads (branches) after you pull, and you’ll need to remember to call “hg rebase” directly (instead of as part of your pull) before you push next time to unify them.
  3. Use “hg shelve” (extension) or simply ‘hg diff > somepatchfile’  to stash your uncommitted changes, then pull –rebase, then unshelve  (or import / patch) your changes in again. However, if you’re considering this option I urge you to try to use MQ instead.

Mercurial Queues – juggle many unfinished changes at once

The important thing to understand about MQ, which I didn’t grasp at first, is that it’s basically just designed to automate the process of using ‘hg diff’ to generate named patch files which you stash away somewhere as an unfinished change. I used to do this all the time in CVS and SVN when working on multiple things that weren’t quite ready to be committed yet – dump my current changes to a patch file, put it on a server or USB stick (so I can apply it on other machines & back it up), revert my changes then start work on something else, pulling the patch file back again when I want to re-start it. You can still use this manual approach with Mercurial of course, but this is also exactly what MQ does, it just has a bunch of explicit commands for it, and keeps track of which patches you have applied already in your local copy.

For example, ‘hg qnew mypatch’ creates a new patch and imports your current uncommitted changes to the patch (after which your ‘hg status’ shows as unchanged since hg is MQ-aware). It’s actually created a patch file of the changes in .hg/patches/mypatch. If you subsequently make changes, hg status shows the changes since you did the hg qnew, and you can update the patch to the latest set of changes by using ‘hg qrefresh’ – note how this just updates that patch file, it doesn’t create any changesets or anything.

If you want to go back to the status of the repo without this patch applied, just call ‘hg qpop’, or ‘hg qpush mypatch’ to bring it back. The reason for the ‘push’ and ‘pop’ concept is that each patch (which you create with qnew, you can call it at any time to spawn a new patch with just your current alterations) is considered to be in an ordered stack, with each one depending on the one underneath (and applying the patch at the top of the stack implies applying all those beneath it too). You can navigate around the stack whenever you like, qrefreshing individual patches (remember, they’re not changesets so they’re mutable – this is how you get around the fact that Mercurial’s history is not changeable), but you’re always in a stack. This is the one thing I’m not so fond of about MQ, I think patches should be in a list of stacks to allow you to have unrelated (and potentially breaking) sequences of patches tracked differently in parallel. The sharp-minded among you will note that git’s equivalent – which is to use local branches – effectively allows this by nature of the branch structure. If you’re really in need of this behaviour, you could try the LocalBranch extension or use local clones instead of MQ.

However, there’s something else I particularly like about MQ over the local branch approach, and that’s what happens when you want to synchronise your work-in-progress changes with other machines (or other people).

Repositories within repositories

When using local branches (in Git or via LocalBranch), you’re always working with regular committed changesets. They’re not pushed to others by default, but if you want to synchronise them you have ways to force them to push (or pull) – for example when you work across multiple machines. You might do this direct, or if the machines are not on at the same time you may do it via an intermediate server or USB stick. The main problems I have with this approach when related to work-in-progress changes are:

  • Inherently these changesets are not finished, and you’re not going to publish them as-is elsewhere. In Git you can collapse them together later of course with the finished state, but it means that even locally you’re synchronising unstable state, which is fragile if you screw up
  • If you need an intermediary for the sync (server or USB stick), it has to be a full repository clone even though you only need your WIP changes.

Of course, you could generate patch files yourself in both cases to remove both of these issues. But that’s why MQ is ideal, because it already does that for you, without you having to track external patch files. And best of all, it has a simple way to synchronise and version control just those mutable patches with others, without ever raising the issue of history modification: it simply creates a Mercurial repository within the .hg/patches directory! This repository only contains your MQ patches, and can be safely published, synchronised without ever looking like a proper ‘history’ of the main repository. Best of all, most of the regular hg commands take a ‘–mq’ option to perform actions on this nested patch repository instead of the main repository.

Putting it all together

Since Mercurial 1.5 you haven’t strictly needed to call ‘hg qinit’ to start using Mercurial Queues, but in order to create a nested repository for sync, you still want to call it, like this:

hg qinit -c

That will create a nested repository in .hg/patches which maintains its own history and can be synchronised with others, independently of the main repository. You might want to take the opportunity to edit .hg/patches/.hg/hgrc and provide a default remote path if like me you want to specify a server you will use to keep all your WIP patches for this repository.
So let’s say you’ve been doing some work on a change, but it’s not ready yet and you want to put a stake in the ground where you are without committing. Simple, you just do:

hg qnew awesome-feature-1 -m"I'm working on an awesome feature"

That says “create me a new patch in .hg/patches/awesome-feature-1 with my current changes, and give it an initial description” – you can change this description later whenever you want when you update the patch with qrefresh, see below. At this point hg considers that you have no local uncommitted changes anymore so you can happily do rebases and merges without the ‘outstanding uncommitted changes’ message. But, there’s more, you may want to do the following so you can sync the current state of this patch with other machines:

hg commit --mq -m"My awesome feature WIP #1"

Woah, woah, I said you didn’t want to commit, right, because this is a work in progress? Notice though, that I called commit with the ‘–mq’ option, which means ‘commit changes to the MQ patches repository’, and not ‘commit working copy changes’. If you look now, you’ll see that you have a patch file called .hg/patches/awesome-feature-1, which was automatically added to the nested patch repository when you called ‘qnew’, and which was committed to that repository in its current state (remember, this patch repository is completely separate from the main repo).

If I wanted to push the current state of my WIP patches to somewhere else, I can do it now (I’m assuming that I’ve already edited .hg/patches/.hg/hgrc to include a default remote path:

hg push --mq

That pushes the commits I’ve made to the patch queue (not the main repo) to that remote location. So for example I use a small area on a local server to track this, but there’s no reason I couldn’t push this somewhere public too if I wanted – again all explicitly separate from my actual repo, which is perfect. All the other main commands take a –mq option too, so you can use “hg outgoing –mq” etc too.

At this point, I could carry on working on my same patch, and every time I wanted to update the patch file in .hg/patches with my local changes I would call:

hg qrefresh

You can use the ‘-e’ command if you want to update the commit message that will eventually be used. And again, if I wanted to commit the changes to that patch, I’d call ‘hg commit –mq’ (and potentially ‘hg push –mq’). If at some point I wanted to quickly work on a different change, I could call ‘hg qnew’ again to create a new patch file based on the current diff, and deal with that separately. To navigate around the patch queue to identify which patch I’m currently working on, I’d use qpop and qpush.

Finally, at some point I’m going to want to turn these patches into real commits. Assuming that my commit message is already correct in the patch (if not, use ‘hg qrefresh -e’ to edit it), and that your current position on the patch stack is the position you want to create a proper commit for, all you need to do is:

hg qfinish --applied

Which will move the patches you currently have applied into the real Mercurial repository history, ready to be officially pushed with everyday work. Alternatively if you want to just pick a revision or range of revisions, you can name them instead of using –applied.

Conclusion

I didn’t really see the advantage of using MQ over just keeping my own patch files generated from my working copy via ‘hg diff’, particularly since MQ makes you work in a stack-based structure instead of via independent patches, until I learned how to use the nested patch repository. Manually handling patch files is fine, but once you get more than a couple of them on the go it can get tricky to manage. Having an automated approach to this with the nested repository works really very well, so I’m now using it as my primary method of managing work-in-progress.

Git users will no doubt think the local branch approach is better, and in at least one way (not having to keep patches in a single stack), it is – however I personally like how MQ mirrors my normal approach to unfinished changes as ‘volatile patches’ rather than treating unfinished changes as regular changesets and relying on history modification to fix them up later, and as someone who regularly moves between machines, MQ’s ability to sync patches separately from the main repository is very valuable. As is ever the case with Mercurial and Git, the perfect solution would probably be some combination of the two.

  • jonnii

    Great article. I’m a git user myself and I like the way local branching works as it means you can work on a feature in isolation, with its own history and prepare the final changeset as a single commit.

    Having said that, it’s nice that in hg you can store the ‘volatile diffs’ in the main repository.

    In git if you wanted to share the code between machines or developers you’d create a remote branch, but you’d need to remember to push your changes up in order to make sure they were up to date.

  • http://www.stevestreeting.com Steve

    Yeah, it’s funny because as someone that hops between machines a lot, I found the local branching less useful because I had to think about how to keep all my machines in sync with that work – like keeping a fork on a private server, but then I had to be careful about which branches I published to that versus which I published to the public, which wasn’t that easy. Syncing local branches requires having to create some kind of remote branch in a full-fat repo, or manually export patches if you don’t want to do that – both require some manual work & clean up when you’re done. I really like MQ’s ‘best of both’ solution where you sync a separate patch structure. That said, it could be better, such as allowing multiple streams of patches instead of one stack. Of course, since Mercurial is eminently extendable (everything’s just a Python module after all) I guess I should do the enhancements myself if want them :)

  • Gregory

    Well, there is git stash

  • http://www.stevestreeting.com Steve

    git stash just does the same as hg shelve, and doesn’t help sync WIP changes between machines at all.

  • blankthemuffin

    In case somebody sees this and is looking for a nice way to move patches around over usb sticks and the like with git, http://progit.org/2010/03/10/bundles.html

  • http://www.stevestreeting.com Steve

    I don’t really consider bundles to be a replacement (Mercurial has them too). They’re basically just binary patches anyway.

    a) They’re manual (see the number of commands & arguments required in that post!)
    b) Once again they only work on commits, which makes them less useful for unfinished changes that aren’t ‘safe’ to refer to as a valid history point – this requires history modification if you want to fix-up changesets that were not quite ready yet. I know git promotes committing ‘unfinished’ work to local branches but this mandates ‘fixing up’ work later (merge/fold). MQ at least recognises that these things are unfinished WIP patches and doesn’t pretend they’re a valid changeset at any point, nor is there any chance of confusing changesets which are unfinished (in MQ) and those in the main repository when you start moving things around (in git once you start pushing your local branches – directly or via bundles – it’s easy to get mixed up about what you are or are not syncing upstream to ‘official’ repos and what is just local syncing) – conceptually I like that strong division.

    As someone how regularly has several threads of unfinished work on the go that I want to move easily between machines without having to remember precisely where I am on all of them, MQ is preferable to manual bundling or patching.

  • http://www.jacmoe.dk/ jacmoe

    Well informed and highly useful – as always.
    Thanks a lot. :)

  • blankthemuffin

    Yeah I was not suggesting it was a mq replacement, more that if you looked at this and wanted something in git to move sets of patches around.

    And there are a number of commands in that post because it describes the whole bundle interface, showing the status of the repositories each step of the way.

    The interface is more like:
    git bundle create
    git bundle verify
    git bundle list-heads

    With the last two just letting you take a look at the bundle before you make use of it.

  • http://paulecoyote.com Paul Evans

    Thanks for documenting all this Steve, it’s making me seriously consider Mercurial next time I have the resources for a fresh start.

  • Bob

    Have you looked at how guards work in MQ? I haven’t messed with them myself yet, but I remember when I looked at them that I got the vaguely puzzled notion that they would allow me to act as if I had multiple queues of patches in a way that…well…might be simpler than deliberately setting up multiple mq repositories. Without testing both approaches, I’m honestly not really sure. I got the vague impression that combining MQ with the Attic extension might produce the simplest sequences of commands for dealing with such cases, once one got used to them, but I’m not too sure about that either. I suspect that these are all further examples of things that will seem amazingly clear and useful AFTER they “click.”

  • http://www.stevestreeting.com Steve

    @Bob: No, I haven’t tried guards. I realised afterwards that the easiest way to manage multiple independent patch streams is just to create branches in the nested patch repository – this is best done after you’ve emptied all the patches out of it, then start a new branch for each patch series, only using default as a ‘reset point’. Works really well.

  • Jon

    > This is the one thing I’m not so fond of about MQ, I think patches should be in a list of stacks to allow you to have unrelated (and potentially breaking) sequences of patches tracked differently in parallel

    You might want to take a look at the qqueue command. It seems to allow you to manage multiple patch queues as a map for those unrelated sets of changes.