SteveStreeting.com

So, I’ve just about completed my practical experiments & review of Mercurial and Git.

In the end, I had far too many separate notes and sets of experiences to post, so I boiled the argument down into the 10 most important factors to me, and scored Mercurial and Git on a scale of 1-5 based on what I’d found when using them. Here are the (annoying) results:

I’ll explain the scores, and my conclusion, after the jump.

Read the rest of this entry »

Git is picky when it comes to converting large, moderately complex Subversion repositories and so far the only option I’ve found that works reliably is using the very latest version on Linux. Forget about using 1.6.5 on Windows via msysGit, at least for the git-svn conversion it’s very, very unreliable. Similarly I found Git 1.5 on Linux very flaky for the svn conversion. This doesn’t give me the greatest confidence in Git but in order to properly explore all the angles, I’ve committed to making it work even if it means I have to monkey about a bit.

So, I installed a fresh Kubuntu 9.04 (and of course, 9.10 went stable a few days later) and tried to build Git 1.6.5 from source. The configure script is unfortunately a bit rubbish and doesn’t bother trying to detect the dependencies though, so for those that don’t want to go through the fail/retry build loop I went through, here are the packages you’ll want to install via apt from a clean version:

• build-essential
• curl
• libcurl3
• tk
• subversion
• libsvn-perl
• cpio
• zlib
• zlibg1-dev
• expat
• perl
• iconv

If like me you use the excellent wajig wrapper around apt, you can also do ‘wajig build-depend git-svn’, but that seems to install things that are not strictly necessary in the default build (but maybe are needed with some non-standard options).

sudo apt-get install curl
   41  sudo apt-get install libcurl
   49  sudo apt-get install tk8.4
   53  sudo apt-get install cpio expat
   55  sudo apt-get install zlib
   61  sudo apt-get install build-essential
   66  sudo apt-get install zlib1g-dev
   72  sudo apt-get install asciidoc
   75  sudo apt-get install xmlto

As  you know I’ve been reviewing DVCSs lately. I’m taking my time doing real use cases on them, and deliberately not doing the sort of feet-first leap into whatever looks best / most popular on the surface because I don’t particularly want to discover unexpected problems down the track. It’s consuming a lot more time than I expected – I’m writing up my findings and may publish the entire results later on if I can find the time to clean them up and format them better, but for the moment I thought I’d share some experiences with the conversion process of a relatively large, long-lived, multi-branch repository (OGRE) from Subversion to Git and Mercurial, because that’s what I’ve been wrestling with in the last few days. I discovered a bunch of additional issues during this process that did not occur when starting from scratch or doing conversions from more trivial repositories, so I thought it might help others to talk about it.

Source Subversion repositiory specifications

Revisions: 9215 (as of today)
Branches:  9 permanent, 22 temporary / experimental
Size: 375 MB

Also of note is that the source repository is still at Subversion 1.3 – this is because Sourceforge was stuck on this version for a long time and we haven’t upgraded the repository since they started supporting newer versions. We never bothered because it requires locking out the repository while you download the whole thing to a local machine, upgrade it and re-upload it, which is a hassle, especially when you have things to do. In practice the server-side version hasn’t been a major issue since you can still use newer clients with it and svnmerge operates regardless.

General Approach

I rsync the OGRE repository down to a local Linux server several times a week, so that was the source of all my conversions, eliminating most of the network time. I tried to convert the repositories using Windows clients in the first instance, because that was easier to use the latest versions of the tools (my Linux Server is on Ubuntu 8.04 LTS and even with hardy-backports available it’s not as up to date – and for simplicity because this is an important server I stick to the official versions). There is a 1Gb network connection between the machines so it could be pretty speedy.

The principle is that I want to preserve all history, all branches, and all tags. In practice I may actually prune off some branches later on, so that the clone process is quicker, but the base principle is that it should be a lossless conversion in the first instance. Definitely no top-skimming of the trunk like some conversion articles advocate – we have stable branches that must be maintained and regularly have work that we want to keep in experimental branches. In particular, post conversion it must be possible to continue committing to and merging from stable branches.

Git Conversion Experience

I’d previously converted some other, small and fairly simple Subversion repositories using git-svn (less than 500 revisions, and 2-3 branches) and it worked fine. However, when trying it against the considerably more complex OGRE repository I hit problems very quickly. On Windows, using msysGit 1.6.4 the process failed after 1900 revisions, just after doing the automatic repository tidy (git gc). The error message was simply ‘fatal error running git-svn’, even though it had been running exactly that command for the last 1900 revisions. Thinking there might be an msysGit issue here, I switched to the Linux server (git 1.5.4) and tried the same thing. This time it fell over at revision 176 with absolutely no error message. In both cases the repository left behind was corrupt so I could not resume the process.

The other thing I noticed was how long the process took on Windows. 1900 revisions took 5 hours (!) and thus I wasn’t in a hurry to retry the process there. On Linux the process was much faster, as far as it got. It’s worth noting that this is not caused by running across 2 machines – not only do I have a very adequate 1Gb link, Mercurial managed significantly faster conversions using the same topology. msysGit’s git-svn conversion is simply incredibly slow.

At this point I decided to try upgrading the Subversion repository, just in case git-svn hadn’t been tested with older repository versions. My Linux server had svn 1.5 on it, so I upgraded the OGRE repository to that locally and re-ran the git-svn process on the Linux machine (as I say, I wasn’t keen on repeating the glacially slow msysGit conversion). Sure enough, this time all 9200-odd revisions converted fine, in only about 1 hour 40 minutes, or about 15 times faster than doing it on Windows.

So, I may have had a few problems, and being forced to upgrade the repository before converting was a bit of a pain, but at least it worked and was fast (on Linux anyway). After that, I started cloning the repository both on Linux and Windows and tried performing some standard operations.

The first thing that surprised me was that when cloning the converted repository, I could only see the ‘master’ branch on the remote machine. It’s common practice for Git not to create any local branches other than master on clone, but usually you can do ‘git branch -a’ to see all the remote branches that are available, which show up as something like ‘origin/v1-6′ – you can then check them out to local branches. However, no branches other than ‘origin/master’ showed up, even though I knew they’d been converted. It turns out that git-svn converts all branches except master into remote branches in the converted repository, referencing the original Subversion URL – so very much like having cloned from another Git repository. That sort of makes sense, but in the context of a full conversion to a repository that is destined to become the upstream master, isn’t that useful. In practice what you need to do is after the git-svn conversion is complete, git checkout each of the branches that you care about in your converted repository, thus creating local branches in that repository which subsequent cloners will be able to checkout themselves.

So, once I’d figured this out I started to check out different branches to test if it had worked. At first it seemed to, when checking out the first branch (switching from master to v1-6 in a local clone from the conversion). When I came to try to switch back to master however, Git complained that I had modified files in my working directory. WTF? I’d only just checked out the clean copy of the v1-6 branch. But sure enough, git status told me I had 5 modified files. Diffing them showed no changes, and “git reset –hard” returned with no error, but git status still showed these files as modified. Bizarre. A git checkout -f still let me switch, but again after completion a set of other files showed up as modified. Switching back and forth (with -f) a few times revealed that the list of modified files after checkout was different each time. Again worried that this was a Windows thing, I tried checking out on my Linux machine instead (so at that stage the entire process, conversion to checkout, was done on Linux). But no, the same problem occurred – a random selection of 5-7 modified files on clean checkout.

This has raised some serious concerns about using Git for me. Firstly the flaky conversion which requires a bunch of extra steps just to get it to work at all, then the post-conversion bizarre behaviour of thinking files are modified when they’re not. I had none of these problems with smaller repositories, created from scratch or converted, which up until now I’d been using for testing (and Git had been winning me over in fact since it had been working well). But the bottom line is that this process needs to work reliably for the OGRE repository. If it doesn’t, it’s pretty much untenable.

Mercurial Conversion Experience

I started off with the in-built ‘hg convert’ process. It all went smoothly and took about 8 hours, and the resulting repository was mostly fine. However, the default behaviour is to process the revisions in an order which “produces the fewest jumps between branches in the commit log”. In practice, I found that this meant the revision log when reviewing multiple branches was badly jumbled and difficult to use; the use of the ‘–datesort’ option resolved this but increased the conversion time to just under 10 hours (still faster than msysGit but a lot slower than git on Linux).

The guys from BitBucket, who I’d talked to to see if they would offer free unlimited hosting for OGRE since we wouldn’t fit in the default 150MB limit (result was that they were super-friendly and offered not only that but lots of advice), suggested that I try hgsubversion instead. I was initially put off by their website suggesting it wasn’t fit for production use (they’ve removed this statement now), but BitBucket told me that was a little out of date, and in fact the Python project is using it for their conversion, which is obviously of major size. So, I gave it a shot and got some good help from the hgsubversion guys, and the results were great – 1hour 40 minutes from the Windows end (coincidentally the same speed Git managed on the local Linux machine), and the log view was properly ordered right off the bat.

The one remaining issue I had (and this is true of git-svn too) is that all of the branches are open-ended on conversion – that is, no record is made of merges that have been done between branches. That means you would have problems continuing a branch and then merging it, because Mercurial would think it has to merge everything from the point the branch was taken. Neither svnmerge or svn:merge properties are taken into account.

One way to resolve this is to manually create a merge point to close off the branches. The easiest way to do this is:

• Grab the default tip
• Open a command line and define a temporary environment variable “HGMERGE=internal:local”. This means that you want to keep the local files and throw away the other source when doing a merge, which is important for our dummy merge
• hg merge <source_branch> -y
  
• commit – only the .hgtags file should be modified, the rest of the commit is merely metadata alteration to close off the source_branch

Once you’ve done that, your branch is joined back to the trunk and you can carry on as before, any new commits to that branch will merge across cleanly. The only downside of this is that the merge is strictly at the wrong point – if you view the history in the trunk it won’t be technically accurate and you’ll need to use your commit messages as the real guide to the actual merges before the conversion.

A better way to do this would be to record the merges during the conversion, that is for merge commits in Subversion to have 2 parents. So far, none of the conversion tools read svnmerge or svn:merge metadata to implement this, but the standard ‘hg convert’ has an option called ‘–splicemap’ where you can specify merge points to be applied during the conversion. Unfortunately I’ve tried to use this twice so far, and both times it hasn’t worked (just silently done nothing). The documentation for –splicemap is not great so it could be I got the URLs wrong. But anyhow, following 2 failed attempts (20 hours! because this was the standard hg convert with –datesort) I decided I’d try to get a similar bit of functionality working in hgsubversion instead, since that’s much faster (1hr 40m a pop). Right now I’m hacking away on it to try to make this work, so far it’s not but I’ll let you know if I eventually succeed. One of the benefits of Mercurial is that it’s all in Python so it’s very easy to modify, compared to Git which runs all kinds of random scripts and executables, including sh and perl so it’s much more tangled to dig into.

Conclusions, so far

I started my DVCS evaluation very pro-Mercurial and very anti-Git. While working through my detailed use cases, a process which I’ve not quite completed yet, Git has grown on me a great deal, and I discovered a few things about Mercurial which I found a bit limiting at first, but which are mitigated via extensions – Rebase, Queues and Transplant particularly. My recent experience with more complicated, full-scale and imported repositories has once again gone in Mercurial’s favour though, and I saw a nastier side of Git – when it goes wrong, it’s a lot more difficult to figure out why. In contrast when I’ve had my Mercurial conversion crash – and I stress this only happened due to my own screw up, once because it ran overnight when my rsync kicked in and changed the repository under its feet, and a few times when I’ve been experimenting with hacking the Python to get the merges done – the reason has always been clear; a nice Python trace, and the repository was always intact anyhow – in the case of the core hg convert the conversion even restarted from where it left off once I’d fixed it.

If I were to graph my relative opinion of the two over the period I’ve been doing this so far, it would look something like this:

Git totally came up from behind and I was really starting to dig it, until it started freaking out on me with the conversion and I started to try to diagnose why and found it mostly unhelpful. Again I stress I’m not done with my tests yet, but I’m perhaps 75% of the way through now and the conversion problems I’ve had with Git in the last few days don’t look good. Bazaar, I’m afraid, is no longer likely to be part of the evaluation – it takes a long time to do these evaluations properly rather than just trivially, and our survey has indicated that it is the least commonly used among our community by a very large margin, so I’m focussing on the ones more users are likely to already be comfortable with.

The evaluation process continues…

A few weeks ago I decided to start seriously investigating switching to a DVCS. I’m currently up to my eyes in work and haven’t really had time to progress that in the last few weeks; however some absolutely abhorrent performance / reliability problems with Sourceforge’s Subversion server made a large merge process so costly to me (in the end I had to commit in small chunks, breaking transactional consistency, and it needed so much babysitting because of the speed / reliability it took me 4 bloody hours just to commit!!) that it bumped it up my agenda a bit. I don’t have to do merges / commits of that size very often – in this case the problem was wholesale license header changes for our MIT switch – but still, it’s totally unacceptable to have to deal with that. I raised a support request with Sourceforge, but I’ve seen other reports of bad SVN performance from several weeks ago from others, so I’m not holding my breath. It’s time to start considering alternative hosting I think.

I’m not done with my evaluation yet, because I just don’t have the dedicated time I really want to spend on this right now. But, here’s my early-stage results.

Git

I’ve discussed before that I don’t like the where Git has come from. It’s overly complicated, Windows support was clearly not a priority, and it switched existing VCS terminology around just for the sake of it a lot of the time. It practically shouts “I’m different, live with it!” at you, which is frankly a typical elitist geek attitude and not one I particularly respect. This attitude permeates the documentation, reinforced by the absolute insistence of most tutorials that you need to understand Git’s underlying data model before you start using it. Er, right – sorry, but when normal people want to learn how to use a new tool, we just want to know how to use it, not how it’s built. If understanding how it’s built is a prerequisite to using it, then I’m sorry, it fails miserably at being user friendly.

Nevertheless, it’s fast, it’s efficient in storage, it’s the most popular & fashionable DVCS (probably due to GitHub) and that has weight. Of particular interest to me is that TortoiseGit has come along in leaps and bounds, and is really quite pleasant to use. Of course, the reason it’s pleasant is because it hides the majority of the nonsensical changes that Git decided to make to existing VCS terminology; for example ‘Revert’ in TortoiseGit does what you expect (undoes your working copy changes), rather than needing to use ‘git reset –hard’, which is only intuitive to those who live on Mars (git revert, in contrast, records a new commit to undo a previous commit – why the hell do you need a special command for that??). Thus, it’s an odd situation – using TortoiseGit is pleasant, but only because it ushers the underlying git behind a curtain and gives you what most people really wanted from it in the first place. The downside is that using TortoiseGit really doesn’t teach you how to use the command line very well, like most other VCS tools do. In fact, it may well mislead you into thinking Git is friendlier than it actually is. For example, it saves you from the ridiculous need to remember the “-a” argument to “git commit” – without which what you actually get in your commit is the state of the file when you did “git add”, not the version in your working copy. If that makes sense to anyone, raise your hand. Thought not.

Hosting & collaboration wise, GitHub seems very good.

Mercurial

Mercurial on the command line is nice. It behaves the same way centralised VCS’s do, except in the cases where it needs to be different. This is pragmatic design – not being different just to make a point, but being different where it needs to be. It doesn’t break old concepts and does what you expect it to, and contrary to what some people think, that’s a very valuable feature.

It’s not all roses though. TortoiseHg is clunkier than TortoiseGit, despite being based on a more intuitive core tool. The UI just feels a bit wrong (like putting action buttons on the toolbar – who does that?), and I’ve sworn at it for being unintuitive more than once. The other problem is that the Mac GUI tools are not really that great either – MacMercurial only allowed me to do a subset of the operations I needed to do, and Murky just crashed when I tried it. GitX in comparison works quite well on the Mac.

So despite a more intuitive command line and core concepts, and a more pragmatic approach generally to DVCS for ‘regular’ people, when it comes to GUIs Mercurial lags a bit now. This was unexpected to me since it is Git that has traditionally been poor on the GUI front. There are also a few other minor issues like branches being totally permanent and needing to be globally uniquely named, which can make local experiments more cumbersome.

Hosting wise, BitBucket seems quite competent, if a little less polished than GitHub.

Bazaar

I’ve only just started experimenting with Bazaar, and so far I’m quite impressed. It has the pragmatic approach of Mercurial, but also has a built-in GUI which is really quite nice to use and leads you through the initial setup and configuration. There’s also TortoiseBzr which feels somewhere in between TortoiseHg and TortoiseGit. I haven’t tried it on the Mac yet. Performance was always the issue listed as the major downside of Bazaar, but this has improved since 2.0 and while it’s not as fast as Git, it seems to be fast enough.

The main downside for Bazaar is adoption. It trails both Git and Mercurial in terms of the number of people using it, and therefore adopting it for a public project would have the disadvantage of making people use a tool they’re less likely to already be familiar with. Also for hosting, Launchpad is quite new; it looks quite good, and has more features than GitHub, but it doesn’t have the option to host private projects (not an issue for Ogre of course) or a graduated commercial plan – you can self-host of course but that’s not as easy.

Conclusions so far

The sad fact is that none of the 3 are an instant win for me; they all have positive and negative aspects. Summary so far:

So, I’m basically in a no-win scenario. If I pick Git, it’ll work fine via the GUIs but it’s too easy to screw things up when using the command line, and I’m bound to get annoyed at the needless obscurity from time to time. But, lots of people will be happy to use it. If I pick Mercurial, I’ll be happier with the overall core concepts & command line, but the rough edges on the GUIs are going to annoy me day to day. But, quite a few people will be happy with it all the same. If I pick Bazaar, I’ll be happy with both the core concepts and the GUIs, but being the least fashionable option almost no-one in the community will be happy that I picked it over the other two, and lots will bitch about having to use another tool.

I’m reluctantly acknowledging that the least of the evils appears to be Git right now, even though I personally hate its underlying interface. Somehow it feels wrong to only like using it when it’s hidden beneath a GUI – I’ve been a regular user of the command line for CVS and SVN for the best part of a decade, and I like being happy with both modes. I can imagine tolerating Git’s command line, but never liking it just because of the unnecessary idiosyncracies (like commit -a).

I still have lots more detailed tests to do anyway, which will have to wait a month or so until I have more time.

Ok, so I’ve posted my initial feelings about tinkering with Mercurial and Git, and that seems to have generated some interest. It’s time to get a bit more formal about how I’m going to evaluate them against each other, to decide which one I like to use most in real, practical scenarios. So, I decided to come up with a list of use cases for the things that I typically have to deal with when managing the repositories for a software project (open source and otherwise), so that I can methodically test them out and see how I feel about each system. I’ve tried to deliberately include use cases for things that you can’treally do on a centralised system, but that I’d want to make use of, as well as the usual nonsense that happens day-to-day on a typical project

I’ve presented my work-in-progress list below, feel free to suggest more use cases I can test. I really want to see what these things are like in practice in the kinds of situations I encounter in real projects, without actually risking a real project in the process! Like a backup / restore strategy, it’s no good doing it for the first time when the sh*t has already hit the fan.

Oh, and for fun, allegedly Git is MacGyver and Mercurial is James Bond .

Due to the nature of a DVCS, all these use cases must be tested both in isolation, and  after pushing those changes to another potentially ’superset’ repository.

1. General
   1. Commit a few changes to local repository ‘A’ but don’t push to central yet. Push a number of different changes to the central repository from a different repository ‘B’. Then, pull ‘B’s changes from the central repos to local repository ‘A’  to bring it up to date, again without pushing A’s outstanding changes yet. This is equivalent to doing an ’svn update’ while you have local uncommitted changes in Subversion, but while using the local commit features of the DVCS. How does this work in practice? [Suggested by Arseny Kapoulkine]
2. Branching
   1. Create a new public, official branch (stable branch)
   2. Create a new long-term feature branch which is intended for public consumption / collaboration
   3. Create lots of short-term development branches (or equivalent structures) intended for local consumption only
      1. What are the size overheads? (Git claims superiority here)
      2. Does this excessively clutter the public repository when pushed?
      3. Is there a better way of handling multiple local changesets which you may or may not decide to push individually, such as testing many patches (Mercurial queues seem very interesting, Git equivalent?)
      4. . Rename a branch? (optional)
3. Merging
   1. Merge changes unidirectionally from one branch to another, without having to manually pick revisions. Make more changes in the source branch and repeat.
   2. Bidirectional merge – a feature branch which is not yet ready to be merged into the trunk wants to resynchronise from the trunk and continue branched development. Merge of this branch into the trunk must be tested later after further changes
4. Tagging
   1. Create a tag against a specific branch; probably at the HEAD but look for the option to specify a revision
   2. Correct / modify / move a tag following a mistake or last-minute revision (pre-release) without having to make duplicate commits or other such spurious activity
5. Firefighting
   1. Screw-up: developer commits change to trunk instead of stable branch. Merge / move it to the stable instead – change can be left in the trunk or can be removed for re-merging, so long as the procedure is clear.
   2. Screw-up: developer commits change to stable branch that is interface-breaking, must be removed and moved to the trunk. Must be removed from the stable branch and moved.
   3. Screw-up: Revert a single change from the repository, that is not at the HEAD
6. External patch submission tests:
   1. Patch file from same branch, no conflicts
   2. Patch file from same branch, with conflicts
   3. Patch file generated on a different branch to the one we want to apply it to (include conflicts)
   4. Pull from third-party repository, entire branch
   5. Pull from third-party respository, specific changes
   6. Patch file generated from non-repository source copy
7. Backup multiple work-in-progress changes on a local machine that are not ready for public consumption; approaches:
   1. Store a patch per local branch (this is how it’s done with SVN, but too much hassle if you use lightweight local branching, DVCSs can do better)
   2. Push to a backup repository on another machine across existing protocols – ssh, https, Samba share (Git can’t do the latter?)
   3. Push to a backup respository on a USB stick (Git can’t do this?)
8. Binary files
   1. Revise a binary file over a few versions, test storage efficiency
   2. Binary file conflict resolution
9. Conversion from Subversion
   1. Import retaining history
   2. Import multiple branches
   3. Import tags
10. Integration
    1. Mailing list / RSS notifications of commits on official repository
    2. Bugtrackers
    3. CIA.vc et al?
    4. Good free & open source GUI clients for all platforms
    5. Line ending conversions between platforms

I’ve been interested in DVCS for a while; having done my fair share of branch management, something which makes that process easier and more transparent is definitely very attractive. I particularly like the way a DVCS makes it easier for people to collaborate in pockets of their own, away from the centralised environment, and track other repositories and keep their local mods up to date more easily – public-branch-on-demand if you will. However, I’m yet to be convinced by Git for everyday use. As noted, I love the idea, but Git comes across as – and this is an ironic thing for me to say – excessively geeky. At one time I might have believed there was no such thing as a plethora of geekery, but I like to think I’m a more rounded individual now

Don’t get me wrong, I’m sure Git is technically excellent. But a number of things about it remind me of the somewhat stubborn purisms of  ‘old Linux’, such as:

• The documentation is hugely daunting. Whenever anyone tried to convince me why Git is amazing, they point me at documentation that just makes me not want to touch it for anything important. The impression I get is akin to some Linux forums: – this thing is powerful, but it gives you plenty of rope to if not hang yourself, then to tie yourself in knots if you don’t know what you’re doing.
• It requires a Unix back end, to the extent that you have to run it under Cygwin on Windows, and its rigid filesystem requirements mean it won’t work on FAT or Windows network drives. I get the impression there’s something rather aloof about the fact that the core team don’t consider it an issue that Git won’t run on native Windows environments. There are external projects working on that, but the fact that it’s not core is a concern, since I prefer all my 3 platforms to be supported equally.
• The UIs still don’t seem very user-friendly, and most responses I see to the question of why that is boil down to “Git is too powerful to be captured in a GUI”. Sorry, but I don’t buy that – no system is too complex to be captured in a user-friendly tool, it just takes the will to do it, and like ‘old Linux’, there seems to be little will to make Git more approachable.

So, I’ve never had great vibes from Git, despite it’s doubtless technical merits. So, I decided to check out Mercurial instead – from what I read, Git is the fashionable DVCS that all the cool kids want to be seen using, but when it comes down to it Mercurial does exactly the same thing – except that it has native support for all platforms out of the box and TortoiseHg is looking pretty mature. I also read that it handles binary files better than Git too, since it does binary diffs everywhere. The documentation seemed much more approachable too, so I figured I’d have a play.

My first impressions are incredibly positive. The most impressive thing of all is that I imported a (relatively small- about 100 revisions) Subversion repository, with all the history intact, about 5 minutes after installing it, and was bouncing that across Linux and Windows immediately afterwards (haven’t tried OS X yet). TortoiseHg is instantly familiar, and despite the fact that the concepts are the same as Git, the presence of some familiarity, together with some distinctly less intimidating documentation, has me feeling far happier than the times I’ve dipped into the Git docs. Mercurial’s approachability is a positive contrast to Git’s Unix-purist, RTFM style I think. I know which I prefer so far.