SteveStreeting.com

Yesterday saw world-plus-dog in the technology sector glued to Apple‘s announcement of their new tablet device, which has now been officially dubbed the iPad. Basically, when you boil it down it’s a super-sized iPod Touch with optional 3G support and a few more apps.

Reaction has ranged, as usual, from the ecstatic “I’ve seen the face of God, and his name is Steve”, to “What a useless piece of junk”, stopping at most points in between. In the more negative camp, lots of talk has centred around what it doesn’t have (multitasking, a camera, a USB port, Flash), and that some people seem to find it hard to grasp the usage conditions of a device that neither fits in your pocket, nor does everything a laptop does.

Personally, I’m cautiously optimistic. The device was never supposed to be a phone or a laptop, so I’m curious why people are comparing it to one – the point is that it’s something else. I can actually think of multiple use cases where a device of this form factor and capability would be useful to me. Here are a few examples:

1. I’ve thought about buying an eReader before, but have always been completely unsatisfied with the existing solutions: current e-ink devices are fine for reading black and white novels, but don’t handle A4 formatted content at all well, can’t do colour, take far too long to flip through pages, and are basically unusable for keyboard input, making searching impractical – and therefore these devices do not satisfy my need for a reader that replaces my bookshelf (physical and virtual) of reference material at all. The iPad, however, looks like it would be able to do that much better.
2. Sometimes I’m in the living room or kitchen and I’d just like to look something up on the web; maybe check some news or look up a recipe maybe – just a 5-10 minute thing. Firing up the laptop just for this is overkill, but the pages are too small to really read properly on a phone. In the end I do one of these things anyway but it’s never ideal. Again a tablet form factor would be perfect for this.
3. When we’re showing photos to family and friends, these days we do it on a laptop because we never print anything. It’s not ideal, even the most elegantly built laptop requires everyone to crowd around the screen behind you or similar – it’s awkward. If I had a tablet to do it, one I can easily hold up and pass around, that would work much better.
4. When I’m in a social situation when it would be useful to have intermittent access to some documents or other information that’s too big to fit on a phone screen comfortably, currently you need a laptop to do it. Laptops are really, really unsociable to have out on a table with others around (say at a meeting), because of the way they need to be used, with a screen forming a psychological barrier between you and whoever else is on the opposite side of the table. This happens all over the place: I strongly feel that laptops are the scourge of coffee shops today, turning a social space into a cluster of virtual mini-cubicles with individuals hunched behind screens not talking to anyone. I also play pen-and-paper RPGs socially, and over the years I’ve tried to use a laptop with many highly useful applications as an accessory, and it’s never, ever worked. Even the smaller laptops are too obtrusive, but a phone is just too small to be useful. I’d love to try using an iPad with some dedicated apps for tracking things.

I’m sure there are other examples. Basically I think people need to get over the fact that it doesn’t improve on what they currently use their phone or laptop for – that’s really not the point. I see the iPad as a ‘gap filler’ – and I can certainly see some gaps for it to fill in my life.

The price is much better than expected too, mostly because it’s an upgrade of an iPod rather than a downgrade of a laptop. I’d skip the 3G option because it’s pointless for me, I’d only use it on wifi, so that makes it not that much more expensive than a top-end iPod Touch.

But, it’s not all roses. The lack of Flash is an issue for web compatibility, although at least video through HTML5 is starting to happen (YouTube added it recently). The lack of multitasking is a bit disappointing, but might be relaxed in an OS update later. The GPU capabilities are a bit unexplored online so far, it seems that it’s probably as powerful as an iPhone 3G, but falling short of the 3GS (so GLES 1.1). I’ve also heard today that iBooks might not be available in non-US countries at launch, which definitely undermines the offering as an eReader.

So, depending on the practicalities when it’s released over here, I may or may not grab one. I can definitely see places in my life where a not-a-phone-or-laptop device would be useful, and frankly, I’m intrigued by the possibilities of where this kind of device may go in future.

Like most members of the male species, and particularly the geekier types, I love gadgets. Complex ones are great, but sometimes the greatest satisfaction can come from simple things that just work really well. Here’s a couple of recent buys for me that fall into this category that I thought I’d share.

Joby Gorillapod

When we’re on holiday I often spend time trying to find places to put the camera so we can do a timer shot with us both in the picture, and when you’re in forests and up mountains finding a level spot is tough. I’ve gotten quite good at it, squinting at rocky outcrops and tree stumps with an almost film director level of interest, but it’s still awkward and sometimes precarious; this year in the Canadian Rockies I placed the camera on a rocky slope and only realised when I had to charge down again how many rocks were between me and the ‘mark’ I had to be at within 10 seconds, and I almost came a cropper, much to the displeasure of my wife.

I’d seen the Gorillapod before but kept forgetting to buy one before we went on holiday, so this time I bought one as soon as I thought about it, even if it’ll be sitting around unused for a while. Basically it’s just a small tripod made from a series of ball joints, each one perfectly stiff under the weight of a camera but easy enough to move, and with rubber surrounds on every joint and on the ends for grip. It’s very bendy and yet very sturdy once it’s set, so you can use it as a regular mini-tripod (but can adjust for uneven surfaces really easily), or you can suspend it from tree branches and poles, secure it up on top of fences or bollards just by bracing it, and all kinds of things. It just clips on to a small tripod mount and folds up really small.

It’s just an incredibly useful little gadget that I wish I’d had for holidays ages ago, and I imagine regular photographers would find it invaluable too.

Bicycle iPod Mounts (for drum kits)

I don’t ride a bike anymore, but after setting up my drum kit I realised I needed somewhere to mount my iPod if I was going to hook it up for practice, rather than having it on the floor or using gaffer tape or something. Surprisingly there didn’t appear to be any standard accessories to do this (a bit of an oversight on Roland’s part I think since this must be a common requirement), so I was nosing around in the VDrums forum and discovered that most people were just using regular old bicycle mountings, and attaching them to one of the cymbal riser arms (since they’re about the same diameter as bicycle handlebars, compared to the main drum frame which is much thicker).

They were cheap so I gave it a try, and sure enough it works beautifully – you wouldn’t know that the mounting wasn’t made entirely for this specific purpose in fact. Score one for the community

I always find Matt Asay’s blog an interesting read – even if I don’t always agree with him, his posts on open source are always thought provoking. Today he was talking about how Wikipedia’s contribution rate is falling and how that has parallels in open source; that the community is no replacement for a centralised, focussed team.

He’s right on the core point – at the heart of every successful open source project there’s always a core team (or individual), and in the really influential ones, that team is usually funded – Mozilla is famously bankrolled almost entirely by Google, the Apache foundation has many, many sponsors including Google, Yahoo and Microsoft, Eclipse has IBM, and so on. Many of the big projects that don’t have more general sponsorship still have a core team funded by a dual-license or other premium software model: MySQL, RedHat/JBoss, Qt etc. Such guidance & direction at the core is crucial – at OGRE we have a core team too, except that we’re not directly funded by anyone in terms of developer time (we have several generous sponsors who cover the majority of our hosting needs); we guide it because we want to, and because we use OGRE ourselves too. My company is probably the closest thing to a core development sponsor, in that I’ll allocate “work time” to doing OGRE development that could otherwise be spent making commercial products or doing consultancy, but it’s by necessity small beer compared to the likes of Mozilla and Apache.

But I do think he underplays the changes that have taken place in the software development world. He asserts that because most headline software development is still focussed at big influential companies, we’ve mostly just rearranged the chairs a bit at the same banquet. I don’t agree with that at all – by nature it still makes most sense to concentrate much of the development in a small team for quality, consistency and organisational purposes, but the point is that where precisely this centre is determined primarily by merit, not by the boundaries of a company’s org chart. While the core team is doing a good job, and accepting reasonable patches and such, people are happy for the show to be run there. The community is still definitely involved in the development, and certainly adds considerably to the end result. Yes, proportionately the central team does more, but crucially, should anything go badly wrong – such as the core going in a direction a lot of people don’t like, or the product being sidelined, if there’s enough of a community a fork will emerge, with another core team to lead it. That’s a critical safety valve that keeps companies more “honest” than they had to be in the past, and is a vital insurance policy for anyone investing their own resources in a piece of software. Matt claims the ‘Command and Control’ setup of software vendors is still in place; I think his view is clouded by the fact that he’s solely focussed on enterprise software, and enterprise customers move at such a glacial pace that any change is largely imperceptible – to the extent that ‘community’ maybe does look a lot like the ‘customers / partners’ relationship of old. But that would be a bad call, completely ignoring the difference in the level of control that is ceded to a community versus the customers of old – sure, many enterprise customers may not wish to leverage that control, and would take a long time to move if someone else chose to do so, but that option is still always there. And not everyone in the world is an enterprise customer – the enterprise usually follows the grass roots eventually.

In practice, it’s really all about balance, the middle ground. Yes, we still need focii of development just to make sure things get done in a reasonable fashion – no-one likes chaos in their software. Yes, it makes most to have that focus funded, in a traditional company model, if that piece of software gets beyond a certain size / popularity. But that doesn’t for a second undermine the value of community participation; in fact the two are deeply interdependent – one without the other is just not sustainable in a sizeable project.

So, people certainly shouldn’t be deluded into thinking that random crowds of people on the internet will create great software without some organisation (the infinite monkeys creating Shakespeare fallacy), but they also shouldn’t think that community is disposable and that we’re in the same situation we were before but with a different label. Nothing could be further from the truth.

One of the things I love about open source is that there’s a huge amount of power in the idea that you can use, and indeed co-operate on, a whole ecosytem of generalised, robust, re-usable components, and then combine, configure and supplement them into something that is greater than the sum of its parts – into a final result which concentrates on being extremely good at a particular task. There was a time that to create something awesome in a particular space, you’d either have to buy in lots of expertise or you’d have to invent it yourself, before you ever got to the interesting bit that mattered. Open source totally flattens the landscape, and at a stroke makes the software world far more interesting as innovations can happen on the back of shared experience.

This also comes at a time when our use of technology is fragmenting – the PC isn’t the sole device through which we view the tech world anymore, we use all kinds of other devices all suspended in the Internet soup, and the devices that win the most favour are those that are again specialised for the task / environment in hand.

So what does this mean? It means making generalist software such as regular beige-box operating systems is rapidly becoming quite uninteresting, and not something you can sell much product based on (unless you’re in the position of being able to leverage a huge existing install base). Generalism belongs in open source libraries, lego bricks which while fairly uninteresting in and of themselves, can be built into vast arrays of much more interesting combinations. As a commercial outfit, being the generalist jack of all trades really doesn’t get you very much attention any more. Wowing people with how well you can address a particular problem is what gets you noticed.

I say all this in the context of Google’s release of the Chromium OS last week. While it’s not being sold in its own right, make no mistake that it’s definitely selling you something – specifically Google hosted app services. And it’s also a prime example of heavy specialisation – it’s only designed to work with hosted apps, nothing local. Some may scoff at that, and indeed it’s not exactly practical right now for all but specific cases, but that’s the point – it’s an angle, it’s a specific pitch at a specific audience, and as such it differentiates itself. Increasingly that’s what interests people about a product, even if they’re not in that particular segment yet. It’s not trying to please everyone, it’s just trying to polish the offering for a particular subset of the market. I see this becoming more rather than less common.

Many have observed that Google is basically just like Apple, but for hosted software rather than hardware. Apple give you a specialised experience which is a combination of hardware and software, mainly so they can sell you some sexy premium hardware. They specialise in making that particular experience the best they can, they don’t try to be all things to all people at once, and it works for them. People notice them as a brand and associate certain things with them inherently – sleek hardware, intuitive interfaces etc. The same goes for Google – they’re known for their free online services and want you to gravitate towards the commercial services they provide either yourself, or your company through the well-understood principle of employees nudging their company to use the stuff they already like.

So it’s interesting to think about where Microsoft sits in this environment. They’re still obscenely profitable of course, more so than a specialist like Apple, but they’re also  increasingly perceived as the ‘dull’ option. On phones, the consumer is raving over the iPhone, and to a lesser extent Palm Pre and Android, while Windows smartphones are increasingly overlooked. Windows 7 is trying to shake things up with some new features, but when it comes down to it, Windows is a workhorse at heart, as is their other flagship, Office. That doesn’t mean they’re bad products – in fact being a decent workhorse for a vast range of uses is hard – but they’re not exactly exciting; you don’t look at a Microsoft product and desire it like you do when you look at the latest Mac laptops or an iPhone (or maybe that’s just me), nor do they get as much community buzz as Google Wave did (deserved or not). Now, given that most of Microsoft’s revenues come from Enterprise users (for whom ‘dull’ is often a desirable feature!) and PC bundles anyway, they aren’t really affected by consumer aspirations, so maybe this lack of a sexy brand is not an issue. Despite years of people saying Microsoft is due to be made irrelevant by Linux or OS X, and releasing a pretty undesirable version of one of their flagship products (Vista), their influence has not waned that much, although it has come down somewhat from what was a frankly unhealthy peak of dominance a few years back. But, nothing ever stays the same forever. Are people in the boardrooms at Microsoft worried that they don’t really have an ‘image’ that ordinary people can relate to positively? The Windows 7 adverts would suggest they do care about this enough to spend quite a lot of money on it, and to create such astoundingly cringe-worthy content as the Windows 7 Party Guides. I hate to think what cocktail of drugs those actors had to take to get them through that particular horror, or what counselling they will need in the aftermath.

But whatever, Microsoft would be right to be concerned about their image, which remains rather vague (mostly because MS seemingly tries to have its fingers in every pie just in case it misses something, but doesn’t excel in very many) . Regardless of all the lengthy and dehumanised procurement procedures, the general inertia which is present at the heart of every large organisation, the inherent resitance to change and dependence on paternalistic vendors that worms its way deep into IT departments; having large numbers of people desire your products on an individual level still matters. It might take a bunch of years to filter through, but that’s where the trends are, at the grass roots – eventually they will drag the enterprise kicking and screaming in that direction too. I’m sure Microsoft knows this, but so far hasn’t really got to grips with a genuine ‘identity’ that appeals to regular people. Arguably the 360 has done best at this, but it’s unlikely that there’s much of an image halo (sic) effect to Windows/Office there, in the same way that you get with the iPod/iPhone and the Mac. Microsoft don’t have much to worry about just yet, but like a trickle of water that carves a canyon eventually, it’s something they must be concerned about long term. Being a decent generalist just doesn’t make waves; it’s time to find a specific vision people can get excited about, then drop everything that’s unrelated to that and concentrate on doing it ridiculously well. Only then will people really know what Microsoft represents versus Apple or Google, or anyone else, and know whether they like that or not. Inertia can only take you so far before you need some more momentum, and that requires direction – which implies a specific direction, not in all general directions at once.

As soon as Macs started running on Intel, they became infinitely more attractive just because suddenly you had the option of using Windows on them too if you needed to. Because let’s face it, as lovely to use as OS X is, and as much as its popularity has grown, the majority of the world still runs Windows. Boot Camp is a great little tool provided by Apple which makes setting up a dual-boot into Windows generally a breeze, barring a few small niggles such as the slightly ropey support for the extended functions of the track pad (two-finger right-clicking and scrolling is very flaky).

I have Vista installed as a secondary boot on my 2007 MacBook Pro, something I’ve come to regret as it’s regularly far more hassle than XP would have been. In practice, I’ve found that I hardly use it, with weeks passing without me bothering to boot up Vista (which makes it worse, because when I do boot it up its intrusive and wholly unstoppable update process starts kicking in in the background, making me grind my teeth at the CPU / RAM loss). Nevertheless, it’s useful at times.

I’ve dallied with virtualisation occasionally, from Parallels to VMWare to VirtualBox, but have been generally disappointed – my day-to-day work and most of my recreation involves 3D acceleration in one form or another, and virtualisation has historically been pretty bad at supporting it, making it an impractical option. VMWare Fusion 3, however, claims support for Shader Model 3, which is good enough for the vast majority of the work that I do (Dx10+ is still a tiny, tiny niche that is still rarely in demand), so I thought I’d give it a shot.

One of the nice things about Fusion 3 is that it lets you boot your existing Boot Camp partitions in a VM, which meant I could test it quickly. Parallels supported this too but I found it didn’t work that well in practice when I tried it. Fusion 3′s support is excellent – just a few minutes of adaptation and the requisite install of VMWare Tools on the Windows partition and it was up and running. The main problem I initially encountered is that the default 1GB memory allocation is, of course, absolutely no use for running the sweating hunk of lard that is Vista. This is the OS that is capable of getting up to 1.2GB on my machine after first boot with only Explorer open, if you’re unlucky enough that it feels it needs to run update processes in the background. I only (only??) have 2GB on my laptop so I had limits, but a quick tweak of the VM to 1.6GB (and also letting it use multiple CPUs) made it run OK and still let me jump back to OS X, albeit a bit slower than usual (but show me a post-millennium Windows OS that can stay usable on 400MB!).

For general non-specialist use, Fusion 3 is as awesome as other virtualisation tools. Either in a window, full screen, or in ‘Unity’ mode, where Windows apps look like first-party OS X windows in your task list (a bizarre feeling to have Explorer and Visual Studio show up in Exposé), it works great. They even fix the track pad issues, it works much better than in Boot Camp natively. I’d definitely want to upgrade the RAM on my machine if I did this regularly, but it’s certainly solid. But, for me, it’s all about the 3D, so let’s get to that.

I ran through a few GPU tests, after having to wait an hour or so for Vista to go through its background update processes so that it stopped sucking CPU and RAM, distorting my performance sampling (grr), and sure enough all the OGRE demos ran ok. They did, however, run pretty slowly; simple fixed-function demos that usually run at 2000fps on this machine ran at about 300fps, and moderately complex SM2 demos usually pegged at 250fps here ran at about 40fps. Highly complex SM3 demos (e.g. OgreSpeedTree) were unfortunately almost unusable. So, you can expect a pretty steep performance penalty of at least a factor of 6 times running virtualised 3D in my experience. Clearly, at this performance degradation, this would be no good for me as a sole Windows option on this machine. But, it still is valuable, since it’s using the Boot Camp partition, I can use this to quickly test things for compile & run-time behaviour, albeit at unrealistic performance levels, and reboot ‘properly’ into Windows for the times where I want to performance test or demo something.

That is, I would be able to if it wasn’t for Windows Activation. As we all know, this is Microsoft’s anti-piracy strategy and involves requiring you to re-activate your copy of Windows within 3 days if a ‘significant’ hardware change is detected. Since a VM basically emulates all of the core system processes, inevitably Windows sees that as a ‘significant’ change and requires you to reactivate. You can do this online, but only 5 times, after which you have to phone someone every time you want to do it. If I used my Boot Camp partition in both modes, as is useful to me, I’d have to reactivate way too often.

There are some threads and articles about this issue on the VMWare site; they claim that provided you follow this article, then the Windows activation monster will be sated and leave you alone when switching between Boot Camp and Fusion 3. No such luck for me, I’ve rebooted many times between Boot Camp and VMWare and have had the VMWare Tools installed since the first boot, and all I continuously get is an activated product when I use Boot Camp, and a ’2 days to activation expiry’ message in VMWare. I’m not the only one: people are reporting this issue in the forums, including with Windows 7.

If they can’t get the activation issue fixed, so that I can choose to use Boot Camp and VMWare interchangeably, this effectively kills the product stone dead for me, as with every other virtualisation product before it. Which is a shame, because apart from this,  it works great. It’s Windows fault of course, rather than VMWare’s, but regardless the effect is the same. This is a perfect example of bad, customer-unfriendly DRM – it’s especially galling to have MS accuse me of stealing when in fact I’ve paid through the nose for many, many copies of Windows before, and am set to do so again with their expensive Windows 7 (Pro+) pricing – where they still seem to stick to this ridiculous principle. Another option would be to free up a spare Windows license from my collection and have separate Boot Camp and VMWare partitions, but I’d really prefer not to burn the time & disk space on maintaining two installs.

Fix the activation issue and I’ll be buying a copy of this plus a 4GB RAM upgrade immediately. Until then, it’s another nice product that’s scuppered by one small, but nevertheless major issue.

Edit: Ok, I’ve resolved this issue. Actually the original KB article on the VMWare site was correct, I just didn’t follow it correctly. In my defence, the key to success is in the small print at the bottom of the article rather than the ordered steps: you have to make sure that you’ve booted once into Boot Camp with the VMWare Tools installed, then afterwards to start the VM under OS X and reactivate Windows. Having done this, both my hosted VM and Boot Camp configuration show up as ‘activated’, which is precisely what I need.

So, now to buy a copy of this puppy and get 2 new sticks of RAM to shove in the laptop.

A few weeks ago I decided to start seriously investigating switching to a DVCS. I’m currently up to my eyes in work and haven’t really had time to progress that in the last few weeks; however some absolutely abhorrent performance / reliability problems with Sourceforge’s Subversion server made a large merge process so costly to me (in the end I had to commit in small chunks, breaking transactional consistency, and it needed so much babysitting because of the speed / reliability it took me 4 bloody hours just to commit!!) that it bumped it up my agenda a bit. I don’t have to do merges / commits of that size very often – in this case the problem was wholesale license header changes for our MIT switch – but still, it’s totally unacceptable to have to deal with that. I raised a support request with Sourceforge, but I’ve seen other reports of bad SVN performance from several weeks ago from others, so I’m not holding my breath. It’s time to start considering alternative hosting I think.

I’m not done with my evaluation yet, because I just don’t have the dedicated time I really want to spend on this right now. But, here’s my early-stage results.

Git

I’ve discussed before that I don’t like the where Git has come from. It’s overly complicated, Windows support was clearly not a priority, and it switched existing VCS terminology around just for the sake of it a lot of the time. It practically shouts “I’m different, live with it!” at you, which is frankly a typical elitist geek attitude and not one I particularly respect. This attitude permeates the documentation, reinforced by the absolute insistence of most tutorials that you need to understand Git’s underlying data model before you start using it. Er, right – sorry, but when normal people want to learn how to use a new tool, we just want to know how to use it, not how it’s built. If understanding how it’s built is a prerequisite to using it, then I’m sorry, it fails miserably at being user friendly.

Nevertheless, it’s fast, it’s efficient in storage, it’s the most popular & fashionable DVCS (probably due to GitHub) and that has weight. Of particular interest to me is that TortoiseGit has come along in leaps and bounds, and is really quite pleasant to use. Of course, the reason it’s pleasant is because it hides the majority of the nonsensical changes that Git decided to make to existing VCS terminology; for example ‘Revert’ in TortoiseGit does what you expect (undoes your working copy changes), rather than needing to use ‘git reset –hard’, which is only intuitive to those who live on Mars (git revert, in contrast, records a new commit to undo a previous commit – why the hell do you need a special command for that??). Thus, it’s an odd situation – using TortoiseGit is pleasant, but only because it ushers the underlying git behind a curtain and gives you what most people really wanted from it in the first place. The downside is that using TortoiseGit really doesn’t teach you how to use the command line very well, like most other VCS tools do. In fact, it may well mislead you into thinking Git is friendlier than it actually is. For example, it saves you from the ridiculous need to remember the “-a” argument to “git commit” – without which what you actually get in your commit is the state of the file when you did “git add”, not the version in your working copy. If that makes sense to anyone, raise your hand. Thought not.

Hosting & collaboration wise, GitHub seems very good.

Mercurial

Mercurial on the command line is nice. It behaves the same way centralised VCS’s do, except in the cases where it needs to be different. This is pragmatic design – not being different just to make a point, but being different where it needs to be. It doesn’t break old concepts and does what you expect it to, and contrary to what some people think, that’s a very valuable feature.

It’s not all roses though. TortoiseHg is clunkier than TortoiseGit, despite being based on a more intuitive core tool. The UI just feels a bit wrong (like putting action buttons on the toolbar – who does that?), and I’ve sworn at it for being unintuitive more than once. The other problem is that the Mac GUI tools are not really that great either – MacMercurial only allowed me to do a subset of the operations I needed to do, and Murky just crashed when I tried it. GitX in comparison works quite well on the Mac.

So despite a more intuitive command line and core concepts, and a more pragmatic approach generally to DVCS for ‘regular’ people, when it comes to GUIs Mercurial lags a bit now. This was unexpected to me since it is Git that has traditionally been poor on the GUI front. There are also a few other minor issues like branches being totally permanent and needing to be globally uniquely named, which can make local experiments more cumbersome.

Hosting wise, BitBucket seems quite competent, if a little less polished than GitHub.

Bazaar

I’ve only just started experimenting with Bazaar, and so far I’m quite impressed. It has the pragmatic approach of Mercurial, but also has a built-in GUI which is really quite nice to use and leads you through the initial setup and configuration. There’s also TortoiseBzr which feels somewhere in between TortoiseHg and TortoiseGit. I haven’t tried it on the Mac yet. Performance was always the issue listed as the major downside of Bazaar, but this has improved since 2.0 and while it’s not as fast as Git, it seems to be fast enough.

The main downside for Bazaar is adoption. It trails both Git and Mercurial in terms of the number of people using it, and therefore adopting it for a public project would have the disadvantage of making people use a tool they’re less likely to already be familiar with. Also for hosting, Launchpad is quite new; it looks quite good, and has more features than GitHub, but it doesn’t have the option to host private projects (not an issue for Ogre of course) or a graduated commercial plan – you can self-host of course but that’s not as easy.

Conclusions so far

The sad fact is that none of the 3 are an instant win for me; they all have positive and negative aspects. Summary so far:

So, I’m basically in a no-win scenario. If I pick Git, it’ll work fine via the GUIs but it’s too easy to screw things up when using the command line, and I’m bound to get annoyed at the needless obscurity from time to time. But, lots of people will be happy to use it. If I pick Mercurial, I’ll be happier with the overall core concepts & command line, but the rough edges on the GUIs are going to annoy me day to day. But, quite a few people will be happy with it all the same. If I pick Bazaar, I’ll be happy with both the core concepts and the GUIs, but being the least fashionable option almost no-one in the community will be happy that I picked it over the other two, and lots will bitch about having to use another tool.

I’m reluctantly acknowledging that the least of the evils appears to be Git right now, even though I personally hate its underlying interface. Somehow it feels wrong to only like using it when it’s hidden beneath a GUI – I’ve been a regular user of the command line for CVS and SVN for the best part of a decade, and I like being happy with both modes. I can imagine tolerating Git’s command line, but never liking it just because of the unnecessary idiosyncracies (like commit -a).

I still have lots more detailed tests to do anyway, which will have to wait a month or so until I have more time.

I can’t believe this is the first time I’ve needed this on OS X, but it came about from needing to write a document for a European customer and suddenly realising I didn’t know how to make an umlaut on my Macbook Pro’s British keyboard. On Windows I might fire up the Character Map, but I didn’t know how to do it on OS X. Here’s what I discovered:

1. OS X friendly apps like Mail, Safari, iCal and even Firefox have a ‘Special Characters’ entry on the Edit menu which brings up an equivalent of Character Map.
2. For less OS X friendly apps (like Open Office), you can add a menu bar item to do the same everywhere under System Preferences, Language & Text, Input Sources – check the Keyboard and Character Viewer option and make sure the Show Input On Menu Bar is enabled. Then you just click the new icon on the menu bar every time you need the character browser.
3. The most common ones have keyboard shortcuts which modify the next character you type afterward – Option-e puts an acute accent on the next character you type, Option-` is a grave accent, Option-u is an umlaut, Option-i a circumflex, Option-n that weird Spanish squiggle While experimenting I found the ® and © symbols too (Option-r and Option-g respectively, Option-c is the cedilla)

So there you go – useful stuff if you’re on a British (or presumably US) keyboard and need to deal with non-English names from time to time. If you already knew this, great – this is just for people in my position who have to use these characters rarely and haven’t encountered it on a Mac yet.

It’s nice when software reflects a programmer’s sense of humour and humility. This message appeared when I restarted Firefox 3.5.3 after an XP crash:

Bravo – thanks for making me chuckle, and thus forgive you instantly for any error (and it might not even have been yours). Bless.

Ok, so I discovered a number of shortcomings in my recent attempt to sync a folder in one direction to Amazon S3 using encryption, the most important of which was that it wouldn’t resume a failed transfer efficiently, which in the case of large transfers wasn’t at all ideal (as I learned to be own cost – damn my 256k upload speed).

So, this is attempt number 2. I decided to completely rewrite the script in Python instead to give me some more flexibility, coupled with the availability of Boto, a nice Python library for accessing all the Amazon Web Services. Rather than rely on just local information, or even date/time stamps, I decided to use hashes to track whether files were different. Amazon already stores the MD5 of the file you upload to them and makes that available without downloading the file, but that’s no use when you encrypt your files before uploading them, because the MD5 is of the encrypted contents rather then the original; so unless you keep the encrypted copies around too, or encrypt the local files again every time just to check the match (expensive if you’re dealing with large files) you won’t be able to compare them – I think this is the reason why ‘s3cmd sync’ currently doesn’t support encryption.

So, I decided to use S3′s ability to store custom metadata in keys, and stored the MD5 hash of the original file against the encrypted contents that I uploaded. That way, I can check the hashes against each other pretty quickly without having to re-encrypt the local files. If the hashes are different, I encrypt and upload. This approach trades a bit of preprocessing against avoiding uploads, so it’s likely to be more efficient on small groups of very large files rather than lots of small files – that’s how I use S3 for my backups of course. It also means I don’t have to worry about timestamp variations, it’s the content of the file that is the driver of whether it’s uploaded or not.

So, here’s the new version. It’s a bit more powerful than the last one – I’m calling gpg myself now so you have the choice between encrypting using public keys (more secure, and the default), or using symmetric encryption with a passphrase. You need to install Boto before you can run it, and it depends on Python 2.5 with hashlib installed. I’ve run it on both Linux and the Mac, it should work on Windows too provided you take the trouble to set up Python and GnuPG, but I haven’t tried; my Linux (apt) and OS X (macports) setups make these things quicker so being short of time I just went with that. Here’s the usage from –help:

Usage: s3putsecurefolder.py [options] source_folder target_bucket gpg_recipient_or_phrase

Options:
  -h, --help            show this help message and exit
  -n, --dry-run         Do not upload any files, just list actions
  -a ACCESS_KEY, --accesskey=ACCESS_KEY
                        AWS access key to use instead of relying on
                        environment variable AWS_ACCESS_KEY
  -s SECRET_KEY, --secretkey=SECRET_KEY
                        AWS secret key to use instead of relying on
                        environment variable AWS_ACCESS_KEY
  -c, --create          Create bucket if it does not already exist
  -v, --verbose         Verbose output
  -S, --symmetric       Instead of encrypting with a public key, encrypts
                        files using a symmetric cypher and the passphrase
                        given on the command-line.

Once again, no warranty is given, MIT license. If you see that I’ve done anything dumb, let me know

Edit: this script is deprecated in favour of a rewritten version 2.

I use Amazon S3 to host large media files which I want cheap scalable bandwidth on, and for expandable offsite storage of important backups. I used to have some simple incremental tar scripts to do my offsite backups, but since I moved to Bacula, I’ve just established an alternative schedule and file set definition for my offsite backups, the critical subset of data I couldn’t possibly stand to lose (like company documents). Since I was refreshing all my procedures and tarring the Bacula volumes no longer made any sense, I rewrote my script for putting the resulting backup data on S3.

The prerequisite in all cases is s3cmd, which is pretty mature now and available on most distros (“apt-get install s3cmd” and you’re done on Ubuntu). s3cmd actually has a ‘sync’ command, but firstly that tries to sync in both directions, which I don’t want (I know in theory it should never overwrite any local version so long as I don’t update the remote copies from somewhere else, but I’m paranoid when it comes to my backups and prefer to be explicit), and secondly it obviously has to connect to S3 to determine the sync status, wheras I always know whether I need to upload new files just from my local environment (and S3 charges per request – not much, but it’s not zero and it’s the principle of the thing). So, I decided not to use the ‘sync’ command, and just determine locally what new files I needed to ‘put’ on the server.

Secondly, encryption is a must, since some of the data is sensitive and I don’t want to trust anyone else with it. I used to manually GPG my tarballs before uploading them, but I noticed that s3cmd supports an encryption option too. It just uses GPG anyway, just in symmetric form rather than asymmetric like my version did (translation: you use the same passphrase for encryption and decryption; a little less secure than using generated public/private keys but still ok so long as you pick a good passphrase and look after it). The default symmetric algorithm in gpg is CAST5 which seems pretty good, although you can change it if you want by editing your s3cmd config file. So, I decided to give it a try – after you configure s3cmd to use encryption, it actually automatically decrypts too when you pull the data back (symmetric key, remember) – being distrustful, I pulled the data back from S3 in a different environment and examined it, and it was indeed complete gibberish, but decipherable with the passphrase. Good stuff.

So, here’s my little script which will upload the encrypted contents of a folder to S3 – just the contents that have been added or updated since the last sync of that folder, and will encrypt them by default. I just run this on a cron schedule now and it seems to work fine. License is MIT, use at your own risk, no warranty is given that it won’t destroy every file on your machine or eat your children. Usage is like this:

s3putsecurefolder /my/source/folder my.s3.bucket

Edit: it was brought to my attention that Amazon have made it easier to create pseudo-folder structures in S3 buckets since I last tried to do it (I swear it used to throw out keys with forward slashes in them, I had to mangle my names last time I did this), so I’ve updated the script to allow nested folders too.