SteveStreeting.com

Our rampantly consumerist world has many facets, pros and cons to it, but one thing has so far been perceived as a universal constant – the quest for the ‘next big thing’. That one product, or class of product, that every man, woman and small furry creature from Alpha Centauri wants to get their hands on. In the technology world, analysts have long since been riding the gravy train of purporting to be able to peer sagely into this murky future in order to extract those world-changing gems that everyone would be invested in.

I don’t think this is the case anymore, at least not to the extent that one product or product category can be seen as ‘the future’. In much the same way as in days of yore there were only a small handful of TV channels that everyone watched, compared to now where everyone has a million channels plus the Internet to cater for their media absorption needs, I think the technology needs of the general public can no longer be stuffed into one universal pigeon-hole as has happened in the past.

Up until recently, everyone felt they needed a PC of some sort, and most of them also felt they needed to use Windows because that’s what they used at work and also what most PC vendors pre-loaded their machines with. Cue a huge homogenous mass of people on the same technology – exactly the sort of ‘next big thing’ that has been common over the last 20 years.

But, as computing resources get smaller and more connected, people are realising that they don’t necessarily need all the things a PC can do, all the time. Things we would traditionally consider to be ‘computing devices’ are fragmenting along functional lines in much the same way as household devices have always been this way – your toaster, your microwave, your TV. The idea that there needs to be a single device that has the ability to do everything is something that increasingly only technophiles will hold on to, because specialism almost always means an improvement in the user experience for the given functional subset. Think about it – sure, you could build a device for the home that let you wash dishes, watch TV and toast bread all in one package – but why would you? The toaster works well doing what it does, taking up only the space it needs to, while the dishwasher and TV are specialised for their tasks too. Plus, you can use them simultaneously for different things. Sure, household devices have a common medium – plumbing, electricity etc – but they’re inherently separate, and all the better for it.

Computing devices are really no different. Over the last 20 years we’ve been conditioned to expect that we all need a common beige box that does everything anyone could possibly need, but in fact our lifestyles don’t agree with that at all. The common medium of the Internet is ubiquitous, but apart from that we all want to do different things with technology, and even within our own lives we need different things in different circumstances. When I want to do intensive tasks like writing code or editing video, I want a full keyboard, a hi-res screen and a lot of processing power. But when I’m just checking my email on the go, I want something small, portable with a good battery life, and I’m willing to sacrifice powerful CPUs, large screens, and full keyboards for that. If I’m on the sofa and want to check a website or read an article, a pad-style device would work best for that – bigger than a phone, but more casual and form-friendly than a laptop. Even in the context of a single user, our lives are not geared to single devices that do everything, and in fact there are hard limits that prevent any one device, even the best smartphone in the world, from fulfilling this – even if you could shoehorn the power into a smartphone, you’ll never replicate the full keyboard or screen short of things seen only in Inspector Gadget. So it’s not at all surprising that now that technology is allowing these devices to morph into more functionally specific roles, people are snapping them up – much to the horror of people that have a vested interest in the PC being the future of everything of course.

Cloud computing may be the only fashionable technology that cuts across all of this (hence why everyone is so scared of Google), but even then, people (particularly businesses) are just not ready to give away control of all their data just yet, and roaming data charges – since you’re not always near a free Wifi spot – are still nowhere near the place they need to be at for people to be able to rely on non-local storage entirely. So again, cloud computing is going to co-exist in the overall technology soup with everything else.

I think the next few years are going to be more interesting in technology than the last 20 have been by quite a long way, simply because of the way it’s going to blend into our lives better. Standard office bureaucracies may well be locked into the standard Microsoft PC / Server model for quite a while yet, and power users (me included) are still going to be buying PCs and laptops in addition to specific devices, but outside of that, things are set for major change in multiple directions. I like that – technology, like fashion, should be a personal choice, tailored to your lifestyle, varied and multidimensional depending on your frequently changing environment and needs. The one-size-fits-all model is dead, and I don’t think many outside of Redmond will mourn its passing.

For 18 months I’ve been told by a succession of doctors and physios that I didn’t have anything structurally wrong with my spine and that my bouts of back pain were simply ’standard non-specific back pain’ – ie muscle problems that I should just take NSAIDs for and exercise more. I’d been a bit skeptical because the problems were occasionally quite extreme and seemed to be always centred on one particular location (the joint just at the bottom of my ribcage), but after getting many opinions and one set of x-rays I went along with it.

Things have been quite good recently, up to mid-February when I had a bit of a relapse for a few weeks after doing a little too much. I raised it with my doctor again, explained that I’d been doing all the exercise and going to the gym as recommended, and yet it still flared up at what I considered to be fairly minor provocation. He scheduled me in for another set of x-rays which I expected to not come back with anything conclusive since the last set didn’t (and you can’t get into the MRI scan here unless you go through this step again first, allegedly). They took more pictures this time but I didn’t expect much given all the opinions so far.

Imagine my surprise therefore that when I got the results today, they actually had a concrete explanation for me. Apparently in my lower thoracic (ie exactly where I’d been pointing all these months) I have some disc degeneration and calcification going on, which is what is causing the stiffness and pain. This is something that happens with age anyway, but given my relative(!) youth (36) they thought it looked like it might be a result of either a trauma such as a sports injury – I can’t think of anything – or sometimes they see it in people who were child gymnasts – again not something I can attest to! Basically, something has happened to make my spine degenerate in that area faster than it should have done for my age. Too many hours spent stressed out at a desk may have been a contributing factor in that, although he thought it would have to be a lot of hours and probably combined with other factors.

So anyway, the ‘good’ news is that I actually have a reason now, an explanation for why I’m so susceptible to strains and stress on my back these days. In a way it’s nice to have something to point at. The bad news is that this isn’t fixable, it can merely be managed via careful exercise and lifestyle changes – many of which I’ve made already but I probably need to go even further. The prognosis is that I should be able to live pain-free so long as I manage it carefully over the long term to stop it degenerating further.

Following this analysis, I’ve been prompted to make a decision which I’ve been reluctantly considering for a while anyway – I’m retiring as OGRE Project Lead. I’ve thoroughly enjoyed my 10 years leading OGRE from unknown personal project to where we are today, but leading an open source project requires an enormous amount of dedication, passion, and above all an awful lot of time spent at a keyboard, most often in addition to a ‘regular job’ with which to pay the bills, and I feel I just can’t give that to the level that’s required any more. It will be with no small amount of sadness that I finally take off the leader’s hat – which by now is quite battered and worn in.

I still intend to be around and involved in the project – I’ll be contributing some code, giving advice when it’s wanted, and overseeing the establishment of an OGRE Foundation to handle the donations and funding side, but the days of me living and breathing OGRE, vetting every change, and being the person with whom the buck stops when there’s a bug, will be over. I’ll basically be contributing what and when I can, but shrugging off the responsibility and expectation that is inevitably associated with being the lead developer.

We have a great team and community around OGRE and I’m sure the project will be fine with me taking a more back-seat role – time for younger and less physically challenged developers to step into the limelight

When I was evaluating Mercurial, I touched briefly on Mercurial Queues (MQ), but at the time felt that they were probably a little more complicated than I needed. After all, one of the things I liked about Mercurial was that it all felt rather straightforward and intuitive, while MQ seemed downright confusing. However, I’ve just discovered a really powerful use case that has meant I’m now committed to using MQ for my every day workflow. But I’ll come back to that.

First, some explanation of why I revisited this. I’ve been working with Mercurial in a live situation now for a few weeks now, and my experience has been uniformly good. Particularly the main thing that used to be a total pain in my ass before – merges – has become infinitely less frustrating, and I even managed to use it to take a client’s highly modified version of Ogre which was forked from quite an old base, and bring it up to date with later versions via 3-way merges in a fraction of the time it would have taken me with Subversion’s relatively brute-force, roundtrip-generating merge system.

However, one thing that has come up is that some other team members have found the ‘uncommitted changes’ error message annoying – that is, there are some things that Mercurial won’t let you do if you have uncommitted changes; mainly merge and rebase, since both of these work on committing (or committed) changesets. Some of this confusion has been caused by me telling people to always use ‘hg pull –rebase’, to avoid unnecessary merge points when parallel development has happened, but the knock-on effect is that using the ‘–rebase’ option makes Mercurial stop if you haven’t committed your current work.

Resolving This Without Mercurial Queues

1. Finish what you’re working on and commit it before pulling new changes (with –rebase). In fact, most of the time this is just fine, unless for some reason you’re dependent on someone else’s work to finish yours. Some people got into the habit of pulling SVN changes every morning regardless of whether they’re in the middle of something or not, but actually you don’t need to, and in fact doing so can introduce unexpected and unwanted variables into your existing work. But, there are some times when you genuinely want to pull other people’s changes before you’ve finished, so..
2. Drop the ‘pull with rebase’ and use a regular ‘pull with update’ instead, ie “hg pull -u”. This is compatible with outstanding uncommitted changes – however if you’ve also got unpushed changesets of your own locally, bear in mind you’ll end up with multiple heads (branches) after you pull, and you’ll need to remember to call “hg rebase” directly (instead of as part of your pull) before you push next time to unify them.
3. Use “hg shelve” (extension) or simply ‘hg diff > somepatchfile’  to stash your uncommitted changes, then pull –rebase, then unshelve  (or import / patch) your changes in again. However, if you’re considering this option I urge you to try to use MQ instead.

Mercurial Queues – juggle many unfinished changes at once

The important thing to understand about MQ, which I didn’t grasp at first, is that it’s basically just designed to automate the process of using ‘hg diff’ to generate named patch files which you stash away somewhere as an unfinished change. I used to do this all the time in CVS and SVN when working on multiple things that weren’t quite ready to be committed yet – dump my current changes to a patch file, put it on a server or USB stick (so I can apply it on other machines & back it up), revert my changes then start work on something else, pulling the patch file back again when I want to re-start it. You can still use this manual approach with Mercurial of course, but this is also exactly what MQ does, it just has a bunch of explicit commands for it, and keeps track of which patches you have applied already in your local copy.

For example, ‘hg qnew mypatch’ creates a new patch and imports your current uncommitted changes to the patch (after which your ‘hg status’ shows as unchanged since hg is MQ-aware). It’s actually created a patch file of the changes in .hg/patches/mypatch. If you subsequently make changes, hg status shows the changes since you did the hg qnew, and you can update the patch to the latest set of changes by using ‘hg qrefresh’ – note how this just updates that patch file, it doesn’t create any changesets or anything.

If you want to go back to the status of the repo without this patch applied, just call ‘hg qpop’, or ‘hg qpush mypatch’ to bring it back. The reason for the ‘push’ and ‘pop’ concept is that each patch (which you create with qnew, you can call it at any time to spawn a new patch with just your current alterations) is considered to be in an ordered stack, with each one depending on the one underneath (and applying the patch at the top of the stack implies applying all those beneath it too). You can navigate around the stack whenever you like, qrefreshing individual patches (remember, they’re not changesets so they’re mutable – this is how you get around the fact that Mercurial’s history is not changeable), but you’re always in a stack. This is the one thing I’m not so fond of about MQ, I think patches should be in a list of stacks to allow you to have unrelated (and potentially breaking) sequences of patches tracked differently in parallel. The sharp-minded among you will note that git’s equivalent – which is to use local branches – effectively allows this by nature of the branch structure. If you’re really in need of this behaviour, you could try the LocalBranch extension or use local clones instead of MQ.

However, there’s something else I particularly like about MQ over the local branch approach, and that’s what happens when you want to synchronise your work-in-progress changes with other machines (or other people).

Repositories within repositories

When using local branches (in Git or via LocalBranch), you’re always working with regular committed changesets. They’re not pushed to others by default, but if you want to synchronise them you have ways to force them to push (or pull) – for example when you work across multiple machines. You might do this direct, or if the machines are not on at the same time you may do it via an intermediate server or USB stick. The main problems I have with this approach when related to work-in-progress changes are:

• Inherently these changesets are not finished, and you’re not going to publish them as-is elsewhere. In Git you can collapse them together later of course with the finished state, but it means that even locally you’re synchronising unstable state, which is fragile if you screw up
• If you need an intermediary for the sync (server or USB stick), it has to be a full repository clone even though you only need your WIP changes.

Of course, you could generate patch files yourself in both cases to remove both of these issues. But that’s why MQ is ideal, because it already does that for you, without you having to track external patch files. And best of all, it has a simple way to synchronise and version control just those mutable patches with others, without ever raising the issue of history modification: it simply creates a Mercurial repository within the .hg/patches directory! This repository only contains your MQ patches, and can be safely published, synchronised without ever looking like a proper ‘history’ of the main repository. Best of all, most of the regular hg commands take a ‘–mq’ option to perform actions on this nested patch repository instead of the main repository.

Putting it all together

Since Mercurial 1.5 you haven’t strictly needed to call ‘hg qinit’ to start using Mercurial Queues, but in order to create a nested repository for sync, you still want to call it, like this:

hg qinit -c

That will create a nested repository in .hg/patches which maintains its own history and can be synchronised with others, independently of the main repository. You might want to take the opportunity to edit .hg/patches/.hg/hgrc and provide a default remote path if like me you want to specify a server you will use to keep all your WIP patches for this repository.
So let’s say you’ve been doing some work on a change, but it’s not ready yet and you want to put a stake in the ground where you are without committing. Simple, you just do:

hg qnew awesome-feature-1 -m"I'm working on an awesome feature"

That says “create me a new patch in .hg/patches/awesome-feature-1 with my current changes, and give it an initial description” – you can change this description later whenever you want when you update the patch with qrefresh, see below. At this point hg considers that you have no local uncommitted changes anymore so you can happily do rebases and merges without the ‘outstanding uncommitted changes’ message. But, there’s more, you may want to do the following so you can sync the current state of this patch with other machines:

hg commit --mq -m"My awesome feature WIP #1"

Woah, woah, I said you didn’t want to commit, right, because this is a work in progress? Notice though, that I called commit with the ‘–mq’ option, which means ‘commit changes to the MQ patches repository’, and not ‘commit working copy changes’. If you look now, you’ll see that you have a patch file called .hg/patches/awesome-feature-1, which was automatically added to the nested patch repository when you called ‘qnew’, and which was committed to that repository in its current state (remember, this patch repository is completely separate from the main repo).

If I wanted to push the current state of my WIP patches to somewhere else, I can do it now (I’m assuming that I’ve already edited .hg/patches/.hg/hgrc to include a default remote path:

hg push --mq

That pushes the commits I’ve made to the patch queue (not the main repo) to that remote location. So for example I use a small area on a local server to track this, but there’s no reason I couldn’t push this somewhere public too if I wanted – again all explicitly separate from my actual repo, which is perfect. All the other main commands take a –mq option too, so you can use “hg outgoing –mq” etc too.

At this point, I could carry on working on my same patch, and every time I wanted to update the patch file in .hg/patches with my local changes I would call:

hg qrefresh

You can use the ‘-e’ command if you want to update the commit message that will eventually be used. And again, if I wanted to commit the changes to that patch, I’d call ‘hg commit –mq’ (and potentially ‘hg push –mq’). If at some point I wanted to quickly work on a different change, I could call ‘hg qnew’ again to create a new patch file based on the current diff, and deal with that separately. To navigate around the patch queue to identify which patch I’m currently working on, I’d use qpop and qpush.

Finally, at some point I’m going to want to turn these patches into real commits. Assuming that my commit message is already correct in the patch (if not, use ‘hg qrefresh -e’ to edit it), and that your current position on the patch stack is the position you want to create a proper commit for, all you need to do is:

hg qfinish --applied

Which will move the patches you currently have applied into the real Mercurial repository history, ready to be officially pushed with everyday work. Alternatively if you want to just pick a revision or range of revisions, you can name them instead of using –applied.

Conclusion

I didn’t really see the advantage of using MQ over just keeping my own patch files generated from my working copy via ‘hg diff’, particularly since MQ makes you work in a stack-based structure instead of via independent patches, until I learned how to use the nested patch repository. Manually handling patch files is fine, but once you get more than a couple of them on the go it can get tricky to manage. Having an automated approach to this with the nested repository works really very well, so I’m now using it as my primary method of managing work-in-progress.

Git users will no doubt think the local branch approach is better, and in at least one way (not having to keep patches in a single stack), it is – however I personally like how MQ mirrors my normal approach to unfinished changes as ‘volatile patches’ rather than treating unfinished changes as regular changesets and relying on history modification to fix them up later, and as someone who regularly moves between machines, MQ’s ability to sync patches separately from the main repository is very valuable. As is ever the case with Mercurial and Git, the perfect solution would probably be some combination of the two.

I’ve had requests to post the OS & browser stats for the OGRE site, which I didn’t include in my previous demographics post, so here we go.

A caveat to start off with – as a programmer-oriented site our users are obviously a little different in their choice of tech to the population at large!

Operating Systems

Not really a surprise there, Windows dominates the landscape, with Linux and OS X pulling up the rear. Personally on the desktop I’m a Windows and OS X user so my visits are contributing to those rows. There’s a decent showing for the mobile platforms too, iPhone and iPod particularly, a fair few on Android and even some early iPad hits.

Windows Breakdown

It’s worth drilling down into the huge Windows stat to see what versions people are using:

XP still rules the roost then, and thank goodness Vista is sinking rapidly to the bottom like the cast-iron turd sculpture it is – Windows 7 has already more than doubled its share. And it’s amusing to see that a couple of crazy b?st#rds are still running ME and NT.

Mac OS X Breakdown

I’d heard that supporting pre-Leopard versions of Mac OS X was increasingly becoming pointless since almost everyone had upgraded, and these stats bear that out – 96% of users are running 10.5 (Leopard) or better.

Even more interesting was the sole visitor running 10.7! Was that an error, or did we have someone from Apple on an unreleased future version visiting the site? And what’s that ‘68K’ entry about – someone running an Apple ][ emulator or something?

The Linux breakdown wasn’t interesting (99% ‘not set’, the rest just kernel versions), so there’s nothing to post for that – don’t email me about  missing out Linux please

Browsers

Firefox is the clear winner here at a huge 50% – this definitely reflects our developer-focussed audience. In fact, I used to be an avid Firefox user until quite recently, when the new version of Chrome added the equivalent to the extensions I relied on in Firefox, at which point I switched because Chrome is more efficient with memory in particular. IE’s share at 18% is definitely not representative of the general user population, but then our users tend to be a bit better informed than that

So there you go, request fulfilled – hope it was interesting.

Ok, so a new clause in the Terms of Service for Apple’s newly announced iPhone OS 4 is understandably causing some consternation around the internet:

“3.3.1 … Applications must be originally written in Objective-C, C, C++, or JavaScript as executed by the iPhone OS WebKit engine, and only code written in C, C++, and Objective-C may compile and directly link against the Documented APIs (e.g., Applications that link to Documented APIs through an intermediary translation or compatibility layer or tool are prohibited).”

The common understanding is that this is a shot across Adobe’s bow, but also aimed at people creating emulation environments. Potential justifications for this could include performance concerns (given the new multitasking feature in 4), wanting to avoid shovelware ports from other platforms with no iPhone-specific features to make their platform stand out, or sheer bloody mindedness and wish to tie developers directly to their APIs to minimise the chance that they’ll deploy on competitors machines.

As a general principle, I don’t like this sort of thing – telling developers what they can and can’t do is stifling. But, I had to take away one positive from it – a company telling people to use native code, instead of the opposite which I’ve seen too much of lately. In recent years, the likes of Microsoft have insisted that developers use their intermediate VM layers to deploy on some devices (XNA, Windows Phone 7) – regardless that these environments have about 20 years less maturity (in terms of libraries and existing code) than what I already have in C & C++. Having them tell me that no, despite all these great battle-tested libraries that I’m used to using, instead I have to use comparatively immature ports and replacements of varying quality, just because they tell me so. That drives me nuts – sure, let’s throw away and re-invent hundreds of functioning & tested libraries just because…well, just because! They’re old and we’re new and awesome! Hmm.

So while Apple telling developers what they can and can’t use is still very wrong from a point of principle, I’m actually glad that someone is championing native code for once, rather than pushing a VM environment. I’d prefer they didn’t mandate anything at all, but I can’t deny a certain urge to fist-bump when native code was the one to get the seal of approval, after getting the impression from other companies that they’d rather no-one had access to the underlying workings of the machine. I like native code, there’s a certain purity about it – and maybe it’s like a sad old gear-head going on about how great the old V8’s used to be, but I don’t care Mostly it’s about my frustration with being forced to discard perfectly good native libraries and look for / build replacements for no good reason.

PS For the record, OGRE on iPhone isn’t affected by this new ToS because we’re 100% native, baby.

[edit]For those pointing out that C# and such eventually run on native code anyway – that’s not the point. The point is that on certain devices – XNA and Windows Phone 7 – you simply cannot use libraries that were not written in .Net originally, meaning that years worth of dev libraries are inaccessible and need to be (pointlessly IMO) rewritten in .Net. And yes, this is exactly the same as Apple are doing here (but in reverse), if you interpret it in the strictest sense that you’re only allowed to use code written in Obj-C, C and C++. I’m just taking a perverse delight in the fact that it’s C/C++ libraries from the last 20+ years that are on the winning side for a change, instead of being the ones that are excluded (which frankly I’m completely sick of).

Almost exactly three years ago, I posted an analysis of the traffic on ogre3d.org and the rough country breakdown of our users, which is always fascinating to me. I hadn’t actually been collecting web stats on the site for about a year (the previous set-up was lost when I had to recreate the server in a hurry, and somehow reinstating it never seemed to rise to the top of my TODO list), but a month ago I finally got around to adding Google Analytics to the site. The results have been very interesting, particularly when compared to 3 years ago, so I thought I’d share some factoids with you.

Visitors still increasing

In 2007, log analysis indicated we were getting a little over 100,000 unique visitors per month; obviously this is not 100% accurate due to shared & dynamic IP addresses, people logging in from multiple sites, etc, but it’s a reasonable order of magnitude indicator. Analytics is typically more conservative in its figures, since it excludes bots better as well as non-Javascript browsers, but still in the past month (actually only 28 days) we’ve had over 120,000 unique visitors to the site – and 1.2M page views – so we’ve sustained and slightly improved our user traffic. And all without any Slashdot posts Also, this doesn’t count visits to Sourceforge, BitBucket or ome static generated HTML like our online documentation pages.

The Meteoric Rise of China

In 2007, China ranked 15th in our league table of visitors. Three years on and they’ve risen to the number two spot, comfortably surpassing Germany – at first I wondered if that was down to users there using fewer proxies, but since the figures for other countries have remained fairly stable I think the majority of this is genuinely a vast increase in the number of Chinese visitors to OGRE’s site. Here’s the top 10 countries (figures are for the number of visits):

The range of countries is demonstrated by how many are in the grey ‘others’ section (38.73%). Except for the massive change in China’s share, most of the other countries have stayed approximately in their relative positions & shares of the user population since 2007.

Region View – Europe still dominates, Asia challenges the Americas

The country view is, however, quite misleading if your aim is to decide where to locate a web server for example, because it naturally biases the figures towards large unified countries (like the USA and China), and doesn’t really show a true regional picture. For that, we have to examine the numbers (again, number of visits) by continent:

Now, even though the continent view includes Russia in Europe which screws up the locality principle a bit, even if you exclude that Europe dominates our community, with close to 1 in every 2 visitors to the site being from Europe. The Americas  and Asia share most of the rest almost equally now, which is a change from 2007 when the Americas were more dominant, and everyone else shares the scraps (3.5%). The Americas figure is made up of about 86% North America and 14% South America, and Asia is predominantly (60%) the Eastern Asia countries (mostly China, but South Korea holds its own too), with South-East and Southern Asia sharing the rest – particular hotspots there are India, Indonesia, Malasia, Turkey and Vietnam.

City Clusters

One thing I like about Analytics is the ability to drill down into countries and look at the local clustering. There are the expected clusters around cities – in the USA, the top 2 cities are unsurprisingly New York and Los Angeles, although Columbus OH takes the number 3 spot, and in the UK the clustering around London is massive – but they typically represent only about 25-30% of the audience, with the rest being scattered pretty much uniformly across most areas of the country in question. It’s fun to be able to point at almost anywhere in Europe, North America and the southern and eastern parts of Asia and to have a pretty good chance of being quite near to someone who has used the OGRE site.

The Monday morning OGRE fix

With OGRE obviously used by a lot of people in their spare time, you might expect that the weekends would be the busiest times for the site, but the opposite is in fact true – Mondays are consistently the busiest days (particularly 6-9am PST), with Saturday being the least busy. Whether this is because people are working with OGRE, or just cheekily surfing in their work time rather than face the Monday workload, is hard to verify!

Final Thought

I get a kick out of looking at these stats so I hope you find them interesting too. It’s really cool to think that there are only a very small number of countries (such as North Korea and Laos) from which we don’t get any (non-proxy) visitors from in a typical month, and it’s very interesting to see how the visitor base is gradually spreading out and diversifying, something which I’m sure every site witnesses but it’s interesting to see it in your own data. The question is – will China keep the current trajectory? At this rate, they’ll take the number one spot from the USA in only a few more years and put Asia second in the regional rankings!

Trends – or as I would call them, rampant fads populated by people looking to leverage the best buzzwords to get VCs to throw money at them – come and go. The one constant is the claim that <insert trend here> is so awesome that will universally and irreversibly replace <insert existing technology here>, to the extent that if you’re using or producing <insert existing technology here>, you are irretrievably lame, and complete strangers will point at you in the street and laugh at your horribly backward ways.

The fact is though, the best that today’s trends can aspire to is to become the existing proven technology that tomorrow’s trends will point and laugh at. That’s if they do well – most will simply evapourate and leave the world as if they never were. It’s rather beautiful in its own way, a sort of karmic circle where the unjustified elitism associated with being part of the ‘hip’ crowd is eventually cruelly punished by the derision of those who replace them.

The current trending darling is cloud computing, following in the wake of the dot com boom, the social networking explosion, and yes even open source . Let’s face it, there are quite a lot of people and companies who participated in open source not because of the fundamentals, but because for a while including open source on your corporate manifesto was a  damn good way to get funding. Now that open source is no longer a leading trend that you can sell to VCs (it’s graduated to ‘mature’ and has therefore lost its sparkle to a certain breed of person), the piranhas have swum elsewhere. Good riddance, I say.

Trends are like the Borg – they’re not happy to be just a part of a diverse technical melting pot, they have to be front-and-centre in everything, and want everyone else to be defined in   terms of themselves. So predictably, now we’re told that everything will eventually run in the cloud, and that the browser will be our only OS, and every company chasing funding right now is trying to shoehorn some cloud aspect into their corporate plans. What a load of old rubbish – while I fully expect cloud computing to be one of the ’stayers’, just like open source, it’s only going to be a part of the whole. I fully expect us to make far more use of hosted & distributed capabilities in the future, but I know for a fact that dedicated platforms are never going to go away – they’ll simply blend.

I could make all kinds of detailed arguments as to why browser based servicing of all needs is not a panacea, but there is one fundamental  issue that is most important - generalised tools and grand unified visions always fail, even when they make perfect sense to a designer or ‘visionary’.

Unified visions and perfect generalised solutions only exist in the head  of one person, usually a designer who has ’seen the future’ and realises that with some adaptation, he can express all things in terms of the model he has in his head, just with some funky parameterisation. Eureka!

But, regular people don’t want generalisation or unification, only designers do. You’ll generally get a good response from developers, technicians and sometimes ‘extreme power users’  if  you pitch highly adaptable generalised toolsets to them (open source anyone?), because they are adapters and creators, but try to package that approach into an end product for the masses and it just won’t work. At the sharp end, all that matters is that a piece of tech does the one or two main things that it’s designed for, really, really well, and everything else is irrelevant – Apple figured this out years ago, and it’s why the iPod crushed its arguably more fully featured competitors. Generalisation is just not a feature regular people want – quite the opposite, they want specialisation.

The idea that in future all things will be done through a general browser to the cloud is a designer’s vision that will never happen. In the same way that the general public is moving away from using a single PC to do everything, and instead likes to use devices that better reflect the use context and purpose (but to have them all connect together), the vision of a unified application (browser) that can do everything is similarly flawed. The iPhone allegedy was originally conceived to use its browser for everything, but in practice most people preferred to use dedicated apps for each purpose (that could talk to the internet anyway) because they’re more functional.

So, who cares about trends anyway?

I recently finished Mass Effect 2 – I was reserving my judgement until the end because Mass Effect 1, while great, failed in a few areas to deliver a KOTOR-beating experience that reviewers had attributed to it. ME2 was looking very promising, but I couldn’t realistically call it until I was done. Now, I can safely say that KOTOR has finally been matched, and in some ways surpassed.

The main thing that I griped about in my Mass Effect review was that the characters were far too vanilla and predictable. The clean-cut military outfit with a couple of playing-to-type aliens (Garrus being the only exception – gotta love his dry wit) might have done it for some, but compared to the richness of KOTOR –  the confusingly ‘grey’ Jedi Jolee Bindo, the twisted but consistent morality of Mandalorian mercenary Canderous Ordo, and the delightfully psychotic and insubordinate-but-faux-subservient assassin droid HK-47 – there really was no contest. Add to that a great plot twist and soak in the juices of the best period of the poster-child sci-fi franchise and it’s a tough act to follow. Mass Effect was really good, but it couldn’t match it.

Mass Effect 2 really ups its game. The canon of Bioware’s universe is rich (sometimes anally so), consistent and built on enthusiastically by this instalment – it’s still got a long way to go before it can challenge the Star Wars universe, but I realised how much I’d gotten into it when I started explaining to my wife about Salarian life-spans and certain elements of their history, and how that explained Mordin’s attitudes in a particular conversation she was listening in on, and why the advert about Elcor versions of Hamlet were hysterical, because, you know, they don’t emote well because of the high gravity on their world making large movement dangerous so they prefix all their statements with emotional contexts, and…. well, as you can imagine she looked at me a bit curiously. I stopped and realised – hell, I really liked this universe. Ok, it was not going to hold the same place in my heart as Episodes IV-VI, plus the early Jedi Knight games and KOTOR (Episodes I-III having been stricken from the record as the frothings of a madman), but I really cared about it.

And the characters – what an improvement. There are way more of them to begin with, and almost without exception they’re interesting. The decision to include a specific side-quest for each character was a great one; some are better than others but there are some real gems that genuinely made me fond of my team, to the extent that when I got to the end I really, really wanted them all to survive for more than just the ‘No one left behind’ achievement. I was devastated when in my first play through of the end sequence, Mordin died, such that I had to play it again to make sure he survived.

Because it’s about more than just this game, or even that Mordin is just an awesome character. Mass Effect 2 reportedly brought forward 700 plot decisions from the first game, which changed how things play out in Mass Effect 2 – I certainly noticed lots of decisions that I made in ME1 having knock-on effects, and that was pretty satisfying – and they’ll be doing the same for Mass Effect 3. If I let Mordin die, he wouldn’t be around for Mass Effect 3! I could not possibly let that happen.

Speaking about the end mission (and without spoilers), I loved how it mattered how loyal your team was following all the side-quests you’d done. The more loyal and better equipped your team, the better your chances. Sure, you could have just powered up yourself and a couple of favourites, but the rest of the team would have been mincemeat and also not able to provide adequate backup. Picking team members for certain assignments and choosing how to split the team based on what you knew about them – it felt like you had a proper, functioning team that you knew something about, rather than just a faceless bunch that you min/max on away missions. I’d like to see them take this aspect further in ME3, maybe making it more of a feature during earlier parts of the game (although it definitely has the most impact at the end).

The graphics are also better than the first one; it’s a trivial thing but the lack of texture pop-in helps immensely and the world just feels more solid generally. And the voice acting is probably the best I’ve heard, stealing KOTORs crown there. The sheer amount of it is insane – not only do the 2 characters (out of 10) that you take with you interject regularly with comments, and obviously there are different sets for each mission for all of them and they often interact, but also in populated areas people are always talking. All the time. You hear them all as you walk past, and they’re often fascinating. And there’s tannoy announcements, news stations, as well as all the people you can talk to normally. Of course it’s peppered with big names too: Martin Sheen, Seth Green, Michael Dorn, Armin Shimmerman, Adam Baldwin, and more.

Ok, there are a couple of boring bits. The mining is dull – compulsive, and it does give you an edge, but still dull. The moral decisions, while more interesting and not quite as clear cut as ME1, are hardly Fallout material. But, this is space opera after all, and broad themes are the order of the day. Still, more convoluted issues such as Mordin’s personal quest about his involvement in the Genophage (engineered disease designed to keep Krogan populations under control), lend a bit more gravitas and grey areas to the story.

The only place where ME2 undershoots is the villain. ME1’s villain Saren was great, an eloquent rogue agent with a twisted but interesting agenda. Harbinger however, is forever distant and impersonal, his will done through the mostly faceless crowd of Collectors, and as such there’s a bit of a hole where a deeper antagonist character should really be. I know they’re supposed to be enigmatic and alien, and that works to a degree, but it feels like there’s something missing there.

On the whole though, an absolutely fantastic game, and I can’t wait for ME3. I actually feel like playing both ME1 and ME2 again as a total ruthless bastard, since I’ve been pretty nice throughout with my main character (barring a couple of ‘Renegade’ conversation interrupts which I just couldn’t resist). Highly recommended.

Writing good documentation is hard. While I happen to think that API references generated from source code can be extremely useful, they’re only part of the story, and eventually everyone needs to write something more substantial for their software. You can get away with writing HTML directly, and separately using a word processor to write PDFs for so long, but eventually you need a proper tool chain with the following characteristics:

• Lets the author concentrate on content rather than style
• Generates multiple formats from one source (HTML, PDF, man pages, HTML Help etc)
• Does all the tedious work for you such as TOCs, cross-references, source code highlighting, footnotes
• Is friendly to source control systems & diffs in general
• Standard enough that you could submit the content to a publisher if you wanted to
• Preferably cross-platform, standards-based and not oriented to any particular language or technology

When I came to write the OGRE manual many, many years ago, I went with Texinfo – it seemed a good idea at the time, and ticked most of the boxes above. The syntax is often a bit esoteric, and the tools used to generate output frequently a bit flaky (texi2html has caused me many headaches over the years thanks to  poorly documented breaking changes), but it worked most of the time.

I’ve been meaning to replace this tool chain with something else for new projects for a while, and DocBook sprung to mind since it’s the ‘new standard’ for technical documentation. It’s quite popular with open source projects now and it’s the preferred format for many publishers such as O’Reilly. In the short term, I want to write some developer instructions for OGRE for our future Mercurial setup, but in the long term, I’d really like a good documentation tool chain for all sorts of other purposes, and Texinfo feels increasingly unsatisfactory these days.

Having spent some time this week establishing a new working tool chain, and encountering & resolving a number of issues along the way, I thought I’d share my setup with you.

Read the rest of this entry »

I’ve been an advocate of digital distribution for a while now; I think packaged physical distribution of a product which is essentially entirely complete as a stream of data is hugely wasteful financially and environmentally. Ever since publishers stopped bothering to give you anything worthwhile in that game case – manuals these days are rubbish, carbon-copy affairs that rightly no-one bothers to read because the in-game tutorials are more interesting, and Ultima-style cloth maps and runes are consigned to history – physical game cases are doing precisely nothing but take up space in my house and making me get up to fiddle with disks when I want to play a particular game. My only possible use for a physical product these days is resale value – but then I don’t often sell on games these days anyway, and publishers are getting wise to ways to make this harder / less attractive (see Mass Effect 2’s free DLC for those with an original one-time only game code).

This was only reinforced by an article in the LA Times which included a chart about the average breakdown of a $60 game:

Basically what this indicates is that, on average, $26 of the $60 price tag, or almost 50%, goes to physical overheads – things that have absolutely nothing to do with making the game itself, and everything to do with shifting bits of plastic around. It’s also interesting to see that the platform royalty (paid to Microsoft, Nintendo or Sony) is only $7, or about 12% of the price (compare this to a typical 30% on current digital distribution). From this you can see why the platform holders are quite keen on cutting the retailer out of the loop, and game buyers would benefit significantly too.

So what would that latest AAA game cost to you and I if it were on digital distribution on launch day? This is hard to judge because currently this only happens on PC where games are already cheaper (via Steam and Direct2Drive, among others); consoles currently only offer smaller or older games so are harder to compare directly. Let’s assume for a second that the platform royalty would be higher (to pay for bandwidth, assuming they host it), but not as high as the typical indie 30% because that can no doubt be negotiated down for big publishers / releases. So let’s assume 20% for sake of argument, still quite a bit more than the return platform holders get now. Working back from that, and assuming that the publisher needs the same absolute amount of money to fund development, marketing etc, that means AAA digital games releases should cost (at launch): 27 / 0.8 = $33.75.

That’s pretty astonishing. I know some people are attached to the physical manifestation of a purely digital product – frankly, I don’t understand this attachment and all the indications are that the new generation of gamers, raised on digital music and grabbing films from the internet, don’t either – but surely if you’re offered this kind of deal for the same game, plus not having to walk down to the shop in the rain, it’s a no-brainer.

This chart actually came from OnLive of course, who propose doing things very differently. I’m not convinced that particular approach will work well enough (particularly since our local internet speeds put us at the arse end of the developed world), but I’m absolutely convinced that digital distribution in some form will be increasingly dominant in the coming years. The only people who will lose out are the retailers, everyone else stands to benefit in both price and convenience.

Of course, what’s likely to happen in the immediate term is that publishers and platform holders will have ridiculous prices for digital distribution that bear no relation to the cost structure above; the primary reasons being that:

1. Big influential retailers will pressure them not to undercut (effectively forming a cartel, which should be illegal, but it happens already)
2. They like to eat money

But, slowly and surely, provided there’s real competition, prices can’t help but come down. Personally I think radicals like OnLive are key to providing this genuine competition – there’s always talk that having competing proprietary consoles means competition is maintained, but I don’t buy it; console manufacturers know full well that they have a captive audience once a consumer has invested in the console (barring the minority that have all of them) so price elasticity for games is very much higher than normal for other products; no other company can publish a game for that platform and compete after all. It’ll take external competition to make downloadable content cost what it actually should, which means the next console cycle (most believe this will kick off in 2012/3) could be very interesting indeed.

I’ll make a prediction here: the company with the best, most practical, and most reasonably priced digital distribution model will have a massive advantage next time around. And, I wouldn’t rule out this being none of the current encumbents either. People scoffed when the iPhone came out, but effective digital distribution had an absolutely massive influence on its success; I’m sure that hasn’t gone unnoticed by the usual suspects.