SteveStreeting.com

I’m 37, and I’ve been a (professional) developer for 16 years. You would have thought that in that time, I’d have figured out an effective work style which delivered the desired outcomes (code cut, products shipped etc) without causing detrimental knock-on effects – but, sadly, you’d be wrong. I think the style in which I practiced my craft for the first 15 years of my career was much the same as every other enthusiastic developer: you put a ton of hours in. 12-16+ hour days, evening and weekend coding marathons, pizza in the keyboard, crunch times, 3am debugging sessions where you just can’t go to bed because you can feel the source of that bug just beyond your fingertips, dammit, desperate last-minute sprints to deadlines where you manage to slot that last piece in, Jack Bauer-like, just before the world goes to hell. If you’re in the demographic I’m talking about, you’re nodding sagely, and probably grinning a little too, reminiscing on past trials and glories. This sort of crazy dedication is respected in our circles, and is pretty much expected of any developer who has claimed to earn their stripes.

But, it turns out this kind of thing is not good for your health – who knew? Those of you who know me or keep up with my blog know that I’ve been dragged kicking and screaming away from my old ways, because of back issues that I initially ignored, then tried to cope with using token accommodations, and finally succumbed to in a big way. Being self-employed, this was a major problem. Crawling out of the pit I dug for myself took a long time and a lot of frustration – I read quite a few productivity books on the subject to try to find answers on how to keep working, and in the end found that the answers you mould for yourself tend to be the best ones. I’d like to share one of the things I learned along the way.

But I’m ‘In The Zone’!!

So, I want to talk about the biggest problem I encountered: concentration periods. I can’t sit at a desk for longer than about an hour at a time now; if I don’t get up and walk around, do some gentle stretching etc, at least this often, I’ll pay for it badly once I do move, and probably over the next few days too. I also can’t realistically work more than a standard 8 hour day without pain any more. The problem with this was that, as a programmer, the style which I developed over 15+ years involved getting gradually ‘Into The Zone’ and coding for very long periods at a time, uninterrupted. This is a common theme among coders, who like to shut themselves away for hours at a time, wear headphones to avoid distractions, have ‘quiet times’ and so on – and it’s also why we tend to react really badly when interrupted. Programming requires concentration, and concentration seems to run on a valve system – it takes time to warm up, and once it’s going, you don’t want to turn it off because starting it up again is a major hassle.

I thought there was no way around this, and had begun to resign myself to just being less productive because of it. However, over the last 6  months in particular, I’ve discovered that, far from being an intractable problem, this ‘slow warm up, long uninterrupted focus time’ approach is to a large degree a learned behaviour, and it’s possible to re-train yourself to cope with things differently. It’s a little like when people learn to adopt polyphasic sleep patterns – it’s not that you can’t do it, it’s just that when you’ve become accustomed to doing things a certain way, changing that is initially very, very hard. But it’s not impossible, given the right amount of motivation and time to adjust.

So, my goal was to acclimatise myself to many shorter work chunks during the day instead of a few very large ones, while still maintaining productivity. The key to this was to learn how to get back ‘In The Zone’ in the shortest time possible – much like the way polyphasic sleepers train themselves to achieve REM sleep more quickly. I’m mostly there now, or at least way better at it than I was, so, what techniques did I use to make this transition?

1. Embrace interruptions
   This is less of a technique and more of a deliberate psychological adjustment which cuts across all the practical approaches I’ll cover next. Instead of being the typical coder who avoids interruptions at all costs, you need to accept them, and learn to manage them better. It’s hard – you have to try to set aside years of resisting interruptions and initially, until you adjust, you’ll feel like you can’t get enough done. Many people will probably want to give up, unless there’s something specific motivating them to push through it – for me, daily pain was a great motivator. My main message here is that the transition is just a phase, and that it is possible to be an interruptable programmer who still gets things done. But you have to learn not to fight against it, hence why this is the first point.
2. Maintain context outside of your head at all times
   Much of the problem with interruptions is that of losing context. When you’re in that Zone, you’re juggling a whole bunch of context in your head, adjusting it on the fly, and maintaining and tweaking connections between issues constantly. Interruptions make you drop all that, and it takes time to pick it all up again. My answer to this was to externalise as much as possible, on as many levels as possible:
   1. Maintain a running commentary on your current task
      I am my very own chronicler. I write notes on what I’m doing all the time, whether it’s adding a comment line to a ticket, committing frequently and writing detailed commit notes (you do use a DVCS to make light commits more practical, right? ) scribbling a drawing on (ordered) pieces of paper. This really isn’t that onerous, and in fact externalising your thoughts can often help you clarify them. Basically the guide is that roughly every 30 minutes, I should have generated some new piece of context which is stored somewhere other than my head. If I haven’t, then that’s context I’d have more trouble re-building mentally if I’m interrupted. It doesn’t take much time to do, and it has other benefits too such as recording your thought & decision process.
   2. Ruthlessly ignore tangental issues
      You might have noticed that in the last bullet, I used the words ‘current task’, singular. Not ‘tasks’. There is no such thing as having more than one ‘current task’ – there is only the one task you’re actually working on, and distractions.
      We probably all use bug trackers / ticket systems to track bugs and feature requests, but when you’re working on a ticket, it’s very common to spot a new bug, or identify an opportunity for improvement, or think of a cool new feature. How many of us go ahead and deal with that right away, because it’s in the area we’re already in, or it’s ‘trivial’, or it’s a cool idea that you want to try right now?  I know I did – but I don’t any more; any tangental issues not related to what I’m currently doing get dumped into the ticket system and immediately forgotten until I’m done with the current task, regardless of their size, relevance or priority. It sounds simple and obvious, and this might even be official procedure in your organisation, but I challenge most coders to say that they actually do this all the time. The benefit is that even the tiniest of distractions add an extra level of context that you have to maintain, which is then harder to pick up again after an interruption.
      For this to work, you need a ticket system which is fast, lightweight, and doesn’t require you to be anal about how much detail you put in initially. You need to be in & out of there in 30 seconds so you can offload that thought without getting distracted – you can flesh it out later.
   3. Always know what you’re doing next
      This is one from GTD (‘Next actions’), but it’s a good one. When you come back from a break or interruption, you should spend no time at all figuring out what you need to be doing next. Your ticket system will help you here, and so will the running commentary that hopefully you’ve been keeping on your active task. If you’ve been forced to switch gears or projects, so long as you’ve maintained this external context universally, you should have no issue knowing what the next actions on each item are. The important thing is to have one next action on each project. If you have several, you’ll have to spend time choosing between them, and that’s wasted time (see the next section on prioritisation). At any one time, you should not only have just one current task, but one unambiguous next action on that task. Half the problem of working effectively is knowing what you’re doing next.
3. Prioritise Negatively
   I mentioned next actions in the previous section, but how do you decide what comes next? A lot of time can be frittered away agonising over priorities, and I used to struggle with it; I would plan on the assumption that I wanted to do everything on the list, and I just needed to figure out which I needed to do first. I discovered that I could cut the amount of time I spent on planning, and also get better, less ambiguous priorities by inverting the decision making process – to assume a baseline that I wouldn’t do any of the tasks, and assessing the negative outcomes of not doing each one. So instead of ‘which of feature A or B is more important to have?’, it became ‘Let’s assume we ship without feature A and B. What are the issues caused by omitting them in each case?’. It might appear to be a subtle difference, but having to justify inclusion entirely, rather than trying to establish a relative ordering assuming they all get done eventually, tends to tease out more frank evaluations in my experience.
4. Recognise the benefits of breaks
   Much of the above is about limiting the negative aspects of taking breaks, but the fact is, that they have many work-related benefits too. I’m willing to bet that all coders have stayed late at work, or late into the night, trying to fix a problem, only to find that they fix it within 15 minutes the next day, or think of the answer in some unlikely place like the shower. The reason for this is very simple – extended periods of concentration seem productive, and can be on operational / sequential thinking, but for anything else such as creative thinking or problem solving, it’s very often exactly the opposite. Not only do tired minds think less clearly, but often the answer to a problem lies not in more extensive thinking down the current path which you’ve been exploring in vain for the last few hours, but in looking at the problem from a completely different perspective. Long periods of concentration tend to ‘lock in’ current trains of thought, making inspiration and strokes of genius all too rare. Creativity always happens when you’re not trying, and it’s an often under-appreciated but vital element of the programming toolbox. Interrupting that train of thought can actually be a very good thing indeed.

There’s more I could talk about, but that’s quite enough for now I think. I hope someone finds this interesting or useful

I only started learning Objective-C and Cocoa in mid-May, and for the first time I think I actually have a tip to contribute to the wider community. It’s about using custom cells in NSOutlineView, but those which you want to design inside Interface Builder rather than drawing manually.

If you’re an iOS developer, you’ll be wondering why this deserves a blog post – it’s easy to do in Cocoa Touch! Well, yes it is easy on iOS, because Apple have specifically allowed you to design table view cells in Interface Builder. When targetting Mac OS X though, it’s actually pretty awkward, and here’s why: in Cocoa Touch, the class which draws the cells in a table is UITableViewCell, which is a subclass of UIView – meaning you can drag one onto the canvas in Interface Builder and lay stuff out right there. In Cocoa, in contrast, the cell is simply represented by NSCell, which is not an NSView subclass. This means Interface Builder will not let you play with them, you draw them by implementing drawWithFrame:inView: instead. I think Apple realised the problem with this design in time for Cocoa Touch but obviously felt they couldn’t break the existing Cocoa interfaces. There are also many differences between the instantiation of NSCell versus UITableViewCell – there’s only one NSCell for all rows in a table / outline view, compared to a type-indexed pool in Cocoa Touch.

The problem boiled down to this: if the contents of your table / outline cell is non-trivial, or if you just don’t want to write a bunch of layout code, it’s a PITA to implement a custom look in NSOutlineView, such as the one in the picture, especially if you want custom controls embedded in it. For my current Cocoa app, I really wanted to design my cells (and I have 2 different styles for group levels and detail levels) in Interface Builder to save me hassle, including using Cocoa Bindings to hook up some dynamic fields within. Many internet searches later, and mostly the answer I found was that it couldn’t be done. Luckily, I’m too stubborn to take no for an answer, and eventually I figured out a way to do it. Details are after the jump, with an example project to show it in action.

Read the rest of this entry »

I came across a couple of interesting issues when I came to do the first pass of writing the text for the user-visible strings I’d been setting up for a Cocoa app I’m writing (painfully slowly as I learn the nuances of the environment), and I thought I’d share them. Full details are after the jump, since I’ve embedded a large script in the post.

The basic principle for text localisation on OS X is that, like most systems, you externalise your user-visible strings in string tables and reference them by keyed aliases in code – in this case using NSLocalizedString. Apple provide a tool called ‘genstrings’ which extracts all these into a template strings file called Localizable.strings which you can then populate per language – localised files are kept in folders called en.lproj, fr.lproj etc and helpfully they’re picked up by default like this.  So far, so good.

Read the rest of this entry »

Giving up the leadership of OGRE was a sad moment for me, but in hindsight it has also been rather liberating. For 10 years I’d spent most of my energy on OGRE or on projects that were related to OGRE. There was an implicit understanding both from the community and from myself that everything I embarked on would in some way tie into OGRE – and indeed my business has always been based on a constant balancing act between how I can make a living while also promoting and advancing OGRE. I’d tended to major on the latter rather than the former most of the time as it happens, because I had an emotional attachment to the project and a feeling of responsibility and custodianship that I took very seriously. So when I finally admitted to myself that my back couldn’t take the ongoing demands of being an open source leader as well as making a living, the big question was: what next?

Over the years I’ve learned a couple of things about choosing what projects to work on – follow your gut, and work on things you’d do even if there was no money in it. Yes, you need to do a business case and convince yourself that there’s a viable market for what you have in mind, but all that’s irrelevant if you don’t feel strongly about what you’re doing, because it’s passion and enthusiasm that will get you through the difficult times. So I sat down and gave some thought to what really excited me these days, what I liked using and what technical directions piqued my interest. I still find 3D and games fascinating, but they’re far from my only interests.

So, I realised that one area that I’ve been dying to get my teeth into properly for ages but had never found the time before, was coding specifically for Mac OS X. In 3 years I’ve gone from a total newcomer to the platform, to a staunch advocate of it. However until now I’d never really had time to play with developing on it, beyond porting cross-platform C++ code and providing / using intermediate libraries. One thing I learned in those 3 years as a user was how much better applications designed for OS X felt to use compared to those that were just ported via a common UI layer (like wxWidgets / Qt), and I’m convinced now that while cross-platform infrastructural code is great, user experiences are far better when designed with the specific platform in mind – increasingly that means OS and physical device now of course. Sure, cross-platform UIs save the developer time, but the result is often a watered-down experience – early on I liked OS X applications that felt like Windows, or ran the same on both platforms – now I  do not. Such carbon-copying applications were helpful while I was unfamiliar with the platform, but now it’s just glaring to me how basic their compatibility with the OS typically is, and how the UI styles clash with the expected standards.

So, I decided I wanted to learn how to target OS X specifically, and had a couple of ideas for projects I could do with it, which meant learning Objective-C. At first, I hated it and tried to escape via more familiar technologies like Objective-C++ and PyObjC. Ultimately I found shortcomings and limitations of those routes and returned to Objective-C, and the more I used it, the more my animosity toward it diminished. In the end I realised the problem was that I needed to adapt to the environment, rather than try to adapt it to my previously learned styles and behaviours. Sure, missing elements like namespaces might still nag me, but on balance the blend of static and dynamic language elements works very well for the intended use. And besides, I really didn’t want to be ‘that guy’ – the programmer that having decided one language / tech is ‘the best’, then tries to apply it everywhere, regardless of suitability; I like to think I’m a bit more flexible & multicultural than that.

I’ve also learned that Cocoa is a very, very smart system. Mad as a bison if you’re used to other systems beforehand, but persevere with it and resist the urge to hide it under some vanilla layer that you’re already familiar with, and you discover it’s really very powerful. Not to mention the Core Animation and Core Graphics frameworks are a lot of fun.

It’s funny, I’ve spent so many years concerning myself with providing compatibility across multiple OS’s, multiple GPUs, multiple render APIs, and multiple drivers, it’s a genuine joy to actually forget all that for a while, and concentrate on an end goal with a finite number of permutations for a change – and not to shy away from using platform-specific features.

While I’m still very much an advocate of open systems, I look at things slightly differently now – that data & protocols should be open, and that we should all re-use & collaborate on common, preferably open source infrastructure (like OGRE), but that the ‘last mile’ to the user is the least suitable for generalisation, because the more specific you can make that interface to what the user expects on their OS & device, the better that experience will be. And at the end of the day to the user, that experience is the application, and thus all that really matters – and I feel that Apple gets that, in a way that very few others do.

So, I’m having a great time learning to be an Apple developer so far, I’m going to see where this takes me for a while. My gut says it feels right, and I’ve learned to listen to my gut I love the platform, it’s a total change of pace and technology, it’s something I’ve had an interest in for a while, and the Mac has quite a thriving community of quality independent app developers that I can try to join – what’s not to like?

When I was evaluating Mercurial, I touched briefly on Mercurial Queues (MQ), but at the time felt that they were probably a little more complicated than I needed. After all, one of the things I liked about Mercurial was that it all felt rather straightforward and intuitive, while MQ seemed downright confusing. However, I’ve just discovered a really powerful use case that has meant I’m now committed to using MQ for my every day workflow. But I’ll come back to that.

First, some explanation of why I revisited this. I’ve been working with Mercurial in a live situation now for a few weeks now, and my experience has been uniformly good. Particularly the main thing that used to be a total pain in my ass before – merges – has become infinitely less frustrating, and I even managed to use it to take a client’s highly modified version of Ogre which was forked from quite an old base, and bring it up to date with later versions via 3-way merges in a fraction of the time it would have taken me with Subversion’s relatively brute-force, roundtrip-generating merge system.

However, one thing that has come up is that some other team members have found the ‘uncommitted changes’ error message annoying – that is, there are some things that Mercurial won’t let you do if you have uncommitted changes; mainly merge and rebase, since both of these work on committing (or committed) changesets. Some of this confusion has been caused by me telling people to always use ‘hg pull –rebase’, to avoid unnecessary merge points when parallel development has happened, but the knock-on effect is that using the ‘–rebase’ option makes Mercurial stop if you haven’t committed your current work.

Resolving This Without Mercurial Queues

1. Finish what you’re working on and commit it before pulling new changes (with –rebase). In fact, most of the time this is just fine, unless for some reason you’re dependent on someone else’s work to finish yours. Some people got into the habit of pulling SVN changes every morning regardless of whether they’re in the middle of something or not, but actually you don’t need to, and in fact doing so can introduce unexpected and unwanted variables into your existing work. But, there are some times when you genuinely want to pull other people’s changes before you’ve finished, so..
2. Drop the ‘pull with rebase’ and use a regular ‘pull with update’ instead, ie “hg pull -u”. This is compatible with outstanding uncommitted changes – however if you’ve also got unpushed changesets of your own locally, bear in mind you’ll end up with multiple heads (branches) after you pull, and you’ll need to remember to call “hg rebase” directly (instead of as part of your pull) before you push next time to unify them.
3. Use “hg shelve” (extension) or simply ‘hg diff > somepatchfile’  to stash your uncommitted changes, then pull –rebase, then unshelve  (or import / patch) your changes in again. However, if you’re considering this option I urge you to try to use MQ instead.

Mercurial Queues – juggle many unfinished changes at once

The important thing to understand about MQ, which I didn’t grasp at first, is that it’s basically just designed to automate the process of using ‘hg diff’ to generate named patch files which you stash away somewhere as an unfinished change. I used to do this all the time in CVS and SVN when working on multiple things that weren’t quite ready to be committed yet – dump my current changes to a patch file, put it on a server or USB stick (so I can apply it on other machines & back it up), revert my changes then start work on something else, pulling the patch file back again when I want to re-start it. You can still use this manual approach with Mercurial of course, but this is also exactly what MQ does, it just has a bunch of explicit commands for it, and keeps track of which patches you have applied already in your local copy.

For example, ‘hg qnew mypatch’ creates a new patch and imports your current uncommitted changes to the patch (after which your ‘hg status’ shows as unchanged since hg is MQ-aware). It’s actually created a patch file of the changes in .hg/patches/mypatch. If you subsequently make changes, hg status shows the changes since you did the hg qnew, and you can update the patch to the latest set of changes by using ‘hg qrefresh’ – note how this just updates that patch file, it doesn’t create any changesets or anything.

If you want to go back to the status of the repo without this patch applied, just call ‘hg qpop’, or ‘hg qpush mypatch’ to bring it back. The reason for the ‘push’ and ‘pop’ concept is that each patch (which you create with qnew, you can call it at any time to spawn a new patch with just your current alterations) is considered to be in an ordered stack, with each one depending on the one underneath (and applying the patch at the top of the stack implies applying all those beneath it too). You can navigate around the stack whenever you like, qrefreshing individual patches (remember, they’re not changesets so they’re mutable – this is how you get around the fact that Mercurial’s history is not changeable), but you’re always in a stack. This is the one thing I’m not so fond of about MQ, I think patches should be in a list of stacks to allow you to have unrelated (and potentially breaking) sequences of patches tracked differently in parallel. The sharp-minded among you will note that git’s equivalent – which is to use local branches – effectively allows this by nature of the branch structure. If you’re really in need of this behaviour, you could try the LocalBranch extension or use local clones instead of MQ.

However, there’s something else I particularly like about MQ over the local branch approach, and that’s what happens when you want to synchronise your work-in-progress changes with other machines (or other people).

Repositories within repositories

When using local branches (in Git or via LocalBranch), you’re always working with regular committed changesets. They’re not pushed to others by default, but if you want to synchronise them you have ways to force them to push (or pull) – for example when you work across multiple machines. You might do this direct, or if the machines are not on at the same time you may do it via an intermediate server or USB stick. The main problems I have with this approach when related to work-in-progress changes are:

• Inherently these changesets are not finished, and you’re not going to publish them as-is elsewhere. In Git you can collapse them together later of course with the finished state, but it means that even locally you’re synchronising unstable state, which is fragile if you screw up
• If you need an intermediary for the sync (server or USB stick), it has to be a full repository clone even though you only need your WIP changes.

Of course, you could generate patch files yourself in both cases to remove both of these issues. But that’s why MQ is ideal, because it already does that for you, without you having to track external patch files. And best of all, it has a simple way to synchronise and version control just those mutable patches with others, without ever raising the issue of history modification: it simply creates a Mercurial repository within the .hg/patches directory! This repository only contains your MQ patches, and can be safely published, synchronised without ever looking like a proper ‘history’ of the main repository. Best of all, most of the regular hg commands take a ‘–mq’ option to perform actions on this nested patch repository instead of the main repository.

Putting it all together

Since Mercurial 1.5 you haven’t strictly needed to call ‘hg qinit’ to start using Mercurial Queues, but in order to create a nested repository for sync, you still want to call it, like this:

hg qinit -c

That will create a nested repository in .hg/patches which maintains its own history and can be synchronised with others, independently of the main repository. You might want to take the opportunity to edit .hg/patches/.hg/hgrc and provide a default remote path if like me you want to specify a server you will use to keep all your WIP patches for this repository.
So let’s say you’ve been doing some work on a change, but it’s not ready yet and you want to put a stake in the ground where you are without committing. Simple, you just do:

hg qnew awesome-feature-1 -m"I'm working on an awesome feature"

That says “create me a new patch in .hg/patches/awesome-feature-1 with my current changes, and give it an initial description” – you can change this description later whenever you want when you update the patch with qrefresh, see below. At this point hg considers that you have no local uncommitted changes anymore so you can happily do rebases and merges without the ‘outstanding uncommitted changes’ message. But, there’s more, you may want to do the following so you can sync the current state of this patch with other machines:

hg commit --mq -m"My awesome feature WIP #1"

Woah, woah, I said you didn’t want to commit, right, because this is a work in progress? Notice though, that I called commit with the ‘–mq’ option, which means ‘commit changes to the MQ patches repository’, and not ‘commit working copy changes’. If you look now, you’ll see that you have a patch file called .hg/patches/awesome-feature-1, which was automatically added to the nested patch repository when you called ‘qnew’, and which was committed to that repository in its current state (remember, this patch repository is completely separate from the main repo).

If I wanted to push the current state of my WIP patches to somewhere else, I can do it now (I’m assuming that I’ve already edited .hg/patches/.hg/hgrc to include a default remote path:

hg push --mq

That pushes the commits I’ve made to the patch queue (not the main repo) to that remote location. So for example I use a small area on a local server to track this, but there’s no reason I couldn’t push this somewhere public too if I wanted – again all explicitly separate from my actual repo, which is perfect. All the other main commands take a –mq option too, so you can use “hg outgoing –mq” etc too.

At this point, I could carry on working on my same patch, and every time I wanted to update the patch file in .hg/patches with my local changes I would call:

hg qrefresh

You can use the ‘-e’ command if you want to update the commit message that will eventually be used. And again, if I wanted to commit the changes to that patch, I’d call ‘hg commit –mq’ (and potentially ‘hg push –mq’). If at some point I wanted to quickly work on a different change, I could call ‘hg qnew’ again to create a new patch file based on the current diff, and deal with that separately. To navigate around the patch queue to identify which patch I’m currently working on, I’d use qpop and qpush.

Finally, at some point I’m going to want to turn these patches into real commits. Assuming that my commit message is already correct in the patch (if not, use ‘hg qrefresh -e’ to edit it), and that your current position on the patch stack is the position you want to create a proper commit for, all you need to do is:

hg qfinish --applied

Which will move the patches you currently have applied into the real Mercurial repository history, ready to be officially pushed with everyday work. Alternatively if you want to just pick a revision or range of revisions, you can name them instead of using –applied.

Conclusion

I didn’t really see the advantage of using MQ over just keeping my own patch files generated from my working copy via ‘hg diff’, particularly since MQ makes you work in a stack-based structure instead of via independent patches, until I learned how to use the nested patch repository. Manually handling patch files is fine, but once you get more than a couple of them on the go it can get tricky to manage. Having an automated approach to this with the nested repository works really very well, so I’m now using it as my primary method of managing work-in-progress.

Git users will no doubt think the local branch approach is better, and in at least one way (not having to keep patches in a single stack), it is – however I personally like how MQ mirrors my normal approach to unfinished changes as ‘volatile patches’ rather than treating unfinished changes as regular changesets and relying on history modification to fix them up later, and as someone who regularly moves between machines, MQ’s ability to sync patches separately from the main repository is very valuable. As is ever the case with Mercurial and Git, the perfect solution would probably be some combination of the two.

Ok, so a new clause in the Terms of Service for Apple’s newly announced iPhone OS 4 is understandably causing some consternation around the internet:

“3.3.1 … Applications must be originally written in Objective-C, C, C++, or JavaScript as executed by the iPhone OS WebKit engine, and only code written in C, C++, and Objective-C may compile and directly link against the Documented APIs (e.g., Applications that link to Documented APIs through an intermediary translation or compatibility layer or tool are prohibited).”

The common understanding is that this is a shot across Adobe’s bow, but also aimed at people creating emulation environments. Potential justifications for this could include performance concerns (given the new multitasking feature in 4), wanting to avoid shovelware ports from other platforms with no iPhone-specific features to make their platform stand out, or sheer bloody mindedness and wish to tie developers directly to their APIs to minimise the chance that they’ll deploy on competitors machines.

As a general principle, I don’t like this sort of thing – telling developers what they can and can’t do is stifling. But, I had to take away one positive from it – a company telling people to use native code, instead of the opposite which I’ve seen too much of lately. In recent years, the likes of Microsoft have insisted that developers use their intermediate VM layers to deploy on some devices (XNA, Windows Phone 7) – regardless that these environments have about 20 years less maturity (in terms of libraries and existing code) than what I already have in C & C++. Having them tell me that no, despite all these great battle-tested libraries that I’m used to using, instead I have to use comparatively immature ports and replacements of varying quality, just because they tell me so. That drives me nuts – sure, let’s throw away and re-invent hundreds of functioning & tested libraries just because…well, just because! They’re old and we’re new and awesome! Hmm.

So while Apple telling developers what they can and can’t use is still very wrong from a point of principle, I’m actually glad that someone is championing native code for once, rather than pushing a VM environment. I’d prefer they didn’t mandate anything at all, but I can’t deny a certain urge to fist-bump when native code was the one to get the seal of approval, after getting the impression from other companies that they’d rather no-one had access to the underlying workings of the machine. I like native code, there’s a certain purity about it – and maybe it’s like a sad old gear-head going on about how great the old V8′s used to be, but I don’t care Mostly it’s about my frustration with being forced to discard perfectly good native libraries and look for / build replacements for no good reason.

PS For the record, OGRE on iPhone isn’t affected by this new ToS because we’re 100% native, baby.

[edit]For those pointing out that C# and such eventually run on native code anyway – that’s not the point. The point is that on certain devices – XNA and Windows Phone 7 – you simply cannot use libraries that were not written in .Net originally, meaning that years worth of dev libraries are inaccessible and need to be (pointlessly IMO) rewritten in .Net. And yes, this is exactly the same as Apple are doing here (but in reverse), if you interpret it in the strictest sense that you’re only allowed to use code written in Obj-C, C and C++. I’m just taking a perverse delight in the fact that it’s C/C++ libraries from the last 20+ years that are on the winning side for a change, instead of being the ones that are excluded (which frankly I’m completely sick of).

Writing good documentation is hard. While I happen to think that API references generated from source code can be extremely useful, they’re only part of the story, and eventually everyone needs to write something more substantial for their software. You can get away with writing HTML directly, and separately using a word processor to write PDFs for so long, but eventually you need a proper tool chain with the following characteristics:

• Lets the author concentrate on content rather than style
• Generates multiple formats from one source (HTML, PDF, man pages, HTML Help etc)
• Does all the tedious work for you such as TOCs, cross-references, source code highlighting, footnotes
• Is friendly to source control systems & diffs in general
• Standard enough that you could submit the content to a publisher if you wanted to
• Preferably cross-platform, standards-based and not oriented to any particular language or technology

When I came to write the OGRE manual many, many years ago, I went with Texinfo – it seemed a good idea at the time, and ticked most of the boxes above. The syntax is often a bit esoteric, and the tools used to generate output frequently a bit flaky (texi2html has caused me many headaches over the years thanks to  poorly documented breaking changes), but it worked most of the time.

I’ve been meaning to replace this tool chain with something else for new projects for a while, and DocBook sprung to mind since it’s the ‘new standard’ for technical documentation. It’s quite popular with open source projects now and it’s the preferred format for many publishers such as O’Reilly. In the short term, I want to write some developer instructions for OGRE for our future Mercurial setup, but in the long term, I’d really like a good documentation tool chain for all sorts of other purposes, and Texinfo feels increasingly unsatisfactory these days.

Having spent some time this week establishing a new working tool chain, and encountering & resolving a number of issues along the way, I thought I’d share my setup with you.

Read the rest of this entry »

It’s precisely 10 years to the day that I registered OGRE on Sourceforge, so in some ways, today could be considered to be OGRE’s 10th birthday. From most other people’s perspective that won’t come until next year though, since I only made the first public release to CVS in May 2001, over a year later, which really kicked the whole thing off. The delay was down to me not really being able to start work in earnest until late 2000 because of a course of study I was on at the time, but I already knew in February 2000 what I wanted to do, it would just be a few months before I could start to realise it.

So OGRE, like the Queen, has two birthdays – a public and a private one I should start planning something for the official public 10th birthday next year, but this anniversary is an important personal one for me, since it represents my mental commitment to the project, the original kernel of the idea, so to speak.

It’s pretty mind-boggling to think it’s been this long, actually. I have more (ok, considerably more) grey in my hair than I had when I started, along with a ‘proper’ beard instead of that half-hearted goatee business – and the project has matured with me. OGRE has gone from an experimental project for personal fun, to a pretty well recognised name in open source real-time rendering with a global team, underpinning a surprising amount of production software across all sorts of sectors; games, public interactives, architecture, simulation, science, advertising, training, hardware devices, and more.

I haven’t even tried to tally up the amount of time I’ve spent on OGRE in the last 10 years, I’m sure it’s a scary number of man-years. Totally worth it though.

Having already disrespected mailing lists, I might as well get all my ranting about old staple communication techniques out of my system, by admitting that I’ve never really liked IRC.

There’s nothing wrong with it per se, particularly as a casual social tool, but I just can’t say I’ve ever received any great value from it in a project sense, primarily because of it’s real-time and unfocussed nature. As a user of a project, I’ve frequently found that the people that are able to answer my questions are not online at the same time as I am. Secondly, even when those people are online, they tend to get mobbed by everyone, and anything more than one or two active discussions turn the channel quickly to a confusing mess. As a project lead, I always dreaded going on IRC precisely because of this “mobbing effect”; the usual outcome was for me to lose a couple of hours answering a ton of questions – which was not an unpleasant activity, it’s nice to talk to your users, but at the same time it’s a terrible time-sink, and unlike some people I’m incapable of multitasking real-time discussions with coding, at least not on anything remotely complex. As such, my IRC attendance slowly dropped off and I now rarely go on any more; I felt a bit guilty about that, but figured the community would rather I got more done than spend time talking.

I realised recently that Twitter has now settled into my life as a more effective replacement for the times when I might have previously found IRC somewhat useful, despite the noise. It’s as close to real-time as matters, but at the same time it’s not a chat system, which for me is a good thing, since it sidesteps by design the major downsides of an open chat system – the tendency for real-time discussions to ramble on, and the implied expectation of a real-time response. You often get that, of course, but there’s no perception that it’s an affront if there’s a delay, even of many hours. As a system that needs to sit alongside ‘real’ work, it’s a lot more practical in its utility. Also, as primarily a ‘pull system’ (you choose to follow people), the signal-to-noise ratio is far higher. People can reply to your posts, and you can reply to theirs, so the same kinds of discussions as IRC tend to spring up, but they tend to be more useful, because they’re among peers more often than IRC was. Sure, other people can @user you in an unsolicited fashion too, unconnected to your feed, but that’s generally considered impolite so it’s rare. There are also no ‘channels’ so I don’t have to be watching many places depending on the subject, channels simply form naturally based on individuals and subject tags. Finally,the 140 character limit does tend to waste less time for the reader – although for the writer time can sometimes be lost trying to shoehorn a coherent point into that space.

As a result, I find I have all of the benefits of IRC (in a project rather than casual social sense), with few or none of the downsides. I have many semi-real time, compact and most importantly useful exchanges with people on the service, all in a very convenient package (after trying a few clients, I settled on TweetDeck to organise things).

This might come across as me wanting to wall myself off from the ‘n00bs’ in my community. That’s not true, it’s just that time is my most valuable asset, and it’s finite; crushingly so. I’m happy to answer questions on the forum – where I can dedicate a known amount of time and tackle as much as possible, regardless of whether the person is currently online or not, and Twitter fills in the more social & real-time aspects without being a burden. IRC by contrast is high maintenance and extremely wasteful with time for the same purpose, and I just can’t justify it.

So farewell IRC,  I really won’t miss you very much.

I’m not blogging as often these days; as you know I don’t traditionally ‘do’ short blog posts – in my book if something is worth blogging about, it’s worth making sure it holds together as an argument, and as a piece of writing generally – and a combined lack of time of anything I’m motivated (or permitted) to talk about has left the site a little  bereft of content. Luckily my OGRE Twitter is stocked with more frequent and less lovingly crafted status updates on what I’m doing there.

So, on to the title of the post. The Internet has been around for a while now, and has evolved rapidly, particularly in the last decade. And yet, particularly in academic and some open source developer circles, there is an attachment to a particularly creaky piece of technology that I can honestly say I do not share - the venerable mailing list.

Now, to clarify the context, I’m referring to the use of mailing lists for multilateral communication for an entire community, including newcomers, as opposed to a simple 1-way notification list (like we use for commit notifications for example). For N-way communication among a small group of core developers, all of whom will want to read every post, I can see the utility and convenience of a mailing list. But as a community communication channel, where people just want to drop in and drop out, I find it a staggeringly inefficient, awkward and archaic approach. I say this primarily as an occasional community member of various projects that use mailing lists, and therefore someone who has a specific interest in a mere subset of the discussions that go on – I have no time or desire to read every single thread, and indeed if I tried to do this for every project I have an interest in, I’d never get anything done. It’s hard enough to keep up with my own open source community!

The simple fact is that mailing lists have an all-or-nothing mindset that is woefully outdated for community interaction on the scale that the Internet has now grown to.  Subscribing means you get bombarded with every single discussion, either individually or in digests, which pretend to be useful but in fact aren’t, because while they cut down on the number of emails you get, it makes replying to specific posts a pain. If you want to read every single mail in the list, I’m sure they work fine – but most people outside the core group do not want to do this. Most members of the community just want to keep a closer eye on a few select threads of discussion that either affect or interest them, and to be able to search and browse through the rest easily – and the mailing list is a woefully inadequate, blunt instrument for this kind of task.

Sure, you can choose not to subscribe, and go through the archives, searching or browsing them. But you can do that with forums too, and there at least you have the advantage of categorised areas of interest, being able to follow certain people, and to watch certain threads. Mailing list archives have a single filter: date, and also lag by a number of hours dependent on the individual setup, so if you’re not subscribed, you get a lesser service.  Another technique is to subscribe completely but tell your email client to archive or filter things for you, so you can dip into your local replica at leisure. Horribly, horribly inefficient, but it does work.

Mailing lists worked in the 90′s when there were small groups of people who wanted to read everything being discussed, and when email was the primary form of communication between people. We’ve moved on. Forum systems and other flexible hosted systems are far superior in their ability to let you watch particular discussions (or all new posts) that you’re interested in and get told when there’s an update. Anyone can search them easily (internally or via Google) and there’s no archive lag. Maybe some people are worried about forum databases being lost, compared to inherently replicated mailing lists, but anyone worth their salt has a server backup strategy.  Honestly, any project that uses mailing lists as their only community discussion channel instantly puts me off getting involved in that community, because I know that as an occasional participant interested in only certain discussions, the experience is going to totally suck.

And, if you insist on loving your mailing lists approach so much, for goodness sake move to Google Groups. They’re still pretty basic, but at least there, those of us who have moved into the browser world can use an interface we find useful and productive, rather than being forced to use 20-30 year old technology designed to replicate posts around a university science department.