SteveStreeting.com

Writing good documentation is hard. While I happen to think that API references generated from source code can be extremely useful, they’re only part of the story, and eventually everyone needs to write something more substantial for their software. You can get away with writing HTML directly, and separately using a word processor to write PDFs for so long, but eventually you need a proper tool chain with the following characteristics:

• Lets the author concentrate on content rather than style
• Generates multiple formats from one source (HTML, PDF, man pages, HTML Help etc)
• Does all the tedious work for you such as TOCs, cross-references, source code highlighting, footnotes
• Is friendly to source control systems & diffs in general
• Standard enough that you could submit the content to a publisher if you wanted to
• Preferably cross-platform, standards-based and not oriented to any particular language or technology

When I came to write the OGRE manual many, many years ago, I went with Texinfo – it seemed a good idea at the time, and ticked most of the boxes above. The syntax is often a bit esoteric, and the tools used to generate output frequently a bit flaky (texi2html has caused me many headaches over the years thanks to  poorly documented breaking changes), but it worked most of the time.

I’ve been meaning to replace this tool chain with something else for new projects for a while, and DocBook sprung to mind since it’s the ‘new standard’ for technical documentation. It’s quite popular with open source projects now and it’s the preferred format for many publishers such as O’Reilly. In the short term, I want to write some developer instructions for OGRE for our future Mercurial setup, but in the long term, I’d really like a good documentation tool chain for all sorts of other purposes, and Texinfo feels increasingly unsatisfactory these days.

Having spent some time this week establishing a new working tool chain, and encountering & resolving a number of issues along the way, I thought I’d share my setup with you.

Read the rest of this entry »

It’s precisely 10 years to the day that I registered OGRE on Sourceforge, so in some ways, today could be considered to be OGRE’s 10th birthday. From most other people’s perspective that won’t come until next year though, since I only made the first public release to CVS in May 2001, over a year later, which really kicked the whole thing off. The delay was down to me not really being able to start work in earnest until late 2000 because of a course of study I was on at the time, but I already knew in February 2000 what I wanted to do, it would just be a few months before I could start to realise it.

So OGRE, like the Queen, has two birthdays – a public and a private one I should start planning something for the official public 10th birthday next year, but this anniversary is an important personal one for me, since it represents my mental commitment to the project, the original kernel of the idea, so to speak.

It’s pretty mind-boggling to think it’s been this long, actually. I have more (ok, considerably more) grey in my hair than I had when I started, along with a ‘proper’ beard instead of that half-hearted goatee business – and the project has matured with me. OGRE has gone from an experimental project for personal fun, to a pretty well recognised name in open source real-time rendering with a global team, underpinning a surprising amount of production software across all sorts of sectors; games, public interactives, architecture, simulation, science, advertising, training, hardware devices, and more.

I haven’t even tried to tally up the amount of time I’ve spent on OGRE in the last 10 years, I’m sure it’s a scary number of man-years. Totally worth it though.

Having already disrespected mailing lists, I might as well get all my ranting about old staple communication techniques out of my system, by admitting that I’ve never really liked IRC.

There’s nothing wrong with it per se, particularly as a casual social tool, but I just can’t say I’ve ever received any great value from it in a project sense, primarily because of it’s real-time and unfocussed nature. As a user of a project, I’ve frequently found that the people that are able to answer my questions are not online at the same time as I am. Secondly, even when those people are online, they tend to get mobbed by everyone, and anything more than one or two active discussions turn the channel quickly to a confusing mess. As a project lead, I always dreaded going on IRC precisely because of this “mobbing effect”; the usual outcome was for me to lose a couple of hours answering a ton of questions – which was not an unpleasant activity, it’s nice to talk to your users, but at the same time it’s a terrible time-sink, and unlike some people I’m incapable of multitasking real-time discussions with coding, at least not on anything remotely complex. As such, my IRC attendance slowly dropped off and I now rarely go on any more; I felt a bit guilty about that, but figured the community would rather I got more done than spend time talking.

I realised recently that Twitter has now settled into my life as a more effective replacement for the times when I might have previously found IRC somewhat useful, despite the noise. It’s as close to real-time as matters, but at the same time it’s not a chat system, which for me is a good thing, since it sidesteps by design the major downsides of an open chat system – the tendency for real-time discussions to ramble on, and the implied expectation of a real-time response. You often get that, of course, but there’s no perception that it’s an affront if there’s a delay, even of many hours. As a system that needs to sit alongside ‘real’ work, it’s a lot more practical in its utility. Also, as primarily a ‘pull system’ (you choose to follow people), the signal-to-noise ratio is far higher. People can reply to your posts, and you can reply to theirs, so the same kinds of discussions as IRC tend to spring up, but they tend to be more useful, because they’re among peers more often than IRC was. Sure, other people can @user you in an unsolicited fashion too, unconnected to your feed, but that’s generally considered impolite so it’s rare. There are also no ‘channels’ so I don’t have to be watching many places depending on the subject, channels simply form naturally based on individuals and subject tags. Finally,the 140 character limit does tend to waste less time for the reader – although for the writer time can sometimes be lost trying to shoehorn a coherent point into that space.

As a result, I find I have all of the benefits of IRC (in a project rather than casual social sense), with few or none of the downsides. I have many semi-real time, compact and most importantly useful exchanges with people on the service, all in a very convenient package (after trying a few clients, I settled on TweetDeck to organise things).

This might come across as me wanting to wall myself off from the ‘n00bs’ in my community. That’s not true, it’s just that time is my most valuable asset, and it’s finite; crushingly so. I’m happy to answer questions on the forum – where I can dedicate a known amount of time and tackle as much as possible, regardless of whether the person is currently online or not, and Twitter fills in the more social & real-time aspects without being a burden. IRC by contrast is high maintenance and extremely wasteful with time for the same purpose, and I just can’t justify it.

So farewell IRC,  I really won’t miss you very much.

I’m not blogging as often these days; as you know I don’t traditionally ‘do’ short blog posts – in my book if something is worth blogging about, it’s worth making sure it holds together as an argument, and as a piece of writing generally – and a combined lack of time of anything I’m motivated (or permitted) to talk about has left the site a little  bereft of content. Luckily my OGRE Twitter is stocked with more frequent and less lovingly crafted status updates on what I’m doing there.

So, on to the title of the post. The Internet has been around for a while now, and has evolved rapidly, particularly in the last decade. And yet, particularly in academic and some open source developer circles, there is an attachment to a particularly creaky piece of technology that I can honestly say I do not share - the venerable mailing list.

Now, to clarify the context, I’m referring to the use of mailing lists for multilateral communication for an entire community, including newcomers, as opposed to a simple 1-way notification list (like we use for commit notifications for example). For N-way communication among a small group of core developers, all of whom will want to read every post, I can see the utility and convenience of a mailing list. But as a community communication channel, where people just want to drop in and drop out, I find it a staggeringly inefficient, awkward and archaic approach. I say this primarily as an occasional community member of various projects that use mailing lists, and therefore someone who has a specific interest in a mere subset of the discussions that go on – I have no time or desire to read every single thread, and indeed if I tried to do this for every project I have an interest in, I’d never get anything done. It’s hard enough to keep up with my own open source community!

The simple fact is that mailing lists have an all-or-nothing mindset that is woefully outdated for community interaction on the scale that the Internet has now grown to.  Subscribing means you get bombarded with every single discussion, either individually or in digests, which pretend to be useful but in fact aren’t, because while they cut down on the number of emails you get, it makes replying to specific posts a pain. If you want to read every single mail in the list, I’m sure they work fine – but most people outside the core group do not want to do this. Most members of the community just want to keep a closer eye on a few select threads of discussion that either affect or interest them, and to be able to search and browse through the rest easily – and the mailing list is a woefully inadequate, blunt instrument for this kind of task.

Sure, you can choose not to subscribe, and go through the archives, searching or browsing them. But you can do that with forums too, and there at least you have the advantage of categorised areas of interest, being able to follow certain people, and to watch certain threads. Mailing list archives have a single filter: date, and also lag by a number of hours dependent on the individual setup, so if you’re not subscribed, you get a lesser service.  Another technique is to subscribe completely but tell your email client to archive or filter things for you, so you can dip into your local replica at leisure. Horribly, horribly inefficient, but it does work.

Mailing lists worked in the 90′s when there were small groups of people who wanted to read everything being discussed, and when email was the primary form of communication between people. We’ve moved on. Forum systems and other flexible hosted systems are far superior in their ability to let you watch particular discussions (or all new posts) that you’re interested in and get told when there’s an update. Anyone can search them easily (internally or via Google) and there’s no archive lag. Maybe some people are worried about forum databases being lost, compared to inherently replicated mailing lists, but anyone worth their salt has a server backup strategy.  Honestly, any project that uses mailing lists as their only community discussion channel instantly puts me off getting involved in that community, because I know that as an occasional participant interested in only certain discussions, the experience is going to totally suck.

And, if you insist on loving your mailing lists approach so much, for goodness sake move to Google Groups. They’re still pretty basic, but at least there, those of us who have moved into the browser world can use an interface we find useful and productive, rather than being forced to use 20-30 year old technology designed to replicate posts around a university science department.

I love it when shots from Ogre just show up in funny places. This time, it’s from a comic strip called Stolen Pixels on the Escapist, where Ogre-powered games Torchlight and Zombie Driver have been used for comedic purposes:

Thanks to BuschnicK for the heads-up on the Torchlight one, I was surprised to see Zombie Driver just days earlier too!

So, I’ve been a little quieter than usual since the new year, and that’s because I’ve been in  a rather reflective mood as I plan out how I’m going to spend my time in 2010. That’s right – planning! Talk about the final frontier

Basically, as you may have gleaned from my previous post, I’ve been looking to make some significant changes to the way I do things in 2010. I spent 2009 reeling from a back injury and trying to figure out how to deal with that given that I’m self-employed (ie I don’t get paid when I’m not working, regardless of the reason), and a leader of an open source project (with the inherent time requirements that comes with). This meant working out on the fly how to stay afloat financially, and still keeping my own interests and open-source plates spinning, without slipping back into the ‘permanent voluntary crunch mode’ style which triggered my back problems. I can’t stress enough how difficult that transition has been for me – it’s not like anyone was forcing me to work/live that way, I did it because I wanted to, but then it suddenly had to stop. When you invest so much of your time and perceived identity in something, backing away from it is very, very hard.

Of course the economic climate wasn’t great either, meaning I spent a lot of time jumping around between many small projects, leading to more overhead dealing with admin & business relations. I ended up just going almost month-to-month on-demand, not  planning very much and just being grateful to be able to work a decent amount at all – which given how unwell I was at the start of the year was definitely something to be glad about. But, now I’m back on my feet and pretty confident of my future health again (within reason – I’m not going to be bungee jumping any time soon!), I’m ready to start being more pro-active again and to map out some plans.

One thing is for sure, there’s no going back to how I used to do things. My days of saying ‘yes’ to almost everything and being at the keyboard until past midnight most days, and most of the weekend, are gone forever. I don’t regret doing it, despite the pain it ended up causing me, because OGRE wouldn’t be here otherwise and I learned a vast amount and had a ton of fun – but I’ll leave that to the under-35s in future; have fun guys From now on, I’m being ruthless and somewhat selfish about what I work on, and concentrating on things that maximise my personal love-growth-cash triangle. It means I’m passing on a lot more projects, and concentrating far more on things that are strategically significant to me, rather than anyone else.

I’m still planning to lead OGRE, so long as the community is happy for me to do so, but by necessity I’m stepping back a bit to let other people take more responsibility where they want to, and to refocus my time on mentoring and advisory roles rather than trying to be everywhere at once. We have some great people in the team and in the wider community, and I hope our MIT license will foster even more in future. Both I and the community have gotten used to perceiving me as the ‘go to guy’ in the first instance, with responsibility for pretty much everything, but in practice for some time now it’s been very much a team & community effort, just one that I happen to lead (and financially support where needed). In fact one of the things I’m quite proud of is the way so many others have picked up on the way I do things, and taken things forward themselves in a way that I wholly approve of. That’s open source in action, and I’m glad to be part of it, even if I can no longer have my fingers in absolutely every pie with an OGRE symbol on it

Here’s to 2010 anyway. It’s going to be different, but change is good.

Zero Punctuation reviewed Torchlight yesterday!

Of course he was both inaccurate (you don’t have to keep clicking at all, you can hold the button down) and overly harsh, but still very funny. It’s odd to enjoy watching something you had a hand in (albeit in a background technology way in my case) being ripped to shreds, but when it’s done in such an amusing way somehow it’s ok. I guess this is why Yahtzee hasn’t had his teeth kicked in by disgruntled game developers yet

As Runic’s Twitter said: “We’ve arrived!”.

It’s now almost a year since I decided to try using Twitter, specifically to post about Ogre development work I’m doing and other Ogre-related things (well, most of the time anyway). Seeing as I totally deride the concept that it’s a good thing to share the inconsequential, tedious minutae of your life with the internet and view it as the absolute pinnacle of sad, narcissistic behaviour, joining Twitter was a hard sell. After all, at least on a blog you have to write enough in a post to naturally filter out anything that’s not worth saying (in theory), while Twitter seemingly encouraged you to share whatever crossed your mind during the day. In the end my reason for joining was that there tended to be things large and small that happened in and around Ogre that many people might like to know about, and these things didn’t always warrant a blog post,  a news article on ogre3d.org or even a forum post. Provided I stuck to that raison d’etre, perhaps it could have value.

And in fact, it’s actually been very useful. I’ve almost stopped blogging about Ogre work unless there’s a significant event or something I feel needs greater analysis, because my Twitter feed is a better way to get the word out about things. It’s also been useful to get feedback on certain technical issues and to keep up to date with what other people are doing. Specifically, I tend to only follow people who post about things I’m interested in, rather than just because I know them.

And this tends to work well – I’ve found that Twitter users, or at least the ones I follow, in general tend to automatically filter their content to things that are actually interesting. This is in contrast to Facebook, which is so chock full of the utterly banal that I lose the will to live every time I try to catch up with the feed – there are usually some things in there I’d genuinely be interested in, but it’s so full of crap I can hardly face spending the time to find it. Much of that is due to its insistence that I’m somehow interested in the events of all the Facebook games people are playing, when in fact I couldn’t give a flying toss what new fish someone has just unlocked in some ridiculous mini-game. I’m close to just deleting my account and forgetting all about it – if you want to be social, grab a coffee / drink with me sometime or something – at least then you’re unlikely to keep interrupting to tell me what your level is in FarmVille.

Computer systems are tools, and can be used for good or ill. I’ve come across lots of people that use Twitter in a genuinely useful and non-intrusive way, and I try to do the same, and as such it’s made a firm place for itself in my day – something I would not have taken for granted when I started using it.

I’ve already posted about my experiences with Git and Mercurial, the end result of which was a vastly increased respect for Git but a basically confirmed preference for Mercurial, based on ease of use, platform consistency and resilience.

Mercurial’s conversion tools are really quite good – the core tools worked fine but I was impressed by hgsubversion‘s speed and that it seemed to just work, in both initial conversion and pulling subsequent updates. It was missing a couple of features that I wanted though – firstly the ability to reflect merge points between branches during the conversion, and secondly to be able to ‘squash’ ancient history down to a simple snapshot to save space.

At OGRE, we’d carried forward all our history from CVS to Subversion and as such have almost 8 years of history, including a couple of file reorganisations. Mercurial’s storage efficiency falls down compared to Git when files are moved around, because a file stored in more than one place in the tree over the history of the project is physically stored multiple times too, whilst Git stores the content only once with pointers from the various locations / history points. Most of this overhead could be removed just by eliminating old history we didn’t need anymore – history that does no harm in Subversion since only the server holds it, but does cause unwanted overheads in a DVCS since every user gets the entire repository. Removal of history is something that Mercurial shuns – rightly so in the case of public repositories but in these rare cases it would be nice if there was a tool for removing old history; again Git allows this but it has to be used with care. In the absence of that, doing it at conversion seemed the best way.

I asked about these things in the hgsubversion community, but the tradition of open source is that if you really want something urgently, you know where the code is Mercurial is really nice when it comes to hacking because it’s all Python; so there’s a nice unified API in one place that you can refer to – that’s one of the reasons I like it over Git which is far more fragmented in technology terms. I’m not a Python guru by any means, but I managed to implement both these features – I did the “mergemap” support a little while ago and added the “skipto” option today – it’s called that because “skipto” was already referred to in the hgsubversion code but it had no implementation.

The result is that the OGRE Mercurial repository with only the last ~3 years of history (back to when the v1.4 branch was created) is now only 74MB, rather than the 206MB of the original, complete conversion (in comparison Git was 116MB for the whole thing). By dropping the history I’ve removed most of the instances of reorganisation which is where most of the space has gone. I  hope eventually that Mercurial adds a utility to deal with stripping ancient history (right now, you can only strip branches) but this solves my primary conversion issue. Since this new repo can be kept in sync in a very lightweight fashion with the existing Subversion repo, I’ll be periodically updating it and doing more tests to reassure myself that the content really is ok.

If you’d like to get my custom version of hgsubversion with these features, it’s here: http://bitbucket.org/sinbad/hgsubversion/. I make no promises that it’s error-free, use at your own risk. It currently assumes that you’re using the standard Subversion layout, are converting from the root of that and have the ‘svn’ command on your path.

So, I’ve just about completed my practical experiments & review of Mercurial and Git.

In the end, I had far too many separate notes and sets of experiences to post, so I boiled the argument down into the 10 most important factors to me, and scored Mercurial and Git on a scale of 1-5 based on what I’d found when using them. Here are the (annoying) results:

I’ll explain the scores, and my conclusion, after the jump.

Read the rest of this entry »