Category Archives: Development

Development OGRE Personal

10 years ago today…

It’s precisely 10 years to the day that I registered OGRE on Sourceforge, so in some ways, today could be considered to be OGRE’s 10th birthday. From most other people’s perspective that won’t come until next year though, since I only made the first public release to CVS in May 2001, over a year later, which really kicked the whole thing off. The delay was down to me not really being able to start work in earnest until late 2000 because of a course of study I was on at the time, but I already knew in February 2000 what I wanted to do, it would just be a few months before I could start to realise it.

So OGRE, like the Queen, has two birthdays – a public and a private one :) I should start planning something for the official public 10th birthday next year, but this anniversary is an important personal one for me, since it represents my mental commitment to the project, the original kernel of the idea, so to speak.

It’s pretty mind-boggling to think it’s been this long, actually. I have more (ok, considerably more) grey in my hair than I had when I started, along with a ‘proper’ beard instead of that half-hearted goatee business – and the project has matured with me. OGRE has gone from an experimental project for personal fun, to a pretty well recognised name in open source real-time rendering with a global team, underpinning a surprising amount of production software across all sorts of sectors; games, public interactives, architecture, simulation, science, advertising, training, hardware devices, and more.

I haven’t even tried to tally up the amount of time I’ve spent on OGRE in the last 10 years, I’m sure it’s a scary number of man-years. Totally worth it though.

Development Internet OGRE

Twitter is my new IRC

Having already disrespected mailing lists, I might as well get all my ranting about old staple communication techniques out of my system, by admitting that I’ve never really liked IRC.

There’s nothing wrong with it per se, particularly as a casual social tool, but I just can’t say I’ve ever received any great value from it in a project sense, primarily because of it’s real-time and unfocussed nature. As a user of a project, I’ve frequently found that the people that are able to answer my questions are not online at the same time as I am. Secondly, even when those people are online, they tend to get mobbed by everyone, and anything more than one or two active discussions turn the channel quickly to a confusing mess. As a project lead, I always dreaded going on IRC precisely because of this “mobbing effect”; the usual outcome was for me to lose a couple of hours answering a ton of questions – which was not an unpleasant activity, it’s nice to talk to your users, but at the same time it’s a terrible time-sink, and unlike some people I’m incapable of multitasking real-time discussions with coding, at least not on anything remotely complex. As such, my IRC attendance slowly dropped off and I now rarely go on any more; I felt a bit guilty about that, but figured the community would rather I got more done than spend time talking.

I realised recently that Twitter has now settled into my life as a more effective replacement for the times when I might have previously found IRC somewhat useful, despite the noise. It’s as close to real-time as matters, but at the same time it’s not a chat system, which for me is a good thing, since it sidesteps by design the major downsides of an open chat system – the tendency for real-time discussions to ramble on, and the implied expectation of a real-time response. You often get that, of course, but there’s no perception that it’s an affront if there’s a delay, even of many hours. As a system that needs to sit alongside ‘real’ work, it’s a lot more practical in its utility. Also, as primarily a ‘pull system’ (you choose to follow people), the signal-to-noise ratio is far higher. People can reply to your posts, and you can reply to theirs, so the same kinds of discussions as IRC tend to spring up, but they tend to be more useful, because they’re among peers more often than IRC was. Sure, other people can @user you in an unsolicited fashion too, unconnected to your feed, but that’s generally considered impolite so it’s rare. There are also no ‘channels’ so I don’t have to be watching many places depending on the subject, channels simply form naturally based on individuals and subject tags. Finally,the 140 character limit does tend to waste less time for the reader – although for the writer time can sometimes be lost trying to shoehorn a coherent point into that space.

As a result, I find I have all of the benefits of IRC (in a project rather than casual social sense), with few or none of the downsides. I have many semi-real time, compact and most importantly useful exchanges with people on the service, all in a very convenient package (after trying a few clients, I settled on TweetDeck to organise things).

This might come across as me wanting to wall myself off from the ‘n00bs’ in my community. That’s not true, it’s just that time is my most valuable asset, and it’s finite; crushingly so. I’m happy to answer questions on the forum – where I can dedicate a known amount of time and tackle as much as possible, regardless of whether the person is currently online or not, and Twitter fills in the more social & real-time aspects without being a burden. IRC by contrast is high maintenance and extremely wasteful with time for the same purpose, and I just can’t justify it.

So farewell IRC,  I really won’t miss you very much.

Development Internet OGRE Open Source

Mailing lists as community channels – ugh

gnu_mailmanI’m not blogging as often these days; as you know I don’t traditionally ‘do’ short blog posts – in my book if something is worth blogging about, it’s worth making sure it holds together as an argument, and as a piece of writing generally – and a combined lack of time of anything I’m motivated (or permitted) to talk about has left the site a little  bereft of content. Luckily my OGRE Twitter is stocked with more frequent and less lovingly crafted status updates on what I’m doing there.

So, on to the title of the post. The Internet has been around for a while now, and has evolved rapidly, particularly in the last decade. And yet, particularly in academic and some open source developer circles, there is an attachment to a particularly creaky piece of technology that I can honestly say I do not share - the venerable mailing list.

Now, to clarify the context, I’m referring to the use of mailing lists for multilateral communication for an entire community, including newcomers, as opposed to a simple 1-way notification list (like we use for commit notifications for example). For N-way communication among a small group of core developers, all of whom will want to read every post, I can see the utility and convenience of a mailing list. But as a community communication channel, where people just want to drop in and drop out, I find it a staggeringly inefficient, awkward and archaic approach. I say this primarily as an occasional community member of various projects that use mailing lists, and therefore someone who has a specific interest in a mere subset of the discussions that go on – I have no time or desire to read every single thread, and indeed if I tried to do this for every project I have an interest in, I’d never get anything done. It’s hard enough to keep up with my own open source community!

The simple fact is that mailing lists have an all-or-nothing mindset that is woefully outdated for community interaction on the scale that the Internet has now grown to.  Subscribing means you get bombarded with every single discussion, either individually or in digests, which pretend to be useful but in fact aren’t, because while they cut down on the number of emails you get, it makes replying to specific posts a pain. If you want to read every single mail in the list, I’m sure they work fine – but most people outside the core group do not want to do this. Most members of the community just want to keep a closer eye on a few select threads of discussion that either affect or interest them, and to be able to search and browse through the rest easily – and the mailing list is a woefully inadequate, blunt instrument for this kind of task.

Sure, you can choose not to subscribe, and go through the archives, searching or browsing them. But you can do that with forums too, and there at least you have the advantage of categorised areas of interest, being able to follow certain people, and to watch certain threads. Mailing list archives have a single filter: date, and also lag by a number of hours dependent on the individual setup, so if you’re not subscribed, you get a lesser service.  Another technique is to subscribe completely but tell your email client to archive or filter things for you, so you can dip into your local replica at leisure. Horribly, horribly inefficient, but it does work.

Mailing lists worked in the 90′s when there were small groups of people who wanted to read everything being discussed, and when email was the primary form of communication between people. We’ve moved on. Forum systems and other flexible hosted systems are far superior in their ability to let you watch particular discussions (or all new posts) that you’re interested in and get told when there’s an update. Anyone can search them easily (internally or via Google) and there’s no archive lag. Maybe some people are worried about forum databases being lost, compared to inherently replicated mailing lists, but anyone worth their salt has a server backup strategy.  Honestly, any project that uses mailing lists as their only community discussion channel instantly puts me off getting involved in that community, because I know that as an occasional participant interested in only certain discussions, the experience is going to totally suck.

And, if you insist on loving your mailing lists approach so much, for goodness sake move to Google Groups. They’re still pretty basic, but at least there, those of us who have moved into the browser world can use an interface we find useful and productive, rather than being forced to use 20-30 year old technology designed to replicate posts around a university science department.

Business Development Games

“Maturing” download games market starts to show retail-like characteristics

Watching the ebbs and flows of the game industry is simultaneously inspiring and outright depressing. As is usual for this stage in a console generation, we’re at the ‘consolidation point’ (pun unintentional)  - where the tech is pretty well understood, even if it is starting to look a bit dated compared to even a modest PC (how much hassle AA is on this console generation is a case in point), but that at least developers can crank out content in a more efficient fashion. This has led to some darned good games.

What’s depressing is what’s happening to the ‘official’ download channels – which were a bastion of independent content a year or two ago, and now are turning more and more into just another channel for the same mainstream developers & publishers we see at retail. XBLA has been the trend maker here, it was first to really embrace and promote downloadable games to a ‘core’ market, and has done extremely well. Now, however, we have a limit of 2 games per week, and all too often those 2 slots are being assigned to either major developers (Shadow Complex, Alien Breed Evo) or shovelware ports with brand recognition but little quality or innovation (I’m looking at Taito in particular: Bubble Bobble and Qix remakes were incredibly lazy, uninspiring affairs). It’s very clear that the team behind choosing which developers are published in XBLA has changed in recent years, and not for the better from my perspective. That’s not to say there’s anything wrong with Shadow Complex and Alien Breed Evo, but if they’re using up the slots it means that publishing route is rapidly being cut off for small developers who are big on ideas and talent, but short on funds and established brands. Alien Breed Evo’s budget was supposedly around $2.5m for goodness sakes – although it’s looking like that’s going to backfire anyway since sales have been poor. Trials HD, ‘Splosion Man and Peggle are pretty much the only games from small studios with modest budgets that I can think of that made a splash on XBLA in 2009 – the rest just read like a whos who of regular retail channels. Indeed many developers who have had games published on XBLA are no longer welcome there, such as PomPom (interview) and Llamasoft. Clearly the message is ‘win big, or get your coat’. This isn’t the right environment for an indie scene to flourish, where experimentation and mistakes are part of the process.

Yes, I know there’s XBL ‘Indie Games’ but that’s the absolute opposite end of the spectrum, hobbled with a niche development environment that’s incompatible with the most established dev libraries and every other platform a developer might want to deploy on (barring PC), and so far almost totally lacking any way for a decent game to effectively ‘rise above the noise’, except via external review sites like XNPlay, which doesn’t work at all for targetting the majority of game players with information.  It’s just not a very good target for those I would call ‘serious indies’ and actually acts as a false argument for not opening primary download channels more; there’s nothing wrong with the concept, it’s just implemented completely wrong.

PSN got started later so has been earlier in the curve of promoting independent content, but they’re going that way too. I guess they’re all just ‘following the money’, and the games industry remains obsessed with hits because of its current top-heavy model. I’d hoped that the downloadable content channels would promote an equivalent to low-budget and art-house cinema, where content can survive and make a profit for the creators, without necessarily having to be the Biggest Thing Ever(tm), encouraging experimentation. But, the giant flaw in this plan is that independent cinema is able to be published and consumed anywhere – while most games consumers remain shackled to console platform holders, who just want to publish a limited number of the very biggest hits, everything else not being worth their time or risking ‘distracting’ the customer with choice. If you’ve read my blog before, you know my opinion of the effectiveness of closed platforms in the long term when it comes to broadening and deepening a medium, but I’ll say it again – closed platforms are bad for the industry in the grand scheme of things. Games will never be as big as film until this changes, they might compete on the blockbuster level, but that’s far, far from the whole story. But, until we’re further along the lifecycle of games when hardware and delivery becomes mostly invisible,  the vested interests aren’t going to allow that to change for a little while yet.

You really need to go to the iPhone/iPod Touch for more prolific indie content these days. But, how long will that last?

I honestly don’t know why platform holders find it so hard to manage an open publishing strategy. All you need is systems that:

  • Allow users to rank content; and nominate ‘trusted reviewers’ such as those from major game review sites
  • Allow wide marketing opportunities – both in-system and cross-site (such as to xbox.com, where you can buy in-browser too)
  • Robust searching, on keywords, categories, user ratings, friends recommendations etc
  • Cross-promotion, aka the ‘You might also like…’ lists

Hell, if Amazon can create a compelling buying experience with millions of products across a diverse range of departments, why on earth do platform holders think a console user can’t handle more than 2 game choices a week? It’s hugely patronising, and says more about the inadequacy of the platform to manage larger amounts of content effectively than about any limit on what consumers are willing to peruse. Saying “we can’t sell as effectively if we have more product available” actually means “we suck at organisation”. This argument stacks up at retail, where there’s a limited amount of shelf space, and customers don’t want to wander around a massive warehouse or to squint at shelves of tightly packed boxes looking for something, but not when you have unlimited shelf space and a cloud full of computers to index it in the blink of an eye for you (and make suggestions), and where a marketing campaign or friend recommendation can bring a customer instantly to the point of sale with the use of a simple link.

For Christ’s sakes platform holders, wake up to the opportunities of the channel. Stop being blinded by what works at physical retail, it’s really not the same. There are people out there already doing it leagues better than you (see pretty much any of the e-commerce leaders), and putting your fingers in your ears and saying it can’t possibly work  is both ignorant and doing a massive disservice to both customers and content creators.

Development Personal

Salad Days

My friend Damien was blogging about his early experiences with computers & programming yesterday, and it reminded me of how I got started. Specifically, it reminded me of an influential magazine I read at the time called “Input”, which taught BASIC programming for the ZX Spectrum and BBC. It was a short-lived, esoteric British thing, but I was astonished to find that not only does Wikipedia have a page on it, but they also linked a TV advert of it which has been lovingly archived on YouTube:

This brought back some serious memories. I remember that as well as the demo snippets there was an adventure game of sorts which was being published bit by bit in the magazine, as an incentive for you to buy them all. Like most magazines of this type there were regular typos that you’d scratch your head over, but in a way that was a good thing since it taught you to debug other people’s code.

I was 10/11 years old when this magazine came out, and I can probably trace my programming beginnings directly to it. It’s probably a bizarre concept to the younger generation, now that they have instant access to almost limitless information on the Internet. What I would have given to have the Internet back then when I was learning, instead of scratching around for information in magazines like this. :)

Development Open Source

Open source is most important to producers rather than consumers

Gartner haven’t exactly been the sharpest tools in the box when it comes to predicting open source trends over the last few years, vastly underestimating it until about 2008, by which time it didn’t exactly take a professional analyst to tell you that it was popular. Still, now they’ve woken up to its potential, occasionally they post something useful. In particular, I liked a recent blog post about how open source is “trending towards customer obscurity” – that is to say that while open source is incredibly important to producers of software, the vast majority of consumers don’t really care how their software is made any more than they care how their car was made.

I support this view, and it’s one I’ve subscribed to for a while (although the somewhat condescending tone of the article is typical Gartner, the point is valid). My own open source software is aimed squarely at developers, where I think it adds value; since the users of my software are themselves making significant development investment in products using it, open source has significant advantages – the openness and participation-friendly nature of the development, the fact that the software can never be taken away from them either by company policy or acquisition, the fact that absolutely nothing is hidden behind any curtains so there aren’t any nasty surprises. When you’re investing your own time building on top of a foundation, there really is no substitute for being able to see all the working parts, should you want or need to.

Developers can be quite a broad church too – enterprises for example often have a need to modify and adapt software and that’s why open source has been very popular there too, even if they’re not actually making products of their own for publication.

But there’s also a vast group of people who are more traditional consumers (personally and in companies) – and they have no reason to care about open source if they’re never going to modify the software. There is a group of people who are philosophically dedicated to using free software (more so than open source) even if they never modify it themselves, but they’re a minority in the grand scheme of things. Most people that use open source do so because they feel it gives them what they need as users. Even personally, I use Linux on my servers not because I’m dedicated to using open source over the alternatives wherever I can (although I probably do have more bias in that direction than the average), but because it does exactly what I want in a server – it’s reliable, unobtrusive, cheap, has low hardware requirements and plenty of good software. Conversely, I don’t use Linux on the desktop much because personally I don’t feel it operates better than the commercial alternatives in that environment. I decide on a case-by-case basis what works best for me, and so do all but the most fanatical of users.

Now, of course open source regularly helps developers make better products (by using mature, reusable and adaptable components), which in the end can result in more users using software made from open source even if they don’t realise it. But the important thing to remember is that open source itself isn’t a marketing bullet point except to developers and the enthusiast minority. Like any gathering of like-minded people who mostly talk to each other, we open source developers / advocates can often forget that our enthusiasms aren’t necessarily shared by the rest of the populace. We have to remember that the end result is everything – and while open source definitely has a positive knock-on effect on product quality, it’s often a means to an end, not the end in and of itself. It’s obvious really, but worth keeping in mind.

Development OGRE Open Source

hgsubversion – dropping old history during conversion (mod)

mercurialI’ve already posted about my experiences with Git and Mercurial, the end result of which was a vastly increased respect for Git but a basically confirmed preference for Mercurial, based on ease of use, platform consistency and resilience.

Mercurial’s conversion tools are really quite good – the core tools worked fine but I was impressed by hgsubversion‘s speed and that it seemed to just work, in both initial conversion and pulling subsequent updates. It was missing a couple of features that I wanted though – firstly the ability to reflect merge points between branches during the conversion, and secondly to be able to ‘squash’ ancient history down to a simple snapshot to save space.

At OGRE, we’d carried forward all our history from CVS to Subversion and as such have almost 8 years of history, including a couple of file reorganisations. Mercurial’s storage efficiency falls down compared to Git when files are moved around, because a file stored in more than one place in the tree over the history of the project is physically stored multiple times too, whilst Git stores the content only once with pointers from the various locations / history points. Most of this overhead could be removed just by eliminating old history we didn’t need anymore – history that does no harm in Subversion since only the server holds it, but does cause unwanted overheads in a DVCS since every user gets the entire repository. Removal of history is something that Mercurial shuns – rightly so in the case of public repositories but in these rare cases it would be nice if there was a tool for removing old history; again Git allows this but it has to be used with care. In the absence of that, doing it at conversion seemed the best way.

I asked about these things in the hgsubversion community, but the tradition of open source is that if you really want something urgently, you know where the code is :) Mercurial is really nice when it comes to hacking because it’s all Python; so there’s a nice unified API in one place that you can refer to – that’s one of the reasons I like it over Git which is far more fragmented in technology terms. I’m not a Python guru by any means, but I managed to implement both these features – I did the “mergemap” support a little while ago and added the “skipto” option today – it’s called that because “skipto” was already referred to in the hgsubversion code but it had no implementation.

The result is that the OGRE Mercurial repository with only the last ~3 years of history (back to when the v1.4 branch was created) is now only 74MB, rather than the 206MB of the original, complete conversion (in comparison Git was 116MB for the whole thing). By dropping the history I’ve removed most of the instances of reorganisation which is where most of the space has gone. I  hope eventually that Mercurial adds a utility to deal with stripping ancient history (right now, you can only strip branches) but this solves my primary conversion issue. Since this new repo can be kept in sync in a very lightweight fashion with the existing Subversion repo, I’ll be periodically updating it and doing more tests to reassure myself that the content really is ok.

If you’d like to get my custom version of hgsubversion with these features, it’s here: http://bitbucket.org/sinbad/hgsubversion/. I make no promises that it’s error-free, use at your own risk. It currently assumes that you’re using the standard Subversion layout, are converting from the root of that and have the ‘svn’ command on your path.

Development Open Source Windows

Microsoft, the good open source citizen

ms_haloWhat a difference a few years can make. For a long time, Microsoft was seen as public enemy #1 of those who liked to promote, produce and consume open source (I’m deliberately not describing it as a ‘movement’ here – that implies political motivations which I assert that only a vocal minority have). It was entirely their own fault of couse; blustery, really quite bizarre tirades from the only two CEOs their company has ever had cemented their position as the McCarthy’s of the modern era. It wasn’t helped, of course, by extremists on the opposite end of the spectrum, but still – the way the company behaved in previous years has at times been utterly shameful.

The reason it wasn’t sustainable is that they started to lose the very people they’ve always done a pretty good job of nurturing – developers. Even reasonable, level headed developers who have few extremist tendencies but who could see the many benefits of open source  (I count myself among them) began to turn away from the company as they seemed hell-bent on protecting their vested interests using whatever means possible, and irrespective of the collatteral damage – mostly through lies and threats.

I developed my early career around the time that Microsoft was rising, with their software replacing the mainframes and minis that were so tricky to work with at times, and I really appreciated them for it. They made my life easier as a developer in the 90′s. In the new millennium though, when they started rattling sabres over open source, and trying to bind me and my products into ever more of a restricted, Microsoft-only environment, they did precisely the opposite. The notion that you could use their really nice tools, so long as you only targetted Windows & Office, and with constant posturing over whether using open source was ‘communism’, drove me and probably plenty of other developers in precisely the opposite direction.

For as long as Steve Ballmer is in charge, I’ll have a healthy amount of skepticism about whether Microsoft can really, genuinely change its stance at its core. Like Bill Gates before him, these are agressive 80′s-style businessmen who  I can never hope to understand or remotely trust. But what’s clear is that either he’s learned how out of step he is with his potential customers, or he has been forced by others in the company to accept a changing stance on open source.

2009 is for me the year that Microsoft became a regular citizen of the open-source environment. Sure, before that they set up Port25 and CodePlex, but these were mostly self-serving and didn’t necessarily demonstrate MS’s ability to play well with others, which is precisely what open source is about. What really changed in 2009 is that Microsoft began to use external open source, intentionally and unintentionally, and crucially played it squarely by the rules with little or no fuss. This is a very big deal.

One of the first steps was Visual Studio using jQuery, which is entirely sensible. Historically Microsoft has had a terrible tendency to reinvent the wheel unnecessarily, which ends up being more hassle for everyone. Re-use of mature components for everyone’s benefit is what open source is about.

This year though, Microsoft has issued code under the GPL, something I’m sure many people thought would never happen. Firstly there was contributing code to Linux for Hyper-V, and most recently they (unintentionally) used some GPL code in a USB/DVD boot tool for Windows 7, an issue that was raised by a third party but which on investigation Microsoft confirmed – leading them to commit to releasing the full code under the GPL to customers.

Of course, this is precisely what they are bound to do legally. But the fact that it is being resolved in an open and completely unemotive manner, in the same way that any other responsible company would deal with it, is quite significant. This is Microsoft, the company that said the GPL was anti-American and borderline communist – openly and contritely resolving a GPL issue in the correct way with no sleight-of-hand or posturing. I respect that a great deal.

Welcome back to the community Microsoft, it’s about bloody time. Congratulations to all the reasonable people inside the corporate beast who are finally managing to turn the supertanker. I really hope you convince Ballmer to retire soon though, he’s a relic of a bygone age and an impediment to the new image you’re trying to create.

Development OGRE

DVCS Score Card

So, I’ve just about completed my practical experiments & review of Mercurial and Git.

In the end, I had far too many separate notes and sets of experiences to post, so I boiled the argument down into the 10 most important factors to me, and scored Mercurial and Git on a scale of 1-5 based on what I’d found when using them. Here are the (annoying) results:

# Criterion Git Hg
1 Ease of use – command line 4 5
2 Ease of use – GUI 4 4
3 Platform support – core 3 5
4 Platform support – GUI 4 4
5 Web Host Functionality 5 4
6 Reliability & error handling 3 5
7 Storage efficiency 5 3
8 Run-time performance 5 5
9 Flexibility 5 4
10 OGRE Community support 5 4
Totals 43 43

I’ll explain the scores, and my conclusion, after the jump.

read more »

Development Linux

Dependencies to build Git from source on [K]Ubuntu 9.04

Git is picky when it comes to converting large, moderately complex Subversion repositories and so far the only option I’ve found that works reliably is using the very latest version on Linux. Forget about using 1.6.5 on Windows via msysGit, at least for the git-svn conversion it’s very, very unreliable. Similarly I found Git 1.5 on Linux very flaky for the svn conversion. This doesn’t give me the greatest confidence in Git but in order to properly explore all the angles, I’ve committed to making it work even if it means I have to monkey about a bit.

So, I installed a fresh Kubuntu 9.04 (and of course, 9.10 went stable a few days later) and tried to build Git 1.6.5 from source. The configure script is unfortunately a bit rubbish and doesn’t bother trying to detect the dependencies though, so for those that don’t want to go through the fail/retry build loop I went through, here are the packages you’ll want to install via apt from a clean version:

  • build-essential
  • curl
  • libcurl3
  • tk
  • subversion
  • libsvn-perl
  • cpio
  • zlib
  • zlibg1-dev
  • expat
  • perl
  • iconv

If like me you use the excellent wajig wrapper around apt, you can also do ‘wajig build-depend git-svn’, but that seems to install things that are not strictly necessary in the default build (but maybe are needed with some non-standard options).

sudo apt-get install curl
   41  sudo apt-get install libcurl
   49  sudo apt-get install tk8.4
   53  sudo apt-get install cpio expat
   55  sudo apt-get install zlib
   61  sudo apt-get install build-essential
   66  sudo apt-get install zlib1g-dev
   72  sudo apt-get install asciidoc
   75  sudo apt-get install xmlto