SteveStreeting.com

For 18 months I’ve been told by a succession of doctors and physios that I didn’t have anything structurally wrong with my spine and that my bouts of back pain were simply ’standard non-specific back pain’ – ie muscle problems that I should just take NSAIDs for and exercise more. I’d been a bit skeptical because the problems were occasionally quite extreme and seemed to be always centred on one particular location (the joint just at the bottom of my ribcage), but after getting many opinions and one set of x-rays I went along with it.

Things have been quite good recently, up to mid-February when I had a bit of a relapse for a few weeks after doing a little too much. I raised it with my doctor again, explained that I’d been doing all the exercise and going to the gym as recommended, and yet it still flared up at what I considered to be fairly minor provocation. He scheduled me in for another set of x-rays which I expected to not come back with anything conclusive since the last set didn’t (and you can’t get into the MRI scan here unless you go through this step again first, allegedly). They took more pictures this time but I didn’t expect much given all the opinions so far.

Imagine my surprise therefore that when I got the results today, they actually had a concrete explanation for me. Apparently in my lower thoracic (ie exactly where I’d been pointing all these months) I have some disc degeneration and calcification going on, which is what is causing the stiffness and pain. This is something that happens with age anyway, but given my relative(!) youth (36) they thought it looked like it might be a result of either a trauma such as a sports injury – I can’t think of anything – or sometimes they see it in people who were child gymnasts – again not something I can attest to! Basically, something has happened to make my spine degenerate in that area faster than it should have done for my age. Too many hours spent stressed out at a desk may have been a contributing factor in that, although he thought it would have to be a lot of hours and probably combined with other factors.

So anyway, the ‘good’ news is that I actually have a reason now, an explanation for why I’m so susceptible to strains and stress on my back these days. In a way it’s nice to have something to point at. The bad news is that this isn’t fixable, it can merely be managed via careful exercise and lifestyle changes – many of which I’ve made already but I probably need to go even further. The prognosis is that I should be able to live pain-free so long as I manage it carefully over the long term to stop it degenerating further.

Following this analysis, I’ve been prompted to make a decision which I’ve been reluctantly considering for a while anyway – I’m retiring as OGRE Project Lead. I’ve thoroughly enjoyed my 10 years leading OGRE from unknown personal project to where we are today, but leading an open source project requires an enormous amount of dedication, passion, and above all an awful lot of time spent at a keyboard, most often in addition to a ‘regular job’ with which to pay the bills, and I feel I just can’t give that to the level that’s required any more. It will be with no small amount of sadness that I finally take off the leader’s hat – which by now is quite battered and worn in.

I still intend to be around and involved in the project – I’ll be contributing some code, giving advice when it’s wanted, and overseeing the establishment of an OGRE Foundation to handle the donations and funding side, but the days of me living and breathing OGRE, vetting every change, and being the person with whom the buck stops when there’s a bug, will be over. I’ll basically be contributing what and when I can, but shrugging off the responsibility and expectation that is inevitably associated with being the lead developer.

We have a great team and community around OGRE and I’m sure the project will be fine with me taking a more back-seat role – time for younger and less physically challenged developers to step into the limelight

Writing good documentation is hard. While I happen to think that API references generated from source code can be extremely useful, they’re only part of the story, and eventually everyone needs to write something more substantial for their software. You can get away with writing HTML directly, and separately using a word processor to write PDFs for so long, but eventually you need a proper tool chain with the following characteristics:

• Lets the author concentrate on content rather than style
• Generates multiple formats from one source (HTML, PDF, man pages, HTML Help etc)
• Does all the tedious work for you such as TOCs, cross-references, source code highlighting, footnotes
• Is friendly to source control systems & diffs in general
• Standard enough that you could submit the content to a publisher if you wanted to
• Preferably cross-platform, standards-based and not oriented to any particular language or technology

When I came to write the OGRE manual many, many years ago, I went with Texinfo – it seemed a good idea at the time, and ticked most of the boxes above. The syntax is often a bit esoteric, and the tools used to generate output frequently a bit flaky (texi2html has caused me many headaches over the years thanks to  poorly documented breaking changes), but it worked most of the time.

I’ve been meaning to replace this tool chain with something else for new projects for a while, and DocBook sprung to mind since it’s the ‘new standard’ for technical documentation. It’s quite popular with open source projects now and it’s the preferred format for many publishers such as O’Reilly. In the short term, I want to write some developer instructions for OGRE for our future Mercurial setup, but in the long term, I’d really like a good documentation tool chain for all sorts of other purposes, and Texinfo feels increasingly unsatisfactory these days.

Having spent some time this week establishing a new working tool chain, and encountering & resolving a number of issues along the way, I thought I’d share my setup with you.

Read the rest of this entry »

I’m not blogging as often these days; as you know I don’t traditionally ‘do’ short blog posts – in my book if something is worth blogging about, it’s worth making sure it holds together as an argument, and as a piece of writing generally – and a combined lack of time of anything I’m motivated (or permitted) to talk about has left the site a little  bereft of content. Luckily my OGRE Twitter is stocked with more frequent and less lovingly crafted status updates on what I’m doing there.

So, on to the title of the post. The Internet has been around for a while now, and has evolved rapidly, particularly in the last decade. And yet, particularly in academic and some open source developer circles, there is an attachment to a particularly creaky piece of technology that I can honestly say I do not share - the venerable mailing list.

Now, to clarify the context, I’m referring to the use of mailing lists for multilateral communication for an entire community, including newcomers, as opposed to a simple 1-way notification list (like we use for commit notifications for example). For N-way communication among a small group of core developers, all of whom will want to read every post, I can see the utility and convenience of a mailing list. But as a community communication channel, where people just want to drop in and drop out, I find it a staggeringly inefficient, awkward and archaic approach. I say this primarily as an occasional community member of various projects that use mailing lists, and therefore someone who has a specific interest in a mere subset of the discussions that go on – I have no time or desire to read every single thread, and indeed if I tried to do this for every project I have an interest in, I’d never get anything done. It’s hard enough to keep up with my own open source community!

The simple fact is that mailing lists have an all-or-nothing mindset that is woefully outdated for community interaction on the scale that the Internet has now grown to.  Subscribing means you get bombarded with every single discussion, either individually or in digests, which pretend to be useful but in fact aren’t, because while they cut down on the number of emails you get, it makes replying to specific posts a pain. If you want to read every single mail in the list, I’m sure they work fine – but most people outside the core group do not want to do this. Most members of the community just want to keep a closer eye on a few select threads of discussion that either affect or interest them, and to be able to search and browse through the rest easily – and the mailing list is a woefully inadequate, blunt instrument for this kind of task.

Sure, you can choose not to subscribe, and go through the archives, searching or browsing them. But you can do that with forums too, and there at least you have the advantage of categorised areas of interest, being able to follow certain people, and to watch certain threads. Mailing list archives have a single filter: date, and also lag by a number of hours dependent on the individual setup, so if you’re not subscribed, you get a lesser service.  Another technique is to subscribe completely but tell your email client to archive or filter things for you, so you can dip into your local replica at leisure. Horribly, horribly inefficient, but it does work.

Mailing lists worked in the 90’s when there were small groups of people who wanted to read everything being discussed, and when email was the primary form of communication between people. We’ve moved on. Forum systems and other flexible hosted systems are far superior in their ability to let you watch particular discussions (or all new posts) that you’re interested in and get told when there’s an update. Anyone can search them easily (internally or via Google) and there’s no archive lag. Maybe some people are worried about forum databases being lost, compared to inherently replicated mailing lists, but anyone worth their salt has a server backup strategy.  Honestly, any project that uses mailing lists as their only community discussion channel instantly puts me off getting involved in that community, because I know that as an occasional participant interested in only certain discussions, the experience is going to totally suck.

And, if you insist on loving your mailing lists approach so much, for goodness sake move to Google Groups. They’re still pretty basic, but at least there, those of us who have moved into the browser world can use an interface we find useful and productive, rather than being forced to use 20-30 year old technology designed to replicate posts around a university science department.

Making a living from open source is hard. Correction – making a living from writing open source software is hard – it’s incredibly easy to make a living from someone else’s open source software of course, which is why that’s what most people do At one time the popular opinion was that pure-play open source companies could make a living from support services, which works to a degree but I know from both my own experience and from that of others that it doesn’t work that well. Again, the best chances of it working are if you’re providing support services for software that someone else writes, because you’re only able to monetise the service, not the development. This actually discourages people from investing in development, and instead merely in deployment and ancilliary services which isn’t actually a good thing for core product development.

The best cases of companies funding open source are where they’re using it to deliver some other product or service which is directly monetised, therefore the open source development comes under their general R&D budget. Google, IBM and others fall firmly under this category, and you can bet that the largest open source software projects are funded this way – Apache, Eclipse, Firefox all pay their core developers like this. But, it requires a fairly significant level of scale to be able to do that, hence why it’s usually the giant corporations that do it rather than smaller companies.

The next favourite option is dual-licensing; the general set-up if you come at this with a commercial hat on, is that you pick a license that a lot of commercial entities will have a problem with extending from (ie GPL), then you sell them an alternative license; the idea being that you get the adoption via the open source license and make money from the commercial license. But, it can be controversial, as most recently discussed by Greg Stein in the Oracle / MySQL case.  The argument is that if your commercial license is just a proprietary license, and can be revoked and otherwise monkeyed with by the issuing company (or perhaps more importantly, its acquirers), you have actually been lured into a honey trap – the lure being that open source comes with certain protections, but that if you rely on the availability of the commercial license you actually have none of those and might as well have bought from a proprietary software vendor.

So, what to do? If you’re a small development company, open sourcing your product will definitely bring more people in, but if you’re not in the hosting / cloud business and don’t want to rely on services to earn your keep (who can blame you), what can you do to earn your keep except abandon open source for your main products (maybe splitting your time between proprietary and open source), or dual-license and face accusations that you’re fibbing about the true nature of your product for your commercial users?

Well, I’ve been wondering whether the problem is that dual-licensing typically falls back on traditional licensing concepts, ie that your commercial license looks very much like a normal proprietary license, which has all the problems of ‘what if my vendor changes the license conditions’ etc – when in fact it really needs to be more like a permissive open source license, with a payment condition. One of the great powers of open source is that it is ‘detached’ from the producer and compeltely predictable and immutable – once the software is out there, it can’t be taken away from the receiver and is always ‘whole’ in terms of the source code so no-one is tied in. There are also cast-iron source & binary redistribution clauses that are known up-front, and are again immutable, which mean everyone knows where they stand, forever. Why can’t the commercial side of a dual-license continue to do this, while at the same time generating a revennue stream for the company?

Maybe I’m being naive. But what about this sort of dual-license set-up for a library or toolset:

• Default is GPL (and obviously free)
• Commercial alternative license available, giving very permissive rights, but with these important rules:
  • The license is irrevocable once issued
  • The right to redistribute unlimited copies of derivative binary works is included with Apache-style conditions
  • The right to redistribute unlimited copies of derivative source to anyone under the GPL (for free) is included
  • The right to redistribute unlimited copies of derivative source under the permissive commercial license conditions is also included, provided the same original license fee is paid per receiver. Critically, the price and conditions surrounding redistribution may not be altered unilaterally by the licensor at any time after the license is issued (so once you’ve bought it once, the conditions and price for non-GPL redistribution are set in stone and cannot be altered unless both parties agree – say if the price is reduced later)
  • All software reverts to the Apache license if the company folds without selling the rights to someone else

This would mean that those choosing to opt for the commercial license would have the same kind of cast-iron guarantee an open source user has that once software is out in the wild and being used under some conditions, that the originator cannot possibly change that, ie take it away or change their right to modify and redistribute under conditions they agreed to at the start. To me, this seems to give the same kind of certainty over not being screwed over in the future as open source does, thus blunting the accusations of proprietary lock-in by the back door, but while generating some revenue for the developer too. It is, in effect, the same as a permissive open source license with the one addition that redistribution of the source to a new party requires either payment to the originator, or reverting to the GPL.

Now, of course there is still potential uncertainty around new versions of the software, but this is no different from open source, where your only guarantee is over what is published right now, not what might happen in future versions.

Does anyone know companies that use this model? My experience is that commercial dual licenses tend to be as restrictive as proprietary licenses, which then can justifiably lead to accusations that the open source license has been used as a shill to get people into a lock-in scenario. Is there really a ‘third way’ or am I missing the point?

Gartner haven’t exactly been the sharpest tools in the box when it comes to predicting open source trends over the last few years, vastly underestimating it until about 2008, by which time it didn’t exactly take a professional analyst to tell you that it was popular. Still, now they’ve woken up to its potential, occasionally they post something useful. In particular, I liked a recent blog post about how open source is “trending towards customer obscurity” – that is to say that while open source is incredibly important to producers of software, the vast majority of consumers don’t really care how their software is made any more than they care how their car was made.

I support this view, and it’s one I’ve subscribed to for a while (although the somewhat condescending tone of the article is typical Gartner, the point is valid). My own open source software is aimed squarely at developers, where I think it adds value; since the users of my software are themselves making significant development investment in products using it, open source has significant advantages – the openness and participation-friendly nature of the development, the fact that the software can never be taken away from them either by company policy or acquisition, the fact that absolutely nothing is hidden behind any curtains so there aren’t any nasty surprises. When you’re investing your own time building on top of a foundation, there really is no substitute for being able to see all the working parts, should you want or need to.

Developers can be quite a broad church too – enterprises for example often have a need to modify and adapt software and that’s why open source has been very popular there too, even if they’re not actually making products of their own for publication.

But there’s also a vast group of people who are more traditional consumers (personally and in companies) – and they have no reason to care about open source if they’re never going to modify the software. There is a group of people who are philosophically dedicated to using free software (more so than open source) even if they never modify it themselves, but they’re a minority in the grand scheme of things. Most people that use open source do so because they feel it gives them what they need as users. Even personally, I use Linux on my servers not because I’m dedicated to using open source over the alternatives wherever I can (although I probably do have more bias in that direction than the average), but because it does exactly what I want in a server – it’s reliable, unobtrusive, cheap, has low hardware requirements and plenty of good software. Conversely, I don’t use Linux on the desktop much because personally I don’t feel it operates better than the commercial alternatives in that environment. I decide on a case-by-case basis what works best for me, and so do all but the most fanatical of users.

Now, of course open source regularly helps developers make better products (by using mature, reusable and adaptable components), which in the end can result in more users using software made from open source even if they don’t realise it. But the important thing to remember is that open source itself isn’t a marketing bullet point except to developers and the enthusiast minority. Like any gathering of like-minded people who mostly talk to each other, we open source developers / advocates can often forget that our enthusiasms aren’t necessarily shared by the rest of the populace. We have to remember that the end result is everything – and while open source definitely has a positive knock-on effect on product quality, it’s often a means to an end, not the end in and of itself. It’s obvious really, but worth keeping in mind.

I picked this story up via Matt Asay and it pretty much summed up the frustrations I’ve had in the last 10 years when talking to certain people about open source – particularly when I was involved in business software. Peter Gyorgy, CIO of GE made this comment in a recent panel discussion:

“I think open source is great for own internal playground type of things but if it’s running vital mission critical applications – networks running on open source for example – then that is a huge, huge risk to the organisation,”

This would be incredibly funny if it wasn’t so damn indicative of so many CIOs, managers and other closed-minded, overly conservative IT people who have long since given up on trying to stay reliably informed and just believe what their vendors tell them. It’s especially amusing given that GE’s healthcare division runs its mission critical software on Linux, which their CIO seems oblivious of. And I would expect the New York Stock Exchange would be considered ‘mission critical’, and it runs on an open source platform (and interestingly the LSE is switching to Linux too) – so clearly not everyone thinks like this.

The one place where he does have a potential point, albeit skewed beyond all recognition is when he says:

“We are not here to be an IT shop, we are here to be the partner of a business and we shouldn’t put businesses operations into risk by running very low cost solutions,”

That’s a very valid point. However, it’s got nothing whatsoever to do with the choice between open source and anything else! This is such a common misconception. Open source has matured – if you need enterprise-level open source there are companies that are quite happy to take your money to remove the hassle and worry of system stability. They’re really no different from the Microsofts and Oracles of this world, except that the software they’re running your system for you on is open source rather than closed. That gives you an additional bit of leverage because if they suck, or if they try to pull a fast one on prices, you can actually get that enterprise management from someone else without having to change your software too. Try doing that when you switch from Oracle to Microsoft or vice versa for services.

You also have the advantage of not having to wait for a central vendor to hear your pleas for feature A or B, or a bugfix that might be low priority for most people but is absolutely critical for you. Instead of hacking workarounds and seething in the wings while you wait for your vendor to get around to addressing something they think isn’t a priority because it’s not affecting that many people, you can pay someone to fix it for you and submit it upstream, where it will undoubtedly get accepted far quicker than it would have got fixed if only a central vendor was looking after it.

You don’t have to be running an IT shop – although if you do, you have the option of trading your own time for monetary savings and greater agility – your support options are just different. Sure, they can be slightly more complicated if you let them be – particularly if you’re looking to save money or drive things in an optimal direction for your company such as tailoring the software – but if you want to have a simple 1-vendor setup using only standard versions you can do that with open source too. Delegating all support to a third party will cost you more but the option is there. It’s all about choice, flexibility, and empowerment – all things a CIO should welcome, not be afraid of (otherwise he/she’s probably in the wrong job).

I think too many IT managers / CIOs have a mental block which prevents them from being really committed to optimising their IT delivery, in terms of both spend, alignment with the business and agility for the future, because they’re locked inside a box of their own making.

I always find Matt Asay’s blog an interesting read – even if I don’t always agree with him, his posts on open source are always thought provoking. Today he was talking about how Wikipedia’s contribution rate is falling and how that has parallels in open source; that the community is no replacement for a centralised, focussed team.

He’s right on the core point – at the heart of every successful open source project there’s always a core team (or individual), and in the really influential ones, that team is usually funded – Mozilla is famously bankrolled almost entirely by Google, the Apache foundation has many, many sponsors including Google, Yahoo and Microsoft, Eclipse has IBM, and so on. Many of the big projects that don’t have more general sponsorship still have a core team funded by a dual-license or other premium software model: MySQL, RedHat/JBoss, Qt etc. Such guidance & direction at the core is crucial – at OGRE we have a core team too, except that we’re not directly funded by anyone in terms of developer time (we have several generous sponsors who cover the majority of our hosting needs); we guide it because we want to, and because we use OGRE ourselves too. My company is probably the closest thing to a core development sponsor, in that I’ll allocate “work time” to doing OGRE development that could otherwise be spent making commercial products or doing consultancy, but it’s by necessity small beer compared to the likes of Mozilla and Apache.

But I do think he underplays the changes that have taken place in the software development world. He asserts that because most headline software development is still focussed at big influential companies, we’ve mostly just rearranged the chairs a bit at the same banquet. I don’t agree with that at all – by nature it still makes most sense to concentrate much of the development in a small team for quality, consistency and organisational purposes, but the point is that where precisely this centre is determined primarily by merit, not by the boundaries of a company’s org chart. While the core team is doing a good job, and accepting reasonable patches and such, people are happy for the show to be run there. The community is still definitely involved in the development, and certainly adds considerably to the end result. Yes, proportionately the central team does more, but crucially, should anything go badly wrong – such as the core going in a direction a lot of people don’t like, or the product being sidelined, if there’s enough of a community a fork will emerge, with another core team to lead it. That’s a critical safety valve that keeps companies more “honest” than they had to be in the past, and is a vital insurance policy for anyone investing their own resources in a piece of software. Matt claims the ‘Command and Control’ setup of software vendors is still in place; I think his view is clouded by the fact that he’s solely focussed on enterprise software, and enterprise customers move at such a glacial pace that any change is largely imperceptible – to the extent that ‘community’ maybe does look a lot like the ‘customers / partners’ relationship of old. But that would be a bad call, completely ignoring the difference in the level of control that is ceded to a community versus the customers of old – sure, many enterprise customers may not wish to leverage that control, and would take a long time to move if someone else chose to do so, but that option is still always there. And not everyone in the world is an enterprise customer – the enterprise usually follows the grass roots eventually.

In practice, it’s really all about balance, the middle ground. Yes, we still need focii of development just to make sure things get done in a reasonable fashion – no-one likes chaos in their software. Yes, it makes most to have that focus funded, in a traditional company model, if that piece of software gets beyond a certain size / popularity. But that doesn’t for a second undermine the value of community participation; in fact the two are deeply interdependent – one without the other is just not sustainable in a sizeable project.

So, people certainly shouldn’t be deluded into thinking that random crowds of people on the internet will create great software without some organisation (the infinite monkeys creating Shakespeare fallacy), but they also shouldn’t think that community is disposable and that we’re in the same situation we were before but with a different label. Nothing could be further from the truth.

What a difference a few years can make. For a long time, Microsoft was seen as public enemy #1 of those who liked to promote, produce and consume open source (I’m deliberately not describing it as a ‘movement’ here – that implies political motivations which I assert that only a vocal minority have). It was entirely their own fault of couse; blustery, really quite bizarre tirades from the only two CEOs their company has ever had cemented their position as the McCarthy’s of the modern era. It wasn’t helped, of course, by extremists on the opposite end of the spectrum, but still – the way the company behaved in previous years has at times been utterly shameful.

The reason it wasn’t sustainable is that they started to lose the very people they’ve always done a pretty good job of nurturing – developers. Even reasonable, level headed developers who have few extremist tendencies but who could see the many benefits of open source  (I count myself among them) began to turn away from the company as they seemed hell-bent on protecting their vested interests using whatever means possible, and irrespective of the collatteral damage – mostly through lies and threats.

I developed my early career around the time that Microsoft was rising, with their software replacing the mainframes and minis that were so tricky to work with at times, and I really appreciated them for it. They made my life easier as a developer in the 90’s. In the new millennium though, when they started rattling sabres over open source, and trying to bind me and my products into ever more of a restricted, Microsoft-only environment, they did precisely the opposite. The notion that you could use their really nice tools, so long as you only targetted Windows & Office, and with constant posturing over whether using open source was ‘communism’, drove me and probably plenty of other developers in precisely the opposite direction.

For as long as Steve Ballmer is in charge, I’ll have a healthy amount of skepticism about whether Microsoft can really, genuinely change its stance at its core. Like Bill Gates before him, these are agressive 80’s-style businessmen who  I can never hope to understand or remotely trust. But what’s clear is that either he’s learned how out of step he is with his potential customers, or he has been forced by others in the company to accept a changing stance on open source.

2009 is for me the year that Microsoft became a regular citizen of the open-source environment. Sure, before that they set up Port25 and CodePlex, but these were mostly self-serving and didn’t necessarily demonstrate MS’s ability to play well with others, which is precisely what open source is about. What really changed in 2009 is that Microsoft began to use external open source, intentionally and unintentionally, and crucially played it squarely by the rules with little or no fuss. This is a very big deal.

One of the first steps was Visual Studio using jQuery, which is entirely sensible. Historically Microsoft has had a terrible tendency to reinvent the wheel unnecessarily, which ends up being more hassle for everyone. Re-use of mature components for everyone’s benefit is what open source is about.

This year though, Microsoft has issued code under the GPL, something I’m sure many people thought would never happen. Firstly there was contributing code to Linux for Hyper-V, and most recently they (unintentionally) used some GPL code in a USB/DVD boot tool for Windows 7, an issue that was raised by a third party but which on investigation Microsoft confirmed – leading them to commit to releasing the full code under the GPL to customers.

Of course, this is precisely what they are bound to do legally. But the fact that it is being resolved in an open and completely unemotive manner, in the same way that any other responsible company would deal with it, is quite significant. This is Microsoft, the company that said the GPL was anti-American and borderline communist – openly and contritely resolving a GPL issue in the correct way with no sleight-of-hand or posturing. I respect that a great deal.

Welcome back to the community Microsoft, it’s about bloody time. Congratulations to all the reasonable people inside the corporate beast who are finally managing to turn the supertanker. I really hope you convince Ballmer to retire soon though, he’s a relic of a bygone age and an impediment to the new image you’re trying to create.

I’ve been involved in open source for a long time – probably what might be considered a ‘generation’ in this industry. I was a fan of open source before I even knew the term existed – during my formative coding years in the early 90’s I was always releasing code for free and encouraging people to tell me why it sucked, and doing the same for them. Of course, most of the discussion went on over FidoNet, BBS-relayed emails, the very early (pre-WWW) internet and code on FTP sites, but the principle was much the same.

It was only when I needed a host for an increasingly large 3D engine I’d been writing in 2000 (which later became the earliest version of OGRE) that I discovered Sourceforge and open-source licenses. At that stage, I didn’t really care much about the minutae and just went with what seemed the most popular choice – the GPL. Fairly soon after this was revised to LGPL because it was clear GPL wasn’t a flexible choice for a library. Over the years I formed the opinion that the LGPL was a good balance for what I was doing – it allowed use in proprietary applications but required that modifications to the open-source part were passed on; and this also, I thought, encouraged contributions back to the core since it’s simpler to have your changes promoted. The downside was that the license is pretty long and full of legalese, and frequently needed explaining & clarifying. Still, I considered that a necessary cost of a license that ‘protected the interests’ of the open source project. I had considered more permissive licenses like zlib and MIT to be too weak -  giving everything away with no conditions encouraging reciprocal contributions or participation.

I only really started to seriously question this long-held opinion recently, shortly after I had to deal with a license violation issue. The details of this are irrelevant (and will not be discussed here), the important thing is that after all the dust had settled, it really made me sit down and take a long, hard look at what open source licenses actually achieve in practice, and which bits I felt were the most important for sustaining my projects.

Purist free software advocates subscribe to the view that all software should be ‘free’ (as in speech), which is, when you boil it down, a philosophical principle. While the intention is often genuinely altruistic – enablement for all developers and users of software – in a practical sense it is also a highly prescriptive, black-and-white approach. Free software licenses like the GPL restrict as much as they enable, on the basis that not to do so would deny future users of the flexibility to alter the software. It’s an understandable point of view, if very hacker-oriented, but it’s not one that I personally sign up to. I’m not involved in open source as a ‘movement’, in order to ensure that all users have access to all the source code of every application they use. Instead, I’m involved in open source because I think it’s a damn good way to get good software written for the benefit of the many, particularly infrastructural elements that can then be used to innovate faster, by smaller teams, to challenge incumbents, and generally shake things up. I’m a firm believer that open source and proprietary software can be good bedfellows, and that each can benefit the other – open source providing solid foundations for proprietary innovation, and proprietary software contributing to the open source projects underpinning it, in terms of code and funds. In that context, the restrictive elements of the LGPL seemed well-placed – the LGPL allows proprietary use, but modifications had to be passed on.

But, after much consideration and hard self-questioning in recent weeks I realised something important. Of all the people that have contributed to OGRE over the years, I can’t think of a single good code contribution that has come about because the license conditions encouraged it. All of the people who have made a significant positive contribution to OGRE have done so because they chose to do so. Either because they wanted to do it for fun, or because they saw that it was in their own interests as users, they sent patches in, created new add-ons, joined the team etc. None of these people required any coersion from a license to do what they did, and all the better for it – because they chose to do it, they tended to be more forthcoming in terms of adhering to standards, answering questions about their contribution, and generally participating in the community rather than just doing a minimal code-dump.

The other thing I considered was the impact of someone modifying OGRE and not publishing their changes. On the one hand, this is a potential for a lost contribution, but in practice the LGPL conditions are only to pass on the source to others, not to contribute back to the core project (which requires a contributor agreement anyway), and as mentioned we’ve already established that the best contributions are from willing sources anyway. Then comes the principle of whether it matters that the end user is not guaranteed to see the modified source of OGRE – free software principles say this is not right, but I have to say that in practice, most end users don’t care and in practice it’s of little consequence for OGRE. Obviously many people in the free software camp would disagree strongly with this. It’s more of an issue in enterprise apps where a customer who receives a significantly customised application from a vendor, and needs to keep maintenance in mind; IMO regardless of whether the software is based on open source the customer should be insisting their vendor delivers source anyway – that’s what contracts are for. But I digress.

So, after much thought I concluded that the most useful pay-backs to an open source project, and thus its community, from a user (in my opinion) were:

1. Code & documentation contributions – which based on my experience come from voluntary sources
2. Community participation – forum support, bug reports etc
3. Publicity.

The ‘restrictive’ elements of the LGPL (and GPL), to which so much confusing license text is dedicated, didn’t seem to contribute to any of those except number 3, and then really just as a side-effect.

It was at this point that I realised that my previous opinions about permissive licenses not providing enough safeguards against exploitation for an open source project were off-base. In practice, open source projects don’t really need protection, because their best contributors are going to be there regardless (yes, I realise the GPL provides more protection to end users who want to get at the source code, that’s not what I’m considering here). ‘Freeloaders’ – people who use or modify the open source project for their own ends but give no code or community contribution back – are always going to exist; even under the GPL it’s easy to freeload, if you make your money from hosting services for example, and thus license choice has little impact on the scale (if not the nature) of the freeloading. Besides the annoyance of ‘that guy took my work and made some money out of it’ – which you have to accept as an inevitable outcome of going open source, so stick to making proprietary software if that bugs you – freeloaders have little negative effect on an open source project, and actually their use can contribute positively to item 3 (publicity). The key is to recognise that in practice you can really just ignore freeloaders, and instead concentrate on maximising the positive contributors in your community.

So, if we acknowledge that the people whose contributions we actually want are those who contribute voluntarily, regardless of license, we quickly come to the conclusion that all that really matters is the size of the community. It’s a fair assumption that for a given project there is a relatively stable percentage of users who will choose to contribute back (the percentage itself varies per project, but is fairly stable per project in my experience), therefore the easiest way to increase your contributors is to just increase your user base. Forget about trying to coerce people into being ‘good’ members of the community, just trust that the percentage will be there and will track your overall numbers.

One way in which to attract more users is to make the licensing simpler and more easier to understand. Programmers hate legalese, and a simple, clear license is bound to be more attractive than our LGPL (with static link exclusion), plus OUL option. It’s for this reason that from OGRE 1.7 we’re switching to the MIT License.

I’d like to thank Matt Asay for his post on the subject, which really jump-started my thought processes on this subject in the last few weeks. This whole process has been a reminder to me that it’s always good to re-examine your previous assumptions and opinions in the light of practical experience, and to be prepared to reach different conclusions than you did before. The whole open source landscape is constantly changing and maturing, why should those of us immersed in it be any different?

Ok, so I’ve posted my initial feelings about tinkering with Mercurial and Git, and that seems to have generated some interest. It’s time to get a bit more formal about how I’m going to evaluate them against each other, to decide which one I like to use most in real, practical scenarios. So, I decided to come up with a list of use cases for the things that I typically have to deal with when managing the repositories for a software project (open source and otherwise), so that I can methodically test them out and see how I feel about each system. I’ve tried to deliberately include use cases for things that you can’treally do on a centralised system, but that I’d want to make use of, as well as the usual nonsense that happens day-to-day on a typical project

I’ve presented my work-in-progress list below, feel free to suggest more use cases I can test. I really want to see what these things are like in practice in the kinds of situations I encounter in real projects, without actually risking a real project in the process! Like a backup / restore strategy, it’s no good doing it for the first time when the sh*t has already hit the fan.

Oh, and for fun, allegedly Git is MacGyver and Mercurial is James Bond .

Due to the nature of a DVCS, all these use cases must be tested both in isolation, and  after pushing those changes to another potentially ’superset’ repository.

1. General
   1. Commit a few changes to local repository ‘A’ but don’t push to central yet. Push a number of different changes to the central repository from a different repository ‘B’. Then, pull ‘B’s changes from the central repos to local repository ‘A’  to bring it up to date, again without pushing A’s outstanding changes yet. This is equivalent to doing an ’svn update’ while you have local uncommitted changes in Subversion, but while using the local commit features of the DVCS. How does this work in practice? [Suggested by Arseny Kapoulkine]
2. Branching
   1. Create a new public, official branch (stable branch)
   2. Create a new long-term feature branch which is intended for public consumption / collaboration
   3. Create lots of short-term development branches (or equivalent structures) intended for local consumption only
      1. What are the size overheads? (Git claims superiority here)
      2. Does this excessively clutter the public repository when pushed?
      3. Is there a better way of handling multiple local changesets which you may or may not decide to push individually, such as testing many patches (Mercurial queues seem very interesting, Git equivalent?)
      4. . Rename a branch? (optional)
3. Merging
   1. Merge changes unidirectionally from one branch to another, without having to manually pick revisions. Make more changes in the source branch and repeat.
   2. Bidirectional merge – a feature branch which is not yet ready to be merged into the trunk wants to resynchronise from the trunk and continue branched development. Merge of this branch into the trunk must be tested later after further changes
4. Tagging
   1. Create a tag against a specific branch; probably at the HEAD but look for the option to specify a revision
   2. Correct / modify / move a tag following a mistake or last-minute revision (pre-release) without having to make duplicate commits or other such spurious activity
5. Firefighting
   1. Screw-up: developer commits change to trunk instead of stable branch. Merge / move it to the stable instead – change can be left in the trunk or can be removed for re-merging, so long as the procedure is clear.
   2. Screw-up: developer commits change to stable branch that is interface-breaking, must be removed and moved to the trunk. Must be removed from the stable branch and moved.
   3. Screw-up: Revert a single change from the repository, that is not at the HEAD
6. External patch submission tests:
   1. Patch file from same branch, no conflicts
   2. Patch file from same branch, with conflicts
   3. Patch file generated on a different branch to the one we want to apply it to (include conflicts)
   4. Pull from third-party repository, entire branch
   5. Pull from third-party respository, specific changes
   6. Patch file generated from non-repository source copy
7. Backup multiple work-in-progress changes on a local machine that are not ready for public consumption; approaches:
   1. Store a patch per local branch (this is how it’s done with SVN, but too much hassle if you use lightweight local branching, DVCSs can do better)
   2. Push to a backup repository on another machine across existing protocols – ssh, https, Samba share (Git can’t do the latter?)
   3. Push to a backup respository on a USB stick (Git can’t do this?)
8. Binary files
   1. Revise a binary file over a few versions, test storage efficiency
   2. Binary file conflict resolution
9. Conversion from Subversion
   1. Import retaining history
   2. Import multiple branches
   3. Import tags
10. Integration
    1. Mailing list / RSS notifications of commits on official repository
    2. Bugtrackers
    3. CIA.vc et al?
    4. Good free & open source GUI clients for all platforms
    5. Line ending conversions between platforms