SteveStreeting.com

Quick check – ok, the sun is in fact not as black as sackcloth. But today, something earth-shattering happened – Microsoft has contributed code to Linux.

I’m sure I’m not alone in thinking that I’d never live to see the day this happened. It’s 20,000 lines of driver code to make Linux run better under Hyper-V, which is of course in their interest (since you have to buy a copy of Windows Server 2008 as the host) , but that’s par for the course for open source contribution (you scratch your own itch!), and it’s a massive watershed regardless. From what I hear there’s still a lot of concern at Microsoft about how to manage contributions across the company boundary (in both directions), so I’m not sure what extra procedures they would have put in place for the developers involved in this process to keep the corporate legal army satisfied – perhaps pre- and post-project selective mind-wipes – but the fact that they managed to make it happen is a big deal.

Microsoft has wielded by far the most acrid rhetoric about open source in the past – we all hear that it’s changing, and I know particularly of specific people at Microsoft (mostly developers) who take a much more open view, but it’s hard to escape the feeling that while the top brass who set the ‘old’ policies remain in situ, substantive change will be difficult. But this move is one of many lately that make me think that just maybe, people higher up the chain are starting to get it. Or at least, they’re starting to defer to people who know better.

I’d argue that very few people in the open source community are inherently anti-Microsoft, they’re just a little more free-thinking when it comes to technology choices, a little more honest with their opinions, and have come to view MS as ‘the enemy’ primarily because of the old rhethoric the company used to use on a regular basis to attack them (and some parts of the company still don’t seem to be getting the ‘openness’ memo – as TomTom found to their detriment). Microsoft, or rather, Mr Gates and Mr Ballmer specifically, effectively made themselves the enemy of the open source community with their often ill-conceived tirades, and that’s something that will take a long time to heal. But, as we all know, actions speak louder than words – and if the company continues to make these kinds of conciliatory moves, they will start to win people in the open source community back, at least those people that judge on facts rather than old predjudices.

Trust takes a long time to be earned, particularly from where MS started from, so it’ll be a long road – but if this is how things are going to develop in future, then bon voyage, MS.

So, after much initial confusion, sabre-rattling, dispair and finally acceptance (sometimes grudging), the world now pretty much groks open source. In addition, we’re all getting better at doing open source – it’s increasingly obvious what the best practices are in terms of growing and developing an open source project, and of how businesses as well as hippie coders can use and produce it effectively. Increasingly, open source is a known beast, and those that don’t embrace it in some fashion are increasingly looking hopelessly luddite. It’s all a far cry from 10 years ago when merely suggesting open source in a serious environment would get you some curious looks  – having been the recipient of many of those looks over the years, it’s satisfying to be able to pull out the ‘I told you so’ line now and again

So, open source is mainstream – that’s good. So, apart from gradually filtering into the last few bastions of old-skool thinking (console vendors, I’m looking at you – how about returning my calls sometime, mm?) , where do we go next, besides more of the same? A wild-eyed revolutionary could get bored with all this mainstream acceptance, after all

I think Matt Asay nailed it in his post today. Open source only works at optimum efficiency when you’re accepting of outside contributors, and the worldwide melting pot / distillery is on full, mixing up and crystallising the best the community has to offer. But, due to the nature of this, all the people involved by nature tend to be technicians – programmers, sysadmins, people with a technical bent with a problem to solve, and some time to contribute to help make it happen. As Matt says (and as I’ve said here before), it’s why Linux is hugely successful on the server and with device builders, but less so on the desktop with real people. He says this:

But the real goldmine is broadening the definition of “developer” to include lay users of your software. The day that I, as a nontechnical software user, can meaningfully participate in an open-source project is the day that open source will truly have won

This is dead-on. There’s a perception that only developers and techs like to contribute to projects like open source, but that’s not true at all. There’s a certain cache of people from all walks of life that like to contribute to creative endeavours, who have ideas and the ability to express them, and their abilities are entirely complimentary to a technical team creating a product. Hell, I’d go as far to say as they are absolutely essential to the construction of any product that’s aimed outside of the technical user market. Anyone who has been involved in a large software project, and who isn’t an elitist prig, will know that you need your ‘lead users’, really, really badly. These are the people that aren’t usually technically minded beyond being a user of technology, and being open to change, but who are smart enough, and experienced enough in the domain, to know what needs to be be built, how things can be improved, and how a wider population will perceive it. They will regularly come up with points that you, as a developer, would never have remotely considered, and will make what you produce far better than it ever would have been without their input.

In large commercial projects these people are usually on the payroll as analysts, liasons, testers, technical writers, and other roles. Yet in open source, they are vastly underrepresented, because open source typically starts with a developer scratching their own itch, then attracting like-minded individuals. There’s a chicken-and-egg issue around getting non-technical people involved – they won’t necessarily be interested unless there’s something they want to use, but the product may not exhibit those aspects in the first place without input from like-minded people. As a developer it’s easy to get on board and complete the value-circuit quickly with a couple of patches, but it’s much trickier for other roles outside of a long-term project arrangement.

I’ve been lucky enough to be involved in projects working alongside really excellent lead users in the past, and I make a point of calling out developers who talk about ‘users’ in derogatory terms. Smart, experienced but non-technical users are like gold dust, you want to hang onto them with both hands, even if they do give you feedback you don’t always want to hear. If, somehow, we could encourage greater participation from these kinds of people in open source, I know things would be the better for it. How? I have no idea yet

There have been cries of joy on the intertubes recently from people who like .Net but who like using it on non-Windows platforms (or believing they have the option to do so), in that Microsoft has extended their Community Promise to Mono, meaning they won’t sue over uses of it. Or so it’s been rather shallowly reported in some circles, more on this below. Firstly I’ll just say that no matter what, this is a positive step. I’ve long thought of Mono as a technically excellent project which an awful lot of uncertainty and legal confusion around it, which was only really made worse by the highly specific arrangement that Novell and Microsoft brokered a while ago.

What this announcement does is bring some clarity to the situation, which is good. However, I think people should hold off putting the bunting out just yet, because as ever the devil is in the detail, and the promise only in fact covers ECMA standards 334 and 335 – that is, C# and the CLI. As anyone who has used .Net knows, there’s an awful lot more to it than that, and this promise is currently equivalent insuring the wiring and plumbing in your house, but none of the contents. So it’s nice that these standards are now explicitly covered, meaning that projects that solely rely on C# and the CLI plus other completely independent components are safe. But if you’ve used anything else, such as any of the recreations of the rest of the .Net framework, you’re still on very uncertain territory, and in some ways you could say it’s even more uncertain, since clearly Microsoft omitted those from this promise for a reason (which may be a deliberate, strategic position, or it might be that they’re just not ready to take that more significant step yet – we don’t know).

Mono intends to cope with this by providing clear dividing lines between what’s covered and what’s not, which is very sensible. However, I do suspect that will result in the vast majority of the stuff that developers actually need to rely on when writing what are generally considered ‘.Net systems’ (such as implementations of core components of the .Net framework) will still be in no-mans land. Thus, while this promise does move things on in a positive way, in many ways it again casts light on how uncertain the legal position of the other elements is. Personally, my position is pretty much unchanged from before – if you’re writing ‘.Net applications’, that is, using the .Net frameworks, then you’re targetting Windows, don’t fool  yourself into thinking otherwise. There’s certainly scope for using C# and CLI in cross-platform scenarios (which was the safest course even before this announcement, since those were the only standardised parts), but on the basis of both legality and conformance I’d say it’s best to stay away from any .Net framework components if you intend to be cross-platform.

Maybe MS will extend this promise to the older .Net framework versions later on; that would seem a smart move, get people on the ladder but keep all the new stuff to yourself?

I can’t remember who made the assertion / joke that if you looked through an infinitely powerful telescope you’d end up seeing the back of of your own head, but I was reminded of that by a certain event today. In the last couple of years I’ve often Googled for a particular subject and ended up with the top hits pointing me back at one of my own posts in the OGRE Forum or on my blog, in a weird self-citing manner. In the worst cases, these posts answer or clarify my own current question, because a thought process I’d had a few years before, and then forgotten about, can often be useful. It’s like having a stack of your old notebooks in the cloud! Or an archived clone of yourself pointing out how age is making you stupid.

The other weird experience is when you download or otherwise get hold of a piece of software, and unexpectedly find that it uses your own code. I’m sure this is common in open source circles, because users of open source don’t have to tell you when they use your code, but nevertheless it’s still an odd experience. It’s especially nice when you like that piece of software – this happened to me today with PixPlant 2.

As I mentioned yesterday, I was reviewing tools for normal/displacement/specular map generation from reference sources, and I’d been evaluating CrazyBump and ShaderMap Pro. Evak in the OGRE Forums suggested I try PixPlant2 because he liked it. So I did, and I was impressed – the texture generation seemed as good as CrazyBump, but it’s cheaper ($175 rather than $299), and it also includes tools for creating tileable textures from original sources, detecting repeating patterns, straightening things, and blending the edges for you.

So, I was already leaning towards this purchasing PixPlant2, but then as I was browsing for textures, I noticed that the PixPlant2 application folder had some familiar files in it – such as OgreMain.dll, and rather familiar material files in the media folders! Checking the docs, sure enough OGRE was credited as a dependency. The application I ended up gravitating towards included software that I wrote!

To cap it all off, they’ve been very nice and offered me a free copy, so my normal map generation needs are entirely satisfied, for far less than I was expecting to pay. It’s not often things work out quite so well!

I reported a few months ago on how pleased I was that Qt was changing license to the LGPL, something I saw as a watershed for Qt adoption. I already had an awful lot of respect for Qt, but the previous GPL/commercial license did mean that adoption was in two quite widely separated camps – those who were already making GPL software, and those that could afford to license it for other cases. Great though Qt is, the price of the commercial license is really quite steep ($3,695 per-developer, per-platform), and that was hard to justify for a small developer.

So, the LGPL move is a big change and opens up a huge opportunity for Qt to be adopted in the ‘middle ground’. But beyond just adoption, Qt are clearly pushing for much higher levels of community participation going forward. A couple of weeks ago I got a call from someone in their strategy group, who is looking to reach out and build relationships with other open source groups & companies in related fields, to see how we might help each other in the future, and that sounded really promising. Today I received an email from the Qt mailing list outlining how they’re structuring their new public code repositories, and how they’ll be accepting external contributions. It all looks well thought out – they’re using Git to make patch submission easier on a large scale for example (something I’ve considered in the past, it’s only unfamiliarity and the lack of good Git UIs that puts me off doing it).

It’s great to see big companies making such solid moves into true open source participation like this, as opposed to some other companies which use open source licenses, but still operate a 100% ‘push’ model of development. Ultimately I’m sure it will pay huge dividends both for them and for the Qt user community. Obviously before the Nokia aquisition, Qt had to generate all its revenue from licensing so it had less scope or impetus to explore openness like this, but still I think there’s scope for many organisations to explore this approach with some aspects of their software products; specifically those parts that underpin the ‘premium’ side of its business or provide other infrastructural elements on which it builds its higher-margin offerings. As more companies explore these kinds of opportunities to blend open source and commerce, both as providers and consumers (and open source offers benefits on both ends of the relationship), I think we’ll see an acceleration of iterative innovation in the industry.

In a past working life, I used Oracle a fair amount – I used Oracle 7 through 10, and they were pretty decent products. The lineup was pretty simple back then – Oracle was the gruff, stoic mercenary who didn’t talk much and cost a fortune, but had it where it counted – if you could get him to do what you wanted; SQL Server was the approachable and gregarious rogue who was a jack of all trades and came fairly cheap, but had a habit of disappearing into the shadows or asking for more money at more sticky moments; and MySQL was the happy-go-lucky bard who was just along for the ride, happy to work for free so long as it was all just a jape and no-one asked him to do any real work.

How things have changed; SQL Server has got more mature (and more expensive), Oracle has bristled with ever more confusing add-on components while the core has got cheaper, MySQL has become a much more serious contender for many businesses and has already been swallowed once recently by Sun for an insane $1bn. However, that’s all going to change again now that Oracle is buying Sun, and thus with it, MySQL. So what does this mean for MySQL?

A lot of people are saying it’ll be curtains, but I’m not so sure. Oracle has already chowed down on several other open source vendors in this space, and perhaps surprisingly not much has changed. In 2005, they bought Innobase, a Finnish company that produced the transactional back-end for MySQL, InnoDB. So essentially from 2005 Oracle controlled the most important part of MySQL anyway, certainly from the perspective of increasing its business use. And yet, really not much happened, except for some rumblings in the community and some uncertainty around MySQL 5 (which was no doubt Oracle’s intention). Then in 2006, Oracle bought Sleepycat, which produced Berkeley DB, an open source embedded database. Again, this continued pretty much unchanged afterwards. So, what will they do with MySQL now?

I’m not even sure it matters. Because the reason that Oracle’s purchase of InnoDB and Berkley DB were effectively a non-event for users is that they were both open source. No matter what Oracle did, it couldn’t change that – if they try to change the license from future versions, a fork will just appear instead and people will move. The key people involved in the project would just leave and work for whoever ends up running the fork (probably a startup) – after all, most of the time these people were in the startups that created it. There is actually not that much ‘control’ at all that you gain from purchasing an open source project like this – you get the copyright, so that means you’re the only one who can change the license for future versions, but the open source license can never be revoked on existing versions. You might own the rights, but you don’t own the customers.

So really, it makes very little sense for Oracle to try to ‘kill off’ MySQL, or to cripple it somehow. With Inno, the one thing they had in their back pocket was the Hot Backup, which was a closed part of the code, but because it’s not the majority of the product there’s nothing to stop someone else developing an equivalent – most of the time the only reason people don’t is that while the company plays fair, there’s no need to. If it’s a higher-end add-on, people tend to accept that the originator company can sell a minority of their product as an add-on, it’s “fair” given all the code they’re giving away as open source. But, if the company acquiring them then tries to exploit that, say by making it prohibitively expensive or withdrawing it completely in order to try to make the core open source product less attractive (maybe in favour of their own proprietary product), then you can guarantee others will enter the space to resolve the issue.

It’s a perfect example of why open source is a hugely valuable insurance policy to anyone using it. Even if mergers and acquisitions change the priorities of those who control the code, the kinds of forced switching and upselling that typically occurs to customers in the proprietary space (I’ve had this happen to me several times) in the wake of such M&A activity just doesn’t generally happen so much with open source products – because if vendors inconvenience their customers, they really do have the viable option to go elsewhere. As it should be!

I enjoyed reading this post at l2admin, celebrating some of the big names in open source development. Of course, we can all argue about names which didn’t make this particular list (personally I think Larry Wall and Guido van Rossum are just two of the important omissions), but what strikes me most – well, except that Mark Shuttleworth is younger than me, which is slightly dispiriting – is how globally representative the list is.

Traditionally the commercial software industry has been mainly associated with the USA, and while of course all those corporations are bursting with international employees on H-1Bs, and have international offices scattered all over, nevertheless the impression that you still get is that the USA drives most of the industry, with a few exceptions. This is of course one of the reasons behind initiatives by other countries to promote their local industries instead, such as China’s Red Flag Linux and the European research funding programme – with mixed results. It’s nice, therefore, to see the range of nationalities called out in lists like this, which reflects the meritocratic and perhaps academic style of the open source field in general.

Of course, a large number of the successful open sourcers end up in the USA eventually anyway, because by and large that’s where the money is – which is what biases where most software originates in the commercial world too of course. Still, I take heart at the diversity in this list of luminaries, which definitely reflects the diversity I see in my own open source community. Believe it or not, great stuff does get done outside of the Valley sometimes

I’ve always been a fan of staying flexible as regards to platform, but it’s especially true these days, since my desktop environment is heterogeneous – I still tend to use Windows most for work, but for personal use I’m most comfy in Mac OS X now. I do have Ubuntu around too although I generally only use it when I have to on the desktop (although I love it to death as a server OS since it takes everything that is great about Debian and updates it a bit). Therefore, when I’m seeking out desktop applications they either need to be cross-platform in themselves, or if the applications are platform-specific – which is ok too, so long as each platform has a decent option available – they have to be able to share data effectively.

Here’s a couple of apps that are nothing new, but which I only started using recently and have been quite positive about.

TrueCrypt

Keeping your personal data encrypted is fairly important these days, particularly for portable devices, but I also have information that I don’t want to lose (passwords etc) but wouldn’t want to be stored in plain text. Operating systems are increasingly implementing this, and both Vista and Mac OS X have encryption features built-in. The problem is that neither of them inter-operate with each other, so if you need to access the encrypted data on the other platform you’re knackered. Previously, I would use the Mac OS X encrypted drive image on the laptop so I could take copies of company documents etc around on it when I was travelling without worrying too much (these didn’t need to be accessed on Windows), and would use GPG for things that needed to be cross-platform, but it got a little bit awkward (keeping keys in sync, finding productive shortcut interfaces on both platforms). The reason I looked at this area again is that I went to a presentation on Windows 7 last week that mentioned the encryption enhancements it has (BitLocker), but that the encryption was still platform specific like Vista, so no good for sharing USB drives. It prompted me to take another look around.

Enter TrueCrypt – it has full support for Windows, Mac OS X and Linux, encrypted images can be mounted equally on all platforms regardless of where they were created, and the user interface is quick, easy and consistent on all platforms.  As an added bonus, because it’s open source you can almost guarantee it doesn’t have any back doors in it, which is always a concern when dealing with proprietary security utilities. It’s not perfect – I couldn’t for example figure out how to create a ‘sparse’ image, ie one that expands depending on how much data is in it, so I think that feature isn’t available yet, you just have to create an image that’s as large as you’ll need.

So far, it seems much better than the platform-specific alternatives for people that frequently hop platforms, and is definitely friendlier than using GPG.

Chandler

I was looking for a better way to track my TODO list, one that could sync across the Internet but wasn’t just web-based. Evernote was an option, but it was more than I needed and I always like to give open source a chance, so I tried Chandler, which has Windows and Mac clients and we synchronisable web version too.

It doesn’t quite do everything I want – it’s a little too obsessed with the ‘triage’ concept and doesn’t support dependencies between tasks, but as a place to dump things from my TODO and categorise, vaguely prioritise, and synchronise them, it works pretty well. It’s fast and unobtrusive at least, feels the same on all platforms, and I guess the lack of complications means it’s less likely to be distracting and become a task all in its own right to manage. I could still wish for a little more granularity, and definitely a little better documentation, but on reflection, it does the job pretty darn well.

Ideally I’d like a cross-platform version of something like Midnight Inbox. Why must some developers of nice programs focus only on one platform?

I’m on the Qt (owned by Nokia now) mailing list since I have a commercial license for a client project, and I got a very interesting email today, telling me that on its release in March 2009, Qt 4.5 will be available under the LGPL.

This is really big news. Up until the current Qt 4.4, your only licensing options are a per-seat and per-platform commercial license (which adds up if you have multiple developers and multiple target platforms, which you will do if you’re using Qt anyway), or alternatively the free option which means you use it under the GPL – meaning all your own code has to also be GPL, with an exception that allows you to publish / use software under other open source licenses too, but nevertheless it all has to be public. There’s a pretty big chasm in between these two options that in my experience a lot of projects fell into.

So, offering an LGPL option from Qt 4.5 is a major development. Qt is a really excellent library, the best cross-platform, native code widget set I’ve seen, but the licensing cost often put people off; this changes everything. I’m actually quite surprised they’re going this far – I think just changing the commercial licensing structure to be a little less multiplicative would have made a lot of people happy, but the LGPL option really turns things upside down. Essentially it means that you get all the cost benefits of the GPL version, except there’s a ‘fire break’ of open source responsibility between Qt and your application, meaning a lot more people can consider it. Nokia’s stated aims are:

• facilitate wider adoption of Qt across industries, desktop, web and embedded platforms
• establish Qt as a de facto standard for application development
• receive more valuable feedback and increased user contributions to ensure that Qt remains the best-in-class, cross-platform framework
• extend Nokia’s existing platform commitment to the open source community

I’m pretty sure this move will achieve all of those goals. It may well take a little wind out of the sails of wxWidgets, which in my experience never seemed quite as polished as Qt, but the licensing was much more favourable – although of course the wxWidgets project is still a lot less ‘corporate’, so will continue to have an appeal from feeling a bit more open.

Interesting times anyway!

Note: I’m going to pick the way I discuss this carefully, since I have a good friend on the LINQ to SQL team (yes, we Guernseymen do get around) and I feel bad to criticise too much in this area; nevertheless I think there are lessons to be learned and I have a definite angle on this, being an ex-business coder and open source enthusiast. My thoughts here reflect pretty much what I’ve already suggested on his blog, but in more detail, so hopefully this won’t offend him!

LINQ was a new feature introduced with Visual Studio 2008, and LINQ to SQL is the lightweight SQL Server implementation. News broke recently that Microsoft have decided to concentrate on the larger and more ‘complete’ Entity Framework instead, and encourage people to move to that for their more advanced needs rather than continuing to expand L2S. They’re not dropping L2S, it’s essentially being ring-fenced on a feature basis and will continue to be supported by the team. However, many people are reading between the lines and assuming that, in practice, LINQ to SQL is now a dead-end, and many of them are very upset about that, having invested development time in adopting it, and who were expecting it to continue evolving.

I can see both sides of this – MS need to have a strategy, they have finite resources, and they feel focussing on EF is the best way forward. However if you’ve adopted technology, and invested your own time in it, you want that investment to be strategically valid. While L2S isn’t being dropped, it is unlikely to escape the perception of being ‘on the bone pile’ if the strategy is to expand & promote EF as the preferred solution in future. When updates to a technology you’ve invested in are wholely controlled by one company, and that company decides it no longer wants to make it the core of their strategy, you basically have to suck it up and accept that. The problem here is that producers and consumers of the technology don’t necessarily agree on the best way forward, so bad feeling is the result.

It really doesn’t have to be this way though. What if Microsoft, on deciding that they wanted to focus on EF, released L2SQL as open source instead? Maybe it isn’t strategically core for them anymore, but those who have invested heavily in it already are bound to feel differently. Popular technologies (and I would venture that L2SQL is probably very popular due to its power & simplicity) tend to foster their own communities, and even if only 1% of the developers using it would actually become contributors to it, that’s still probably more people that Microsoft would want to dedicate to the effort from an internal team over the long term.

I’m not even sure what MS would lose from doing that – the technology is free already, so it’s not a revenue generator for them, and it would do wonders for community relations. What they would lose is control, which is perhaps what they’re afraid of – maybe that they’d have to compete with an open source L2SQL – but competition is good for customers. I get the impression from my MS contacts that they feel it’s not viable, that people wouldn’t want it to be open source, that they like the simple, spoon-fed, ‘MS knows best’ approach – but personally I’d say the existence of the Mono project is an indication that this isn’t doing the community justice. I think this poor view of the community model is formed because the open source communities around extending MS products tend to be much, much smaller than those elsewhere (like Java, Linux, Apache etc), but I’d counter that that’s precisely because of the tone that MS sets; ie that you dance to their tune. Environments that are open to more extensive external involvement tend to attract more active contributors, so saying there wouldn’t be enough people when you’re operating in a closed way is actually a self-fulfilling argument.

In the days of the ‘new open Microsoft’, I can’t see a downside to them open sourcing key parts of their framework in practice. Developers these days are a lot more savvy than 5-10 years ago about how they spend their time, and open source is winning a lot of favour not because it’s free, and not always because it’s cross-platform, but because it’s open, and cannot be taken away from them. A number of my customers use Ogre specifically because they’ve been with proprietary systems in the past, which have fallen victim to the companies controlling them deciding they didn’t want to support them anymore, or didn’t want to take them in the direction the customer wanted, or wanted to force them to move to something else when they didn’t want to – and they found themselves restricted / railroaded by a system they’d invested a lot of their own development money in.

Open source for these people means ‘taking the chains off’ – they know that no matter what, there’s no one company that can tell them what to do with their own investment. They’ll stay on the main track while it’s beneficial to them, they can benefit from the core development just like proprietary software customers can, but if at any point they disagree with the ‘central’ decisions, they can do what the like. That’s a very powerful insurance policy in development circles, something I hear time and again from customers as a major positive, and personally I think Microsoft needs to think about that more. They are very much still in the ‘producer/consumer’ mindset (or ‘cathedral’ if you will), but the world around them is rapidly changing to a model of iteration, collaboration, and distributed control. Fewer and fewer customers like to be dictated to by a central oligarchy anymore, and many are going to places they feel more empowered – which is usually the open source camp. Microsoft simply cannot afford to ignore this for much longer.