SteveStreeting.com

Having already disrespected mailing lists, I might as well get all my ranting about old staple communication techniques out of my system, by admitting that I’ve never really liked IRC.

There’s nothing wrong with it per se, particularly as a casual social tool, but I just can’t say I’ve ever received any great value from it in a project sense, primarily because of it’s real-time and unfocussed nature. As a user of a project, I’ve frequently found that the people that are able to answer my questions are not online at the same time as I am. Secondly, even when those people are online, they tend to get mobbed by everyone, and anything more than one or two active discussions turn the channel quickly to a confusing mess. As a project lead, I always dreaded going on IRC precisely because of this “mobbing effect”; the usual outcome was for me to lose a couple of hours answering a ton of questions – which was not an unpleasant activity, it’s nice to talk to your users, but at the same time it’s a terrible time-sink, and unlike some people I’m incapable of multitasking real-time discussions with coding, at least not on anything remotely complex. As such, my IRC attendance slowly dropped off and I now rarely go on any more; I felt a bit guilty about that, but figured the community would rather I got more done than spend time talking.

I realised recently that Twitter has now settled into my life as a more effective replacement for the times when I might have previously found IRC somewhat useful, despite the noise. It’s as close to real-time as matters, but at the same time it’s not a chat system, which for me is a good thing, since it sidesteps by design the major downsides of an open chat system – the tendency for real-time discussions to ramble on, and the implied expectation of a real-time response. You often get that, of course, but there’s no perception that it’s an affront if there’s a delay, even of many hours. As a system that needs to sit alongside ‘real’ work, it’s a lot more practical in its utility. Also, as primarily a ‘pull system’ (you choose to follow people), the signal-to-noise ratio is far higher. People can reply to your posts, and you can reply to theirs, so the same kinds of discussions as IRC tend to spring up, but they tend to be more useful, because they’re among peers more often than IRC was. Sure, other people can @user you in an unsolicited fashion too, unconnected to your feed, but that’s generally considered impolite so it’s rare. There are also no ‘channels’ so I don’t have to be watching many places depending on the subject, channels simply form naturally based on individuals and subject tags. Finally,the 140 character limit does tend to waste less time for the reader – although for the writer time can sometimes be lost trying to shoehorn a coherent point into that space.

As a result, I find I have all of the benefits of IRC (in a project rather than casual social sense), with few or none of the downsides. I have many semi-real time, compact and most importantly useful exchanges with people on the service, all in a very convenient package (after trying a few clients, I settled on TweetDeck to organise things).

This might come across as me wanting to wall myself off from the ‘n00bs’ in my community. That’s not true, it’s just that time is my most valuable asset, and it’s finite; crushingly so. I’m happy to answer questions on the forum – where I can dedicate a known amount of time and tackle as much as possible, regardless of whether the person is currently online or not, and Twitter fills in the more social & real-time aspects without being a burden. IRC by contrast is high maintenance and extremely wasteful with time for the same purpose, and I just can’t justify it.

So farewell IRC,  I really won’t miss you very much.

I’m not blogging as often these days; as you know I don’t traditionally ‘do’ short blog posts – in my book if something is worth blogging about, it’s worth making sure it holds together as an argument, and as a piece of writing generally – and a combined lack of time of anything I’m motivated (or permitted) to talk about has left the site a little  bereft of content. Luckily my OGRE Twitter is stocked with more frequent and less lovingly crafted status updates on what I’m doing there.

So, on to the title of the post. The Internet has been around for a while now, and has evolved rapidly, particularly in the last decade. And yet, particularly in academic and some open source developer circles, there is an attachment to a particularly creaky piece of technology that I can honestly say I do not share - the venerable mailing list.

Now, to clarify the context, I’m referring to the use of mailing lists for multilateral communication for an entire community, including newcomers, as opposed to a simple 1-way notification list (like we use for commit notifications for example). For N-way communication among a small group of core developers, all of whom will want to read every post, I can see the utility and convenience of a mailing list. But as a community communication channel, where people just want to drop in and drop out, I find it a staggeringly inefficient, awkward and archaic approach. I say this primarily as an occasional community member of various projects that use mailing lists, and therefore someone who has a specific interest in a mere subset of the discussions that go on – I have no time or desire to read every single thread, and indeed if I tried to do this for every project I have an interest in, I’d never get anything done. It’s hard enough to keep up with my own open source community!

The simple fact is that mailing lists have an all-or-nothing mindset that is woefully outdated for community interaction on the scale that the Internet has now grown to.  Subscribing means you get bombarded with every single discussion, either individually or in digests, which pretend to be useful but in fact aren’t, because while they cut down on the number of emails you get, it makes replying to specific posts a pain. If you want to read every single mail in the list, I’m sure they work fine – but most people outside the core group do not want to do this. Most members of the community just want to keep a closer eye on a few select threads of discussion that either affect or interest them, and to be able to search and browse through the rest easily – and the mailing list is a woefully inadequate, blunt instrument for this kind of task.

Sure, you can choose not to subscribe, and go through the archives, searching or browsing them. But you can do that with forums too, and there at least you have the advantage of categorised areas of interest, being able to follow certain people, and to watch certain threads. Mailing list archives have a single filter: date, and also lag by a number of hours dependent on the individual setup, so if you’re not subscribed, you get a lesser service.  Another technique is to subscribe completely but tell your email client to archive or filter things for you, so you can dip into your local replica at leisure. Horribly, horribly inefficient, but it does work.

Mailing lists worked in the 90’s when there were small groups of people who wanted to read everything being discussed, and when email was the primary form of communication between people. We’ve moved on. Forum systems and other flexible hosted systems are far superior in their ability to let you watch particular discussions (or all new posts) that you’re interested in and get told when there’s an update. Anyone can search them easily (internally or via Google) and there’s no archive lag. Maybe some people are worried about forum databases being lost, compared to inherently replicated mailing lists, but anyone worth their salt has a server backup strategy.  Honestly, any project that uses mailing lists as their only community discussion channel instantly puts me off getting involved in that community, because I know that as an occasional participant interested in only certain discussions, the experience is going to totally suck.

And, if you insist on loving your mailing lists approach so much, for goodness sake move to Google Groups. They’re still pretty basic, but at least there, those of us who have moved into the browser world can use an interface we find useful and productive, rather than being forced to use 20-30 year old technology designed to replicate posts around a university science department.

It’s now almost a year since I decided to try using Twitter, specifically to post about Ogre development work I’m doing and other Ogre-related things (well, most of the time anyway). Seeing as I totally deride the concept that it’s a good thing to share the inconsequential, tedious minutae of your life with the internet and view it as the absolute pinnacle of sad, narcissistic behaviour, joining Twitter was a hard sell. After all, at least on a blog you have to write enough in a post to naturally filter out anything that’s not worth saying (in theory), while Twitter seemingly encouraged you to share whatever crossed your mind during the day. In the end my reason for joining was that there tended to be things large and small that happened in and around Ogre that many people might like to know about, and these things didn’t always warrant a blog post,  a news article on ogre3d.org or even a forum post. Provided I stuck to that raison d’etre, perhaps it could have value.

And in fact, it’s actually been very useful. I’ve almost stopped blogging about Ogre work unless there’s a significant event or something I feel needs greater analysis, because my Twitter feed is a better way to get the word out about things. It’s also been useful to get feedback on certain technical issues and to keep up to date with what other people are doing. Specifically, I tend to only follow people who post about things I’m interested in, rather than just because I know them.

And this tends to work well – I’ve found that Twitter users, or at least the ones I follow, in general tend to automatically filter their content to things that are actually interesting. This is in contrast to Facebook, which is so chock full of the utterly banal that I lose the will to live every time I try to catch up with the feed – there are usually some things in there I’d genuinely be interested in, but it’s so full of crap I can hardly face spending the time to find it. Much of that is due to its insistence that I’m somehow interested in the events of all the Facebook games people are playing, when in fact I couldn’t give a flying toss what new fish someone has just unlocked in some ridiculous mini-game. I’m close to just deleting my account and forgetting all about it – if you want to be social, grab a coffee / drink with me sometime or something – at least then you’re unlikely to keep interrupting to tell me what your level is in FarmVille.

Computer systems are tools, and can be used for good or ill. I’ve come across lots of people that use Twitter in a genuinely useful and non-intrusive way, and I try to do the same, and as such it’s made a firm place for itself in my day – something I would not have taken for granted when I started using it.

I have to hand it to the guys at the Clan of the Gray Wolf, who are doing a 1,000 song Rock Band marathon for the charity Childs Play, all streamed live on the Internet. Presumably this is linked with the fact that Rock Band itself recently crested the 2009 target of having 1000 in-game tracks – and a month earlier than their deadline.

.

At the time of writing they’re 46 hours in which given that they’ve tackled 615 songs so far, represents not quite two thirds of the way. Even though there’s 6 of them taking shifts (3 playing, one commenting), this is still an ambitious thing to be trying – we haven’t even tackled the 80-odd song Endless Setlist 2 in Rock Band 2 yet, and probably never will! I can’t imagine how bad their blisters are going to be after this, not to mention their vision – my guess is that the whole world is going to look like it’s scrolling upwards for them in the next week.

Anyway, much respect – I encourage you to donate if like me you respect this kind of crazy endeavour which is nevertheless brimming with geek cool.

My broadband connection was on the blink this morning, which affected me less than it would usually would have because I had a dentist appointment, so I didn’t think too much of it. I heard on the radio when driving to said appointment that the whole island was affected so that made me feel a little better, and everything came back about an hour after I returned.

However a friend of mine works at one of the local telecoms companies (and which is also the broadband wholesaler to the others – kind of like our local version of BT) phoned me at lunchtime to ask if my connection was back, since he hadn’t seen me on Skype (I’d actually just forgotten to turn it back on). He informed me that the reason for the technical difficulties was a massive spike in internet traffic caused by everyone playing Call of Duty : Modern Warfare 2

Yes, it seems lots of people decided to take the day off today to play the new release; even in our small juristiction over 100 people queued outside the local HMV at midnight to get their copy early, so it seems that many more people bought it at 9am and went home to fire it up. Given that the game had a day 1 patch (at least for PS3 for a trophy bug, I don’t know about 360) I presume that plus lots of people jumping online all at once was a bit much for the system to handle.

It appears I’m the only person in the world who isn’t obsessed with COD:MW2

There’s one problem with having a relatively public presence online, even in such a niche that I’m in, and that’s how to deal with unsolicited friend requests. I’m a happy user of LinkedIn, I have a Gamertag you can see on the right hand side there, and I’m also a reluctant and infrequent member of Facebook. As well as letting people who know me connect, it also means that on some occasions, I get friend / contact requests on these systems from people I’ve never heard of before. In the case of a Gamertag in particular, I’ll sit there trying to figure out whether I know the person or not, then agonise about whether I should just reject it (causing potential offense), initiate a ‘Who the hell are you’ discussion which might end up that way anyway, or worse open a dialog with someone I wish I hadn’t, or just accept on systems I don’t care so much about because it’s less hassle.  I’ve gravitated towards ‘reject’ as a default response now.

Here’s a tip: if you want to connect with me or anyone else on an online system, at the very least email me to introduce yourself first, or include in your friend request why we might know each other already, or have something in common. I’m fully aware that I ‘know’ far more people than I realise half the time, since I talk to people every day in forums etc via the obscurity of nicks. And if we haven’t spoken before, that might be ok too; I like ‘meeting’ new people, but you really have to talk to me first.

Just firing a friend request at me with no introduction is akin to throwing your business cards at random strangers in the street; it’s just not polite. Going through introductions might seem old fashioned in today’s fast-moving world, but it really does break the ice and makes the difference between a genuine friend request and a ’stalking request’.

Ok, so I discovered a number of shortcomings in my recent attempt to sync a folder in one direction to Amazon S3 using encryption, the most important of which was that it wouldn’t resume a failed transfer efficiently, which in the case of large transfers wasn’t at all ideal (as I learned to be own cost – damn my 256k upload speed).

So, this is attempt number 2. I decided to completely rewrite the script in Python instead to give me some more flexibility, coupled with the availability of Boto, a nice Python library for accessing all the Amazon Web Services. Rather than rely on just local information, or even date/time stamps, I decided to use hashes to track whether files were different. Amazon already stores the MD5 of the file you upload to them and makes that available without downloading the file, but that’s no use when you encrypt your files before uploading them, because the MD5 is of the encrypted contents rather then the original; so unless you keep the encrypted copies around too, or encrypt the local files again every time just to check the match (expensive if you’re dealing with large files) you won’t be able to compare them – I think this is the reason why ’s3cmd sync’ currently doesn’t support encryption.

So, I decided to use S3’s ability to store custom metadata in keys, and stored the MD5 hash of the original file against the encrypted contents that I uploaded. That way, I can check the hashes against each other pretty quickly without having to re-encrypt the local files. If the hashes are different, I encrypt and upload. This approach trades a bit of preprocessing against avoiding uploads, so it’s likely to be more efficient on small groups of very large files rather than lots of small files – that’s how I use S3 for my backups of course. It also means I don’t have to worry about timestamp variations, it’s the content of the file that is the driver of whether it’s uploaded or not.

So, here’s the new version. It’s a bit more powerful than the last one – I’m calling gpg myself now so you have the choice between encrypting using public keys (more secure, and the default), or using symmetric encryption with a passphrase. You need to install Boto before you can run it, and it depends on Python 2.5 with hashlib installed. I’ve run it on both Linux and the Mac, it should work on Windows too provided you take the trouble to set up Python and GnuPG, but I haven’t tried; my Linux (apt) and OS X (macports) setups make these things quicker so being short of time I just went with that. Here’s the usage from –help:

Usage: s3putsecurefolder.py [options] source_folder target_bucket gpg_recipient_or_phrase

Options:
  -h, --help            show this help message and exit
  -n, --dry-run         Do not upload any files, just list actions
  -a ACCESS_KEY, --accesskey=ACCESS_KEY
                        AWS access key to use instead of relying on
                        environment variable AWS_ACCESS_KEY
  -s SECRET_KEY, --secretkey=SECRET_KEY
                        AWS secret key to use instead of relying on
                        environment variable AWS_ACCESS_KEY
  -c, --create          Create bucket if it does not already exist
  -v, --verbose         Verbose output
  -S, --symmetric       Instead of encrypting with a public key, encrypts
                        files using a symmetric cypher and the passphrase
                        given on the command-line.

Once again, no warranty is given, MIT license. If you see that I’ve done anything dumb, let me know

Oh, please let this come to pass soon. TechCrunch reports that YouTube is due to drop support for IE6 ’soon’, pointing users at Chrome (obviously), IE8 and Firefox 3.5. Finally, one of the worst pieces of software ever to pollute the Internet with its presence is getting taken out to the barn with a double barelled shotgun, and not a moment too soon.

Sure, Digg already said they might do this, but YouTube is far more significant; if YouTube stop supporting IE6, then in practice it means I can too Bye bye IE6, please do let the door hit your ass on the way out, preferably hard enough to fracture your pelvis. It’s the least you deserve for making life hell for hundreds of thousands of web site maintainers over the last few years.

Oh, this is so ripe for satire I really can’t believe Microsoft didn’t see it coming. Or, perhaps they did and just ran with it anyway, for funsies. It appears Microsoft’s Australian website is encouraging people to switch to IE8 by offering an online treasure hunt, where a series of clues will lead you to a site identifying the location of the $10k (AU$ presumably), which can only be viewed with IE8. They gleefully point out:

“But you’ll never find it with old Firefox. So get rid of it, or get lost.”

So, let’s stack up the issues here:

• Microsoft has resorted to offering a monetary incentive to encourage people to use its free browser. Is that an admission that based on just the merits of the product itself, IE8 probably wouldn’t be the user’s first choice? I’d guess that people who actually choose their browser (rather than accepting what they get preinstalled) are not that likely to pick IE8.
• ‘old’ Firefox? Last I checked, IE predated Firefox by some years, and the latter has a new version coming out in mere weeks. Resorting to empty name-calling now? Dear me.
• Websites that only work in IE? Wow, welcome back to 1999 guys. ActiveX, Outlook Express bindings – ah, the memories. The horrible, eye-watering memories.

A Mozilla dev has already fired back a response, but really I don’t think it needed one. I think the fact that this promotion exists at all, and the tone which it takes, speaks volumes about how much the browser landscape has changed in recent years.

I’ve harped on many times about how I think centrally controlled services like Facebook are the antithesis of what the Internet was supposed to be about – a distributed, decentralised place with authority controlled at the leaves by those with most interest in maintaining it, rather than some corporate hub holding all the cards.

Well, it seems like a small bunch of companies are starting to latch on to this idea too, a welcome respite from the huge number of ventures that just want to be the new singular nexus of your internet life. Google Wave certainly ‘gets it’, if the reality reflects the stated vision where the open-source software can be run anywhere, not just on Google’s servers. And Opera Unite is making the right kind of noises for me too, even if right now the service is embryonic.

In essence, it’s a semantically richer, more secure version of BitTorrent – the ability to share files, photos and media within interfaces dedicated to that purpose, serve web pages, and chat, but by making direct connections with your peers rather than going through a centralised hosting service. Opera Unite provides the software to perform the hosting from your own devices, and provides the discovery and network trust systems to allow people to hook up.

There are lots of issues with this approach of course – such as whether you trust the hosting software not to punch holes in your local security, whether you really want to have the bandwidth issues of self-hosting, what happens when your machine is off, etc. Right now, I don’t think it’s that workable as a replacement for centralised systems, but that’s not the point – the point is that the principle of entrusting all your unencrypted data to a single online entity is eventually not going to be good enough anymore, and we need to be developing alternative approaches. If the future is truly in the cloud, we need far more than what the cloud offers right now – which is to say services that while user-friendly, require you to give up far more control over your data than is feasible for anything remotely important. Sure, you’re happy to put photos on Facebook, and Twitter about all those things that you don’t mind the world knowing, but that’s a very specific, non-critical subset of the data we all increasingly need to hold. Would you be happy to scan your bank statements and put them on Facebook, even if you set them to private? Of course not – but if the cloud is to realise its potential, these are the kinds of harder applications we need to try to address.

I’m not saying Opera Unite addresses that – not even close. But the fact that people are exploring alternative approaches to the 100% centralised model is a positive sign to me. We need to start tackling how we use entirely public transport & repository systems (ie the cloud) to securely store and exchange important and sensitive data, and I say that’s impossible to address with an entirely centralisd model, because a centralised model focusses control in too few hands. Encryption gives us the ability to store and transport secure information in plain sight, but it’s traditionally a very tricky thing to make easy to use for the general public, particularly when multiple parties and ‘controlled’ sharing is required. Thus, one approach is to focus on securing the transport instead (which is easier, and why SSL is ubiquitous) and lock down access to the leaves more tightly. Opera Unite is an experiment in the leaf model and may well inform the process, leading to more innovation in this area down the road.