SteveStreeting.com

I’ve harped on many times about how I think centrally controlled services like Facebook are the antithesis of what the Internet was supposed to be about – a distributed, decentralised place with authority controlled at the leaves by those with most interest in maintaining it, rather than some corporate hub holding all the cards.

Well, it seems like a small bunch of companies are starting to latch on to this idea too, a welcome respite from the huge number of ventures that just want to be the new singular nexus of your internet life. Google Wave certainly ‘gets it’, if the reality reflects the stated vision where the open-source software can be run anywhere, not just on Google’s servers. And Opera Unite is making the right kind of noises for me too, even if right now the service is embryonic.

In essence, it’s a semantically richer, more secure version of BitTorrent – the ability to share files, photos and media within interfaces dedicated to that purpose, serve web pages, and chat, but by making direct connections with your peers rather than going through a centralised hosting service. Opera Unite provides the software to perform the hosting from your own devices, and provides the discovery and network trust systems to allow people to hook up.

There are lots of issues with this approach of course – such as whether you trust the hosting software not to punch holes in your local security, whether you really want to have the bandwidth issues of self-hosting, what happens when your machine is off, etc. Right now, I don’t think it’s that workable as a replacement for centralised systems, but that’s not the point – the point is that the principle of entrusting all your unencrypted data to a single online entity is eventually not going to be good enough anymore, and we need to be developing alternative approaches. If the future is truly in the cloud, we need far more than what the cloud offers right now – which is to say services that while user-friendly, require you to give up far more control over your data than is feasible for anything remotely important. Sure, you’re happy to put photos on Facebook, and Twitter about all those things that you don’t mind the world knowing, but that’s a very specific, non-critical subset of the data we all increasingly need to hold. Would you be happy to scan your bank statements and put them on Facebook, even if you set them to private? Of course not – but if the cloud is to realise its potential, these are the kinds of harder applications we need to try to address.

I’m not saying Opera Unite addresses that – not even close. But the fact that people are exploring alternative approaches to the 100% centralised model is a positive sign to me. We need to start tackling how we use entirely public transport & repository systems (ie the cloud) to securely store and exchange important and sensitive data, and I say that’s impossible to address with an entirely centralisd model, because a centralised model focusses control in too few hands. Encryption gives us the ability to store and transport secure information in plain sight, but it’s traditionally a very tricky thing to make easy to use for the general public, particularly when multiple parties and ‘controlled’ sharing is required. Thus, one approach is to focus on securing the transport instead (which is easier, and why SSL is ubiquitous) and lock down access to the leaves more tightly. Opera Unite is an experiment in the leaf model and may well inform the process, leading to more innovation in this area down the road.

Now that I’ve had my 360 fixed & returned, I’m being pestered with requests to fill in a survey about my experience. Ignored the first two, since I was neither ecstatic nor furious about my support experience, so it would make a particularly tedious ‘ok I guess’ response. But, they’re insistent with their damnable reminder emails, so I tried to do it.

I got right to the end screen, and then got this:

Microsoft OLE DB Provider for ODBC Drivers error ’80040e31′

[Microsoft][ODBC SQL Server Driver]Timeout expired

/CSatCalSrvy/LibProcess.inc, line 216

Heh. So, where do I fill in a survey about how badly the survey system worked?

[Edit] Ah well, 4th attempt worked at least. And to be fair, they farm this out to a 3rd party research firm anyway. But, still funny.

Many people have declared email to be dead in the past, and they’ve all been wrong. The typical play has been from instant messenger advocates, and most recently from Facebook. But, while these options have been a valid all-encompassing solution for teenagers and students, I haven’t met a single serious modern IT user whose life isn’t still driven primarily by email. There’s a reason that Outlook and Exchange are such consistent cash cows for Microsoft, and so many business people own Blackberrys. IM, Facebook & Twitter may represent certain facets of your online existence, but if push really came to shove, and you were only allowed to use one electronic service, I bet you every gadget I own that almost everyone will opt to keep their email over anything else.

I certainly could not operate without my email, but after watching the demo of Google Wave, I saw for the first time something that could genuinely be better, without leaving me with a gaping functionality hole in the name of ‘progress’. In retrospect, the idea seems incredibly obvious, but I’m sure the implementation was tricky.

In essence, Google Wave is basically a fusion of email and IM functionality. You still compose emails, reply to them, and include people in the threads, but the whole functionality set can also operate basically in real-time, just like an IM client. Whether something is instantly transmitted interactively depends on whether the person you’re sending it to is online, and some preferences of your own. There were some nice geeky demo things like instant translation via a bot too, but the most important thing to me was how holistic it was. One of the major problems I have is connected with using a mixture of IM and email communication, particularly with clients but also friends. I might remember that I had a conversation with X about Y, and want to go refresh my memory about it, but I don’t remember whether we talked about it on email, or on IM (and whether it was Skype, GTalk, MSN, and which machine I was on at the time). Looking up this information is a pain, because my email is one island of information, and my IM conversations are many separate islands. Being able to search across the whole thing in one swoop, from any PC/device, and see all the conversations, both deferred email-like and instant, all in their original threaded contexts, would be absolutely fantastic. It would give me value in my working life, instantly, right now.

They could go even further, by supporting VOIP calls through it too, and have the option to use voice recognition to transcribe it (or record it as well), or even just to log that the conversation happened and allow me to add a few manual notes to it. I’d imagine this would be early on the list of extensions, by Google themselves or by external developers, since they’re encouraging people to use the API. And I can imagine that linking all this up with Google Maps, Google Calendar, Google Docs etc will have a multiplicative effect.

So I’m pretty excited about Wave. It’s the first collaboration tool I’ve seen that could genuinely replace my email (and IM), although I’d then have to tackle the very real question of whether I really want to give Google control of such critical parts of my electronic life.

I, like many people, viewed the Bing marketing video last week, which promised not to create a search engine, but to create a ‘decision engine’ – if you winced at this blatant attempt at the ‘game changing switcheroo’, congratulations, you can join me on the ‘jaded technology observer’ bench. Despite my distaste at having to swim through the murky waters of marketing blurb in this video, the demonstration looked pretty nice – showing how the ‘decision engine’ picks out flight details, product reviews and other things out of your search terms and provides context-sensitive recategorisations such as price and specifications of digital cameras if that’s what you were searching for etc. I could definitely see a use for that sort of thing, provided the ‘decision’ part of the engine didn’t get too zealous, start finding unintended patterns, and obscuring the regular search results.

Now that I’ve had a few minutes to play with the real thing today, I needn’t have worried. Because it seems that despite what I type in to the search engine, it just gives me regular search results – even if I try to recreate some of the examples in the demo from last week, like searching for digital cameras. Is the ‘decision’ part of the engine just turned off right now? Because as it stands, what I see is a pretty competent Google rip-off, which technically is no mean feat, but the world doesn’t need another Google. We already have one.

Maybe someone will flip a switch and it’ll suddenly start doing the magical things the marketing video talked about.

Edit: ok, I see that the filtering and review features are in the Shopping section, but Google has these in it’s own Shopping section too. Someone tell me, exactly what is Bing doing that Google hasn’t been for the last several years? Am I just blind to the genius of the ‘decision engine’?

So, the intertubes are awash today with people venting their spleens about Twitter’s decision to stop sending replies by people you do follow, but to people you don’t follow, to your main Twitter feed. Previously you had the option either way, and now some people are getting their panties in a bunch about it.

There are two things to say about this issue:

1. Personally, I don’t want to see all the random replies to other people I don’t follow. I already deliberately only follow a small number of people, beacuse frankly I don’t have time to sift through a huge list of tweets every day. I have absolutely no idea how anyone copes with following more than about 10 people who tweet regularly, and still get something done in the day, nevermind seeing all the secondary replies. Am I just inefficient at processing large numbers of posts, or do I just have a staggeringly lower level of patience than the average Twitter user? The way I have things right now, I read every one of the posts from people I follow, because I consider them interesting, and that takes little time. I couldn’t do that if I was following 100 people and their replies to other contacts too, so I’d either have to lie (ie stick to etiquette and follow them, but then filter out most of what they say on the client), or just spend all day reading Twitter. So personally, this seems a sensible choice – you can always use the Twitter web if you really have nothing better to do but surf Twitter, or browse your friends ‘following’ list if you’re desperate to mine the system for new contacts.
2. Twitter is free. If you paid nothing for a service, you are entitled to offer your constructive feedback which the providers may choose to listen to, but you are not entitled to have a major tantrum about it. As Matt Asay suggests, if you care about the service that much, then you should probably be paying for it – and God knows, Twitter needs a business model other than the typical Web 2.0 “Attract viewers …….  profit!” fantasy right now. On the whole, the Internet needs a slap to wake up its users from the bloated sense of entitlement they’ve developed over the years, fueled by a huge number of startups that delude people into thinking they can expect everything for nothing. 100% free models don’t work (yes, I know, I’m an open source advocate, but that doesn’t mean I believe that you can give everything away) – they are a complimentary aspect, or a stop-gap until you can develop a real model or pursuade some sucker to acquire you before the hype train grinds to a halt. Eventually, these cycles of pretending that you can get premium service for free will end, and everyone will have to face up to the reality that ‘freeloaders’ have a place (building momentum, awareness etc), but ultimately they’re at the bottom of the food chain. Plankton are vital to the oceanic ecosystem, but no-one asks them for their opinion.

Keeping web software up to date is a pain, but failure to do it can result in significant ramifications. Some bits of software are easier to keep up to date than others, but one thing I never like doing is using web-based upgraders. They may be convenient, but for a start they require that you give the web server far more file permissions than any sane person would want to during the upgrade process, plus the fact that any kind of ‘black box’ upgrade makes me nervous.

WordPress is fairly easy one to update, but even so requires some manual fiddling if like me you shy away from fully web-based upgraders. So, I wrote a simple script to automate it, including backing up previous files and the database in case anything goes horribly wrong. I’ve tested this on 2 sites so far (in 3 environments, my local test server and 2 separate live sites) and it upgraded WordPress installs running both 2.5.x and 2.7.x to the latest version without any problems. Certainly saved me time once I’d written it and removed the element of random human error (replacing it with predictable automated error ). I figured someone else might find it useful.

Disclaimers: this script is presented AS-IS and I take no responsibility for any effects of using it, use entirely at your own risk. In particular, it only backs up the core WordPress tables and assumes that you used the default ‘wp_’ table prefix, so if you have plugins which need extra tables, or used a different prefix, you will need to alter the script if you want a valid database backup. It’s also only applicable to Linux servers, although that’s the most common setup for WordPress anyway.

#!/bin/bash

# WordPress upgrader, run from the root of your WordPress install like this:
# /path/to/wpupgrade.sh /path/to/wordpress-x-x-x.zip
# (tar.bz2 / tar.gz archives also supported)

if [[ "$1" == "" ]]
then
	echo Required: WordPress archive parameter
	exit 1
fi

if [ ! -e $1 ]
then
	echo Archive $1 does not exist
	exit 1
fi

if [ ! -e wp-config.php ]
then
	echo This script must be run from the WordPress root directory
	exit 1
fi

# Back up old files
echo Backing up DB...
rm wordpress_db_backup.sql.bz2
read -pDatabase: db
read -pusername: user

mysqldump -u$user -p $db wp_comments wp_links wp_options wp_postmeta wp_posts wp_term_relationships wp_term_taxonomy wp_terms wp_usermeta wp_users > wordpress_db_backup.sql || exit 1
bzip2 wordpress_db_backup.sql

echo Backing up files...
rm wordpress_file_backup.tar.bz2

tar -cjf wordpress_file_backup.tar.bz2 wp-includes wp-admin wp-content wp-*.php index.php xmlrpc.php || exit 1

rm -rf tmpwordpress
mkdir tmpwordpress

case $1 in
	*.zip) unzip $1 -d tmpwordpress/
	;;
	*.tar.bz2) tar -xvjf $1 -C tmpwordpress/
	;;
	*.tar.gz) tar -xvzf $1 -C tmpwordpress/
esac

rm -rf wp-includes
cp -R tmpwordpress/wordpress/wp-includes ./

rm -rf wp-admin
cp -R tmpwordpress/wordpress/wp-admin ./

cp -f tmpwordpress/wordpress/*.php ./
cp -f tmpwordpress/wordpress/*.html ./
cp -f tmpwordpress/wordpress/*.txt ./ 

rm -rf tmpwordpress

echo Files updated, now go into WordPress admin to finish the upgrade

As an aside, WordPress 2.7.1 has a bug in it that can cause saving posts to fail with a message about htmlspecialcharacters_decode sometimes (such as when using the syntax highlighter in the above code). If you hit it too, the fix is in this bug report.

Obviously I’ve been doing a lot of web work this week, getting ogre3d.org back to how I want it. In a lot of cases that’s meant changing the software wholesale – something you really don’t want to be rushed into, but in this case I had no choice.

It’s been one of those weeks where I feel like I’m desperately doing a headlong sprint, only to run into an invisible brick wall two feet from the finish line. A number of the pieces of software I’ve tried using have been in the ‘almost, but not quite’ bracket. We used Gallery for years for example, but had never upgraded past version 1.5. Given that I’m making sure we’re not languishing on any old versions any more, I obviously picked up Gallery 2.3, spent a fair amount of time re-importing all of our galleries with the metadata attached, integrating it with WordPress via WPG2, and styling / configuring it the way I wanted, only to find that when I put it on the live site, the memory usage was completely unacceptable, even with all non-essential plugins disabled and full optimisation turned on. We’re a reasonably high-traffic site; nothing compared to some of course, but still higher than a lot of places), and it was clear fairly quickly that Gallery 2.x was going to be a dead weight – literally as soon as I enabled it the server’s available memory would plummet like a stone. I wish I had a representative performance testing set up to try these things out beforehand – sometimes I miss having the resources of a larger organisation to tap into in cases like this!

Rather annoyed at the time I’d wasted, I then tried NextGEN Gallery for a little while, but found that inadequate too. Again it was almost good enough, but unfortunately the navigation between albums was not very good, and would require some modding to make it work well. And that was even before I knew how it would perform – I suspected better than Gallery, since it’s a lot less functional, but still, it was an unknown, and I didn’t want to spend time fixing it up the way I wanted (like I did with Gallery) only to find I have to toss it out.

So finally, I settled on hosting our gallery on Flickr instead of self-hosting, and exposing them on the main site via Slickr Gallery. Slickr has a really nice navigation mode and looks very slick (it’s aptly named), and using Flickr for hosting obviously has bandwidth and performance advantages.

The one problem with this solution though was that the lightbox that Slickr uses is a bit weak. It can only handle a fixed-size window (although it will resize to the aspect ratio of the picture), which means the image is almost always too small on a decent sized screen. You can increase the size, but then it’s too large for smaller screens and you have to scroll, so it just ends up feeling unsatisfactory. So, instead I wanted to use Shadowbox.js, which resizes dynamically to the client area of the web page, which is ideal. Unfortunately Slickr doesn’t have an option to change the lightbox, and there wasn’t much information online about it. So, after some tinkering, I discovered how to do it:

1. You should have Slickr, Lightbox-2 and Shadowbox.js installed.
2. In wp-content/plugins/slickr-gallery/slickr/slickr.php, replace all instances of ‘lightbox’ with ‘shadowbox’
3. In wp-content/plugins/slickr-gallery/slickr/slickr.js, replace “initLightbox();” with “Shadowbox.setup();”

That’s it.  It’s fairly simple, but it took me a bit of trial and error to get completely right even though the end result was trivial (Shadowbox’s code is a little hard to read). I originally made more changes thinking I had to do more than this, but ended up breaking things in IE, or in certain navigation cases. This simple change seems to work in all reasonable cases (IE7, FF3, Opera9, haven’t tried Safari yet). You could probably make more changes to remove the Lightbox-2 dependency entirely (disabling the plugin breaks Slickr still), but I have other things to be getting on with for now. Like uploading and tagging all the old screenshots!

Edit: here’s the result. Yes, I know the page title says ‘Page Not Found’ (something to do with the way Slickr redirects page entries), will try to figure out how to fix that later.

So, now that I have basic service back at my newly reinstalled server, it’s time to start re-downloading clean packages for the software that makes up ogre3d.org. Imagine my surprise when I went to phpBB.com and saw this:

Looks like I’m not the only one having a bad week

Current priority is getting the forums back online, watch that space.

Great. I got a call from the provider of the server on which ogre3d.org runs today saying that the server had been compromised, and that they were going to have to resinstall it. Seems that an unpatched flaw in a Joomla! component allowed a backdoor to be created which was spamming.

My fault – I should have patched Joomla! more often. But, I’m busy. What’s annoying is that even though I can 100% guarantee that the compromise could not have extended beyond a certain defined scope, since SELinux prevents that at the kernel level, the host has insisted on taking a tough line on this – their unshakeable policy says that a completely fresh reinstall is required, and it has to be done today. I can understand that, but I see now it’s been pointless for me to use SELinux in this case then. They’ll charge me extra for the reinstall too The only benefit is that at least while I’m doing it I can hop onto RHEL 5 which will mean MediaWiki can be upgraded at last.

This is going to cause some major downtime on ogre3d.org, firstly for the physical reinstall, and secondly for the time it takes me to get the ton of application software reinstalled. First priority will go to getting the forums running again, obviously. I’ll have to decide what to put on the main site page – it’ll be a placeholder at first. It’s very likely to be a few days before everything is completely back to normal again.

Don’t expect me to be doing much except getting that sorted out for a little while. I’m expecting to be stressed. Once I’ve fought these particular fires and got things on an even keel again, I’m considering moving host again since clearly the ‘premium’ support that I pay for is actually a pretty blunt instrument (you can say  you told me so now).

Update: ogre3d.org is offline as of now for reconstruction. Subversion and downloads are still available on Sourceforge, once the server has basic functionality back I’ll put up a placeholder letting people know what’s going on.

I’m a bit of a grump when it comes to a lot of the Web 2.0 startups of recent years. I still dislike Facebook – originally it was just an in-principle reaction based on their rather irritating child-CEO and his ability to attract vast amounts of investment based on a business plan made entirely of arm-waving and wet tissue paper, but now having used it for a while, I dislike it on its own merits. Even with my relatively small friends list it seems dedicated to the task of swamping me with as much useless information, stupid applications, and other such nonsense as it possibly can. Occasionally there will be a nugget of something in there, but it’s drowned in so much crap it’s hardly worth the effort to filter it. Sure, it allows me to keep up with the activities of ‘friends’, but in a much more shallow and sterile way than actually meeting them or talking on the phone/skype/in-game chat. Anyone who thinks Facebook is a good way to develop actual friendships is deluded. As a supplement for real-world or voice meetings (e.g. arranging get-togethers, sharing photos etc), I can see the utility, but I attest that it’s really not even the most efficient way to do that, what with all the junk on there. So generally, pass.

LinkedIn I like, because it sticks to the point. Every business person needs their Rolodex, and that’s what LinkedIn is – a business tool. It doesn’t claim to be central to your life, or that you can build / sustain real friendships there or other such twaddle. The focussed nature of it means there’s less noise, so it’s easier and quicker to get what you need. Contacts there are not personal, they’re a reference for it you’re looking for work, a contractor, a partner etc – which is appropriate for the medium in a way that personal relationships are not. I don’t feel the need to constantly update it or prod my contacts, or keep track of all their statuses; it just does what I need and gets out of the way.

I’ve avoided Twitter because again, it just looked like a way to pour time down the toilet. It seemed like blogging without the in-built filter that says ‘is this worth blogging about’?. Besides, I already actively participate in forums, mailing lists, blogs etc – I really didn’t feel I needed yet another dimension to have online discussions in. And I don’t really want to tell the entire world what I’m doing at any one time – I strain out the small number of actually interesting things already and post them up on this blog, and the rest of the minutae rapidly decreases in value the more degrees of separation away from me you get; so in fact real-world communication works far better as a filtering mechanism for that. Clay Shirky described why information filters in the real world have historically done a much better job with these kinds of things.

However, I decided to open a Twitter account anyway, but only for a very narrow subset of ‘what I’m doing’ – that is, what I’m currently doing in core Ogre. Obviously I blog about that sometimes, and I post in the forum for things that people need to know about, and keep the roadmap on the wiki up to date, but sometimes there are low-level things that don’t warrant that, but people are still interested in. I used to use IRC for this, but over the years drifted away from it, as it was too time consuming to loiter in channels for extended periods looking for topics in real-time, and people just bombarded me with questions all the time anyway so it was a little too much like hard work. So, for things that don’t warrant a blog or forum posting, I’m going to try to keep Twitter informed. There’s also a widget on the right there.

The focus will be kept deliberately very tight – so nothing about my other work commitments, personal life etc are going on there. If any of those things merit comment, they’ll get a more considered entry in my blog. Consider this a sort of ‘pre commit log’ of sorts! If I continue with it, anyway…