SteveStreeting.com

Keeping web software up to date is a pain, but failure to do it can result in significant ramifications. Some bits of software are easier to keep up to date than others, but one thing I never like doing is using web-based upgraders. They may be convenient, but for a start they require that you give the web server far more file permissions than any sane person would want to during the upgrade process, plus the fact that any kind of ‘black box’ upgrade makes me nervous.

Wordpress is fairly easy one to update, but even so requires some manual fiddling if like me you shy away from fully web-based upgraders. So, I wrote a simple script to automate it, including backing up previous files and the database in case anything goes horribly wrong. I’ve tested this on 2 sites so far (in 3 environments, my local test server and 2 separate live sites) and it upgraded WordPress installs running both 2.5.x and 2.7.x to the latest version without any problems. Certainly saved me time once I’d written it and removed the element of random human error (replacing it with predictable automated error ). I figured someone else might find it useful.

Disclaimers: this script is presented AS-IS and I take no responsibility for any effects of using it, use entirely at your own risk. In particular, it only backs up the core Wordpress tables and assumes that you used the default ‘wp_’ table prefix, so if you have plugins which need extra tables, or used a different prefix, you will need to alter the script if you want a valid database backup. It’s also only applicable to Linux servers, although that’s the most common setup for Wordpress anyway.

#!/bin/bash

# Wordpress upgrader, run from the root of your Wordpress install like this:
# /path/to/wpupgrade.sh /path/to/wordpress-x-x-x.zip
# (tar.bz2 / tar.gz archives also supported)

if [[ "$1" == "" ]]
then
	echo Required: Wordpress archive parameter
	exit 1
fi

if [ ! -e $1 ]
then
	echo Archive $1 does not exist
	exit 1
fi

if [ ! -e wp-config.php ]
then
	echo This script must be run from the Wordpress root directory
	exit 1
fi

# Back up old files
echo Backing up DB...
rm wordpress_db_backup.sql.bz2
read -pDatabase: db
read -pusername: user

mysqldump -u$user -p $db wp_comments wp_links wp_options wp_postmeta wp_posts wp_term_relationships wp_term_taxonomy wp_terms wp_usermeta wp_users > wordpress_db_backup.sql || exit 1
bzip2 wordpress_db_backup.sql

echo Backing up files...
rm wordpress_file_backup.tar.bz2

tar -cjf wordpress_file_backup.tar.bz2 wp-includes wp-admin wp-content wp-*.php index.php xmlrpc.php || exit 1

rm -rf tmpwordpress
mkdir tmpwordpress

case $1 in
	*.zip) unzip $1 -d tmpwordpress/
	;;
	*.tar.bz2) tar -xvjf $1 -C tmpwordpress/
	;;
	*.tar.gz) tar -xvzf $1 -C tmpwordpress/
esac

rm -rf wp-includes
cp -R tmpwordpress/wordpress/wp-includes ./

rm -rf wp-admin
cp -R tmpwordpress/wordpress/wp-admin ./

cp -f tmpwordpress/wordpress/*.php ./
cp -f tmpwordpress/wordpress/*.html ./
cp -f tmpwordpress/wordpress/*.txt ./ 

rm -rf tmpwordpress

echo Files updated, now go into Wordpress admin to finish the upgrade

As an aside, Wordpress 2.7.1 has a bug in it that can cause saving posts to fail with a message about htmlspecialcharacters_decode sometimes (such as when using the syntax highlighter in the above code). If you hit it too, the fix is in this bug report.

Valve are awesome. They’ve made a string of excellent games, many of them including elements that have significantly progressed the medium, like the Half-Life series’ in-game storytelling, Team Fortresses class systems, Portal’s FPS without guns and Left 4 Dead’s reinvention of the co-operative gameplay experience (yes, I know some of these became Valve when they absorbed other teams, but they had the vision to nurture and promote them). Then there’s the fact that they’re almost single-handedly helping to keep PC gaming relevant in the modern world with Steam. It may have had it’s problems initially, and some people don’t like the lack of resale value on games, but there’s simply no other system as slick as it for getting content and updates, and connecting with friends on PC, and the opportunities it offers for independent developers is fantastic.

It was just announced that the Left 4 Dead Survival Pack will be free, both on PC and 360, which is great and another reason to say ‘yay Valve’. The one criticism you can level at Left 4 Dead is that it’s a little short on content, given that a single playthrough of the entire game could maybe take you 6-8 hours. Of course the replay value is great though, very much like Horde on Gears 2, which we’re playing a ton, but more content is always welcome (note: this pack doesn’t include new campaigns, but a new play mode called ‘Survival’ and enables Versus mode for the 2 maps that didn’t have it - not that we play Versus). Multiplayer games have always had this benefit of course, but in my case I haven’t been a fan of playing against other random players from around the world for a few years, simply because I don’t want to burn lots of time practicing, and in that case the experience is too unpredictable to guarantee a fun time, every time - I really don’t want to have a 50:50 chance of being frustrated every time I play. Conversely when I play co-op with friends against the AI in Horde or Left 4 Dead, I can guarantee that it’ll be fun every time because we can peg the difficulty the way we like, and still have fun as a group.

For this reason I don’t feel bad that I’m buying my 3rd copy of Left 4 Dead this week. We already have 2 copies on PC, but we decided to buy the 360 version too so we can play with non-PC owning friends. These games definitely work best when played with real-world friends rather than ‘randoms’ (as we’ve come to call them) from the Internet, and the sad fact is that more of our friends play on 360 than PC these days. We’ve had so much fun on Horde lately we want to have another similar experience to alternate with.

Important: the subject matter and parties involved with this legal issue are deliberately not mentioned here; if you are aware of their identities, I ask you not to mention them publicly here in comments, or anywhere else.

As some of you are already aware, over the last few months there has been an ongoing legal issue with a 3rd party having allegedly used OGRE code without respecting the license conditions. I hate getting involved in legal disputes, there are so many more useful things to do with time, money, and emotional energy, but nevertheless as custodian of OGRE it falls to me and my company to take charge of situations like this, however reluctantly.

Happily, the issue was finally resolved today. Details are confidential (so don’t ask), but what I can say is that a satisfactory resolution was achieved, and without me having to remortgage to fund the legal bill. The Free Software Foundation Europe were very instrumental in advising me on the case, and directing me towards good legal representation, so I really can’t thank them enough for their help; but I hope that the donation I’ll be making will go some way to repaying my debt to them.

On the whole though, I’m just glad it’s all out of the way, it’s one less distraction

My new chair arrived today, packed into a surprisingly small box thanks to the marvels of German engineering (or something). A deft bit of allen key action later, and it was ready to go. Initial impressions are that it’s going to take some getting used to. The gas lift does indeed go high enough to use with a standing height desk, and it’s very solidly built and very stable, which is good when you’re perched that high.

It’s adjustable in more ways than I can get my head around right now so will take some time to get right, and my jury-rigged elevated desk isn’t so great with this set-up since your legs are cramped by the original desk surface (which is now at thigh height), so I’ll need to look for a more permanent setup. And the saddle seat - well, it certainly does alter your posture so that it’s like a horse riding or skiing pose (bend zee knees!) rather than sitting, but the initial downside of the saddle-like configuration is that it can make your inner thighs a little sore to begin with - but at least that prompts a return to the standing posture. At least it doesn’t bounce around like a real horse does though - I know from experience how sore that can make you after a few hours! I’ll see how it goes.

Oh, and I have a hundred and one things to catch up on this week after last week’s unplanned diversions, so bear with me if I’ve missed things in the forum or am otherwise slow to respond. There just aren’t enough hours in the day to make up for the time lost in a short time, especially when my back still limits the amount of desk time I have available.

Obviously I’ve been doing a lot of web work this week, getting ogre3d.org back to how I want it. In a lot of cases that’s meant changing the software wholesale - something you really don’t want to be rushed into, but in this case I had no choice.

It’s been one of those weeks where I feel like I’m desperately doing a headlong sprint, only to run into an invisible brick wall two feet from the finish line. A number of the pieces of software I’ve tried using have been in the ‘almost, but not quite’ bracket. We used Gallery for years for example, but had never upgraded past version 1.5. Given that I’m making sure we’re not languishing on any old versions any more, I obviously picked up Gallery 2.3, spent a fair amount of time re-importing all of our galleries with the metadata attached, integrating it with WordPress via WPG2, and styling / configuring it the way I wanted, only to find that when I put it on the live site, the memory usage was completely unacceptable, even with all non-essential plugins disabled and full optimisation turned on. We’re a reasonably high-traffic site; nothing compared to some of course, but still higher than a lot of places), and it was clear fairly quickly that Gallery 2.x was going to be a dead weight - literally as soon as I enabled it the server’s available memory would plummet like a stone. I wish I had a representative performance testing set up to try these things out beforehand - sometimes I miss having the resources of a larger organisation to tap into in cases like this!

Rather annoyed at the time I’d wasted, I then tried NextGEN Gallery for a little while, but found that inadequate too. Again it was almost good enough, but unfortunately the navigation between albums was not very good, and would require some modding to make it work well. And that was even before I knew how it would perform - I suspected better than Gallery, since it’s a lot less functional, but still, it was an unknown, and I didn’t want to spend time fixing it up the way I wanted (like I did with Gallery) only to find I have to toss it out.

So finally, I settled on hosting our gallery on Flickr instead of self-hosting, and exposing them on the main site via Slickr Gallery. Slickr has a really nice navigation mode and looks very slick (it’s aptly named), and using Flickr for hosting obviously has bandwidth and performance advantages.

The one problem with this solution though was that the lightbox that Slickr uses is a bit weak. It can only handle a fixed-size window (although it will resize to the aspect ratio of the picture), which means the image is almost always too small on a decent sized screen. You can increase the size, but then it’s too large for smaller screens and you have to scroll, so it just ends up feeling unsatisfactory. So, instead I wanted to use Shadowbox.js, which resizes dynamically to the client area of the web page, which is ideal. Unfortunately Slickr doesn’t have an option to change the lightbox, and there wasn’t much information online about it. So, after some tinkering, I discovered how to do it:

1. You should have Slickr, Lightbox-2 and Shadowbox.js installed.
2. In wp-content/plugins/slickr-gallery/slickr/slickr.php, replace all instances of ‘lightbox’ with ’shadowbox’
3. In wp-content/plugins/slickr-gallery/slickr/slickr.js, replace “initLightbox();” with “Shadowbox.setup();”

That’s it.  It’s fairly simple, but it took me a bit of trial and error to get completely right even though the end result was trivial (Shadowbox’s code is a little hard to read). I originally made more changes thinking I had to do more than this, but ended up breaking things in IE, or in certain navigation cases. This simple change seems to work in all reasonable cases (IE7, FF3, Opera9, haven’t tried Safari yet). You could probably make more changes to remove the Lightbox-2 dependency entirely (disabling the plugin breaks Slickr still), but I have other things to be getting on with for now. Like uploading and tagging all the old screenshots!

Edit: here’s the result. Yes, I know the page title says ‘Page Not Found’ (something to do with the way Slickr redirects page entries), will try to figure out how to fix that later.

I live on an island that often gets bad press for being a ‘tax haven’. Those in the local financial services industry don’t like that term of course, pointing out how standards-compliant the finance industry is, and how many information exchange agreements we have with other countries (the line ‘the lady doth protest too much’ bubbles to the surface in some people’s minds I’m sure at this stage). So, we’re not technically a tax haven according to the OECD definition, but we’re certainly a place for people to stash their money and avoid paying tax on the income they derive from it in the juristictions in which they live. There’s no getting around the wider political debates about whether that’s a morally respectable position to take, particularly that the reason taxes can be so low is that the state doesn’t have to pay for defense, which is sponged from the UK government, provides precious little in the way of healthcare, and generally takes a right-wing policy route that seems to actively promote social division, but let’s leave that for the moment. Obviously being somewhat of a left-leaner I find it all slightly distasteful and am quite grateful my career path has so far kept me doing other things. I don’t deride people for choosing to be involved in that business - it’s their choice, and certainly it pays better than most of the alternatives in a small locality, but for a little while now (certainly since developing my own political opinions rather than inheriting them from the community around me) I’ve felt much better to be ‘aligned’ with the goals of whatever organisation I’m spending my working time with. Seems to me that we spend so much of our lives working, it probably ought to be for something we actually believe in, and I personally can’t say I have any significant motivation to help people avoid paying tax. While I can, I’ll keep doing other things, although increasingly our local government doesn’t seem very interested in seriously promoting much else.

However, I have been dismayed with one of the latest developments locally which are supposedly ‘branching out’ beyond financial services, because it’s actually worse - our IP law has been revised now so that patents from any juristiction can be re-registered locally to obtain the same protection (previously, it only applied to IP registered in the UK  I believe). Simple enough, except that articles in our local news have been chest-beating about it specifically with reference to the fact that now, patents that wouldn’t be valid in the UK can now be registered, so long as they are valid in another juristiction - and in particular they singled out business method patents as registered in the US, which are currently not allowed in the UK. They’re happy that ‘asset holders’ can now ‘bolster their protection’ by re-registering their ‘IP’ even though the UK would have thrown it out as worthless.

Ugh. One of the things I was proud about in the UK is that bullshit patents on business methods weren’t valid. I was happy that total nonsense like the Amazon 1-Click patent and it’s ilk were deemed not to be valid inventions, for they are widely acknowledged to suppress innovation and play directly into the hands of patent trolls. The world is blighted by people who register widely known techniques as patents with a registrar who is so ignorant and/or compromised by conflicting interests that they’re incapable of acknowledging the prior art, and an entire industry wastes precious resources either fighting patent spam, or building their own equally rancid pile of patents as a self-defense mechanism, all instead of actually inventing significant things, or you know, making great products that stand up in their own right. If even half the time that went into the overheads of establishing, debating, licensing and fighting low-brow, pointless patents was spent on the creative process, who knows how much we’d actually advance the human race. Instead, that effort gets spent on lawyers instead - it’s no wonder that the people lauding these ‘advances’ are from that particular profession.

At a time when everyone else, even big companies like Microsoft in the US, are recognising that software / business method patents are proving corrosive to the industry, locally our law makers are puffing themselves up over having allowed such nonsense to happen here too.

But hey, it’ll make a quick buck, both for the registrar and for our local legal firms, so it’s ok right? Favouring the financially convenient over the holistically respectable seems a common line around here. I despair sometimes.

Despite my initial positive reaction to working standing up, since I made that post my back has gone steadily downhill again, starting with a dodgy weekend that I hoped was a blip, but grumbled on for pretty much all last week, before taking a rapid decline this week. This morning I’ve been in the kind of pain I haven’t had since late last year. So, coupled with the ogre3d.org hack, colour me extraordinarily pissed off this week.

One aspect is that I hadn’t been out walking this week until this morning, since it’s been icy and since my back was already bad, I didn’t feel it was worth risking a slip. I’ve been out this morning but it hasn’t helped - it was even too painful to do my stretching exercises afterwards. But I know it’s not just that, because it was dodgy last week too and I was going out every day. Still, the steep decline this week clearly indicates I can’t get back to normal work yet, even when an emergency prompts me to try.

So, I dunno. I’m trying to track it, trying to observe cause and effect, but it still seems elusive. I’m just going to have to keep trying (what other option is there?). If there’s one thing I hate, it’s illogical things, and this is like hooking yourself up to a 20,000 volt cable with a random number generator controlling the on switch.

A small bit of musing while I wait for another back-up to run…

Reinstalling a server from scratch sucks. Obviously. Not being able to use direct dumps of the old system itself because of concerns of how far a malicious attack got, and how long ago (even though we’re running SELinux) means that everything has to be constructed afresh. How much fun I’m having.

But if there’s one silver lining here, it’s that at least Linux stores every shred of its configuration in a simple, plain text format, and in one dedicated subtree of the file-system. Even though the server itself had to be taken down, the old disk was mounted so that I could look at previous configuration files easily, and carry across relevant ones (checked manually, natch) directly. It still takes a lot if time, and the fact that I jumped OS versions at the same time has complicated it (but, if there’s any time to do that, it’s now), but it would have been much worse if I couldn’t reference the old system.

One thing I found annoying at times about doing admin on Windows servers (in a past life) was that they generally hid their settings away from you - the common assumption was that you use a GUI to edit everything, and accessing settings without that GUI was frequently difficult, if it was even possible at all. Although a GUI is friendly for many uses, it also does a pretty good job of hiding things from you. Sometimes that’s useful (drawing attention to the most important things), sometimes it’s very unhelpful (trying to find which tab / dialog a particular option is in). One thing it definitely fails at is making it easy to extract / summarise all the necessary information to audit it, or to recreate an entire setup. Even when you have the server still running its a pain, but if that server has been taken offline it’s extremely hard to extract information from it without booting the thing up again (which if it’s damaged or compromised, might not be desirable or possible). Settings were often scattered among the registry, proprietary repositories, application specific places, and sometimes in custom data formats. If you’re lucky you might be able to extract the settings some way, but usually you have to have thought of it before the machine was out of action, and the process is often specific to a given application - so even assuming you remember to do it for all of the different server apps you’re running, the results are disparate, hard to organise and very often not human-readable - something you really want when you’re auditing a machine or creating a variant. The result was, in my experience, relying heavily on binary machine images for reference setups, test servers etc. That works well enough, but it’s a bit opaque and doesn’t help you much when you want variations (unless you have MxN images, or a tree of derivative images).

In comparison on Linux, I know I just have to look at plain text, readable configuration files in /etc/, which I can do on pretty much any device without actually having to have any of the old software running - I can just mount the disk with minimal permissions. By and large the text files are extremely well commented and contain pretty much every option you might need, just commented out when defaulted. I can search quickly for settings just like I can in any text file, and search across the entire configuration if necessary. Being text, it’s very easy to create a standardised configuration template that you can roll out, in a much more configurable way than a raw machine image. The visibility of all the settings certainly helps, and you can do all sorts of nice things such as generating configurations from variable options, should you want to. Text configuration might look less friendly at a surface level, but over time, I’ve found that in practice it’s actually a considerably more productive way of doing things for many admin tasks - especially the more difficult ones.

Tweaking an existing installation is still probably easier with a GUI though, and less intimidating to beginners or occasional admins (and I actually count myself in the latter). The best solution is to have both - a GUI for casual admins and core text config storage underneath - and of course there are plenty of options  about to do that on Linux too.

I’m not sure if Windows Server 7 does anything different here - I haven’t administered a Windows server since 2003 so I may well be out of date. I just thought I’d break out some love for the often unappreciated plain text config file. Sometimes simplicity is the best choice.

So, now that I have basic service back at my newly reinstalled server, it’s time to start re-downloading clean packages for the software that makes up ogre3d.org. Imagine my surprise when I went to phpBB.com and saw this:

Looks like I’m not the only one having a bad week

Current priority is getting the forums back online, watch that space.

Great. I got a call from the provider of the server on which ogre3d.org runs today saying that the server had been compromised, and that they were going to have to resinstall it. Seems that an unpatched flaw in a Joomla! component allowed a backdoor to be created which was spamming.

My fault - I should have patched Joomla! more often. But, I’m busy. What’s annoying is that even though I can 100% guarantee that the compromise could not have extended beyond a certain defined scope, since SELinux prevents that at the kernel level, the host has insisted on taking a tough line on this - their unshakeable policy says that a completely fresh reinstall is required, and it has to be done today. I can understand that, but I see now it’s been pointless for me to use SELinux in this case then. They’ll charge me extra for the reinstall too The only benefit is that at least while I’m doing it I can hop onto RHEL 5 which will mean MediaWiki can be upgraded at last.

This is going to cause some major downtime on ogre3d.org, firstly for the physical reinstall, and secondly for the time it takes me to get the ton of application software reinstalled. First priority will go to getting the forums running again, obviously. I’ll have to decide what to put on the main site page - it’ll be a placeholder at first. It’s very likely to be a few days before everything is completely back to normal again.

Don’t expect me to be doing much except getting that sorted out for a little while. I’m expecting to be stressed. Once I’ve fought these particular fires and got things on an even keel again, I’m considering moving host again since clearly the ‘premium’ support that I pay for is actually a pretty blunt instrument (you can say  you told me so now).

Update: ogre3d.org is offline as of now for reconstruction. Subversion and downloads are still available on Sourceforge, once the server has basic functionality back I’ll put up a placeholder letting people know what’s going on.