I hate Mondays – ogre3d.org hack

Great. I got a call from the provider of the server on which ogre3d.org runs today saying that the server had been compromised, and that they were going to have to resinstall it. Seems that an unpatched flaw in a Joomla! component allowed a backdoor to be created which was spamming.

My fault – I should have patched Joomla! more often. But, I’m busy. What’s annoying is that even though I can 100% guarantee that the compromise could not have extended beyond a certain defined scope, since SELinux prevents that at the kernel level, the host has insisted on taking a tough line on this – their unshakeable policy says that a completely fresh reinstall is required, and it has to be done today. I can understand that, but I see now it’s been pointless for me to use SELinux in this case then. They’ll charge me extra for the reinstall too :( The only benefit is that at least while I’m doing it I can hop onto RHEL 5 which will mean MediaWiki can be upgraded at last.

This is going to cause some major downtime on ogre3d.org, firstly for the physical reinstall, and secondly for the time it takes me to get the ton of application software reinstalled. First priority will go to getting the forums running again, obviously. I’ll have to decide what to put on the main site page – it’ll be a placeholder at first. It’s very likely to be a few days before everything is completely back to normal again.

Don’t expect me to be doing much except getting that sorted out for a little while. I’m expecting to be stressed. Once I’ve fought these particular fires and got things on an even keel again, I’m considering moving host again since clearly the ‘premium’ support that I pay for is actually a pretty blunt instrument (you can say  you told me so now).

Update: ogre3d.org is offline as of now for reconstruction. Subversion and downloads are still available on Sourceforge, once the server has basic functionality back I’ll put up a placeholder letting people know what’s going on.

  • Dark Sylinc

    So spamers aren’t just parasites, they are also taking your time AND money. Specially to someone from the community (you)
    That pisses me off.

  • volca

    That just sucks. I wish you that the reinstall will go well at least. PHP coders seem never to learn :(

  • kinjalkishor

    give ur time but dont get stressed. You certainly don want another health problem. Things will be restored. Though this is such a problem. You definetly should think of another host. I thought the good ones are cheaper as they do more buisness.

  • http://www.stevestreeting.com Steve

    Hosting in the US is cheaper than the UK, but I prefer having the server legally close to me. We’ve had cheap hosts before and outgrown them, three times.

    With hosts you do get what you pay for – cheaper hosts have more sites to deal with, so support is worse in the event of a problem. With previous hosts, the server always seemed to go down when I was away and no-one monitored it. Uptime has been a lot better since moving to this host since they monitor it and investigate quickly, perform full backups for me etc. And, I can talk to a real engineer on the phone, not some call centre.

    But it’s expensive and the support isn’t quite as detailed as I’d like – basically they will do app-specific support, but only things that are supported by RHEL. In practice, we have to stray outside this too much so we’re on our own in too many places. So, I may look for something cheaper, but I sure as hell won’t go for cheap & dirty.

    That’s for another time though, immediate concern is getting things back up.

  • Eric

    Ugh, that sucks man. Hope you’re able to get that taken care of as painlessly as *possible*. Good luck.

  • Assaf Raman

    Wow, that is the worst new. If there is any thing I can do to help – e-mail me and I will do it to save you time.
    I was just about to post on the multi-screen patch and saw your e-mail.
    Here is a screen shot of the multi-screen layout – at least you will be able to look at something that will put a smile on your face:
    http://assafogre3dforumfiles3.googlepages.com/multi1.jpg
    http://assafogre3dforumfiles3.googlepages.com/multi2.jpg

  • Dark Sylinc

    Yeah don’t get stressed
    SVN and Ogre3D downloads are working so it’s not such a big deal.
    Server problems affect even big enterprises with big budgets and it’s own server farm.
    True users of OGRE will be patients.
    Those who are not probably don’t even understand well the phylosphy behind Ogre in the first place.

    Just for you to know, your Torus Knot webpage seems down too. Don’t forget to check that out.

    Cheers ;)
    Dark Sylinc

  • haffax

    Good idea with wordpress replacing joomla. Managing wordpress is easy, and the news section is like a blog anyway. These few static pages are just as easy to set up and there are lots of gallery plugins for wordpress available.

  • Roz

    Steve, I don’t understand your post in the least and don’t need too. But if you expect to be stressed you will be. Expect to be positive and productive and for things to go to plan :) Lecture over.

  • http://www.stevestreeting.com Steve

    @Assaf: thanks, nice pics.
    @Dark: yes, I know the TKS site is down too, it’s on the same server, hence why I’m keen to get them both back up. That will be faster to restore because it’s simpler, if only I could get into the reinstalled server right now (grr). I could have switched the DNS over to another server temporarily, but the time lag of doing that (and returning it) and given how close I thought I was to getting a placeholder going, I haven’t done it. If I could just get someone to walk into the datacentre and fix the sshd problem, I could probably be back up at a basic level in about 30 minutes. My only guess is that they’re understaffed because of all the travel disruption in the UK right now.
    @Roz: I appreciate the thought, but after 15 years doing this I’ve learned the one constant is that things never go to plan. In life Im an optimist, but I’m also a realist when it comes to technical things ;) Sods law says that what can go wrong, will go wrong – like having these sorts of problems on the very day that the datacentre is understaffed.

  • Game_Ender

    That sucks Steve. I do a little adminning on the side, although you appear to better at it then I, and I know how hard it can be to get everything setup just perfectly.

    Good luck getting the new site up.

  • http://www.g-boot.com Luis

    sorry to hear that! I hope software reinstall will not take too much time ;)

  • http://www.jacmoe.dk/ jacmoe

    I recommend CMS Made Simple, even though I use ModX.
    It’s lean and quite capable.
    This really sucks. :|

  • http://blog.shadypixel.com/ btmorex

    That sucks. It seems this sort of thing happens all the time now, I guess because almost everyone is using CMS software (joomla, wordpress, drupal). I used to use drupal, but I found that wordpress had much less painful upgrades. Especially with 2.7, it’s basically all automated, which makes it much easier to keep on top of things.

  • Pingback: SteveStreeting.com » Blog Archive » Part high-chair, part bronco simulator

  • Noid the Proid Loid

    Black-hat spammers are parasites. Ogre is one of the good guys making decent software. Anyone targetting a project like Ogre must be a total jerk with no life. I wish you the best of luck getting the new site back up, and look forward to another year of great Ogre advances.

  • Pingback: SteveStreeting.com » Blog Archive » Wordpress upgrade script