Category Archives: Web

OGRE Tech Web

Apache to Nginx & PHP-FPM : part 1

Background
The Ogre3D website has been running on a dedicated server for about 7 years now; this is relatively expensive, but when we moved away from the shared hosting that Sourceforge generously provided, but which we had outgrown, our initial foray with a VPS (at the time lighttpd on Linode) proved inadequate for our needs, so after a month of futile tuning we gave up and went fully dedicated.

Time has moved on of course, and virtualisation technology is considerably better than it was in 2005. I’d intended to try again soon anyway to reduce Ogre’s overheads but our Adsense revenue was still covering the cost and I hadn’t got around to it yet. Then suddenly, Google pulled our ads after a mistaken (I believe automated) conclusion that we were hosting copyrighted material – a few users had posed test binaries of their own work on MediaFire and similar ‘red flag’ download sites – and all of a sudden we were leaking money. The misunderstanding was sorted out with Google within a few days, but even so it illustrated that we should probably look to move to a cheaper solution if we can so we have less exposure.

The Ogre site’s main issue with performance is Apache’s memory usage under load, so given a VPS is more constrained I wanted to address that. Enter Nginx, stage right.

read more »

Personal Tech Web

Blog revamp

As I promised, I’ve given this blog a bit of an overhaul in anticipation of posting more often again. For those who are interested, here’s a run-down of the updates:

  1. New responsive design
    Responsive design is all the rage these days; in summary, it’s all about making your site adapt fluidly to the reading environment so it looks good on a variety of devices, even resizing images so they always fit. Try resizing your browser window, and you’ll see that the layout adapts, dropping the sidebar when it gets too thin and so on. I’d love to take credit for it, but monkeying about with CSS is one of my least favourite things in the world so I just used/tweaked an existing theme which took my fancy. Clearly, I’m in a minimalist phase right now.
  2. Commenting via Disqus
    Previously I’d used the standard WordPress commenting system plus a reCAPTCHA plugin to cut down on spam. That worked well enough, but Disqus has some advantages, chief of which is that you can comment using existing accounts such as Twitter, Google, OpenID and Facebook, which is much faster and frictionless. Also if you have a Disqus account, it collates all your comments across all websites so you can reference them more easily – I’ve found this useful myself in the past if I’ve commented on a post I found on Twitter and then forgot where it was. Importing all the blog’s existing comments to Disqus was easy, and allegedly all comments in future will be stored in both places (edit: confirmed, this is working), so you don’t have to worry about being exposed to external data loss, you always have all the comments in your own database too.
  3. Replaced many scattered plugins with Jetpack
    I used to use a bunch of different plugins for twitter feeds, subscriptions and social sharing buttons, but then I found Jetpack, which packages a bunch of them in one plugin.  Seems to work well, and it beats having to upgrade a bunch of separate plugins.

So, I hope you like the new design. Apologies for any lingering issues, I’m sure there will be some tweaks to do in the coming days but generally I think it’s a big improvement.

Cocoa Development Objective C Web

Automatically get latest download link from Sparkle in PHP

This is just a quick post to help someone out on Twitter, and the blog seemed the best place to post it.

If you make Mac apps, you probably use Sparkle to manage your auto-update process, outside the Mac App Store anyway. And so you should, it’s awesome, and makes keeping software up to date much easier. But what about the download link on your website for new users? Wouldn’t it be nice not to have to manually update that every time?

For SourceTree, I use a PHP script which just parses my Sparkle appcast and reads the latest release from there, generating a download link automatically. So whenever I update the Sparkle appcast (and I have scripts to do this from XCode, which I may share in a subsequent post if there’s interest), the download link for new users is always up to date immediately.

The full code is after the jump if you want it. As usual, feel free to use this for whatever you want, but there are no warranties for anything whatsoever; don’t blame me if it doesn’t work, or dissolves your website into a pool of steaming alien slime.

read more »

Internet Tech Web

Hosting services: my recommendations

After hearing on Twitter how an acquaintance’s new hosting provider went ‘mammaries skyward’ this week, much to their understandable annoyance, it occurred to me that I have some recommendations I can make on this subject. While I don’t host that many sites, I’ve been doing it for long enough and had experience of both personal and medium-traffic sites that I’ve experienced the highs and lows quite a few times already.

The Golden Rule: Support > Everything

When it comes to hosting, the most important thing to look for, beyond what all the statistics of how much space and bandwidth you get, beyond even quoted up-times, is the quality of the support service. The big question is: when things go wrong – and if you host long enough, eventually they will even in the best possible hosting environment – how quickly are problems resolved, and how responsive are the support engineers during the process. Literally nothing is more important than this, and unfortunately it’s the one thing that you’ll only really learn with experience, unless you’re hosting a site big enough that you can get a formal SLA. Assuming you’re not going big enough for that, the only way to judge this is by being with a provider for a while, or knowing someone who has been with them, or possibly looking at online review sites – although frankly these are often highly unreliable, polluted as they are with inaccuracies and omissions either because of ignorance (people who post glowing reviews after being with the site 2 weeks) and unfortunately by frequent shill reviews.

I’ll post a couple of hosts I’ve had good experience with over many years later in the post, through good times and bad.

Know Your Bandwidth

Personally, I instantly rule out any host that claims ‘unlimited bandwith’. This is a crock – there is no such thing, and to claim there is just means the host is already lying to you before you even start – they have to pay for their bandwidth, so they can’t possibly allow everyone truly unlimited bandwidth and stay in business. If you really need unlimited bandwidth, i.e. you have a high-traffic site with lots of media files, then you will quickly bump into the way that these sites offer this ‘unlimited’ deal – via throttling. You may not have an absolute physical cap on your bandwidth, but if the tap is locked off to a slow dribble beyond a certain usage, it’s really worthless. In practice, ‘unlimited bandwidth’ is just a marketing point that they hope will draw in people who will only actually use a tiny amount of bandwidth, but will somehow favour them because the offer looks good. Don’t be one of those dumb guys.

Really you need to establish your bandwidth requirements and head for a host that can fulfil them for a reasonable price. For example, ogre3d.org uses between 125GB and 250GB per month, which is a reasonable amount, compared to my personal site here which only needs 5-10GB per month.

If you have ‘spiky’ bandwidth, i.e. occasionally you need to be able to distribute large amounts of data, but it’s not a constant stream, it would be best to go for a lower monthly limit and host high-bandwidth items elsewhere. I often use Amazon S3 for this purpose which can be made to look like a sub-domain of your own site, and which charges for bandwidth at a very fine granularity so matches your demand closely – it’s more expensive than buying a monthly allowance if you use it a lot, but for on-demand spikes it works very well.

Shared, Dedicated or Virtual/Cloud?

I currently use two shared hosts and one dedicated host, to match the demands of each site. Personally, I’m still very skeptical about virtual private servers and cloud hosting, due to a bad experience I had a few years back when we tried running ogre3d.org on a VPS. We lasted not much more than a month before we moved the server to a dedicated machine because the VPS simply didn’t deliver on its promises – performance was unpredictable and to be honest you had the worst of both worlds – you had to admin your own server but you still didn’t have a 100% guarantee that no-one else would be screwing with something on the machine, or that the disk arrays wouldn’t be hammered by someone else (regardless of CPU assignment), or some other balancing issue. Virtualisation has evolved in the last few years so this may not be an accurate representation anymore, but personally I wouldn’t go for a VPS again any time soon, unless it was a machine I controlled entirely and partitioned myself into virtuals – at least with shared and dedicated servers you know exactly what you’re getting – either a low-maintenance but shared resource environment, or total control & power. VPS claims to offer a middle ground but in my experience it didn’t deliver.

So, who do I use?

For my shared hosting, I’ve been using Hosting Matters for about 10 years now. I went through a couple of other hosts before them and had terrible experiences, but since I switched to them I’ve been very happy. I can count the number of hours downtime (that I’ve been aware of) over those years on one hand, and whenever there’s an issue they’re incredibly fast to respond – they have both community forums and support tickets depending on the urgency. It’s also very reassuring to see the same names cropping up in the support responses over the years.

Their offerings are pretty standard, nothing that would make them jump off the page for anyone looking for a stellar feature list or super-cheap pricing. But they’re very reasonable, they’re honest about what they’re offering (like bandwidth), and as I said before, support > everything.

For dedicated hosting, since 2007 I’ve used Dedipower. They’re based in Reading, their support staff are all local and are on the end of a phone if you need them (no call centres). Having been through a UK dedicated server comparison twice in the last 3 years (once again just recently), Dedipower came out as the most competitive for the service they were willing to offer, and I’ve been happy with the support service. In once instance in fact, when I moved a sub-site off the server, they were quickly on the phone to me within 10 minutes to tell me it was ‘down’ – at which point I had to explain it was expected & apologise for not notifying them in advance. You really can’t complain about that.

I hope that’s useful to someone. In case I need to point this out, I’m not getting paid or receiving discounts to promote either of these hosts, they’re just the two I’ve been most happy with over the ~10 years I’ve been hosting sites. YMMV but they’ve worked well for me :)

Personal Web

Why I won’t be updating Facebook anymore

I’ve never really liked Facebook, as regular readers of the blog will be all too aware of, but I’ve been a user of it in the last couple of years, mostly at the cajoling of friends. During this same period of time I also started using Twitter, a service which I was also skeptical about initially. Previously, I’d always relied on my blog, forums and official sites to do my interacting, and I wasn’t sure I needed anything else.

What’s happened in the last couple of years is that Twitter has risen inexorably in my daily usage due to being genuinely useful as a networking aid, while Facebook has not fared that well at all. I tried posting updates to it from Twitter for a while via Selective Tweets, which led to a considerable blip in my Facebook usage (even though I wasn’t using the site) but honestly I can’t be bothered with it anymore. And not just because of the rubbish that sloshes around Facebook.

Facebook is ‘sold’ on the idea that you can realistically make / be friends with people, or even keep in meaningful personal touch with people you do genuinely know, via short messages and photos on an online service. My personal opinion after trying to use it as such for a while, is that this is a massive crock of elephant dung. Relationships are built on meaningful contact, and Facebook provides – is only capable of providing – only meaningless, trivial contact. Of all the ways you can keep in touch with people, it’s about the very worst you could possibly use, IMHO. It’s convenient, sure, but convenience doesn’t make for good personal contact. Honestly, if you can’t be bothered to make the effort to meet someone in person or talk on the phone (and home video calls aren’t that hard in the 21st century), then really you’re not that bothered about keeping in genuine contact with them. It’s better just to be honest – you’re kinda interested in knowing what’s up in their life, but not interested enough to have coffee with them, or otherwise make any genuine time for them.

In this context, Facebook is a fallacy. The whole point of Facebook versus Twitter or any other public internet medium is that Facebook is ‘private’ (barring any changes in rules they might feel like making to make a buck on your data of course). But if you don’t care enough to speak to someone other than in Facebook, and vis-a-vis you’re not really that much of a friend, then why would you want them to know things you wouldn’t post publicly? This is behind many the Facebook fo-pahs that seem to crop up with increasing regularity. Twitter is honest about the level of interaction  you’re having with people. Facebook tries to make you think it’s more significant than it is.

Maybe you’re a highly gregarious person who genuinely has over 100 real friends that you’d share your private information with. Maybe you feel that personal relationships are upheld and maintained by Facebook messages. I don’t, and I’m going to stop pretending that it’s serving a useful purpose in my life. I have a small collection of friends and family with whom I exchange information by meeting and speaking to them – these are the only people I’d consider exchanging genuinely personal information with. There’s a second much larger collection of people whom I don’t meet personally but who I’m interested in keeping in contact with, and by and large they’re all on Twitter – which also does a far better job of delivering that information without bombarding me with all kinds of other nonsense. There’s really no useful middle ground for Facebook to fill, for me. And using Facebook as a supplement with people I already know? Frankly it’s far more entertaining to talk to them about their week’s exploits than read about them on Facebook – in fact Facebook can take the wind out of a good anecdote :)

Really, you should never post anything on the Internet that you wouldn’t mind being public, since Facebook hardly has an unblemished record for actual privacy (whether that’s via Facebook themselves, or friends or friends-of-friends). I think that a lot of people use Facebook like I use Twitter, except there’s far more scope for dangerous misunderstandings about what is appropriate to post. I think non-digital social protocols dealt with sensitive & private information much better, and that online services are best used only for non-private things. From now on, that’s what I’m going to do, and I’m not the only one.

Internet OGRE Web

OGRE OS & Browser Stats

I’ve had requests to post the OS & browser stats for the OGRE site, which I didn’t include in my previous demographics post, so here we go.

A caveat to start off with – as a programmer-oriented site our users are obviously a little different in their choice of tech to the population at large!

Operating Systems

Not really a surprise there, Windows dominates the landscape, with Linux and OS X pulling up the rear. Personally on the desktop I’m a Windows and OS X user so my visits are contributing to those rows. There’s a decent showing for the mobile platforms too, iPhone and iPod particularly, a fair few on Android and even some early iPad hits.

Windows Breakdown

It’s worth drilling down into the huge Windows stat to see what versions people are using:

XP still rules the roost then, and thank goodness Vista is sinking rapidly to the bottom like the cast-iron turd sculpture it is – Windows 7 has already more than doubled its share. And it’s amusing to see that a couple of crazy b?st#rds are still running ME and NT.

Mac OS X Breakdown

I’d heard that supporting pre-Leopard versions of Mac OS X was increasingly becoming pointless since almost everyone had upgraded, and these stats bear that out – 96% of users are running 10.5 (Leopard) or better.

Even more interesting was the sole visitor running 10.7! Was that an error, or did we have someone from Apple on an unreleased future version visiting the site? And what’s that ’68K’ entry about – someone running an Apple ][ emulator or something?

The Linux breakdown wasn’t interesting (99% ‘not set’, the rest just kernel versions), so there’s nothing to post for that – don’t email me about  missing out Linux please ;)

Browsers

Firefox is the clear winner here at a huge 50% – this definitely reflects our developer-focussed audience. In fact, I used to be an avid Firefox user until quite recently, when the new version of Chrome added the equivalent to the extensions I relied on in Firefox, at which point I switched because Chrome is more efficient with memory in particular. IE’s share at 18% is definitely not representative of the general user population, but then our users tend to be a bit better informed than that :)

So there you go, request fulfilled – hope it was interesting.

Internet OGRE Web

OGRE web demographics, revisited

Almost exactly three years ago, I posted an analysis of the traffic on ogre3d.org and the rough country breakdown of our users, which is always fascinating to me. I hadn’t actually been collecting web stats on the site for about a year (the previous set-up was lost when I had to recreate the server in a hurry, and somehow reinstating it never seemed to rise to the top of my TODO list), but a month ago I finally got around to adding Google Analytics to the site. The results have been very interesting, particularly when compared to 3 years ago, so I thought I’d share some factoids with you.

Visitors still increasing

In 2007, log analysis indicated we were getting a little over 100,000 unique visitors per month; obviously this is not 100% accurate due to shared & dynamic IP addresses, people logging in from multiple sites, etc, but it’s a reasonable order of magnitude indicator. Analytics is typically more conservative in its figures, since it excludes bots better as well as non-Javascript browsers, but still in the past month (actually only 28 days) we’ve had over 120,000 unique visitors to the site – and 1.2M page views – so we’ve sustained and slightly improved our user traffic. And all without any Slashdot posts ;) Also, this doesn’t count visits to Sourceforge, BitBucket or ome static generated HTML like our online documentation pages.

The Meteoric Rise of China

In 2007, China ranked 15th in our league table of visitors. Three years on and they’ve risen to the number two spot, comfortably surpassing Germany – at first I wondered if that was down to users there using fewer proxies, but since the figures for other countries have remained fairly stable I think the majority of this is genuinely a vast increase in the number of Chinese visitors to OGRE’s site. Here’s the top 10 countries (figures are for the number of visits):

The range of countries is demonstrated by how many are in the grey ‘others’ section (38.73%). Except for the massive change in China’s share, most of the other countries have stayed approximately in their relative positions & shares of the user population since 2007.

Region View – Europe still dominates, Asia challenges the Americas

The country view is, however, quite misleading if your aim is to decide where to locate a web server for example, because it naturally biases the figures towards large unified countries (like the USA and China), and doesn’t really show a true regional picture. For that, we have to examine the numbers (again, number of visits) by continent:

Now, even though the continent view includes Russia in Europe which screws up the locality principle a bit, even if you exclude that Europe dominates our community, with close to 1 in every 2 visitors to the site being from Europe. The Americas  and Asia share most of the rest almost equally now, which is a change from 2007 when the Americas were more dominant, and everyone else shares the scraps (3.5%). The Americas figure is made up of about 86% North America and 14% South America, and Asia is predominantly (60%) the Eastern Asia countries (mostly China, but South Korea holds its own too), with South-East and Southern Asia sharing the rest – particular hotspots there are India, Indonesia, Malasia, Turkey and Vietnam.

City Clusters

One thing I like about Analytics is the ability to drill down into countries and look at the local clustering. There are the expected clusters around cities – in the USA, the top 2 cities are unsurprisingly New York and Los Angeles, although Columbus OH takes the number 3 spot, and in the UK the clustering around London is massive – but they typically represent only about 25-30% of the audience, with the rest being scattered pretty much uniformly across most areas of the country in question. It’s fun to be able to point at almost anywhere in Europe, North America and the southern and eastern parts of Asia and to have a pretty good chance of being quite near to someone who has used the OGRE site.

The Monday morning OGRE fix

With OGRE obviously used by a lot of people in their spare time, you might expect that the weekends would be the busiest times for the site, but the opposite is in fact true – Mondays are consistently the busiest days (particularly 6-9am PST), with Saturday being the least busy. Whether this is because people are working with OGRE, or just cheekily surfing in their work time rather than face the Monday workload, is hard to verify!

Final Thought

I get a kick out of looking at these stats so I hope you find them interesting too. It’s really cool to think that there are only a very small number of countries (such as North Korea and Laos) from which we don’t get any (non-proxy) visitors from in a typical month, and it’s very interesting to see how the visitor base is gradually spreading out and diversifying, something which I’m sure every site witnesses but it’s interesting to see it in your own data. The question is – will China keep the current trajectory? At this rate, they’ll take the number one spot from the USA in only a few more years and put Asia second in the regional rankings!

Internet Tech Web

Who cares what’s trending?

Trends – or as I would call them, rampant fads populated by people looking to leverage the best buzzwords to get VCs to throw money at them – come and go. The one constant is the claim that <insert trend here> is so awesome that will universally and irreversibly replace <insert existing technology here>, to the extent that if you’re using or producing <insert existing technology here>, you are irretrievably lame, and complete strangers will point at you in the street and laugh at your horribly backward ways.

The fact is though, the best that today’s trends can aspire to is to become the existing proven technology that tomorrow’s trends will point and laugh at. That’s if they do well – most will simply evapourate and leave the world as if they never were. It’s rather beautiful in its own way, a sort of karmic circle where the unjustified elitism associated with being part of the ‘hip’ crowd is eventually cruelly punished by the derision of those who replace them.

The current trending darling is cloud computing, following in the wake of the dot com boom, the social networking explosion, and yes even open source . Let’s face it, there are quite a lot of people and companies who participated in open source not because of the fundamentals, but because for a while including open source on your corporate manifesto was a  damn good way to get funding. Now that open source is no longer a leading trend that you can sell to VCs (it’s graduated to ‘mature’ and has therefore lost its sparkle to a certain breed of person), the piranhas have swum elsewhere. Good riddance, I say.

Trends are like the Borg – they’re not happy to be just a part of a diverse technical melting pot, they have to be front-and-centre in everything, and want everyone else to be defined in   terms of themselves. So predictably, now we’re told that everything will eventually run in the cloud, and that the browser will be our only OS, and every company chasing funding right now is trying to shoehorn some cloud aspect into their corporate plans. What a load of old rubbish – while I fully expect cloud computing to be one of the ‘stayers’, just like open source, it’s only going to be a part of the whole. I fully expect us to make far more use of hosted & distributed capabilities in the future, but I know for a fact that dedicated platforms are never going to go away – they’ll simply blend.

I could make all kinds of detailed arguments as to why browser based servicing of all needs is not a panacea, but there is one fundamental  issue that is most important - generalised tools and grand unified visions always fail, even when they make perfect sense to a designer or ‘visionary’.

Unified visions and perfect generalised solutions only exist in the head  of one person, usually a designer who has ‘seen the future’ and realises that with some adaptation, he can express all things in terms of the model he has in his head, just with some funky parameterisation. Eureka!

But, regular people don’t want generalisation or unification, only designers do. You’ll generally get a good response from developers, technicians and sometimes ‘extreme power users’  if  you pitch highly adaptable generalised toolsets to them (open source anyone?), because they are adapters and creators, but try to package that approach into an end product for the masses and it just won’t work. At the sharp end, all that matters is that a piece of tech does the one or two main things that it’s designed for, really, really well, and everything else is irrelevant – Apple figured this out years ago, and it’s why the iPod crushed its arguably more fully featured competitors. Generalisation is just not a feature regular people want – quite the opposite, they want specialisation.

The idea that in future all things will be done through a general browser to the cloud is a designer’s vision that will never happen. In the same way that the general public is moving away from using a single PC to do everything, and instead likes to use devices that better reflect the use context and purpose (but to have them all connect together), the vision of a unified application (browser) that can do everything is similarly flawed. The iPhone allegedy was originally conceived to use its browser for everything, but in practice most people preferred to use dedicated apps for each purpose (that could talk to the internet anyway) because they’re more functional.

So, who cares about trends anyway?

Linux Open Source Tech Web

s3putsecurefolder

Edit: this script is deprecated in favour of a rewritten version 2.

I use Amazon S3 to host large media files which I want cheap scalable bandwidth on, and for expandable offsite storage of important backups. I used to have some simple incremental tar scripts to do my offsite backups, but since I moved to Bacula, I’ve just established an alternative schedule and file set definition for my offsite backups, the critical subset of data I couldn’t possibly stand to lose (like company documents). Since I was refreshing all my procedures and tarring the Bacula volumes no longer made any sense, I rewrote my script for putting the resulting backup data on S3.

The prerequisite in all cases is s3cmd, which is pretty mature now and available on most distros (“apt-get install s3cmd” and you’re done on Ubuntu). s3cmd actually has a ‘sync’ command, but firstly that tries to sync in both directions, which I don’t want (I know in theory it should never overwrite any local version so long as I don’t update the remote copies from somewhere else, but I’m paranoid when it comes to my backups and prefer to be explicit), and secondly it obviously has to connect to S3 to determine the sync status, wheras I always know whether I need to upload new files just from my local environment (and S3 charges per request – not much, but it’s not zero and it’s the principle of the thing). So, I decided not to use the ‘sync’ command, and just determine locally what new files I needed to ‘put’ on the server.

Secondly, encryption is a must, since some of the data is sensitive and I don’t want to trust anyone else with it. I used to manually GPG my tarballs before uploading them, but I noticed that s3cmd supports an encryption option too. It just uses GPG anyway, just in symmetric form rather than asymmetric like my version did (translation: you use the same passphrase for encryption and decryption; a little less secure than using generated public/private keys but still ok so long as you pick a good passphrase and look after it). The default symmetric algorithm in gpg is CAST5 which seems pretty good, although you can change it if you want by editing your s3cmd config file. So, I decided to give it a try – after you configure s3cmd to use encryption, it actually automatically decrypts too when you pull the data back (symmetric key, remember) – being distrustful, I pulled the data back from S3 in a different environment and examined it, and it was indeed complete gibberish, but decipherable with the passphrase. Good stuff.

So, here’s my little script which will upload the encrypted contents of a folder to S3 – just the contents that have been added or updated since the last sync of that folder, and will encrypt them by default. I just run this on a cron schedule now and it seems to work fine. License is MIT, use at your own risk, no warranty is given that it won’t destroy every file on your machine or eat your children. Usage is like this:

s3putsecurefolder /my/source/folder my.s3.bucket

Edit: it was brought to my attention that Amazon have made it easier to create pseudo-folder structures in S3 buckets since I last tried to do it (I swear it used to throw out keys with forward slashes in them, I had to mangle my names last time I did this), so I’ve updated the script to allow nested folders too.

Internet Tech Web

YouTube putting the bullet in IE6′s head?

Oh, please let this come to pass soon. TechCrunch reports that YouTube is due to drop support for IE6 ‘soon’, pointing users at Chrome (obviously), IE8 and Firefox 3.5. Finally, one of the worst pieces of software ever to pollute the Internet with its presence is getting taken out to the barn with a double barelled shotgun, and not a moment too soon.

Sure, Digg already said they might do this, but YouTube is far more significant; if YouTube stop supporting IE6, then in practice it means I can too :) Bye bye IE6, please do let the door hit your ass on the way out, preferably hard enough to fracture your pelvis. It’s the least you deserve for making life hell for hundreds of thousands of web site maintainers over the last few years.