Streaming media from Amazon S3

Business, Political, Tech, Travel, Web 9 Comments

Thanks John for the reminder to investigate S3 as a business media hosting service, it works like a charm!

Now that I have far fewer bandwidth worries (max $0.17 per GB), the Torus Knot site includes a nifty dynamic selector so you can pick low, medium or high quality - the latter is at a higher resolution too, clocking in at about 100Mb. I may well use S3 for future public commercial downloads in the future too. It’s altogether more convenient than the block bandwidth allocations you get with regular hosting packages, since it scales dynamically at a very fine level of detail depending on demand. And don’t be fooled by ‘unlimited’ bandwidth offers, all hosting companies have to pay for bandwidth and there’s no such thing as ‘unlimited’ resources; you’ll actually find your bandwidth being throttled or cut off via a ‘reasonable use’ clause in the small-print; ‘unlimited’ is simply a marketing lure. If you want truly scalable guaranteed bandwidth, you have to pay for it.

Getting S3 media hosting working wasn’t that hard, but did require a few discrete steps. Firstly, you need to create a bucket in your S3 account which is all in lower case, is globally unique and is DNS-compatible; so for example I created a bucket called ‘media.torusknot.com’.

Then to make it all look nice you need to create a DNS CNAME entry to map a sub-domain of your site to that S3 bucket; in my case I mapped ‘media.torusknot.com’ to ‘media.torusknot.com.s3.amazonaws.com’. That allows me to access any files I upload to that S3 bucket via ‘http://media.torusknot.com/somefile.jpg’. You do just need to set the ACLs on the files & the bucket to make sure public access is allowed.

Finally, if you want to stream video files via a Flash player from S3 to another domain, you also have to tell Flash that it’s ok for the content to be pulled in from a different domain. Create a file called ‘crossdomain.xml’ in the bucket, with these contents:

<cross-domain-policy>
<site-control permitted-cross-domain-policies="all"/>
</cross>

That allows the media to be accessed from anywhere - you can be more specific if you want but this is the simplest approach.

Once again I’m using the excellent FlowPlayer; my only issue with it is that the ‘buffering’ animation seems to not work all the time (so be patient if you’re viewing the high quality version).

Gotta love this cloud computing business :)

The Terabyte Club

Tech No Comments

It’s an eternal truth that no matter how much hard drive space you have, you always manage to fill it with something.

In my case, it’s mostly tools, SDKs and most importantly build environments. I have to maintain at least 3 active development copies of OGRE at any one time (the previous stable branch, the current stable branch and the development branch), plus clean build environments for at least 3 compilers in order to do releases. Add to that the occasional backport scenario for clients, plus build variants (static linking, multiple threading options, double precision mode), plus project builds for customers and my own spin-offs like OgreSpeedTree and it all starts to mount up, particularly as a full build of OGRE will creep up to about 3-4GB of data (mostly intermediate files).

To be fair, hard drives do tend to last longer and longer every time. I bought a 250GB hard drive about 4 years ago and was amazed at the amount of space I had free (with my secondary backup drive, I had 330GB at my disposal). However, over the last month or two I’ve been pushing at the limits and so I decided to upgrade.

Terabyte hard drives are getting quite cheap these days, although the cheaper ones (like the Seagate Barracuda) have been getting some poorish reviews, so given I do some pretty HDD intensive work I looked for something with decent performance (barring going OTT with an array of Velociraptors) . The Western Digital Caviar Black was getting some good reviews, so I went with that - the only one that seemed to compete at this size was the Samsung Spinpoint F1, but according to the reviews I read, while it did very well at bulk sustained transfer rates, its performance didn’t scale well to lots of random I/O requests, something that’s pretty important when doing compilations with lots of small files being accessed in many threads. Thus, I chose the Caviar Black which is a touch slower than the Spinpoint on bulk transfers, but is a clear winner in the kind of random access workloads I do most.

They come in 750GB and 1TB variants; I decided to go with the 750GB since the Caviar Black is a more expensive drive (£80 for the 750GB, £110 for the 1TB), and I intend to keep my 250GB as a secondary drive anyway, so together they push me over the magic psychological 1TB barrier (in theory, of course you never see all that space in reality).

It feels like moving into a new house with loads more room - you just want to do cartwheels on the empty space (metaphorically). It’s great not to have to worry about disk space again as I fire off a new build, and I don’t personally notice the noise that the reviews mentioned as a downside of the Caviar Black, but then my existing drives were 4+ years old and probably noisy by comparison to modern equivalents anyway. In a way, I find the slight noise reassuring, particularly as I don’t have a HDD light on the front of my machine (it’s behind the door panel, in a rather odd design choice by ThermalTake).

I wonder if it will be another 4 years before I fill this one up…

DLC Took My Lunch Money

Games, Tech 9 Comments

For some reason I was suddenly curious as to how much money I’d spent since last December on digital content for the 360, such as XBox Live Arcade titles and more recently Rock Band DLC. Of course you buy things in Microsoft Points on the 360, which like Wii Points and Disney Dollars are designed precisely to disguise how much money you’re actually spending. The PSN has my respect in this regard for taking the brave step of actually pricing things in units of real money. Quite why Microsoft and Nintendo chose to go against the precendent set by every other marketplace in the developed world (except Disneyland, but that’s intentionally ‘wacky’) I’m not sure - I doubt we’d take most high street retailers very seriously if they required us to buy things in ‘Starbucks Bucks’ or ‘HMV Quatloos’.

Anyway, when I totalled it all up, I’ve chugged my way through 13,000 Microsoft Points in 9 months so far, which in the real world is £110.50, and judging by the dates, about half of that has been on Rock Band DLC, the rest being XBLA titles. I’ve spent more on digitally delivered content on my 360 than I have on boxed games (just software, excluding plastic peripherals) - some of that is because I received most of my boxed games as presents, and I picked up a back catalogue from eBay, but even so, I do think my own habits are a sign of how quickly digitally delivered content is becoming accepted.

Small purchases (micropayments is an often used term, although that usually refers to even smaller amounts) are just easy to mentally justify, even if you end up making enough of them to exceed a larger pruchase that you would perhaps think about more carefully. It’s really easy to slap down £1.36 for a new Rock Band track (that’s less than 2 tubes of Pringles), or £6.80 for an XBLA title (Geometry Wars 2 is particularly a no-brainer); it really doesn’t take much to convince you, particularly when you know exactly what you’re getting - after all you can check out the Rock Band tracks on RockBandContent.com and play demos of every XBLA game before you buy.

Taking out the overhead of the retailer and physical distribution makes products cheaper - that’s obvious. Games are, in general, very overpriced - we pay £40 for a boxed game which required the same budget to make as a Hollywood blockbuster I can pick up across the aisle for a tenner. One obvious reason is that market is smaller, another reason is the silly situation we have where a console platform holder takes a huge slice of the pie just for letting developers deploy on their platform.  All these things are interlinked - the audience is smaller partly because the content is so expensive, which leads to content marketed more at the core audience which spends that money, and larger margins required to make back console hardware development costs, etc etc. Nintendo has broken out of that to some degree, but they’ve mostly appealed just to the mass market, leaving most of the core audience on 360 and PS3. Ideally we’d have a situation where the whole spectrum of game players (’core’ and ‘mass market’ are the most talked about but there are lots of graduations) existed in one place, just like you have with movies, and obviously this is the holy grail that certainly MS and Sony are trying to chase, even though I have serious doubts that we’ll ever get there until the industry rids itself of the counterproductive market segmentation that multiple proprietary consoles creates. I do think that digital distribution helps though, because it disrupts the status quo, creates a more fluid situation that just can’t exist very easily elsewhere, and makes a space for people to experiment more - both as producers and consumers - and to see what works.

I’d love to know the bigger picture of how much money is spent on the likes of XBLA, Steam, PSN etc. I’m sure it’s generally smaller than retail sales, but I’d be interested in knowing the trajectory of those numbers, and in particular which kinds of players they are. Anecdotally I get the impression that digital distribution tends to be good for those ‘ex hardcore’ gamers like me - we don’t buy a ton of games anymore, but like quality bite-sized content and are willing to pay for it. We’re not casual, but we’re not hardcore anymore either.

Anyone else got interesting comparisons of their physical / digital purchase numbers?

Gates & Seinfeld - funny in whose dimension?

Comedy, Tech, Windows 11 Comments

I don’t know if they’re actually airing these adverts Stateside, or whether they’re a web-only phenomenon for the moment, but Penny Arcade drew my attention to them today. Colour me unimpressed. If the intention was to shake off Vista’s sales blues, or to generally ‘connect’ with the wider consumer in a way that Apple does so well but Microsoft almost never does, but I’d have to classify this effort as a failure of sizable proportions.

Maybe it’s me; maybe I just don’t ‘get’ Seinfeld-style humour, or maybe it’s that Bill Gates really doesn’t remotely inspire or entertain me (making skiploads of cash every nanosecond might inspire some people to revere him, but not me). From my perspective though this is a fairly poor attempt at deadpan humour which leaves me with an impression of Gates being even more of an arrogant but incredibly dull grey suit than I thought before. Worse - it seems his thinking is that acting like a dull grey suit in wacky surroundings will somehow make him endearing and amusing, which couldn’t be further from the truth. It’s equivalent to your Uncle Henry who, despite bearing an unnatural affinity with tweed, thinks he can transform into John Travolta when he hits the wedding reception dance floor. Deeply painful to watch.

Tip to Microsoft - if you want endearing consumer-friendly ads, write Gates out of the script next time, he’s a dead weight.

*edit: I know this is old news but I’m behind a little, very busy lately!

Great video on making money as a startup

Business, Tech 1 Comment

David Heinemeier Hansson is famous for being the guy that invented Ruby on Rails and running 37Signals; I  have mixed feelings about Rails personally (great for some things, not so great for others, but then that applies to pretty much every technology), but this presentation he did on making money as a tech startup is very good indeed - insightful yet very amusing.

He presents in an online context for the most part but as he says himself, the principles apply to all kinds of product. It also dovetails in nicely with what I was saying a few days ago about open source and business, in that there are similar arguments about not believing the hype we’re often sold by high-profile business news stories.


Found via Matt Asay.

Microsoft patents pagination calculations

Political, Tech 6 Comments

More patent silliness from those idiots in the US Patent Office, as they get exploited by soulless corporate types again:

US Patent 7415666: Method and system for navigating paginated content in page-based increments

I really can’t imagine how messrs. Sellers, Grantham and Dersch can sleep at night, having officially claimed that calculating how far to advance down a document when you hit the PageDn is a significant innovation that warrants the protection of 20-year exclusivity that a patent brings. It beggars belief that an engineer could possibly think that way - I’m guessing a company-sponsored discount lobotomisation scheme, or perhaps it’s enough to run internal training courses such as ‘TKNGTHPSS101: Stifling innovation by patenting the bleeding obvious’.

Time to stop this nonsense. Now.

Tim O’Reilly agrees with me about Facebook?

Tech, Web 5 Comments

I came across Tim O’Reilly’s post entitled Open Source and Cloud Computing today, and I was pretty happy to see that his thoughts reinforced what I was saying a couple of months ago about how I thought isolated, corporate-owned islands in the ‘cloud’ were not a beneficial model for the Internet long-term, despite the short-term convenience in today’s society.

I was also very interested to see from his links in that article that some in the open source community are already forming plans to address it. It’s early days yet, but I look forward to a day when we can have all the convenience of sites like Facebook provide without having to cede control of our data to a centralised corporate entity searching desperately for a way to dissect that data & our browsing behaviour for revenue opportunities.

It’s funny because when talking about this issue to a friend recently, I made the comparison between sites like Facebook now, and the ‘gated communities’ of AOL / Compuserve in the past, before the Internet took off. They too used their sealed community to generate revenue, but when people finally had the option to become free of that and get the same functionality, users deserted in their droves to the liberated environment of the Internet. Tim used the same analogy in his article, so maybe I’m not crazy after all.

Strange coincidences

Internet, Political, Tech 9 Comments

I read today that ‘Pentagon uber-hacker’ (if you believe the US authorities, who presumably don’t want you to think that their security systems are akin to wet tissue paper) Gary McKinnon has lost his appeal in the Lords against his extradition to the USA. I think we can all feel sorry that a misguided but definitely non-malicious geek is going to get the book thrown at him.

Coincidentally, we also watched Sneakers last night, after I finally got around to buying it on DVD. It’s still one of my favourite films, even though occasionally it errs on being film-friendly rather than technically realistic (accoustic couplers in 1992?). The cast is fantastic, the script is great, and the appeal of a bunch of non-conformist, philanthropic hackers coming out on top is enduring.

What a shame life doesn’t imitate fiction a little more often.

Ballast for your mouse?

Tech 12 Comments

My venerable MX510 has served me well but after a few years of concentrated (ab)use it was somewhat the worse for wear - the scroll wheel died a few months back, and more and more of the low-friction feet had become detached, making it start to limp like a wounded animal. I finally caved this week and took it out to the barn with a 12-gauge.

I thought about getting a wireless mouse this time around, but I concluded that I really can’t be arsed to charge yet another wireless device or keep it stocked with batteries, so I stuck with a traditional wired rodent. I’ve always preferred the design of Logitech mice so I just went with a fairly conservative choice of the G5, because it has the same basic shape I’m used to.

It’s nothing particularly special, a high-precision laser product as you’d expect these days, and you can vary the sensitivity dynamically which is a bit of a gimmick for me since I’m hardly a pro gamer. However, the cool thing that actually is useful is that it’s not a fixed weight. The base mouse is quite light, but undernearth there’s a slot for a ‘ballast cartridge’, which is just a piece of plastic with 8 round slots in it. In the box they give you a metal tin with tiny metal weights in it - 8 x 1.7g and 8 x 4.5g - which are quite nicely designed. You just snap in whatever combination of weights you want; this way you can vary the weight by up to 36g in 1.7g increments. It’s a nice idea actually, since weight does have a bearing on how comfy a mouse is to use and you’re not forced into a one size fits all setup. I have mine set to 30.4g :)

The Gates Legacy

Tech, Windows 10 Comments

I think this is a very interesting and balanced write-up of the life and times of William Henry Gates III. I still can’t believe we gave him an honoury knighthood though. :?