OGRE Tech Web

Apache to Nginx & PHP-FPM : part 1

Background
The Ogre3D website has been running on a dedicated server for about 7 years now; this is relatively expensive, but when we moved away from the shared hosting that Sourceforge generously provided, but which we had outgrown, our initial foray with a VPS (at the time lighttpd on Linode) proved inadequate for our needs, so after a month of futile tuning we gave up and went fully dedicated.

Time has moved on of course, and virtualisation technology is considerably better than it was in 2005. I’d intended to try again soon anyway to reduce Ogre’s overheads but our Adsense revenue was still covering the cost and I hadn’t got around to it yet. Then suddenly, Google pulled our ads after a mistaken (I believe automated) conclusion that we were hosting copyrighted material – a few users had posed test binaries of their own work on MediaFire and similar ‘red flag’ download sites – and all of a sudden we were leaking money. The misunderstanding was sorted out with Google within a few days, but even so it illustrated that we should probably look to move to a cheaper solution if we can so we have less exposure.

The Ogre site’s main issue with performance is Apache’s memory usage under load, so given a VPS is more constrained I wanted to address that. Enter Nginx, stage right.

read more »

Business Open Source Productivity

Oh no, not a ‘Best of’ collection?

Well, yes – and my apologies if you’ve already seen these.  In celebration of the new blog and before I’ve polished any new entries for it – I often write & refine my posts over several sessions, I find the content is better that way – I thought I’d flag up three posts from this blog that I’m particularly satisfied with, and that I think resonated well with people.

  1. Work 2.0 – the interruptible programmer
  2. How to make decisions
  3. My evolving view of open source licenses

Hope you enjoy!

Personal Tech Web

Blog revamp

As I promised, I’ve given this blog a bit of an overhaul in anticipation of posting more often again. For those who are interested, here’s a run-down of the updates:

  1. New responsive design
    Responsive design is all the rage these days; in summary, it’s all about making your site adapt fluidly to the reading environment so it looks good on a variety of devices, even resizing images so they always fit. Try resizing your browser window, and you’ll see that the layout adapts, dropping the sidebar when it gets too thin and so on. I’d love to take credit for it, but monkeying about with CSS is one of my least favourite things in the world so I just used/tweaked an existing theme which took my fancy. Clearly, I’m in a minimalist phase right now.
  2. Commenting via Disqus
    Previously I’d used the standard WordPress commenting system plus a reCAPTCHA plugin to cut down on spam. That worked well enough, but Disqus has some advantages, chief of which is that you can comment using existing accounts such as Twitter, Google, OpenID and Facebook, which is much faster and frictionless. Also if you have a Disqus account, it collates all your comments across all websites so you can reference them more easily – I’ve found this useful myself in the past if I’ve commented on a post I found on Twitter and then forgot where it was. Importing all the blog’s existing comments to Disqus was easy, and allegedly all comments in future will be stored in both places (edit: confirmed, this is working), so you don’t have to worry about being exposed to external data loss, you always have all the comments in your own database too.
  3. Replaced many scattered plugins with Jetpack
    I used to use a bunch of different plugins for twitter feeds, subscriptions and social sharing buttons, but then I found Jetpack, which packages a bunch of them in one plugin.  Seems to work well, and it beats having to upgrade a bunch of separate plugins.

So, I hope you like the new design. Apologies for any lingering issues, I’m sure there will be some tweaks to do in the coming days but generally I think it’s a big improvement.

Cocoa Development Objective C OS X Personal Travel

See you at WWDC!

Apple kept everyone on tenterhooks this year by announcing WWDC 2012 very late – the second latest announcement ever in fact.

Like many other people (11,000 of them I hear, which is alarming given that there are only 5,000 tickets to the event) I signed up to WWDC Alerts,  which sent me an SMS message while I was having lunch, only a few minutes after the tickets went on sale. That I was lucky enough to bag myself a ticket has a lot to do with that – about 90 minutes later, they were all gone – so big thanks to fellow Brits Anthony Herron and Aaron Wardle for running that, completely free of charge too. Legends.

Apple picked a surprising time to announce the tickets, being as it was about 2am Pacific Time. Perfect for people like me in Europe – I’d expected to get the call late at night – but I imagine there are a lot of people on the west coast who are seething about this choice of timing.

I’ve never been to WWDC before, but this year was a perfect time to go, with the continued growth of SourceTree and the fact that I’m attending Atlassian Summit, which is held a couple of weeks before in San Francisco. So I’ll be heading over for Summit, hanging out in the Atlassian SF Office for a week, then heading down to WWDC. A pretty efficient trip :)

So, if you’re going to WWDC, Summit or are in the SF area generally around that time and want to say hi, look me up!

I’m also thinking about getting a short-term pre-paid SIM (preferably with data) for the 3 weeks I’m out there, if you have any suggestions on that front I’d love to hear them.

 

Personal

Realisation: blogs are still where it’s at

I’ve been shamed by Scott Hanselman into realising that I’ve been neglecting my blog recently, and using Twitter as a poor surrogate for more detailed commentary, something I plan to address in the next few months. Google+ acted as a sort of halfway house for a while too, but a combination of their lack of mobile updates, a terrible new desktop design, and the realisation that my blog does a better job of forming a permanent archive of my musings mean that I’m unlikely to use it very much beyond linking now.

I do want to improve the experience here though, so I plan to upgrade the theme and am also considering replacing the embedded comments system with Disqus. The main advantage of that is that people can comment via their Twitter / Facebook  etc accounts, reducing friction but also it lets people manage their own comments better across many websites, while having have richer threaded discussions. The downside is that comments then become dependent on external data. Still, I think it’s an overall win, if you have any thoughts on that subject, I’d be glad to hear them!

Development Personal

Why I’m a software developer

How often do you stop and think about why it is you do what you do for a living? Maybe it’s a mid-life crisis thing, but of late I’m acutely aware of the finite nature of time, and that there are an infinite number of ways I could spend that time. I’m also aware that ‘software developers’ are a quite diverse bunch of people, despite the persistent stereotype of math geeks huddled around technical toys talking in obscure acronyms (OK, we do that too). So I put some thought into why I choose to spend my time making software.

For me, it’s really simple: I like making things that people enjoy. That’s a pretty broad definition, but creation and connection is absolutely at the core of my motivation. It’s not really about the technical or logical challenges for me; at least, not any more – that might have been more of an issue earlier in my career. There’s something indescribably satisfying about creating something from nothing, sculpting and refining it from an image in your head into a functioning, tangible product. It doesn’t really matter what it is, just that it didn’t exist before, and now it does, purely because of your will. “I made that” is a satisfaction universal to all languages and cultures. That I’m a software developer rather than a sculptor, writer, musician or painter is down to a combination of circumstance and natural tendencies, but I don’t think my motivation is limited to this technical sphere at all. In fact, I think we all have this creative spark, it just gets drummed out of a lot of us after childhood.

I also think that you have to make things that speak to you first and foremost. Obviously you hope they will resonate with others too, and it’s pretty much guaranteed that they will (it’s just the degree that’s variable). I’ve made products entirely for other people before, indeed I spent a large portion of my career doing that, and it’s a bit of a lucky dip whether that turns out to be enjoyable and fulfilling or not. I’m at my happiest when I’m scratching my own itch, eating my own dog food, and building a community of people who feel the same way. It’s where both Ogre and SourceTree came from, which are the pieces of work I’m most proud of, and are also the most successful products I’ve created so far. That can’t be a coincidence, right?

This may sound like woolly, new-age thinking, but what feels right normally is right. I’m convinced that when you spend your time doing things which sync up with who you are as a person, better outcomes are more likely.  Have you thought about why you do what you do lately?

    Cocoa Development Objective C OS X

    Follow-up: OS X privilege escalation without using deprecated methods

    I posted a few months ago about the problems I’d encountered with performing privileged actions from a Mac OS X app – in my case, installing a command line utility in /usr/local/bin – and that all the examples of this that I’d come across used an approach which was now deprecated. You can find my original post here:
    Escalating privileges on Mac OS X securely, and without using deprecated methods.

    I had failed to produce a shrink-wrapped working example to go with the discussion, primarily because extracting it into a standalone example would take a while and I made the post a couple of days before I went on holiday. I also didn’t know whether anyone else actually cared about the subject enough for it to be worth me doing it!

    Well, perhaps I should have known better, because I’ve had quite a few requests for such an example since then :) I finally got around to doing this at the weekend – and actually when I came to do it I understood why people had pestered me for it, because it took me a while to get things configured just right in a fresh project! Mostly, it’s that there are quite a few things that can go wrong outside the code, both in the project settings and the plists because of the code signing requirements.

    So anyway, here’s the project: PrivilegedHelperExample on Bitbucket. Please make sure to check the ReadMe.txt – despite being shrink-wrapped, you will need to add your own code signing identity before you can compile the code, and you will need to reflect the name of your certificate in a few places, which I’ve listed. I’ve also tried to point you at the relevant pain points you may encounter when replicating the result in a different project.

    The majority of this code is just the Apple example code from BetterAuthorizationSample and SMJobBless, grafted together, de-duplicated and tweaked. All the changes I made can be considered public domain.

    Enjoy!

    Personal

    Farewell 2011

    It’s that time of year again, the end of that artificial construct we call a ‘calendar year’ that prompts so many of us to cast our minds back over the last 12 months. So, apart from rocketing helplessly through space at 107000 km/h, only to return to where we started (relatively speaking, ignoring where Sol and the Milky Way have moved since then), what’s up?

    As I talked about in my review of 2010, my goal had been to simplify and take back more control in my professional life, revolving around making my own products and cutting down contracting to just single, more significant projects. To that end I’d created SourceTree – this surprised many people, who asked “Why would you go from making a 3D renderer to writing a source control system?? Isn’t that boring?”. Well, actually no – I like learning new things, and I like solving problems, particularly the ones that I have myself, and so SourceTree ticked those boxes, giving me the opportunity to do some native Mac development and scratch my itch for a DVCS tool that worked the way I wanted at the same time. It also made perfect sense from a business perspective, because it was self-contained, deliverable with my modest resources, and there was a proven market for selling independent Mac apps.

    Another subconscious reason that I’m aware of more now was that I needed to prove to myself that I could do something unconnected to my (long!) history on Ogre and make it work. I was starting to wonder whether I was capable of repeating the (popular) success I had with Ogre elsewhere, and particularly with a commercial product – was being free & open source the only reason I’d managed to build such an audience? Could I compete when I’m asking people to pay, and without that previous backdrop where many people feel they owe me a beer for all the free code & help I’ve given them in the past? I kinda needed to know.

    On the whole, it went a lot better than I expected. At the beginning of 2011 SourceTree was just starting to get noticed, and was slowly growing, with lots of really nice feedback from people (even when it was critical). It was still a very nervous time; numbers were still well below where they needed to be and success was far from certain, and I’d already invested a fair amount, so I took a contracting job for a few months in parallel to help replenish the coffers. This was actually a really cool project, a Mac app based on Ogre that simulating lighting rigs for music concerts for a UK company that ran the real things, and I loved doing it. But following the launch on the Mac App store, and with word of mouth recommendations building – I’m not very good at marketing, so I really appreciated my users helping me do that – within a few months SourceTree was self-funding, just in time for that contract to finish. I hadn’t expected to start breaking even until about the 18 month point, so this was a lovely surprise. Not wanting to get complacent though, I started planning other projects, with the intention of running one in parallel with SourceTree to make sure I wasn’t completely reliant on it.

    Everything changed though when Atlassian approached me about acquiring SourceTree, which completed in October – I’ve already talked about this in detail so I won’t repeat here. So now, by a curious twist of fate, I’m in the unexpected position of developing a free product (at least for now) again! I’ve certainly enjoyed the spike in users that’s provoked, and I love being able to tell people they can just go grab it for nothing, but it’s also nice to know that people were willing to pay for it too, before those nice people at Atlassian subsidised it for everyone.

    In summary, 2011 has been really good to me. I’ve had lots of new experiences and learned a hell of a lot, which alone I count as a very positive thing. Managing to build up a successful new product and to go through my first acquisition in 12 months was pretty demanding, but very satisfying and confidence building at the same time, something I’m sure I’ll benefit from in future.

    Much of the tech media would have you believe that creating a technology business requires you to court VCs, move to Silicon Valley and do something ‘fashionable’ (which right now means having ‘social’ in the brief somewhere), but that’s simply not true. Sure, if you’re expecting to rake in millions of dollars in seed funding and expect to retire on a private yacht in your 30s then that might be the only way you can do it; assuming you’re happy with running that gauntlet with a chance you’ll become one of the many roadkill that the press don’t talk about. But if you just want to make a good living and prefer to do it working on projects of your own creation, it definitely is possible, even if you’re just one guy living on a rock in the ocean. 2011 reinforced that belief in me, and for that, above all, I’m very grateful.

    I hope you had a good 2011 too :) Best wishes for the holiday season, and have a great New Year.

    Cocoa Development Objective C OS X

    Escalating privileges on Mac OS X securely, and without using deprecated methods

    This week I implemented a much-requested feature in SourceTree for the upcoming 1.3 release (beta 1 went out on Monday, this will make it into beta 2) – a command-line tool so you can quickly pull up SourceTree for the repository you’re in from a terminal. Writing the command-line tool was trivial, but when I came to implement the menu item which would install it in /usr/local/bin, which inherently needs privilege escalation, it turned out to be a lot more complicated than I expected.

    How so? Surely lots of people have done this sort of thing before? Well, that’s true, they have – but the problem is that just about all of the existing examples of this use the Authorization Services API call AuthorizationExecuteWithPrivileges – and this method is deprecated in OS X 10.7 (Lion). Now, of course that doesn’t stop you using it (yet), provided you’re willing to turn off the warning that building against the 10.7 SDK gives you, but any programmer worth his salt should take deprecation as a hint that they should be looking for another way.

    There are basically 3 ways to escalate privileges on OS X, and only one of them is now recommended:

    1. Use a helper tool which has its setuid bit set so that it runs as root. Risky if that tool gets compromised, and the setuid bit can be lost, needing reinstatement by another privileged task.
    2. Execute a command as root via AuthorizationExecuteWithPrivileges. As mentioned above, this is now deprecated, and again if a hacker compromises either the app or the tool being launched, bad things can happen.
    3. Ask Launch Services to install a privileged helper tool via SMJobBless. This helper is subsequently run by launchd as root when invoked via a Unix socket, and can perform privileged tasks. Importantly, code signing is verified at both ends by Launch Services at install time to prevent tampering with either binary.

    Clearly option 3 was the way to go – the only ‘downside’ about it is that is does require that you have the ability to sign your application and helper tool. I already have valid code signing certificates because I deploy on the App Store, so this isn’t an issue (even though this functionality won’t actually be in the App Store version of SourceTree because Apple disallow installer behaviour there, I still sign with the same certs). In fact, the fact that I know my app and helper tool can’t be interfered with without the code signatures becoming invalid is very reassuring. Given that it only costs $99 per year to be on the Mac Developer Programme which allows you to get certificates (even if you don’t deploy on the App Store), it’s something serious developers should consider strongly.

    SMJobBless is ideally suited to installing daemons, but it’s perfectly acceptable to install tools which run simple one-off tasks too. When setting up the plist for the helper, you specify that it is ‘OnDemand’, with no ‘KeepAlive’ which means it’s not started by Launch Services at startup, only when a Unix socket is opened, and shuts down very quickly if there’s no activity. Unfortunately the SMJobBless example doesn’t do anything except to show you how to install a tool, it doesn’t tell you how to implement that tool to do anything useful, or how to call it from your main application.

    To see how to do that, you need to refer to BetterAuthorizationSample , which includes a re-usable library for this. Ironically though, this example uses AuthorizationExecuteWithPrivileges to install its helper (this example pre-dated its deprecation in favour of SMJobBless). So you have to remove all the code associated with installation; you won’t need it anyway since SMJobBless does that function better. You keep the rest which gives you a framework for implementing and calling the helper. So here’s what I did:

    1. Implement the installation of a privileged helper, based directly on the SMJobBless example. I needed to change the bundle IDs and the certificate CN’s to match my setup of course.
    2. Extend the plist files from SMJobBless to register the helper with a socket. This was basically a case of copying the settings from the BetterAuthorizationSample plists, which already does this.
    3. Bring in BetterAuthorizationSampleLib.c/.h to assist with implementation of the helper, and the code for calling it in the app, but remove everything in the ‘Installation’ section. This eliminates all the references to AuthorizationExecuteWithPrivileges – we’re doing the install with SMJobBless so don’t need that.
    4. Follow the BetterAuthorizationSample for the implementation of the helper, and the bit in the application where you call the helper to perform privileged operations.

    So in my case, the following happens when you click ‘Install Command Line Tool’ in SourceTree:

    1. A privileged helper is installed in Launch Services using SMJobBless. OS X checks the code signatures on both ends to ensure that the helper and the application asking to install it are valid (must be signed with my cert, and that cert must be issued by Apple).
    2. A connection is opened to the privileged helper over a socket which causes launchd to start it up
    3. I ask the helper via the BetterAuthorizationSampleLib to install the command-line tool in /usr/local/bin. As an additional check, the helper validates via ‘codesign -v  -R=”conditions”‘ that this tool is code signed with my cert (again, must be issued by Apple) – this is to prevent anyone else sniffing out this socket and trying to use it to install other things. If that passes, it installs the command.

    This is quite a long-winded process compared to just calling a ‘cp’ command via AuthorizationExecuteWithPrivileges, but it’s also a lot more secure, since a malicious person can’t alter any of the moving parts without invalidating the code signatures. You’re also insulated from future changes when inevitably AuthorizationExecuteWithPrivileges is removed entirely.

    I apologise for the lack of a pre-packaged example here – I haven’t had time to extract one from my own implementation yet. However, as described above if you start with the SMJobBless sample and add-in the BetterAuthorizationSample, removing from the latter everything associated with installation, you’re basically there. If I get chance later I’ll post a shrink-wrapped example.

    I hope that helps someone – I found there to be little information on this subject that was up-to-date, and lots of older information that was misleading so maybe this will save someone some time. Ideally, I hope Apple will combine the SMJobBless and BetterAuthorizationSample some time to produce a 10.7-compliant official example.

    Business Personal

    On being acquired

    A lot of you will already know, but SourceTree, a Mac client for Git and Mercurial I created over the last 18 months, has just been acquired by Atlassian. There’s a press release, articles on TechCrunch and VentureBeat, and an official FAQ on the SourceTree site. But this is my personal blog, and I’ve had a few requests for a personal angle on this, so here you go.

    I said in a previous post that in my experience, the best opportunities often come along when you’re not looking for them, and that was certainly the case here. I wasn’t even thinking about looking for acquisition opportunities for SourceTree – sure, the idea had crossed my mind as something I might want to consider eventually, but it certainly wasn’t an active line of thought this early in the product lifecycle. SourceTree had grown to become a viable business for me, and I was very much enjoying the process of just creating a software product that I used every day myself too.

    So, when the Atlassian opportunity came up, I wasn’t at all prepared for it, and I had to make some decisions. I was enjoying being master of my own destiny, and was managing just fine – so my initial knee-jerk reaction was to be very cautious. However, the more I thought about it, and the more I learned about Atlassian, the more I realised what a huge opportunity I’d be turning down, both personally and for SourceTree, if I said no.

    Any acquisition kicks off with a financial offer (don’t expect details, they won’t be forthcoming ;) ), but that’s far from the whole story. In my case I didn’t have any pressing need to sell, and I’ve learned from experience that being happy about what you do is extremely important. I also have a strong attachment to the products I create – that’s why I stayed with Ogre for 10 years and it was/is still a wrench to leave – and that’s the case with SourceTree too; not to mention that I’m a daily user of it myself. So if I was going to sell, it had to be to the right company who would look after it just as well, or better, than I did.

    Luckily for me, I discovered that Atlassian was about as perfect a fit for SourceTree as I could have asked for. Atlassian lives and breathes developer tools – that’s their entire product focus, which in itself is a good start. They’re investing heavily in DVCS tools – hence the 2010 Bitbucket acquisition and its recent enhancement to handle Git as well as Mercurial (which of course SourceTree does too) – again spot-on on the compatibility chart. I learned, particularly when I visited their HQ in Sydney, that everyone at Atlassian really ‘gets’ developers (well, most of them are developers after all), and care a lot about giving them good products. Development tools permeate the entire company – the CEO is a regular user of SourceTree, and people in marketing understand when you talk about version control. Even though they’re quite a big company now, it retains a startup feel. Then there’s their corporate values, which are very much in evidence when you talk to people there – things like “no bullshit” and “don’t f**k the customer”. And it’s not just on the wall, it’s really how people in the company make decisions. These are the kind of people I can relate to, and definitely the kind of people who can add a lot to the future of SourceTree.

    Another thing I found reassuring is that at no time was there any question that Atlassian would want to railroad developers into their own tools at the expense of others. Clearly Atlassian already owns Bitbucket, and SourceTree supports Bitbucket, GitHub and Kiln already. It was made abundantly clear to me that no-one at Atlassian took the view that restricting developer choice to favour Atlassian tools was a good idea. Their ideology is to make developer’s lives better by giving them choice, and of course they’re going to want to offer good Atlassian options in there, but if a developer wants to use an alternative, no-one is going to stop them.  The view I got from everyone was that giving developers a positive experience that reflects well on Atlassian, including giving them their own choice of integration, is far more valuable than artificially chaining them in. Obviously, I concur.

    My final reason was that while I really, really enjoyed creating and supporting SourceTree myself, the workload is quite high, and was increasing. It’s not a continuous death-march, but the availability requirements are very high – since I was developer, webmaster, sales, customer support and everything else all rolled into one, taking a day off was basically impossible. Making sure the website was still up, and making sure customers got a quick response to support calls, was a 24/7 responsibility. After a while, that gets tiring, even just checking on things all the time means you never have ‘proper’ downtime. A big advantage of joining Atlassian is that I get some extra backup. That’s good for my health & mental wellbeing, and I’m sure that will be good for SourceTree too long-term. I really didn’t want to start resenting SourceTree for preventing me having a proper holiday occasionally ;)

    So based on all these factors, I decided that the future for both myself and SourceTree would be better within Atlassian than continuing alone. I learned a lot along the way to this acquisition – dotting all the i’s and crossing all the t’s turned out to be more time consuming and stressful than I expected, so I wouldn’t say it’s a process for the faint hearted, but if you’re as lucky as I was to be approached by the right company, it can lead to a really great outcome.

    I’m still fully committed to developing SourceTree, like I was before, but now it has a more robust support structure around it. Taking things to the next level, both in terms of user base and features, is so much more practical now within Atlassian. I’m very confident that they’re the right company to take SourceTree forward – our thinking is very similar, I respect their values a great deal, and the people are great. My decision was a lot easier than it might have otherwise been because of this!