Tag Archives: Development

Development Tech Windows

Friends don’t let friends use ClickOnce

I’ve had a fair amount of experience with Windows-based installers in the past, including non-Microsoft Installer based systems like NSIS and the open source WiX, but most of the time I’d been working with one-off installers for native code projects, like the Ogre3D SDKs. I pride myself in not pre-judging the best toolset to use for any given problem – which is why I switch languages a lot – so when I came to write SourceTree for Windows, which is based on .Net, I checked out the landscape again.

I discovered that ClickOnce was Microsoft’s recommended approach to .Net application deployment , and had the advantage of including both the installer tech and an auto-update system, together with Visual Studio integration, a simple deployment model and so on. I’d seen it used on a couple of other products, and during testing both internally and with our batch of beta testers all seemed to be well, so that’s what we went with for the release.

Hindsight is a wonderful thing, and it turns out that ClickOnce has a number of problems which only really show up when you use it at scale, by which time it’s a bit too late and you can find yourself with a lot of support requests. Here are some of the issues:

  1. No support for proxy servers
    It turns out that ClickOnce just doesn’t work at all with secure proxy servers which can’t authenticate automatically. It also turns out that a lot of people have this kind of setup. There’s a (nasty) workaround for the auto-update system, but this can’t work for the first install. Microsoft have known about this problem for some years, but have decided not to bother fixing it.
  2. Packaging format is just a folder of many files
    ClickOnce puts all of its files in a folder per version and just tacks ‘.deploy’ onto the end of each. Simple, but it has many problems:

    • Some corporate firewalls will detect that some of those files are executable binaries, and will flag them as dangerous. Some will totally block the file making the install/update fail with a download error, some will ‘sanitise’ the file and send that on, making the install fail later with an ambiguous hash mismatch error.
    • If you use a CDN to distribute your installers, distribution of many files is very difficult to make atomic, potentially leading to temporarily incomplete results being visible
    • Having the download (first install and update) as a multi-file affair makes failures due to interrupted connections etc much more problematic, vs just resuming the download of a single package.
  3. You can’t separate the update metadata from the update itself
    That is to say, ClickOnce forces you to use a single URL for both the metadata where it finds that there is an update, and where it gets the actual update from (the installation files are in a subfolder). This might seem sensible, but in practice it’s very limiting. If they’re separate, you can point the metadata URL at a location which you can update in-place and which will never change, and you can put the update files themselves wherever you like (a CDN usually). ClickOnce forces you to put both in the same URL, which is limiting when it comes to reorganisation, and also it can mean you have to put the metadata on the CDN too, and CDNs don’t often like files that update in-place. It’s just an awkward and inflexible system.
  4. Random weird sh*t in a small percentage of cases
    We’ve found over the last couple of months that occasionally a user’s machine refuses to install or update despite the ClickOnce installer being fine for everyone else, and in circumstances that were not explainable by the other issues listed above. It’s not that common, but even if it’s only 0.5% of cases the numbers were starting to add up. We often managed to work around the problems manually, but it’s just not something you want to spend your or your customer’s time on.
  5. Other niggling limitations customers don’t like
    None of these are deal-breakers in their own right, but combined with everything else they’re worth mentioning:

    • You can’t choose where to install the product – ClickOnce chooses an esoteric location for you
    • ClickOnce’s reliance on the logged in user means you can’t install once for everyone on a PC, or distribute a bulk install as admin
    • No offline installer – while you can zip up the deployment it didn’t work for everyone

So, all these things have been building up , tempered with workarounds but nevertheless a constant thorn in our side. In the end, we had to make the call that ClickOnce just wasn’t salvageable as a deployment tech to use at scale. Instead we’ve moved to a standard Microsoft Installer based tech stack, using Advanced Installer to build the installer and the update system. I really liked Advanced Installer because not only does it address all the issues above, it saved me a ton of time over building something myself with WiX (and having to solve the prerequisites bootstrapper and auto-update system), and was also just complex enough to do what I needed; it’s a pro tool but doesn’t require 3 months training to master it, you can pick it all up in a few days.

I wrote this blog post because it’s what I wish I’d been able to read before I decided to use ClickOnce. It’s one of those technologies that looks fine on the surface and in limited testing but then reveals a bunch of horrid edge cases and random unreliability issues  in the wild, just when you don’t need it.

Development Personal

Why I’m a software developer

How often do you stop and think about why it is you do what you do for a living? Maybe it’s a mid-life crisis thing, but of late I’m acutely aware of the finite nature of time, and that there are an infinite number of ways I could spend that time. I’m also aware that ‘software developers’ are a quite diverse bunch of people, despite the persistent stereotype of math geeks huddled around technical toys talking in obscure acronyms (OK, we do that too). So I put some thought into why I choose to spend my time making software.

For me, it’s really simple: I like making things that people enjoy. That’s a pretty broad definition, but creation and connection is absolutely at the core of my motivation. It’s not really about the technical or logical challenges for me; at least, not any more – that might have been more of an issue earlier in my career. There’s something indescribably satisfying about creating something from nothing, sculpting and refining it from an image in your head into a functioning, tangible product. It doesn’t really matter what it is, just that it didn’t exist before, and now it does, purely because of your will. “I made that” is a satisfaction universal to all languages and cultures. That I’m a software developer rather than a sculptor, writer, musician or painter is down to a combination of circumstance and natural tendencies, but I don’t think my motivation is limited to this technical sphere at all. In fact, I think we all have this creative spark, it just gets drummed out of a lot of us after childhood.

I also think that you have to make things that speak to you first and foremost. Obviously you hope they will resonate with others too, and it’s pretty much guaranteed that they will (it’s just the degree that’s variable). I’ve made products entirely for other people before, indeed I spent a large portion of my career doing that, and it’s a bit of a lucky dip whether that turns out to be enjoyable and fulfilling or not. I’m at my happiest when I’m scratching my own itch, eating my own dog food, and building a community of people who feel the same way. It’s where both Ogre and SourceTree came from, which are the pieces of work I’m most proud of, and are also the most successful products I’ve created so far. That can’t be a coincidence, right?

This may sound like woolly, new-age thinking, but what feels right normally is right. I’m convinced that when you spend your time doing things which sync up with who you are as a person, better outcomes are more likely.  Have you thought about why you do what you do lately?

    Cocoa Development Objective C OS X

    Follow-up: OS X privilege escalation without using deprecated methods

    I posted a few months ago about the problems I’d encountered with performing privileged actions from a Mac OS X app – in my case, installing a command line utility in /usr/local/bin – and that all the examples of this that I’d come across used an approach which was now deprecated. You can find my original post here:
    Escalating privileges on Mac OS X securely, and without using deprecated methods.

    I had failed to produce a shrink-wrapped working example to go with the discussion, primarily because extracting it into a standalone example would take a while and I made the post a couple of days before I went on holiday. I also didn’t know whether anyone else actually cared about the subject enough for it to be worth me doing it!

    Well, perhaps I should have known better, because I’ve had quite a few requests for such an example since then :) I finally got around to doing this at the weekend – and actually when I came to do it I understood why people had pestered me for it, because it took me a while to get things configured just right in a fresh project! Mostly, it’s that there are quite a few things that can go wrong outside the code, both in the project settings and the plists because of the code signing requirements.

    So anyway, here’s the project: PrivilegedHelperExample on Bitbucket. Please make sure to check the ReadMe.txt – despite being shrink-wrapped, you will need to add your own code signing identity before you can compile the code, and you will need to reflect the name of your certificate in a few places, which I’ve listed. I’ve also tried to point you at the relevant pain points you may encounter when replicating the result in a different project.

    The majority of this code is just the Apple example code from BetterAuthorizationSample and SMJobBless, grafted together, de-duplicated and tweaked. All the changes I made can be considered public domain.

    Enjoy!

    Business Development Tech

    Why ‘software engineering’ is a misnomer

    These days I’m a free agent, and I’m lucky enough to be able to choose what projects I work on, but in a past life, I was what I suppose is properly referred to as an ‘enterprise software developer’. Yes, I once functioned in an environment where terms like ‘mission-critical’, ‘project life-cycle’, ‘stakeholders’ and ‘change management’ came up quite a lot. I’m grateful for the experience I gained over 12 years of doing that, but I’m also very glad to be free of it now.

    One term which was bandied around a lot in this environment was ‘software engineering‘. The implication of this term, and the expectation of many people using it, is that designing and creating software is very much like physical engineering disciplines, such as erecting a bridge or building a block of flats. When a software project starts to go off the rails, and fingers are starting to be pointed, a common question is why large physical engineering projects can come in on time an budget, when most software projects beyond a certain size don’t? Surely if the people who create software claim to be ‘engineers’, they should be able to adhere to the same standards of other engineering professions?

    At face value it sounds like a fair argument, and I’ll be the first one to admit that software engineers are very often not as rigorous as others bearing the title. But there’s a good reason – the belief that software projects are akin to building bridges is completely misplaced, and it is a belief which too often leads people to think that software projects should be run using techniques such as fixed prices, up-front design, and waterfall life-cycles. This is despite decades of case studies indicating that such techniques rarely work, and are more often a collective delusion adopted because the reality – that you’re never going to be able to predict a large software project accurately up-front – is too terrifying to be contemplated. The term ‘software engineering’, while indicating some of the responsibilities of such a person, is an incomplete definition of what building software actually entails, and just reinforces these misconceptions.

    So, why is making software so different to building a bridge?

    1. Software is intangible.
      You can see and touch a bridge, it has a physical presence you can measure and its purpose is absolutely obvious to any observer, as are its success and failure conditions (getting people across a gap without falling down). Even though there are parts of the bridge the user will never see, their purpose can be easily unambiguously defined – carrying utility cables/pipes, providing maintenance access, etc. Software is intangible, and even the part that is seen by those defining its purpose is often hard to evaluate until it can be seen and used directly, never mind the internal working processes. Evaluating and defining software ahead of time is like trying to describe a dream seen through a keyhole – incomplete and everyone has a slightly different impression, and regardless of how much you try to write it down, you can’t capture or communicate its essence completely to others, or guarantee that they interpreted it the same way.
      People are just naturally very bad at defining and evaluating intangible things. There’s no point pretending they can with 300-page requirements documents, you’re just wasting everyone’s time – accept that people won’t know what they like until they can play with it – iterate!
    2. Interaction complexity
      There’s only a small number of ways to use a bridge, and they’re each quite straight forward. Yes, there’s much complexity to the structure of the bridge itself, but that really doesn’t matter to a person driving or walking across it – that doesn’t need input or feedback from people except for whether they like the look of it.
      Software is basically defined by complex interaction, which is often context- and data-sensitive. The number of variations mean that even if every one is bug free (yeah, right), there’s the question of whether it’s solving the correct problem, which is a function of people and their interpretation / opinion. While building projects can be complex, those complexities are most often a function of the engineering challenge, whilst most complexity in a software project is a function of the involvement of people, which are far more difficult to pin down.
    3. Change is a constant factor
      Change happens everywhere of course – maybe the bridge builder discovers that the ground isn’t as firm as they thought, or suddenly someone wants to make an extra deck available for trains. Big disruption, and an effect on time and budget – all understandable. The problem when it comes to software is that the perception of change is very different. If suddenly you have to sink massive extra foundations or double the amount of steel you need to use, people understand the magnitude of that intuitively and accept the effect on the schedule. Changes in a software project tend to be perceived as ‘just a little tweak’ by those requesting it, which tends to mean tolerance for schedule impact is lower, and indeed the potential for interaction between changes in unexpected ways is often overlooked. Now, I’m not for a second advocating that change should not be allowed, but often the mechanisms that have been established for managing a project – up-front estimation, fixed feature sets and so on – prevent the efficient integration of change, or at least obscure its impact to a point where things become critical. It’s important to appreciate that change is more common and expected in software projects, by nature of the issues raised in the previous points, and therefore setting your expectations based on a discipline where change is less common is not a smart move.
    4. Lack of hard limits
      Building  a bridge involves respecting some hard, unchallengeable limits. As a wise man once said “You canna change the laws of physics, Cap’n!”. When you’re building software, however, there are very rarely any unbreakable limits, and even the soft limits that there are tend to be invalidated very regularly as new technology comes on stream. No-one can realistically stand up and say ‘this is impossible’ when faced with a request, because almost nothing is impossible. Hard limits in physical projects make all stakeholders ‘get real’ – you can’t use more land than you have, you can’t build a skyscraper out of paper, there are non-negotiable rules for safe load-bearing and so on. Barring some really crazy and ground-breaking structures (which BTW, rarely come in on time & budget), these limits ground everyone involved in the project to realistic expectations. In a software project, reining in those feature requests, or those ideas that the implementors have for a new tech they could use, is much more abstract affair, and everything is up for debate.

    All these things put together mean that if you think you can use the same kinds of techniques to run a software project as you would use for a building project, you’re kidding yourself. They’re so very different in fundamental ways, it’s like trying to catch a shark with a shrimping net (because it’s just a fish, right?).

    So please, stop with these delusions:

    1. that estimating the cost of any non-trivial software project (that isn’t just a carbon copy of something else) is any more sophisticated than mildly informed guesswork.
    2. that it’s reasonable to think that a fixed-price can be realistic for any software project longer than a couple of weeks
    3. that ‘change control’ is the about exceptions, rather than the norm

    Of course, agile approaches attack these particular delusions rather well. But try to get those adopted in very large organisations where the ‘software is like a bridge’ mentality still thrives :)

    Business Cocoa Development Objective C OS X Personal

    SourceTree, your Mac Git & Mercurial GUI, is 40% off this week

    Since I’m trying to spread this news as far and wide as I can, I might as well say it here too :)

    Since the approval light just went green on the Mac App Store, I’m happy to announce the launch of SourceTree 1.2! In celebration, I’m having a crazy-bonkers 40% off sale just for one week, so get it while it’s hot!

    There’s loads of things that are new or improved in this release, but here are the headlines:

    • Support for GitHub, Bitbucket and Kiln APIs, so you can see your hosted projects inside SourceTree, clone from them, link them as remotes, and even create new projects if you want.
    • Streamlined and polished user interface – I specifically dedicated a lot of extra time in this release on making SourceTree easier on the eyes, and to streamline the layout and workflows better.
    • Performance - I thought SourceTree was already pretty fast, but I managed to find quite a few more places to trim the fat, and also parallelised more activities to make things feel more responsive. Everything feels snappier, and complex repositories benefit especially.
    • New Sidebar - I had previously resisted the need to emulate iTunes here, but once I had implemented it, I had to admit that I was wrong, and in fact this worked great. Provides lots of shortcuts to navigating and operating on branches, tags and remotes.
    • Stashing and Shelving - oft requested, now delivered :)
    • Customise Git and Mercurial – you can now use your system Git / Mercurial instead of SourceTree’s standard versions (which have been updated), and enable additional Mercurial extensions (at your own risk).
    • French and Japanese translations – local versions for our friends in far away (and not so far away) places, likely to be more to come in future. Big thanks to tuan_kuranes and mzch for their help with these two!
    • And the rest – just lots of little refinements too numerous to list. Examples: copying text from the diff panel, ‘git commit –amend’ support, close branches in Mercurial, switch tracking branches in Git

    It’s quite a big update – one user remarked to me that they’d normally expect developers to charge an upgrade fee for something like this, but like all other SourceTree updates this is free to existing customers. I have no plans for any paid upgrades for some time yet, I just want to keep making SourceTree better, and hope that more people come onboard. Maybe it’s my open source background, but I like to keep iterating and continually improving things, based on what I want to do (I’m a daily SourceTree user myself), and on what people tell me they’d like to see. SourceTree 1.2 certainly won’t be the last update by far :)

    When I look back 6 months at SourceTree 1.0, it’s incredible how much better it is as a product now, both visually and functionally. I’ve learned a ton of things while I’ve been developing it, and I continue to learn more all the time, and I can’t think of anything I’d rather be doing right now. Also, my wife Marie re-designed many of the icons for 1.2 (and I think you’ll agree they’re a lot nicer) – that was fun to do as a joint project, even if I am a picky ‘customer’ ;)

    I hope you enjoy the new release!

    Business Development OS X

    Mac user base by country: my figures so far

    As many of you probably know, almost a year ago now I decided to take the plunge and move my primary development activities to the Mac. I taught myself Objective-C, got properly to grips with Cocoa at last, and started a new Mac OS X-specific project which would eventually become SourceTree, learning a ton along the way (a process which is by no means complete!).

    Happily, things have turned out very well – SourceTree continues to sell, reassuring me that there’s enough interest out there for me to keep expanding and improving it (I’m looking forward to getting the next major release in people’s hands soon), and I’ve also been getting some Mac/Ogre-based contract work which I’ve enjoyed a great deal. In short, my Macbook Pro and I are now pretty inseparable, Windows 7 is powered off 99% of the time, and I saved hundreds of pounds by not upgrading to Visual Studio 2010 ;)

    Out of interest, I thought I’d share some of my SourceTree sales information, in terms of the country distribution. Mac use is typically associated primarily with the USA, and while that’s certainly reflected in my absolute numbers, there’s some quite interesting figures revealed when you take into account population size. SourceTree is aimed at developers of course, so all numbers reflect this audience alone (and of course those that chose to buy it) but in practice I suspect that the proportions of developers to non-developers is fairly uniform in most developed countries.

    So, firstly the absolute distribution:

    No surprises there, the USA is the single largest source, followed by Germany and the UK (who are constantly scrapping over second place!). To me though, Switzerland stuck out as the most interesting, because it’s up there in 4th place yet has a relatively small population (under 8m). So I wondered – what would the chart look like if I scaled it by the population size? Here’s the result:

    And there you go – as expected Switzerland jumps right to the top, and to my surprise Luxembourg and Denmark are up there too, beating the UK which I expected to come in second. Quite a lot of European countries are punching above their weight in per-capita Mac development, if SourceTree sales are any indication. In fact, on a per-capita basis, the USA is only just sneaking into the top 10, despite it being by far my best overall customer in sheer sales volume.

    I’m aware that scaling by population isn’t all that scientific, since it is sensitive to the proportion of non-developers (and even non-computer users), but as I say, I think in developed countries at least, the comparisons are reasonably valid.

    So the perception that Mac development is more popular in the USA than elsewhere may be inaccurate, based on my numbers at least (which of course are not massive in the grand scheme of things, but still a statistically relevant sample I think). Sure, simply because of the sheer population size of the USA it’s bound to dominate anyone’s sales numbers, but if you asked the question ‘of 1000 randomly selected developers in a country, what percentage are using a Mac?’, the result may not be skewed in the way you might expect. It was surprising to me, anyway.

    I wonder if anyone else who has been doing this for longer has had similar results?

    Business Cocoa Development Objective C Personal Tech

    Introducing: SourceTree

    I’m pleased to announce that I’m finally ready to make my first fully-fledged commercial Mac OS X application available to the world!

    SourceTree is a user-friendly Mac OS X front-end for Mercurial and Git, the two most popular distributed version control systems used today. The goal was to create a single tool which could deal with both systems efficiently, and to give a developer quick and intuitive access to the things (s)he needs to just get on with building software.

    I thought I’d answer a few background questions on this that I get asked on occasion:

    Why Mercurial AND Git?

    Other apps tend to concentrate on just one version control system, so why am I supporting two? Well, as a developer I’m regularly coming across projects from both sides of the fence, and in practice I find I need to use both fairly regularly. I personally chose Mercurial for my own projects (and discussed why here), but I still use Git when dealing with other projects, and spend a fair amount of time hopping between the two. It struck me that even though they have their differences, they are both based on the same distributed principles, so having to use two separate tools was just unnecessary. I wanted a single tool which provided a common interface where that made sense, while still exposing the things they do differently where that was useful too. SourceTree 1.0 is my first attempt at that.

    Why only Mac OS X?

    There were actually multiple reasons for this choice:

    1. I wanted to learn Objective-C and Cocoa on a real project
    2. I know from experience that designing for multiple platforms can be a distraction, with more time spent on compatibility issues, and less on functionality – and that’s before you even consider the compromises  you have to make, particularly on UI conventions which are far from uniform across platforms. I’ve been a multi-platform developer for more than 10 years, and for a change I just wanted to focus on the end user results and nothing else. I’m aware that schedules slip very easily when you overcomplicate, and I’m already supporting multiple DVCS systems (something I consider to be an important feature point), so I deliberately chose to keep this element simple.
    3. Mac OS X has become my own platform of choice for most things now. The combination of stability, user-friendliness, Unix underpinnings and well designed hardware match my current needs perfectly. I’m done with the ‘some assembly required’ PCs that I loved tinkering with over the past 15 years

    What about Subversion?

    A few people have asked me if I plan to add Subversion support too. I actually did intend to originally, until I realised how much time it was going to take to just do a decent job on Mercurial and Git. Within the time constraints, I focussed on the subject areas that I felt I could contribute most to – there are already quite a few Subversion tools out there for Mac OS X, but Mercurial and Git are much less well served, so that’s where I focussed my efforts.

    I still have Subversion support tentatively on my work plan, but it’s not top of the list. I think it’s better to do your most important features well before diversifying. Plus, there are problems with Subversion – it’s very, very slow compared to Mercurial and Git, so to match the performance in SourceTree of things like the interactive searches and dynamic refreshing / log population I’d probably have to do a ton of extra caching just so the user wasn’t sat tapping their fingers.

    Edit: I made my decision on this: I don’t plan to support local Subversion, but to support operating with Subversion servers with Mercurial and Git locally via hgsvn and git-svn.

    Why didn’t you make it open source?

    Sorry folks, while I love contributing to open source (I’ve done a bit on SourceTree too, sending a patch back to BWToolkit), making it work as a business is very hard indeed. I half-killed myself trying to combine being an open source project leader and doing other commercial activities at the same time, so now I’m trying a more traditional approach. One thing I learned in the last few years is that there are some sectors & application types where being an open source maintainer is very compatible with also running a business based on that project, and there are others where you can really only do one or the other simultaneously without flaming out. Sucks, but there it is ;)

    What’s Next?

    I have a public, official roadmap for SourceTree and encourage users to suggest things they think should be on there, via the support system. I learned from running an open source project for 10 years that being open about your plans can be a big benefit – users like to know where things are likely to be going, and often have better ideas than the developer on what could do with a bit more spit and polish. They can also tell you what’s important to them, which is crucial for prioritising – as developers we tend to get carried away with things we want to work on, but in the end, it’s scratching the customer’s itch that matters most.

    And while I’m really quite proud of SourceTree 1.0, there are plenty of features I’d like to continue to add, and definitely more room for some totally unnecessary beautification which I didn’t have time for in the first release. Hey, this is OS X ;)

    SourceTree is available now on a 21-day trial license. Go get it already :)

    Development Personal Productivity

    Work 2.0 – the interruptible programmer

    I’m 37, and I’ve been a (professional) developer for 16 years. You would have thought that in that time, I’d have figured out an effective work style which delivered the desired outcomes (code cut, products shipped etc) without causing detrimental knock-on effects – but, sadly, you’d be wrong. I think the style in which I practiced my craft for the first 15 years of my career was much the same as every other enthusiastic developer: you put a ton of hours in. 12-16+ hour days, evening and weekend coding marathons, pizza in the keyboard, crunch times, 3am debugging sessions where you just can’t go to bed because you can feel the source of that bug just beyond your fingertips, dammit, desperate last-minute sprints to deadlines where you manage to slot that last piece in, Jack Bauer-like, just before the world goes to hell. If you’re in the demographic I’m talking about, you’re nodding sagely, and probably grinning a little too, reminiscing on past trials and glories. This sort of crazy dedication is respected in our circles, and is pretty much expected of any developer who has claimed to earn their stripes.

    But, it turns out this kind of thing is not good for your health – who knew? Those of you who know me or keep up with my blog know that I’ve been dragged kicking and screaming away from my old ways, because of back issues that I initially ignored, then tried to cope with using token accommodations, and finally succumbed to in a big way. Being self-employed, this was a major problem. Crawling out of the pit I dug for myself took a long time and a lot of frustration – I read quite a few productivity books on the subject to try to find answers on how to keep working, and in the end found that the answers you mould for yourself tend to be the best ones. I’d like to share one of the things I learned along the way.

    But I’m ‘In The Zone’!!

    So, I want to talk about the biggest problem I encountered: concentration periods. I can’t sit at a desk for longer than about an hour at a time now; if I don’t get up and walk around, do some gentle stretching etc, at least this often, I’ll pay for it badly once I do move, and probably over the next few days too. I also can’t realistically work more than a standard 8 hour day without pain any more. The problem with this was that, as a programmer, the style which I developed over 15+ years involved getting gradually ‘Into The Zone’ and coding for very long periods at a time, uninterrupted. This is a common theme among coders, who like to shut themselves away for hours at a time, wear headphones to avoid distractions, have ‘quiet times’ and so on – and it’s also why we tend to react really badly when interrupted. Programming requires concentration, and concentration seems to run on a valve system – it takes time to warm up, and once it’s going, you don’t want to turn it off because starting it up again is a major hassle.

    I thought there was no way around this, and had begun to resign myself to just being less productive because of it. However, over the last 6  months in particular, I’ve discovered that, far from being an intractable problem, this ‘slow warm up, long uninterrupted focus time’ approach is to a large degree a learned behaviour, and it’s possible to re-train yourself to cope with things differently. It’s a little like when people learn to adopt polyphasic sleep patterns – it’s not that you can’t do it, it’s just that when you’ve become accustomed to doing things a certain way, changing that is initially very, very hard. But it’s not impossible, given the right amount of motivation and time to adjust.

    So, my goal was to acclimatise myself to many shorter work chunks during the day instead of a few very large ones, while still maintaining productivity. The key to this was to learn how to get back ‘In The Zone’ in the shortest time possible – much like the way polyphasic sleepers train themselves to achieve REM sleep more quickly. I’m mostly there now, or at least way better at it than I was, so, what techniques did I use to make this transition?

    1. Embrace interruptions
      This is less of a technique and more of a deliberate psychological adjustment which cuts across all the practical approaches I’ll cover next. Instead of being the typical coder who avoids interruptions at all costs, you need to accept them, and learn to manage them better. It’s hard – you have to try to set aside years of resisting interruptions and initially, until you adjust, you’ll feel like you can’t get enough done. Many people will probably want to give up, unless there’s something specific motivating them to push through it – for me, daily pain was a great motivator. My main message here is that the transition is just a phase, and that it is possible to be an interruptable programmer who still gets things done. But you have to learn not to fight against it, hence why this is the first point.
    2. Maintain context outside of your head at all times
      Much of the problem with interruptions is that of losing context. When you’re in that Zone, you’re juggling a whole bunch of context in your head, adjusting it on the fly, and maintaining and tweaking connections between issues constantly. Interruptions make you drop all that, and it takes time to pick it all up again. My answer to this was to externalise as much as possible, on as many levels as possible:

      1. Maintain a running commentary on your current task
        I am my very own chronicler.
        I write notes on what I’m doing all the time, whether it’s adding a comment line to a ticket, committing frequently and writing detailed commit notes (you do use a DVCS to make light commits more practical, right? ;) ) scribbling a drawing on (ordered) pieces of paper. This really isn’t that onerous, and in fact externalising your thoughts can often help you clarify them. Basically the guide is that roughly every 30 minutes, I should have generated some new piece of context which is stored somewhere other than my head. If I haven’t, then that’s context I’d have more trouble re-building mentally if I’m interrupted. It doesn’t take much time to do, and it has other benefits too such as recording your thought & decision process.
      2. Ruthlessly ignore tangental issues
        You might have noticed that in the last bullet, I used the words ‘current task’, singular. Not ‘tasks’. There is no such thing as having more than one ‘current task’ – there is only the one task you’re actually working on, and distractions.
        We probably all use bug trackers / ticket systems to track bugs and feature requests, but when you’re working on a ticket, it’s very common to spot a new bug, or identify an opportunity for improvement, or think of a cool new feature. How many of us go ahead and deal with that right away, because it’s in the area we’re already in, or it’s ‘trivial’, or it’s a cool idea that you want to try right now?  I know I did – but I don’t any more; any tangental issues not related to what I’m currently doing get dumped into the ticket system and immediately forgotten until I’m done with the current task, regardless of their size, relevance or priority. It sounds simple and obvious, and this might even be official procedure in your organisation, but I challenge most coders to say that they actually do this all the time. The benefit is that even the tiniest of distractions add an extra level of context that you have to maintain, which is then harder to pick up again after an interruption.
        For this to work, you need a ticket system which is fast, lightweight, and doesn’t require you to be anal about how much detail you put in initially. You need to be in & out of there in 30 seconds so you can offload that thought without getting distracted – you can flesh it out later.
      3. Always know what you’re doing next
        This is one from GTD (‘Next actions’), but it’s a good one. When you come back from a break or interruption, you should spend no time at all figuring out what you need to be doing next. Your ticket system will help you here, and so will the running commentary that hopefully you’ve been keeping on your active task. If you’ve been forced to switch gears or projects, so long as you’ve maintained this external context universally, you should have no issue knowing what the next actions on each item are. The important thing is to have one next action on each project. If you have several, you’ll have to spend time choosing between them, and that’s wasted time (see the next section on prioritisation). At any one time, you should not only have just one current task, but one unambiguous next action on that task. Half the problem of working effectively is knowing what you’re doing next.
    3. Prioritise Negatively
      I mentioned next actions in the previous section, but how do you decide what comes next? A lot of time can be frittered away agonising over priorities, and I used to struggle with it; I would plan on the assumption that I wanted to do everything on the list, and I just needed to figure out which I needed to do first. I discovered that I could cut the amount of time I spent on planning, and also get better, less ambiguous priorities by inverting the decision making process – to assume a baseline that I wouldn’t do any of the tasks, and assessing the negative outcomes of not doing each one. So instead of ‘which of feature A or B is more important to have?’, it became ‘Let’s assume we ship without feature A and B. What are the issues caused by omitting them in each case?’. It might appear to be a subtle difference, but having to justify inclusion entirely, rather than trying to establish a relative ordering assuming they all get done eventually, tends to tease out more frank evaluations in my experience.
    4. Recognise the benefits of breaks
      Much of the above is about limiting the negative aspects of taking breaks, but the fact is, that they have many work-related benefits too. I’m willing to bet that all coders have stayed late at work, or late into the night, trying to fix a problem, only to find that they fix it within 15 minutes the next day, or think of the answer in some unlikely place like the shower. The reason for this is very simple – extended periods of concentration seem productive, and can be on operational / sequential thinking, but for anything else such as creative thinking or problem solving, it’s very often exactly the opposite. Not only do tired minds think less clearly, but often the answer to a problem lies not in more extensive thinking down the current path which you’ve been exploring in vain for the last few hours, but in looking at the problem from a completely different perspective. Long periods of concentration tend to ‘lock in’ current trains of thought, making inspiration and strokes of genius all too rare. Creativity always happens when you’re not trying, and it’s an often under-appreciated but vital element of the programming toolbox. Interrupting that train of thought can actually be a very good thing indeed.

    There’s more I could talk about, but that’s quite enough for now I think. I hope someone finds this interesting or useful :)

    Cocoa Development Objective C

    Cocoa tip: using custom outline view cells designed in IB

    I only started learning Objective-C and Cocoa in mid-May, and for the first time I think I actually have a tip to contribute to the wider community. It’s about using custom cells in NSOutlineView, but those which you want to design inside Interface Builder rather than drawing manually.

    If you’re an iOS developer, you’ll be wondering why this deserves a blog post – it’s easy to do in Cocoa Touch! Well, yes it is easy on iOS, because Apple have specifically allowed you to design table view cells in Interface Builder. When targetting Mac OS X though, it’s actually pretty awkward, and here’s why: in Cocoa Touch, the class which draws the cells in a table is UITableViewCell, which is a subclass of UIView – meaning you can drag one onto the canvas in Interface Builder and lay stuff out right there. In Cocoa, in contrast, the cell is simply represented by NSCell, which is not an NSView subclass. This means Interface Builder will not let you play with them, you draw them by implementing drawWithFrame:inView: instead. I think Apple realised the problem with this design in time for Cocoa Touch but obviously felt they couldn’t break the existing Cocoa interfaces. There are also many differences between the instantiation of NSCell versus UITableViewCell – there’s only one NSCell for all rows in a table / outline view, compared to a type-indexed pool in Cocoa Touch.

    The problem boiled down to this: if the contents of your table / outline cell is non-trivial, or if you just don’t want to write a bunch of layout code, it’s a PITA to implement a custom look in NSOutlineView, such as the one in the picture, especially if you want custom controls embedded in it. For my current Cocoa app, I really wanted to design my cells (and I have 2 different styles for group levels and detail levels) in Interface Builder to save me hassle, including using Cocoa Bindings to hook up some dynamic fields within. Many internet searches later, and mostly the answer I found was that it couldn’t be done. Luckily, I’m too stubborn to take no for an answer, and eventually I figured out a way to do it. Details are after the jump, with an example project to show it in action.

    read more »

    Development OS X

    OS X Localisation: incremental genstrings and UTF-8 files

    I came across a couple of interesting issues when I came to do the first pass of writing the text for the user-visible strings I’d been setting up for a Cocoa app I’m writing (painfully slowly as I learn the nuances of the environment), and I thought I’d share them. Full details are after the jump, since I’ve embedded a large script in the post.

    The basic principle for text localisation on OS X is that, like most systems, you externalise your user-visible strings in string tables and reference them by keyed aliases in code – in this case using NSLocalizedString. Apple provide a tool called ‘genstrings’ which extracts all these into a template strings file called Localizable.strings which you can then populate per language – localised files are kept in folders called en.lproj, fr.lproj etc and helpfully they’re picked up by default like this.  So far, so good.

    read more »