SteveStreeting.com

Ok, so a new clause in the Terms of Service for Apple’s newly announced iPhone OS 4 is understandably causing some consternation around the internet:

“3.3.1 … Applications must be originally written in Objective-C, C, C++, or JavaScript as executed by the iPhone OS WebKit engine, and only code written in C, C++, and Objective-C may compile and directly link against the Documented APIs (e.g., Applications that link to Documented APIs through an intermediary translation or compatibility layer or tool are prohibited).”

The common understanding is that this is a shot across Adobe’s bow, but also aimed at people creating emulation environments. Potential justifications for this could include performance concerns (given the new multitasking feature in 4), wanting to avoid shovelware ports from other platforms with no iPhone-specific features to make their platform stand out, or sheer bloody mindedness and wish to tie developers directly to their APIs to minimise the chance that they’ll deploy on competitors machines.

As a general principle, I don’t like this sort of thing – telling developers what they can and can’t do is stifling. But, I had to take away one positive from it – a company telling people to use native code, instead of the opposite which I’ve seen too much of lately. In recent years, the likes of Microsoft have insisted that developers use their intermediate VM layers to deploy on some devices (XNA, Windows Phone 7) – regardless that these environments have about 20 years less maturity (in terms of libraries and existing code) than what I already have in C & C++. Having them tell me that no, despite all these great battle-tested libraries that I’m used to using, instead I have to use comparatively immature ports and replacements of varying quality, just because they tell me so. That drives me nuts – sure, let’s throw away and re-invent hundreds of functioning & tested libraries just because…well, just because! They’re old and we’re new and awesome! Hmm.

So while Apple telling developers what they can and can’t use is still very wrong from a point of principle, I’m actually glad that someone is championing native code for once, rather than pushing a VM environment. I’d prefer they didn’t mandate anything at all, but I can’t deny a certain urge to fist-bump when native code was the one to get the seal of approval, after getting the impression from other companies that they’d rather no-one had access to the underlying workings of the machine. I like native code, there’s a certain purity about it – and maybe it’s like a sad old gear-head going on about how great the old V8’s used to be, but I don’t care Mostly it’s about my frustration with being forced to discard perfectly good native libraries and look for / build replacements for no good reason.

PS For the record, OGRE on iPhone isn’t affected by this new ToS because we’re 100% native, baby.

[edit]For those pointing out that C# and such eventually run on native code anyway – that’s not the point. The point is that on certain devices – XNA and Windows Phone 7 – you simply cannot use libraries that were not written in .Net originally, meaning that years worth of dev libraries are inaccessible and need to be (pointlessly IMO) rewritten in .Net. And yes, this is exactly the same as Apple are doing here (but in reverse), if you interpret it in the strictest sense that you’re only allowed to use code written in Obj-C, C and C++. I’m just taking a perverse delight in the fact that it’s C/C++ libraries from the last 20+ years that are on the winning side for a change, instead of being the ones that are excluded (which frankly I’m completely sick of).

The trouble with developing a project which is not only cross-platform, but is old mature & wizened enough to have users dotted across a whole history of incompatible versions of the same tool, is that you end up having to maintain a ton of project files. Linux makefiles, 3+ versions of Visual Studio, Code::Blocks, XCode, Eclipse – it all gets a little unmanageable. I’ve been meaning to look at cross-environment build managers like CMake for a long time, but the time investment required to port & test, let alone learning how to do it, was daunting enough that it never seemed to make it to the top of my TODO. One of the problems was that despite CMake being well respected (projects like KDE and VTK use it), the documentation is a little impenetrable – most of the online stuff being a pretty dry reference that’s difficult to get started with. There is a book, but it’s hard to get hold of and the prices are just crazy even for a technical book – Amazon sellers were asking 70 quid for it used! As such I was never sure if it was going to work out and didn’t really have the spare time to try it.

But, as is so often the case, here’s where the open source community steps in and helps out. CABAListic (aka Holger) has been working really hard not only to get OGRE building with CMake, but doing a damn fine job of it too, with the kind of attention to detail that really makes all the difference. He’s done all the heavy lifting, and by looking at his work I’ve been able to get up to speed fairly quickly, both with the principles and some good practices, and I’m really starting to like CMake a great deal. Right now, it exists in parallel with the other builds in trunk, but relatively soon we’re expecting to make CMake the primary build system – I’m even skipping updating the old project files for the new paging component I’m writing right now and going straight to CMake, and it’s really very nice to use. Sure, you have to edit text files, but they’re easy once you have a pattern defined, and the way CMake checks them before building & refreshes the generated configuration is pretty slick. And I certainly won’t miss poring through page after page of project settings in VS, sometimes for each build configuration, to manually tweak options for multiple projects.

As well as not needing to update so many project files (the exception is XCode which will be separate for now, since I’ve yet to figure out if CMake can do everything I need it to, given the oddities of frameworks and universal binaries on OS X), it’s nice to be able to give people an interface to creating custom OGRE builds (e.g. enabling threading modes, double-precision, static linking etc), and the ability to build them into external directory structures from one source release. Having the equivalent of a ‘configure’ step (as seen on Linux) everywhere is also very nice, for performing sanity checks on dependencies and the like. Also, the fact that the source build collates the final result in a separate location means we can finally unify the directory structures as viewed by 3rd parties between the SDK and source releases, which will smooth things for people moving between the two or publishing add-ons for people using both.  We can also provide CMake files to make it easier for people to locate OGRE when they want to build against it.

Doing something as major as replacing the entire build mechanism for a project is pretty significant, but I can already see the benefits. Huge thanks to CABAListic for getting this rolling!

I’m on the Qt (owned by Nokia now) mailing list since I have a commercial license for a client project, and I got a very interesting email today, telling me that on its release in March 2009, Qt 4.5 will be available under the LGPL.

This is really big news. Up until the current Qt 4.4, your only licensing options are a per-seat and per-platform commercial license (which adds up if you have multiple developers and multiple target platforms, which you will do if you’re using Qt anyway), or alternatively the free option which means you use it under the GPL – meaning all your own code has to also be GPL, with an exception that allows you to publish / use software under other open source licenses too, but nevertheless it all has to be public. There’s a pretty big chasm in between these two options that in my experience a lot of projects fell into.

So, offering an LGPL option from Qt 4.5 is a major development. Qt is a really excellent library, the best cross-platform, native code widget set I’ve seen, but the licensing cost often put people off; this changes everything. I’m actually quite surprised they’re going this far – I think just changing the commercial licensing structure to be a little less multiplicative would have made a lot of people happy, but the LGPL option really turns things upside down. Essentially it means that you get all the cost benefits of the GPL version, except there’s a ‘fire break’ of open source responsibility between Qt and your application, meaning a lot more people can consider it. Nokia’s stated aims are:

• facilitate wider adoption of Qt across industries, desktop, web and embedded platforms
• establish Qt as a de facto standard for application development
• receive more valuable feedback and increased user contributions to ensure that Qt remains the best-in-class, cross-platform framework
• extend Nokia’s existing platform commitment to the open source community

I’m pretty sure this move will achieve all of those goals. It may well take a little wind out of the sails of wxWidgets, which in my experience never seemed quite as polished as Qt, but the licensing was much more favourable – although of course the wxWidgets project is still a lot less ‘corporate’, so will continue to have an appeal from feeling a bit more open.

Interesting times anyway!

I’m going to risk being called a dunce for not picking up on this until now, because I think there are probably other people with this annoying problem too. When you’re debugging in VC++, by default raw pointers only display a single item when you expand them, like this:

Now, if you know this pointer is actually a series of items and not just one, you really want to inspect them all – you can add array indices (pFoo[n]) but this is awkward if you want to browse rather than cherry-pick single items. I’ve sometimes used the ‘memory’ view to get around this, which is especially useful if you’re looking at an array of bytes, but arrays of floating point values aren’t that readable via the memory view.

The way to force VC++ to display raw pointers as arrays is to add ‘,n’ to the end of the watch expression, like this:

And if you want to skip a bunch of items, just use pointer arithmetic as usual, such as (pFoo + 50),100 showing items 50-150.

If you knew this already and think I’m a muppet for only just figuring this out, bully for you (although why didn’t you tell me, you bastard? ). I’m hoping to help the poor saps like me who have found this clunky in the past.

Well, this had me baffled for a while. I’ve been beavering away on allowing custom, user-supplied memory allocators in Ogre, hopefully for inclusion in the upcoming 1.6, and I came across a very weird problem in OS X. The wrapper for customising regular allocations (ie new/delete as opposed to STL allocations, those are in a different std::allocator compatible class), looks like this:

template <class Alloc>
class AllocatedObject
{
protected:
    static Alloc smAllocPolicy;

Pretty obvious stuff, we clearly redirect new/delete operators via that allocator policy. We then go on to define some default allocators (which can be replaced), something like this:

typedef AllocatedObject<StdAllocPolicy> DefaultAllocatedObject;
typedef DefaultAllocatedObject SomeObjectAlloc;

.. and then in our first-party class definition:

class SomeObject : public SomeObjectAlloc
{

All fairly straight forward – all that typedefing is just a compact way of allowing per-class user configurable allocators while still keeping the type duplication to a minimum. I can also place those typedefs in such a way that they can be easily replaced just by user headers or preprocessor blocks. Now, obviously I need to instantiate that smAllocPolicy member, otherwise I’m in trouble. So, here’s what I initially tried to do, in a cpp:

template <> DefaultAllocationPolicy DefaultAllocatedObject::smAllocPolicy;

Seems ok, right? Again I’ve used the typedefs here so that the instantiation will work even if I changed the basis for the types. This worked fine on Visual C++ 2005, Ubuntu 7.1 and in release mode on OS X. But, in debug mode on OS X, I got this:

ld: Undefined symbols:
__ZN4Ogre15AllocatedObjectINS_14StdAllocPolicyEE13smAllocPolicyE
/Users/steve/projects/Shoggoth/ogre/Mac/Ogre/../build/Ogre.build/Debug/Ogre.build/Objects-normal/ppc/OgreEntity.o reference to undefined __ZN4Ogre15AllocatedObjectINS_14StdAllocPolicyEE13smAllocPolicyE
/usr/bin/libtool: internal link edit command failed

WTF? I double-checked that the .cpp was included in the target, that the .LinkFileList file included the right object file, that ZeroLink was disabled (not that that would have any effect but to delay any link errors until runtime anyway). I tried getting rid of the typedefs and making it explicit, no dice. It had me scratching my head for some time, I was convinced I was instantiating this static and the object file was getting linked, so why the problem? In the end, the solution was to to change the instantiation to this:

template <> DefaultAllocationPolicy DefaultAllocatedObject::smAllocPolicy = DefaultAllocationPolicy();

I’m really not sure why in debug mode I was forced to include an explicit construction & assignment of this policy class rather than it being constructed by value in place, as seems to have happened with the other compilers and indeed in release mode under OS X. I usually don’t hold complex types statically by value, so usually I’d have an explicit initialisation in code like this to init pointers to 0 or numerics to something reasonable, so I wouldn’t have tried this particular thing on OS X before, but I did not expect to have to construct a complex type manually (and copy-construct effectively, although that’s likely to be optimised out). The fact that it worked in release mode and on other compilers makes me wonder how idiosyncratic that behaviour is. Maybe I’m just too tired and need to read Stroustrup again. If anyone has a rational explanation, I’d be glad to hear it.

When you’re talking with some programmers, particularly younger ones, you can’t help but run into the ‘great language debate’ at some point or another. That is, that many programmers have a language which they feel is superior to all the others, and they’ll put up a ton of resistence should you suggest that they use something else. It happens in other areas too of course – preferred operating systems, databases, apps etc, but as coders the language issue always tends to come up most, closely followed by IDEs.

I’m increasingly finding myself thoroughly tired of those discussions. Over the years, I’ve used a lot of languages – the main ones in approximate order of ‘first encounter’ (excluding assembly, which I don’t count as comparable): BASIC, C, C++, COBOL, Smalltalk, Visual Basic, Java, Prolog, Javascript, Python, Perl, PHP, C#, Lua, Ruby, Actionscript. At one point in my career, I would have obsessed over things like the verbosity of COBOL and Visual Basic (with all those superfluous Dim, As and line continuation bits), how useful closures are, how C++’s template system exceeds ‘generics’ systems, or Perl’s regular expression integration. But over the years while I still have elements I like and dislike about one language or another, and appreciate it when some ‘nice to have’ features are present, I’ve come to the conclusion that none of it really matters. I reckon I could pick up just about any modern programming language and be productive and content with it, because at the end of the day, language nuances are mostly icing – they don’t fundamentally affect the development process at the macro level at all.

Of course, that’s not to say that you’d want to pick up any random language for a given problem; of course not. Each has its own strengths and weaknesses for particular tasks, and crucially most often that’s not even related to the language itself but rather the surrounding ecosystem; platform / operating system support, availability of libraries and frameworks, performance characteristics at build and runtime, integration with existing software etc. This fact is what makes language allegiance so inappropriate in fact, along with unqualified, generalised statements like ‘language A is so much better than language B’. As engineers, which all good programmers should be at heart, we should recognise that each tool has its strengths and weaknesses.

Personally, if I’m looking to do a system automation tool or other portable scripted system, I’d probably look at Python or Perl, or maybe Ruby (except that I’m not as familiar with it), because they’re suited to that kind of task. If I’m looking to do something very intensive where performance and memory footprint is paramount, C or C++ would likely be a good choice. For an ultra-simple web app, PHP might be fastest, or moving up a little Ruby (on rails) might be good. If I need to make a moderately complex Windows-based tool fast, C# might attract me. If I want to make a more complex portable web app, Java is likely to be my choice. Horses for courses.

I think many programmers expend too much emotional energy on the languages that they like. Languages are just a way of expressing the implementation – they’re the very lowest level consideration of any software project. Sure, it’s what the programmer works with every day so maybe they’re entitled to obsess over it a bit, but in my mind that’s short sighted. Really your choice of language should be based based on the surrounding environmental conditions (like requirements for portability, speed, memory usage, deployment, integration, framework needs etc), not preferance for given syntax, since most syntax elements can be expressed in different ways anyway. Let those needs guide / boundary your choice rather than your language preference.

I think it’s actually far more important to think about all the things that apply regardless of implementation language – things like design, use of patterns & frameworks, use of techniques to improve software quality & maintainability like aspect-oriented programming, dependency injection, test driven development, contract-based programming, iterative agile development, all that good stuff. In the end, all of those considerations will impact the quality and maintainability of your final deliverable far more than what language you used.

In short, I think we should stop loving our languages and see them for what they are – simple tools which should demand no more allegiance than spanners, screwdrivers and hammers. Making software is about so much more than the individual characters we type into an editor, and I think we risk losing sight of what’s really important when we’re geeking out over the language minutae – just making good software. I for one don’t consider myself a C++ coder, or a Java coder, or whatever – I make software, and to do that I use a bunch of different tools depending on the circumstances. I actually think the more flexible you can be about those tools, the better.

Visual C++ Express 2008 came out while I was away I see, which inevitably was going to draw calls for a VS9 version of our SDKs (I always buy the Pro version of Visual Studio myself, but I keep an Express version for testing too).

I’ve always been pretty relaxed about this particular version though, because unlike many previous versions it holds precious little for the native C++ developer. Sure, you can use Vista native controls, which will undoubtedly be useful for some people, and if you’re a purely Windows-targetting .Net user then of course the new features in that area are no doubt going to get you all excited, but for me I can only think of one feature that I really hankered for: multiprocessor builds for all files rather than just projects.

VS8 (aka 2005) does make use of multiple processors, but only between separate projects. If you have a solution with many projects in it, you can get some parallelism there, but typically you’ve got dependencies between projects which forces serialisation. For me, OgreMain was the bottleneck, but once that was built all the plugins and samples could then fire off separately. It helped, but it wasn’t perfect. VS9 introduces fully parallel builds of files which means every project can take advantage of multiple processors.

It’s a fair amount to pay for an upgrade just for that though, so I’d pretty much accepted that I’d leave the upgrade for a while and just live with it. However, the ever-resourceful Paul (tuan kuranes) linked in the Ogre Forum a plugin called MPCL for Visual Studio 2005, which adds fully parallel builds without upgrading. I installed it, and my build time for the OgreMain project (debug only) on my 2-core Core 2 Duo E6750 (2.66Mhz) dropped from 6 mins 45 sec to 4 mins 29 sec – ie shaving a third off the build time. Not double the performance, but not to be sniffed at either.

So that seals it – I won’t be upgrading to VS9 just yet. Those who use VS9 can (as I suspected) use the VS8 OGRE SDK binaries anyway, because the native compiler really didn’t change that much, in marked contrast to VS8. Of course you can only use the MPCL plugin if you use the Pro version of VS so I guess those using Express will upgrade to 9 fairly swiftly, so that’s their path unless they want to use VS9 OGRE SDKs created by third parties (I see those are already springing up – may host those if they work). Anyone reading the OGRE forums regularly already knew about this of course but I just thought I’d post it here too, because being away for 2 weeks I missed it entirely and only picked up on it because someone updated the thread today.

I used to use remote debugging as my primary form of debugging many years ago (we’re talking the early to mid 1990’s now), simply because it was impossible to debug code that was tinkering directly with the VGA registers to do things like Mode X any other way. These days I still find it useful although less frequently – I’ve used it to diagnose full-screen issues that couldn’t easily be tracked via simple log output, and at times I’ve used it to debug server applications (J2EE, via Eclipse that time) where the behaviour could not be replicated anywhere but a super-duper beast of a machine. However, I hadn’t set it up in VS2005 yet.

I have plans to set up a couple of extra machines in the office here in the near future to handle compatibility testing, one AGP machine and one PCIE (and one of them with an embedded GMA chipset hopefully), and I already have my wife’s machine for ATI testing, so it was likely I was going to need it again sometime. It came to a head in that I did some work on sRGB support for OGRE recently as part of a contract project I’m involved with, which worked on my 8800 but is borked on ATI machines in GL for reasons unknown right now. It’s a pain to go backwards and forwards with new binaries using simple logging to diagnose so I decided to get remote debugging set up again.

The basic instructions can be found on the MSDN site. It’s pretty much the same as my experience with older versions, except they’re a bit tighter on security now (but I’m on a small  LAN so I turned all that off and ignored the warnings). However there’s one sticking point – the Visual Studio 2005 Debug CRT. The remote machine won’t have the debug runtime unless it has VC installed (unlikely) so you can’t run full debug builds remotely without getting the debug runtimes installed, or changing the way you build your binaries which is undesirable. The debug runtimes are now (of course) using the side-by-side assembly setup, so you can’t just drop the msvcr80d.dll et al in the local folder, you have to install them (unless you do it as a private assembly which is not ‘officially’ recommended and again requires build changes). Whilst it’s easy to install the release CRT using the redist package, the debug runtime has no redist package because, well, you’re not allowed to redistribute it. Internally for testing it’s allowed but there’s no prebuilt package to help you get it set up.

You can supposedly do it manually from the merge modules by running this on the remote machine:

msiexec /i Microsoft_VC80_DebugCRT_x86.msm

Where the .msm is obtained from your c:\Program Files\Common Files\Merge Modules folder. However, I tried that and it didn’t seem to work – it appeared to set up the appropriate folders in the WinSxS area, but looking at it, it did not appear to follow the redirections to the newer DLL versions set up by VC SP1, probably because the .msm didn’t change with SP1 but the external profiles did. That approach just left me in familiar ‘The application failed to initialise’ territory.

So instead, I built a simple Setup and Deployment Project on my dev machine, added the Debug CRT from Project > Add > Merge Modules, built it and ran the resulting setup program on the remote machine. Hey presto, I can now hit F5 and see OGRE fire up on my wife’s machine across the room, pause, debug and resume it all without moving from my chair. Splendid.