SteveStreeting.com

I’m going to risk being called a dunce for not picking up on this until now, because I think there are probably other people with this annoying problem too. When you’re debugging in VC++, by default raw pointers only display a single item when you expand them, like this:

Now, if you know this pointer is actually a series of items and not just one, you really want to inspect them all - you can add array indices (pFoo[n]) but this is awkward if you want to browse rather than cherry-pick single items. I’ve sometimes used the ‘memory’ view to get around this, which is especially useful if you’re looking at an array of bytes, but arrays of floating point values aren’t that readable via the memory view.

The way to force VC++ to display raw pointers as arrays is to add ‘,n’ to the end of the watch expression, like this:

And if you want to skip a bunch of items, just use pointer arithmetic as usual, such as (pFoo + 50),100 showing items 50-150.

If you knew this already and think I’m a muppet for only just figuring this out, bully for you (although why didn’t you tell me, you bastard? ). I’m hoping to help the poor saps like me who have found this clunky in the past.

I posted recently that we were having some mipmapping issues with NVIDIA’s newest drivers, the 175.x series, on both Windows and Linux when using GL. Thanks to help from the nice chaps at NVIDIA these issues are now resolved for the moment - there does indeed appear to be a bug in some aspects of the hardware mipmap generation implementation in these drivers, but the workarounds exposed a couple of bugs of our own in software mipmap generation code - which hasn’t been used by Ogre on most modern hardware for several years, since we’ve opted for hardware mipmap generation for a long time.

One was that we had been retrieving the internal format of a GL texture after construction using glGetTexLevelParameteriv with GL_INTERNAL_FORMAT, in accordance with recommendations. However we made a mistake in that for cubic textures we were passing the type enum GL_TEXTURE_CUBE_MAP instead of the target enum GL_TEXTURE_CUBE_MAP_POSITIVE_X et al. Subtle, and for whatever reason it must have worked fine on older drivers, but the 175.x drivers became more strict and were throwing an error. Unfortunately our other sin was that we weren’t checking the error code after that call (after all, what could go wrong with something that simple? ) so we completely missed it, and the internal format, which we then used later on to generate software mipmaps, was nonsense.

As it happens, I’ve been trying out Graphic Remedy’s gDEBugger again recently - I’d previously tried it a couple of years ago and couldn’t get it to work properly with Ogre for some reason, and my post in their forum about it had gone without a resolution, so I let it expire without buying it. However, I really have come to miss having some way to easily diagnose GL issues, like you do with the D3D debug runtime (and various GL intercept / dump tools are ok, but generate so much data it’s hard to sift through it), and it’s clear to me that the error above may well have been highlighted by a tool such as this before the driver started to bork on it. So, I emailed GR to see if they’d be willing to give me an extended trial license and some assistance with resolving the problems I was having. They got back to me really quickly, and via our correspondence identified an issue that they will resolve in the next version of gDEBugger, but which I could also work around in Ogre for now. I haven’t had a lot of time with it yet but I intend to give it a workout over the next couple of weeks, and buy a permanent license if it turns out to be as useful as I expect.

I really appreciate being able to get help like this when I need it, I’m in debt to the people at NVIDIA (especially Kevin) and Graphic Remedy for the assistance. Thanks chaps!

Well, this had me baffled for a while. I’ve been beavering away on allowing custom, user-supplied memory allocators in Ogre, hopefully for inclusion in the upcoming 1.6, and I came across a very weird problem in OS X. The wrapper for customising regular allocations (ie new/delete as opposed to STL allocations, those are in a different std::allocator compatible class), looks like this:

template <class Alloc>
class AllocatedObject
{
protected:
    static Alloc smAllocPolicy;

Pretty obvious stuff, we clearly redirect new/delete operators via that allocator policy. We then go on to define some default allocators (which can be replaced), something like this:

typedef AllocatedObject<StdAllocPolicy> DefaultAllocatedObject;
typedef DefaultAllocatedObject SomeObjectAlloc;

.. and then in our first-party class definition:

class SomeObject : public SomeObjectAlloc
{

All fairly straight forward - all that typedefing is just a compact way of allowing per-class user configurable allocators while still keeping the type duplication to a minimum. I can also place those typedefs in such a way that they can be easily replaced just by user headers or preprocessor blocks. Now, obviously I need to instantiate that smAllocPolicy member, otherwise I’m in trouble. So, here’s what I initially tried to do, in a cpp:

template <> DefaultAllocationPolicy DefaultAllocatedObject::smAllocPolicy;

Seems ok, right? Again I’ve used the typedefs here so that the instantiation will work even if I changed the basis for the types. This worked fine on Visual C++ 2005, Ubuntu 7.1 and in release mode on OS X. But, in debug mode on OS X, I got this:

ld: Undefined symbols:
__ZN4Ogre15AllocatedObjectINS_14StdAllocPolicyEE13smAllocPolicyE
/Users/steve/projects/Shoggoth/ogre/Mac/Ogre/../build/Ogre.build/Debug/Ogre.build/Objects-normal/ppc/OgreEntity.o reference to undefined __ZN4Ogre15AllocatedObjectINS_14StdAllocPolicyEE13smAllocPolicyE
/usr/bin/libtool: internal link edit command failed

WTF? I double-checked that the .cpp was included in the target, that the .LinkFileList file included the right object file, that ZeroLink was disabled (not that that would have any effect but to delay any link errors until runtime anyway). I tried getting rid of the typedefs and making it explicit, no dice. It had me scratching my head for some time, I was convinced I was instantiating this static and the object file was getting linked, so why the problem? In the end, the solution was to to change the instantiation to this:

template <> DefaultAllocationPolicy DefaultAllocatedObject::smAllocPolicy = DefaultAllocationPolicy();

I’m really not sure why in debug mode I was forced to include an explicit construction & assignment of this policy class rather than it being constructed by value in place, as seems to have happened with the other compilers and indeed in release mode under OS X. I usually don’t hold complex types statically by value, so usually I’d have an explicit initialisation in code like this to init pointers to 0 or numerics to something reasonable, so I wouldn’t have tried this particular thing on OS X before, but I did not expect to have to construct a complex type manually (and copy-construct effectively, although that’s likely to be optimised out). The fact that it worked in release mode and on other compilers makes me wonder how idiosyncratic that behaviour is. Maybe I’m just too tired and need to read Stroustrup again. If anyone has a rational explanation, I’d be glad to hear it.

As a developer, there’s an polarity of choice when it comes to frameworks regarding whether you would rather lean towards more compact code where most things are implied, or more verbose code where most things are laid out explicitly - most obviously in the web development sphere.

Ruby on Rails is continuing to be a hot property simply because it’s quite elegant and very compact for the functionality it delivers. If you’re making a relatively simple web-based, database-driven application there’s probably no faster way to do it right now - although significant concerns about practical scalability remain, perhaps Ruby 1.9 will make things better once Rails supports it. However, in order to deliver all this power, Rails makes a lot of assumptions about the way a typical user is going to want to organise things, such as driving the structure of the model from an existing database structure and auto-wiring views and controllers etc. It means that when you look at some Rails code for the first time, it can often be startlingly short and leave you wondering where the rest of it is.

In terms of rapid development, this is great. But, brevity does come at a cost, which is that when you look at the code, you’re not seeing the whole picture. That’s always the case to some degree when you’re working with OO or layered software, in that implementation hiding and the ‘onion skin’ effect mean that implementation details are always hidden somewhere. However, one difference is that in a less adaptive application development structure, the base implementation, the part you don’t necessarily see unless you start to heavily dig, is a constant - it’s invariant across all applications. Anything specific to your application will be there in the code, particularly the ‘assembly’ aspects - how it’s all tied together. When you bring in implicit configuration too, this is no longer the case - there’s some ‘magic’ in there that makes things ‘just work’, which is very cool, but it also kind of makes me feel a little nervous. Because what if it ceases to ‘just work’? How do I figure out what went wrong? If I’m looking at code someone else developed a year ago, will I be able to piece all the potential contributing factors together quickly, compared to developing a new app myself?

I had an experience a number of years back with a design tool that let you create complex (fat) GUI applications by dragging, dropping & wiring components visually. It was really impressive when building new things, and made it possible to create complex GUIs with appserver integration incredibly fast, writing almost no code at all. As time went on though, we found that while we could build things quickly, coming back to the more complex designs later to resolve issues became a bit of a nightmare, simply because picking apart all the ‘magic’ bits that were happening was problematic. When it worked, great, but when it didn’t it took a lot longer to figure out why. We found that in many ways it was like running into a tar pit - at first you were flying through it, but after a certain level of critical mass and complexity had been achieved you had a tendency to get bogged down. The metrics of effort versus output over time were not favourable, considering that a typical application spends far longer in maintenance than it ever does in initial development. From then on I’ve always been a little cautious when presented with tech that purports to make building applications fast, because I want to know that it also makes maintaining them fast too. I don’t have a handle on what Rails applications are like to maintain over a significant amount of time because I’ve never done it, obviously the ‘build’ phase is fast, but does it suffer from the problems I had with visual design tools to any degree, or does the high level of convention mitigate that (assuming your application fits into that slot neatly)?

On a purely developer level, I like things to be quick and no more verbose than they need to be, but I also dislike having to be too trusting. I’m never really comfortable when something ‘magically’ happens in technology and it’s less than obvious as to why - I tend to prefer a slightly more explicit line from cause to effect, simply because if I feel that if I can’t obviously see why something worked, how point A got connected to point B, I have a blind spot  which may well come up to bite me later. It’s also comforting to browse some source and see all the pieces slot into place, without any glaring gaps where ’stuff happens’. On balance then, I tend to prefer systems that are explicit, but at the same time I don’t advocate mindless verbosity - there’s a balance here. I quite like Spring in that regard, there’s a lot of prebuilt functionality packaged in there, but while the implementations are wrapped and out of the way it’s all wired up explicitly (unless you use auto-wiring, which I avoid) which for me is confidence building, I can browse the configuration and ‘connect the dots’ easily, dipping in deeper where I want to. It’s not what you’d call a rapid web development environment though; it’s convenient, powerful and very solid, but you’re unlikely to get a ‘20-minute app’ screencast out of it, which seems to be the primary measure of worth these days

I’ve played with Rails a little, but have never used it for anything serious mostly because I’m afraid of the scalability issues I might encounter when it’s too late to do anything about it (except rewrite using something else) - once those scalability issues are resolved I may come back to it. In the meantime, if I had a need for truly rapid web development I’ve got my eye on alternatives such as Pylons or TurboGears, which share a lot with Rails but are based on Python which is somewhat more mature & proven for sizeable deployments, and they’re also a tad more explicit about some elements (like model definition) which I prefer. Whatever option I might choose, I would prefer to dig into the framework source to better understand how the automation that is there functions, rather than just accept it on face value. Rails, Pylons and TurboGears  are all open source of course so that’s at least possible - I don’t think I would ever use them if they were closed source, because ‘magical and opaque’ is the very worst combination I can think of. Not everyone feels like that I know, many people are happy to just trust that something works and move on (and in a closed framework like .Net, that’s your only real option), but over the years I’ve bashed my head against frameworks & libraries enough to dislike trusting things I can’t examine, trace through and even fix / extend when the chips are down. 

Any opinions?

For those on this side of the Atlantic and therefore not in bed, many of the OSDN sites were down all morning including Slashdot, Freshmeat and most importantly for me, Sourceforge. Sourceforge occasionally has some downtime, something that some people moan about, but since they provide a ton of bandwidth and facilities for free (except for the optional yearly subscription that I happily pay) I say we can’t complain. However this time, absolutely everything was offline, CVS and Subversion servers, their own cached site, everything. Yikes.

All seems back now at least. Sometimes you forget how much you depend on resources like this….

I migrated the OGRE CVS repository over to Subversion this weekend, something I’ve resisted in the past due to some problems I’d had when using cvs2svn with our rather old and branch-littered repository. In all honesty, some of the problems were probably self-inflicted since I’d experimented with branch aliases a few years ago which cvs2svn previously didn’t like very much, but luckily the latest version has coped acceptably. For some bizarre reason when imported into Sourceforge the conversion decided to ressurrect a ton of folders & files that had been deleted long ago in CVS, which hadn’t happened when I tested this whole process locally, but all the other files did seem to be correct so some swift purging of the cheeky Lazaruses resolved it.

Subversion’s advantages over CVS are well known - the local diffs, the unified changesets & global revision numbers, nice tools like svnmerge. It’s not perfect though - it’s slower than CVS when checking out and committing, occasionally has a habit of corrupting the local working copy (hence the need for svn clean), and its approach to tags is plain dumb. The Subversion developers clearly liked the fact that they can abstract almost every action into a ‘copy’, but in practice this isn’t nearly as clever as they think it is, and doesn’t actually work well at all for tags. For a start, you should never be able to commit to a tag, and yet you can in Subversion - configuration managers the world over tend to find this to be a breaking of holy scripture. Secondly, because the source of the tag has no knowledge of it (because it’s just the source of a copy), you can’t look through the log of the branch from which the tag was taken and see an easy-to-read tag point there, as you can in CVS.  That actually applies to branches too in fact,  the origin point of a branch is only stored in the branch itself, not in the trunk (or whatever other branch it was spawned from), so from the source of the branch you can’t easily see where it splits off. Yes, you can solve it by relying on the fact that the start of the branch/tag will have a repository-wide revision number, and therefore you can look at the SVN revision log for the trunk / originating branch and infer that because the start of the branch comes between commits X and Y on the trunk, that’s where the branch/tag was taken. Sucks though, it’s much nicer to be explicit about these things, rather than using the ‘copy’ blunt instrument for everything - this is precisely why the ‘Revision Graph’ is so damn slow on TortoiseSVN even for a single file, when it’s very quick on TortoiseCVS. My hope is that they’ve addressed this in Subversion 1.5 along with branch merge tracking.

On balance though, Subversion is generally the most convenient for everyday development tasks so it’s still worth using despite the annoying niggles. Perforce is an even better choice if you can afford it but most won’t choose to (it’s free for open source use, but my guess is most people feel safer from any future changes in that policy on Subversion), and alternatives like Visual SourceSafe and AlienBrain are wholly inadequate in modern times, both because of their chronic inability to work well remotely, and their primitive support for concurrent development practices.

Of course, in the eyes of the open source world, any centralised repository system is strictly a bit ‘old skool’ now, even the popular Subversion. Distributed version control systems (DVCS) are all the rage now, and I can see why, since they most directly represent the open source development methodology. The idea is that there is no central repository (although in practice, there are bound to be those which are considered more authorative), and every developer is capable of synchronising / merging with every other developer in a peer-to-peer, ad-hoc fashion. Tools like Darcs, Git and Mercurial implement this methodology, and in theory, I love the idea. Certainly I can think of several cases where it would have been useful to allow third parties to synchronise their working copies with each other more smoothly & traceably - say, a small ad-hoc subgroup trying out a new technique which may take some time before it’s ready for the central version but people need to collaborate on it, sometimes without the direct involvement of the central team. Also, improved context-aware branching and merging as supported by these tools would definitely be useful to me as in a number of cases, quite large patches can get out of date while lenghty testing goes on, increasing the chances of conflicts when finally applied.

So, in theory I love DVCS - it probably wouldn’t be very natural for traditional corporate teams but for open source, and for some of the distributed commercial teams I’ve worked on, it could be very useful. The problem I see with them right now is that they’re lacking in good toolsets; most are command-line only and some were clearly designed with primarily Linux in mind (e.g. Git, which requires Cygwin to run on Windows). They also tend to be more complex, by nature of the many-to-many synchronisation approaches and high level of flexibility. I think given some more time, they will eventually supercede Subversion in the open source space, and perhaps even some corporate use, but right now I see them very much as I saw Subversion 5 years or more ago - some great ideas but most people will prefer to stick with the mature & well understood solution for now. Once we have a stable and mature TortoiseGit and Eclipse / VS integration, mainstream adoption will follow I’m sure. I would venture that most developers would be loathe to give up their slick Explorer / IDE integration for the kind of flexibility DVCS offers, because in most cases they need the integration more than they need the uber-branching; as useful as that is, it’s probably something that comes up only rarely except in the most popular open source projects.

Most people who have used a source code management (SCM) system in any serious way will have found the need to create multiple branches of their repository at some point. Some people avoid using branches because they don’t understand them, don’t feel they need them, or because they’re a little afraid of the complexity they might bring - however, branch management is something that all serious developers should be comfortable with. I was reviewing my Subversion branch procedures recently (more on why another time) and since I’ve talked to other developers before who find this sort of stuff daunting, I felt I could probably share some insight on the subject, and tools that I’ve found useful.

Read the rest of this entry »

It’s official - Vista’s UAC was designed to annoy users. The idea was to piss them off enough that they’d beat on the ISV’s to make them change their software so it didn’t trigger UAC prompts anymore. So that’s at least one feature in Vista that’s a rip-roaring success then.

To be fair, it’s a bitter pill that had to be swallowed. Microsoft built OSs for decades that were concentrated on flaunting all their wares to developers with little or no protection; because you know, security just gets in the way and all those rookie developers get upset when things are difficult. It was only a matter of time before such a weak design bit them on the arse, and as anyone knows fixing a fundamental design / architecture issue is by far the most painful thing to have to do. They can blame the ISVs, but they set the tone and the philosophy in the first place and the sheep just followed, they can’t really complain about that. Mainframe developers I knew 15 years ago would shake their head in desperation when they saw the kind of architectures Microsoft were building and advocating, and here we are now with all the ‘old’ techniques finally coming back in - hypervisors, VMs, secure rings, non-privileged users. It’s really not that these things are new, it’s just that for 20 years some people thought they didn’t need them. They were wrong, and all the mainframe / Unix guys can sit back and be smug about it now.

The ‘implement now, design later’ approach is popular with Finance departments hacking together Excel/VBScript-based ‘applications’ that the IT department has to try to rescue 3 years later when the gaffer tape and string finally can’t hold it together anymore (and if you’ve had to do this too, we can cry into our beer about it together sometime), but developers should be smarter than this. Yes, agile development is important, and turnaround is important, but there’s a minimum level of design  and good engineering you need to do to make sure you’re not building on sand. Microsoft has always been pretty weak in this area IMO, whether it’s the OS, code samples, the .Net ‘framework’ (too many low-level code tricks, not enough framework - which they now finally seem to be realising with MVC, Linq for Entities). This particular sand has held up for far longer than it really should have, but ultimately it’s just made the eventual transition more painful. I only hope they’ve learned from it.

A few weeks ago I posted a rant about how companies keep way more code to themselves than makes any rational sense, under the pretense of protecting their competitive advantage. I asserted that in a large number of cases, what they’re keeping to themselves is actually exactly what everyone else is also developing internally, and also keeping to themselves in the hope that it’s worth something. Result - a ton of duplicated effort that is worth very little. Open source of course frees people from developing the same thing over and over again via open collaboration, and allows them to concentrate on the 10-20% that really makes their application unique from their peers (assuming here that you have to sell a commercial product rather than being able to live off services, which despite the hype isn’t feasible in every sector).

I was therefore glad to see a Register article today quoting others that were saying basically exactly the same thing:

"The irony is that organizations increase their maintenance costs when they take open source code from projects like Mule in-house and add their own code. In all but a few areas companies are duplicating efforts made elsewhere, and wasting time and effort in repeating boring infrastructure programming, under the illusion they are adding competitive advantage. "There’s so much duplicate effort," Zorro said, echoing Red Hat’s Whitehurst, who claimed last month that "billions" of dollars are wasted each year in internal, non-commercial software development that re-invents the wheel."

Amen. I still think that the primary reason that so many businesses don’t ‘get’ this is that the decision making is too far removed from the people at the coalface who can see (and lament about) this wasted effort, and that the accepted wisdom that has been built up over the past couple of decades (fueled by the success of huge proprietary corporates like Microsoft) is that keeping everything to yourself is the only way to make money. Lawyers, vice presidents etc don’t generally have a clue how to differentiate between IP that’s genuinely valuable enough to invent yourself and protect like rabid wolf, and that which is a totally derivative reinvention of something that’s already out there and just a big waste of time and effort, both from the point of view of original creation and ongoing maintenance. Thus, if you have no way to measure one set of IP against another, just clutch all of it to your chest, knuckles white and teeth bared at anyone who would dare to want to look at it.

Such attitudes are woefully outdated and as open source continues to penetrate into corporate environments, such prehistoric attitudes are going to have to change. Ironically what these companies consider to be competitive advantage is in many cases precisely the opposite - all these largely worthless internal reimplementations of common functionality are in fact unnecessary baggage that’s slowing them down, and making them less competitive. Knowing how to separate the wheat from the chaff - the IP that is truly unique and differentiating you from others versus that which is common and functionary - is something that most experienced developers who know their field can do instinctively in very little time - if they are allowed to. More companies should allow their development leads to make this kind of call, in their own best interests.

Open Season is a podcast about open source issues, weighted towards the practical rather than the philosophical, and as such I tune into it regularly. Some are better than others, but I found the latest Episode 13 quite interesting for a number of reasons. They had an analyst from RedMonk on board this time, which was fascinating - RedMonk are (AFAIK) the only research firm that release their results openly rather than charging a few thousand for detailed papers, so they’re quire interesting. They talked a little about the practicalities of the Microsoft ‘we’re open, honest’ announcement of course, and largely reinforced my own thoughts on it - that while you have the ‘non-commercial’ clause in there it’s nice to have, but not really a practical shifting in the overall commercial landscape. The Zuckerberg interview came up too, as did how Grandmas really can use Ubuntu.

I also enjoyed their musings on remote working too - since I’m now working almost 100% remotely myself. The open source movement has been a standard-bearer for how remote working can work really well when done right, and I do think that with the right organisation, and the right people, it can work just as well for many tasks as cramming everyone into an office. You do lose a certain amount of social contact, which is an adjustment for sure, but I think communicating effectively and team-building across remote links can definitely still be done, it just requires a different approach and a little more effort. Open source communities that work well (and I’ll venture that Ogre’s qualifies) have built up expertise in the mechanisms for doing this, and the same approach is slowly becoming more common in commercial environments too. I don’t think you can run all projects long-term with no personal contact, but you can certainly run many of them for a long time that way, provided you find the right people, and the organisation can cope with the logistics.

The main project I’m involved in right now is almost 100% remotely developed, for example, our team members are in 6 different locations in 4 countries, and it’s actually working really well. I think it helps a lot that everyone on the lead team is a remote worker, even the project manager - so there’s no feeling of a 2-tier system. Time zones can be a positive and negative thing - on the minus side you’ve got the fact that you might only sync up for maybe 4 hours a day on IM, but on the plus side different team members can be working on something while the others are away from work, meaning you can get into work the next day and find you don’t have to wait for whatever it was they were working on - so staggered effort has its advantages too. What’s most important is that each team member has specific areas of responsibility and are mostly autonomous, so cross-dependencies are minimised. It really requires that each team member is an experienced developer & team worker already - it really wouldn’t work with junior staff who might need more assistance, training and so forth. What you lose on the social end, you gain on the ‘uninterrupted concentration’ end - something which anyone who’s had to work in an open-plan office at some point will understand. With today’s technology you’re not entirely disconnected either - IM can in some ways be more productive than over-the-desk crosstalk, in that it a) doesn’t bother anyone else, and b) can be deferred & caught up with later if you’re in the middle of something. It’ll never be as good as a chat over the coffee machine of course, but nevertheless the detachment is not as total as one might imagine.

The boundaries that this arrangement breaks down are not to be underestimated either. It can’t work for a lot of businesses that need a critical mass of people in one place, but for discrete project work, being able to cherry-pick any talent at all no matter where they’re located is a very powerful model - in the past it’s just been the logistics that have proven difficult. It tends to require a more experienced and flexible workforce (freelancers / contractors rather than salaried), and it requires absolute buy-in from the project management, but for many situations it’s ideal.