Taming of the FBO

· by Steve · Read in about 3 min · (625 Words)

I had the day off today, with the intention of getting through some of the outstanding work needing to be done in time for Dagon (OGRE 1.2.0) to hit RC1. Well, it turned into one of ‘those’ days - the ones where you seem to spend eternity not getting very much done at all. Although luckily this one does have a happy ending.

The crux of the bug that I had on my hitlist today was the fact that stencil shadows were misbehaving when used in combination with the compositor framework (full screen post processing effects, doable for a while in OGRE but very, very easily scriptable in Dagon to produce groovy effects like this), but only in OpenGL. I pored over the compositor code for quite a long time, which in itself was worth it since I managed to squeeze a few small optimisations out of it whilst I was at it. I did find a stencil state leak which was duly fixed, but the shadows problem was elusive - I tried any number of things to tease out what I thought was another state leak, but to no avail.

And lo, I was most perplexed and sought solice in ice cream.

On returning, I did some more tests and discovered that any render to texture didn’t seem to have a functioning stencil buffer under GL. It occurred to me to check the FBO code (frame buffer object - the latest and greatest in a long line of ways to create offscreen surfaces in GL), and I discovered that none of the formats were reporting stencil support. Hmm.

The problem was this - FBO is a very forward-thinking specification. You can attach (and in fact share) render buffers to the current frame buffer, to the colour, depth and stencil channels separately. The trouble is, no graphics card manufacturer implements separate depth and stencil buffers, they are always combined. This means that creating separate depth & stencil buffers the way the FBO spec likes always results in an unsupported stencil format. Once again, GL is forward thinking and quite well designed, but way ahead of where hardware is today.

Luckily, another extension has arisen specifically because of this problem. “GL_EXT_packed_depth_stencil” allows you to fudge the binding to the depth & stencil channels and use the same buffer. My stencil support came back in FBO, and I was a happy man. Unfortunately I’m not sure if ATI have implemented this extension yet - I hope they have. Just in case they haven’t, I also added a configuration option allowing you to make GL fall back on PBuffers (or even nasty old glCopyTexSubImage2D - yuck) in case your FBO support doesn’t offer you this option and you need stencils.

The bottom line is always the same - in DirectX it tends to be easier to get at hardware features like VBOs and RTTs, but the API is very much designed to the current hardware implementations and thus ends up changing a lot or suffering from poor compromises. OpenGL tends to have the more robustly designed API, but a standard API consensus either lags behind the hardware features that are available (whilst the features bed down), or they ‘design ahead’ such that hardware support is flaky or non-existent for a while. Must be hard to get the balance right. I like aspects of both APIs (GL’s design, D3D’s close correlation with common hardware), so no comment crusades for either one here please 😉

At least that’s done. I had time to test the VC8 SDK build script, and that seems to work fine. Tomorrow will be wholly dedicated to the few remaining patches (and applying the ones for addons), and documentation so we’re all ready for Sunday’s release.