1 line mistake == 2 hours pain

· by Steve · Read in about 2 min · (326 Words)

I just spent 2 hours looking for an obscure memory leak in OGRE which only occurred in very, very specific circumstances - you had to be using modulative texture shadows (one of OGRE’s 4 in-built shadow types), a spotlight, and have provided a custom shadow receiver material (e.g. for performing a funky shadow effect like percentage closest filtering). Take all 3 together and slowly the app leaked memory, vary any one of them and it didn’t.

After carefully checking all the shadow code (and there’s a fair amount of it) I was starting to get puzzled, when all of a sudden this little gem hit me in the face:

mShadowReceiverPass->createTextureUnitState("spot_shadow_fade.png");

.. which should have been:

targetPass->createTextureUnitState("spot_shadow_fade.png");

Once you know of course, it’s obvious. The difference between mShadowReceiverPass and targetPass in the context is that targetPass is only ever not mShadowReceiverPass when using custom receiver materials. This line is only used when spotlights are in use, it’s used to fade out the edges of the shadows so that they don’t appear outside the penumbra. And, this technique is only used in modulative shadows mode since modulative shadows are an unrealistic (but faster) hack, additive texture shadows don’t need it since they use a more physically correct model which adds light only where the spotlight shines, masked by the shadow texture. The result was it was constantly repeating this line every frame since the rest of the code was checking targetPass properly to see if this step had already been done. Obvious. But I must have looked past it 20 times or more.

One line, one tiny mistake, 2 hours down the pan. Such is my life 🙁

[edit] Strictly speaking it wasn’t a memory leak, since all memory was freed at shutdown - had it been a memory leak it might have been easier to spot since our leak tracker would have pointed at the line that caused it. It was a ‘boundless memory consumption’ error ;)[/edit]