So, I predictably had a ton of stuff to catch up with when I got back, which I’m getting close to clearing now, barring the forums which I’m woefully behind on now. Clearly the discovery of some problems with the Ogre server didn’t help at all, and I spent far too much time at the weekend trying to diagnose and resolve that so the server stayed up.
Turns out that someone somehow was deliberately or accidentally forcing Apache into an infinite loop, or at least a very long process, probably because of a bug in some software we’re running, which meant that server process then became unavailable to future requests, causing spawning of new processes. Eventually this would exceed the maximum number of server processes and the server would be hung. My suspicion is that it’s deliberate, because it isn’t a gradual thing, and we never saw it before the end of last week. Monitoring the processes showed that they were completely stable until they suddenly ramped up in the period of about 30 minutes, so it’s clear that the condition isn’t hit by anyone else for hours at a time, then suddenly it’s repeatedly hit until it becomes fatal. Tracking what precisely triggered this has proven very difficult since the attackers were not issuing a flood of requests or something else which would obviously single them out among the large amount of traffic we see, and there are no errors in the requests; so basically they’re hidden in the crowd.
So the strategy has been to put counter measures in place that automatically detect the condition and resolve it, and also to make sure we have all the latest security fixes on manually updated software. This handled a couple of cases of it overnight Saturday and I had some more ideas for tracking the perps through this automatic script but the attack has not recurred in the last 36 hours. We’ll see. Sorry I haven’t been back in the forum much over the weekend but this has just eaten a ton of my time (and made me extremely grumpy, so they’ve pissed my wife off too), so blame the SOB who is responsible.