wiredfool

Turbulence

Flow of smoke from incense
Flow of smoke from incense
Flow of smoke from incense
Flow of smoke from incense

I got slashdotted today. Well, not exactly slashdotted, and not my site, but a server I run. But none the less, one of my servers was hammered due to an external media event. And since there’s no mushroom cloud over the hill, you can tell that everything survived.

Loads on that machine were running 3-4 times normal peak loads, effectively maxing out the machine for 30 minutes at a time. Sometimes it was churning smoothly but a little sluggishly, and sometimes it was dragging with far too many queued requests for good quality of service. Looking through the logs, the difference in load level between these two modes was minimal. In the most extreme case took less than a minute to switch from one mode to the other. Or rather, it looks like it took one 15 second page load to slow enough other requests that the requests bogged down in a chain reaction accident for the next 30 minutes.

One request out of thousands. One driver hits the brakes. One pebble in the stream. One puff of air. All enough to take a supercritical flow and kick it from laminar orderly flow to turbulent chaotic flow. Like the fluids gurus, we can find the little chaos generators and painstakingly remove them from the system. But Chaos is a difficult mistress. She shows up in simple systems, and no matter how hard we work, we can only push back the boundaries of the chaotic zone.

Not that I’m going to call my web apps simple systems. Or that I’m going to give up on removing turbulators. A cached value here, a super-plasticizer there, and a little bit of Moore’s law over in the corner. There’s progress to be made shoveling back the chaos.

No comments

No comments yet. Be the first.

Leave a reply

You must be logged in to post a comment.