Riding out the Storm
Posted by ibukun on February 25th, 2009 @ 8:08 pmPremature optimization, they say, is the root of all evil.
When server load started to increase unexpectedly a few months ago and response times began to be unreasonable, I responded by doing aggressive optimization. MySQL query caches, Apache MPM tuning, PHP accelerators from Memcache to eAccelerator to APC, and every little trick I could find under the sun.
When things only got worse, I decided that surely a newer, faster server would do the trick. Fortuitously I was offered a newer server option by my hosting provider that kept my costs the same while offering newer hardware and a much faster CPU. As I transferred, I began to optimize even more aggressively, cutting out some web site features and eliminating certain mail services I considered expensive. Well, the situation only got worse until the server basically crashed. Not to worry, these days daily offsite backups are performed — I learned my lesson a long time ago.
After about 12 hours of intensive gnashing of teeth, it became clear that the only option was to wipe the server clean and restore again. Portions of my file system and configuration files had been corrupted by broken disk I/O operations. While performing an rsync restore operation, I noticed that a certain MySQL comments table was unusually large. Bloody spammers! I cleaned out the table, and suddenly everything’s snappy again.
Watch out for comment spammers. They are more evil than you think.