Server Trouble
As you might have noticed, my server was down tonight. This is a quick status update on what's going on.
I'm having a few stability issues with the server for some time now. It seems to freeze for an unknown reason (about once every two months). This time however it was a bit worse: the disk was full. For some reason, lighttpd wrote 42GB to it's error log with the following line:
2009-08-18 06:25:59: (mod_fastcgi.c.2861) wait for fd at connection: 1023
This line occurs several thousand times per second. Google didn't tell me anything useful about it though.
After purging the log and rebooting the server, several MySQL tables were corrupted (a side effect of a full disk). Nice to know, the following command will attempt to repair all tables:
#> mysqlrepair -p<yourpass> --all-databases
Next problem: when starting lighty again I noticed it consuming a lot of RAM. Nearly 3GB and growing. Some googling around revealed this old thread which lead me to try disabling each of my sites step by step to see if any of my scripts might causing this. For now I disabled my rafinfo site which seems to fix the memory problem for now.
However the server is still experiencing quite a high load for a Tuesday morning and I'm not quite sure why. Traffic seems to be quite normal at least. MySQL seems to be quite busy currently with mytop showing about 300 queries per second (but with 98% cache efficiency).
Kernel and package data:
Linux gir 2.6.26-2-amd64 #1 SMP Fri Aug 14 07:12:04 UTC 2009 x86_64 GNU/Linux ii lighttpd 1.4.19-5 ii mysql-server 5.0.51a-24+lenny1 ii php5-cgi 5.2.6.dfsg.1-1+lenny3
All this of course while I'm supposed to work of course 
If you have any hint what I should look at or if you had similar problems, please let me know in the comments.