The speed is a real problem and I'm working on that. Right now, this server also hosts my personal site which is getting an insane amount of traffic. I upgraded the server resources and that helped some...
Right now I'm on a Media Temple VE 2GB server but the next step up in resources costs quite a bit more.
@neonglass - thank you. It is still a work in progress but it is getting there.
@Beast Lee - I'm going to be installing a reverse web proxy this weekend to (hopefully) speed things up. I am going to be using Nginx but it taking a little bit of work to figure how to get it to work on this server which is hosting two different sites.
The server resources are freeing up. Stumble Upon was sending a few thousand visitors per hour to an un-optimized wordpress install. I made a few other tweaks to the worker threads and that is helping some too.
I have Nginx working but it isn't playing nicely with Apache. I'll keep trying as I have time. I have a GMAT scheduled for Friday the 13th of august so I'm not going to have too much free time until then, sadly.