Q:

Improving on server configuration

I am trying to improve the performance of a Drupal (6) installation under load. One web server, one db server (MySQL5) both are Single Socket Quad Core Intel Xeon 2.5GHz with 16GB RAM. On the web server, I am running APC to improve PHP's performance. Running a memcached server with 6 bins on the DB server. Using Apache Solr to power search, Solr lives on the DB server as well.
The bottleneck appears to be Apache on the web server, as it runs out of memory under the load. No single process appears to be hogging too much of the available system resources.
Any input would be greatly appreciated. If there are questions I can answer about the setup, please ask away.
Thank you!

A:

Since we're talking PHP I assume you're using the Apache prefork MPM. Given Rasmus Lerdorf's attitudes on multithreading, you should continue to use Apache prefork or a similar non-threaded setup with another HTTP server (such as nginx).
4 quick wins come to mind. I can't say from OPs original question if this has already been done -- if not, these 4 improvements should bring a substantial improvement.
a) Move static file serving (CSS, images, etc) away from your PHP Apache instances. With the Prefork MPM, when you're sending a single image to a user, a whole Apache child with the full PHP runtime is kept occupied to send that single image. This could be a 20-40 MB Apache process occupied for 0.1 to 3 seconds while the user downloads that image -- in the same time that Apache instance could have served tens or hundreds of dynamic pages. Suggestion: Set up a dedicated server for static resources, or use a cheap CDN like Amazon CloudFront.
b) Turn off HTTP keepalives to the Apache instance that serves dynamic (Drupal) content. HTTP keepalive is great and has its uses, but again, with keepalives on & Apache prefork MPM a single end user browser will keep 'bloking' (i.e. occupying) an Apache child for lengthy periods.
c) Verify that you're sending the proper HTTP headers, especially that static content such as images, CSS etc is cache-able. Otherwise the user's browser will re-downlod this static content on each page view, which is a total waste. Use Yahoo! YSlow, Mark Nottingham's RED or similar tools.
d) Set up another web server, and just use plain DNS round robin to distribute load between the two web servers. I assume Drupal can keep session state stored on the MySQL database (most PHP apps can)? If so, then the session store is taken care of, and you can use DNS round robin.
DNS round robin is not ideal. It does not handle a failure of a server, it doesn't guarantee perfect load distribution. But it's easy to set up, and usually works well enough for a simple setup with 2 or maybe 3 web-servers.

