How to get Apache to run fine without stampeding occurring in high-traffic, low-resource situations.
Hello again. This time, I'll show you how to make a Web server running Apache and Linux survive heavy loads.
Before we go on, you should know something: this is not an article about securing Apache. This is an article about making Apache behave under heavy load conditions.
Okay, now that we're here, let's discuss scalability.
Scalability
Scalability is simply the ability of a server to withstand heavy loads. If you tried to read the last article, Hardening a Linux server in 10 minutes, you probably noticed that this server was down.
That's a scalability fault.
Let's put it in another light. This server has 512 MB of RAM. The surge of traffic (thanks to LinuxToday links pointing to this site) caused the server to fail (more accurately, the MySQL server appeared to hang). Brag all you want about Linux's ability to survive these events, nothing will help you against a misconfigured server.
It all boils down to configuration
In this particular case, the misconfiguration was Apache's. Weighing 13 MB per
httpd process (though some of it is shared with other processes), it's pretty simple to understand that a runaway Apache server can bring your server down completely. When your Apache server starts serving a lot of requests, all those processes quickly fill the available memory (physical and virtual). When your Linux server runs out of RAM, it will start killing processes it deems 'memory hogs'. Usually the first ones to go down are the MySQL processes. If you're serving dynamic pages, that's a disaster.On to Apache configuration
By default, Apache comes preconfigured to serve a maximum of 256 clients simultaneously. This particular configuration setting can be found in the file
/etc/httpd/conf/httpd.conf(though the location of the file may vary, depending on the Linux distribution you use).
Whip your favorite text editor out and open that file (remember that you should be doing this as
root — the administrative account on the majority of Linux servers out there).
Look for
MaxClients. It will probably look like this:# prefork MPM # StartServers: number of server processes to start # MinSpareServers: minimum number of server processes which are kept spare # MaxSpareServers: maximum number of server processes which are kept spare # ServerLimit: maximum value for MaxClients for the lifetime of the server # MaxClients: maximum number of server processes allowed to start # MaxRequestsPerChild: maximum number of requests a server process serves <IfModule prefork.c> StartServers 4 MinSpareServers 3 MaxSpareServers 10 ServerLimit 256 MaxClients 256 MaxRequestsPerChild 10000 </IfModule>
That's the configuration section for the
prefork module. 99% of the Apache servers out there use the prefork module to serve requests, so unless you have an exotic configuration, you'll be changing these settings.
Time to calculate a good value for the
MaxClients directive. Find out how much memory your Apache processes use. Using top, check the RES column. That's the resident memory size. It should say the size in megabytes that your Apache processes are taking. In my example, it's 22m.
Figure out a good value. If your server has 512 MB of RAM (in my case, this is true), and you're sharing your server with MySQL and Sendmail (true in my case, as well), you'll want to reserve about half of it for Apache (256 MB). Divide that by the resident memory each process takes up, and you'll have a number of processes (say, 11). That's the maximum amount of processes you can run without resorting to virtual memory. Resorting to virtual memory (swap) will make your server thrash and become extremely slow.
It's, of course, all about balance. If you have one gigabyte of swap, you may want to raise the number of Apache processes. Raising it too much will cause heavy traffic to spawn lots of Apache processes, bringing your server down.
Setting the MaxClients and StartServers directive
You now have your start value (in our example, it was 11). Change the
MaxClients and the ServerLimit directives to it. Save the file and restart Apache (/sbin/service httpd restart does that trick in Fedora Core).
Now it's time to start testing. Keep a root login open to that server. Using your favorite testing tool (
ab and wget are good at this), start a storm of connections (more than 1024 simultaneous requests) directed to a page served by your Apache server (ideally, one that exercises the server, like dynamic pages with lots of queries). Issuing the uptime command in your root login should not yield a load average above 1, and the server should respond to commands quickly.[rudd-o@amauta2 conf]$ uptime 15:54:18 up 1:41, 3 users, load average: 0.86, 0.70, 1.50
Tuning the configuration
That's great. Once the test is finished, duplicate
MaxClients and StartServers, and try your storm test again. The load average should be low.
Keep tuning until you hit your maximum desired load average. For servers used interactively often, having a load above 3 is way too much to use the server comfortably. For servers used mostly as real servers, a maximum load average of 10 should be acceptable. More than that, and you'll find yourself needing to reboot the server when experiencing heavy traffic conditions, because no terminal or remote console will respond quickly to commands, and managing the server will be impossible.
Conclusions
That's it! With practice, you'll be able to skip the memory math and learn the ideal setting for any server. Other tuning options you may try (in order of diminishing returns):
- Eliminating unnecessary Apache modules from the configuration (perhaps uninstalling them altogether, by use of RPM or your favorite distribution's packaging tool)
- Recompiling Apache, optimizing for memory consumption (the
-Osoption ofgcc) - Recompiling Apache, building modules in instead of having them run as modules
Remember: if you have any questions or suggestions, please leave them as comments below. Happy hacking!
