Customized Logging in Apache 2.2

Posted by Ryan Coyner on July 28, 2008 under Apache - 0 Comments

I have been fairly busy this last month, but during the last couple of days I found some time to play with Apache. I have been using Apache for as long as I can remember, but digging into its functionality and capabilities had never been my top priority. I finally forced myself to sit down and learn more logging techniques in Apache to be able to handle and analyze active feedback from the server more proficiently.

Customized Logging

You can specify the type of information Apache processes and records in the logfiles with the LogFormat directive. Here is the list of format strings used by the directive. The most basic and most commonly used format is the common log format which Apache uses by default. Its format strings look like this:

LogFormat "%h %l %u %t \"%r\" %>s %b" common

Place the following in your configuration file to use the common log format:

CustomLog "/path/to/logs/access.log" common

The second most common format is the combined log format. This format was introduced to include further useful information that became available as the HTTP protocol advanced. These additions are the Referer and the User-agent formats. The format strings look like this:

LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined

Place the following in your configuration file to use the combined log format:

CustomLog "/path/to/logs/access.log" combined

Another important directive is LogLevel. This directive allows you to control the verbosity of the error log which can be used for debugging. There are several hierarchical levels of error logging which are listed here. In order to record a lot of information appropriate for debugging purposes, set the level to Debug:

LogLevel Debug

Logging Compression Ratios

If you use the mod_deflate module to compress output, you can record the compression ratio and the amount of bytes you compressed in the logfiles. This is particularly useful if you want to see or gather statistics of how much bandwidth you are saving by compressing your output. Place the following in your configuration file:

# Some form of compression must take place for logging to work.
AddOutputFilterByType DEFLATE text/html text/plain text/xml

DeflateFilterNote Input in
DeflateFilterNote Output out
DeflateFilterNote Ratio ratio

LogFormat '%{out}n/%{in}n (%{ratio}n%%)' deflate
CustomLog "/path/to/logs/deflate.log" deflate

The output will look like this:

"GET / HTTP/1.1" 15/13 (115%)
"GET /favicon.ico HTTP/1.1" 636/1048 (60%)

Rotating Logfiles

Breaking up logfiles into smaller, manageable pieces is an incredibly useful technique for debugging. This technique is called log rotation, where logfiles are automatically rolled over at specific times without the need to stop and restart the server. With log rotation you do not have to deal with over-sized logfiles when trying to debug a problem, which can save a lot of time. Log rotation is accomplished very easily by piping the information to rotatelogs. The script automatically opens a new file at the designated time and starts writing to it:

CustomLog "| /path/to/rotatelogs /path/to/logs/access-%Y-%m-%d.log 86400" combined

The first argument is the base name of the logfile. It is allowed to contain the strftime format string, as in the above example. The second argument is the interval in seconds between rollovers. The above example is set to a 24-hour interval, which is 86400 seconds. The rollover itself occurs whenever the system time is a multiple of the interval's value. In this case, a new logfile will be created every night at midnight since the system time would be a multiple of 24 hours (it is based at representing midnight on 1 January 1970).

Logging POST

I would not recommend doing this other than for debugging purposes, because this technique outputs a lot of noise and does not work unless the LogLevel directive is set to Debug. To record data submitted via POST, simply enable mod_dumpio and put the following in your configuration file:

LogLevel Debug
DumpIOInput On

You would not need to do this if you are using Django since their default WSGI development server already prints out the contents of POST. However, this can come in handy if you're using Apache as your development web server.

Closing Notes

None of these are complicated techniques and most of them were already familiar to me. That being said, these logging techniques are without a doubt the most useful ones that I use on a regular basis.