System logging explained in Linux

In linux, a variety of log files are maintained. And its understanding is often vital for troubleshooting system problems. The centralized logging is provided by two daemons :

• syslogd
• klogd
  

For example, running the 'ps' command on my system gives the following output ...

$ ps aux|grep [sk]*logd
root      2281  0.0  0.0   1624   612 ?        Ss   13:12   0:00 /sbin/syslogd
root      2287  0.0  0.0   1576   380 ?        Ss   13:12   0:00 /sbin/klogd -x

The log files generated by these daemons as well as the log files generated by the applications like apache, squid etc are stored under the /var/log directory.

Some of the more important log files that are a part of system administrator's forte are as follows:

/var/log/dmesg - This log file is written upon system boot. It contains messages from the kernel that were raised during the boot process. You can also view them using the command:

# dmesg

This log file can be viewed by any non-privileged user. But the next three log files are readable only by root.

/var/log/messages - This is the standard system log file, which contains messages from all your system software, non-kernel boot issues, and messages that go to 'dmesg'.

/var/log/maillog - This log file contains messages and errors from your sendmail.

/var/log/secure - This log file contains messages and errors from security related systems such as login, tcp_wrappers, and xinetd. This log file is very useful in detecting and investigating network abuse.

As I said earlier, syslogd and klogd daemons provide centralized logging in linux. The configuration file for syslogd is /etc/syslogd.conf

System logging is provided by syslogd and klogd intercepts kernel messages and provides them to syslogd daemon.

/etc/rc.d/init.d/syslog script controls both syslogd and klogd daemons.

Messages can be logged to files, broadcast to connected users, written to the console, or even transmitted to remote logging daemons across the network.

By default, the messages of emergency or higher (more severe) are broadcast to all users, and most other messages are written to /var/log/messages file, which is where you should look for non-kernel boot errors, error messages from most application-level services, such as automount, login services etc. After system boot, kernel messages are also written to this file.

Usually a system administrator will run the command :

# tail -f /var/log/messages

at the console, to get a running display of the messages logged by the syslogd daemon. Thus he will be able to keep track of any system wide errors that may be generated.