log: Linux Log Files Explained

You've just deployed an application, and something isn't working. You SSH into your server, run a few commands, and realize you have no idea where to look for errors. The application is crashing, but the logs are nowhere to be found. This is where /var/log comes in.

Every Linux system generates logs. These files contain critical information about what's happening on your server—successful operations, failed attempts, errors, warnings, and system events. Learning to read and understand /var/log is one of the most valuable skills for any system administrator or DevOps engineer.

What is /var/log?

/var/log is a standard directory on Unix-like operating systems that contains variable data files. The "var" stands for "variable," meaning these files are not static—they change as the system runs. Unlike /etc (configuration files) or /usr (programs), /var/log is specifically designed for log files that grow over time.

Most Linux distributions organize logs in /var/log with a consistent structure. The directory typically contains subdirectories for different types of logs, such as system logs, authentication logs, application logs, and kernel messages.

Common Log Files in /var/log

Let's look at the most important log files you'll encounter.

/var/log/syslog or /var/log/messages

This is the main system log file on most Linux distributions. It contains messages from the kernel and various system services. On Debian-based systems, you'll find /var/log/syslog. On Red Hat-based systems, it's /var/log/messages.

# View the last 50 lines of the system log
tail -n 50 /var/log/syslog
 
# Search for error messages in the last 100 lines
grep -i error /var/log/syslog | tail -n 100

/var/log/auth.log

Authentication-related logs go here. This file records successful and failed login attempts, SSH connections, sudo commands, and other authentication events. It's your first line of defense when investigating security issues.

# Check for failed login attempts in the last day
grep "Failed password" /var/log/auth.log | tail -n 50
 
# See all SSH login attempts
grep "sshd" /var/log/auth.log | tail -n 100

/var/log/dmesg

The kernel ring buffer is stored in /var/log/dmesg. It contains messages generated by the kernel during boot and runtime. These messages are useful for diagnosing hardware issues, driver problems, and kernel errors.

# View kernel messages from the last boot
dmesg | tail -n 50
 
# Search for specific error types
dmesg | grep -i "error\|fail"

/var/log/kern.log

On some distributions, kernel messages are logged to /var/log/kern.log instead of /var/log/dmesg. This file contains similar information to dmesg but is written to disk rather than stored in the kernel ring buffer.

# View kernel log entries
tail -n 100 /var/log/kern.log

/var/log/apt/history.log

If you're managing a Debian-based system, /var/log/apt/history.log records all package installations, updates, and removals. This is useful for tracking changes to your system.

# View recent package installations
grep "install" /var/log/apt/history.log | tail -n 20

/var/log/nginx/access.log and /var/log/nginx/error.log

If you're running Nginx as a web server, these files contain access logs (requests) and error logs (server errors). These are essential for debugging web application issues.

# View recent Nginx access logs
tail -f /var/log/nginx/access.log
 
# Check for 404 errors
grep " 404 " /var/log/nginx/access.log | tail -n 50

Log File Rotation and Size Management

Log files grow indefinitely. If left unchecked, they can consume all available disk space. Linux uses log rotation to manage this.

Understanding logrotate

The logrotate utility automatically rotates, compresses, and removes old log files. Its configuration is typically in /etc/logrotate.conf and /etc/logrotate.d/.

# Check the logrotate configuration
cat /etc/logrotate.conf
 
# View the Nginx logrotate configuration
cat /etc/logrotate.d/nginx

A typical logrotate configuration might look like this:

/var/log/nginx/*.log {
    daily
    rotate 14
    compress
    delaycompress
    notifempty
    create 0640 www-data adm
    sharedscripts
    postrotate
        [ -f /var/run/nginx.pid ] && kill -USR1 `cat /var/run/nginx.pid`
    endscript
}

This configuration tells logrotate to:

Rotate logs daily
Keep 14 days of logs
Compress old logs after the first rotation
Not rotate empty files
Create new log files with specific permissions
Send a signal to Nginx to reopen log files

Checking Log Rotation Status

# Check if logrotate is running
systemctl status logrotate
 
# Manually run logrotate to test configuration
sudo logrotate -f /etc/logrotate.conf

Analyzing Log Files Effectively

Reading raw log files can be overwhelming. Here are practical techniques to extract useful information.

Using tail and head

# View the last 100 lines of a log file
tail -n 100 /var/log/syslog
 
# View the first 50 lines
head -n 50 /var/log/syslog
 
# Follow logs in real-time (useful for monitoring)
tail -f /var/log/syslog

Using grep for Pattern Matching

# Search for specific patterns
grep "error" /var/log/syslog
 
# Case-insensitive search
grep -i "error" /var/log/syslog
 
# Search for multiple patterns
grep -E "error|warning|fail" /var/log/syslog
 
# Count occurrences of a pattern
grep -c "error" /var/log/syslog

Using journalctl (systemd systems)

Modern Linux systems use systemd, which provides journalctl for querying logs.

# View all logs from today
journalctl --since today
 
# View logs for a specific service
journalctl -u nginx
 
# View logs with a specific priority
journalctl -p err
 
# Follow logs in real-time
journalctl -f

Filtering by Time

# View logs from the last hour
journalctl --since "1 hour ago"
 
# View logs from a specific time range
journalctl --since "2026-03-12 09:00" --until "2026-03-12 10:00"
 
# View logs from yesterday
journalctl --since yesterday

Common Log Analysis Use Cases

Investigating Application Crashes

When an application crashes, check both the application logs and system logs.

# Check system logs for recent errors
grep -i "error\|fail" /var/log/syslog | tail -n 100
 
# Check application-specific logs
tail -f /var/log/your-app.log
 
# Look for segmentation faults
grep -i "segmentation fault" /var/log/syslog

Monitoring Security Events

Track authentication attempts and suspicious activity.

# Check for failed SSH login attempts
grep "Failed password" /var/log/auth.log
 
# Check for successful root logins
grep "root@pts" /var/log/auth.log
 
# View recent sudo commands
grep "sudo" /var/log/auth.log | tail -n 50

Troubleshooting Network Issues

# Check for network errors
grep -i "network\|connection" /var/log/syslog
 
# View recent DHCP events
grep "dhcp" /var/log/syslog
 
# Check for firewall issues
grep "iptables\|ufw" /var/log/syslog

Performance Monitoring

# Look for high CPU usage warnings
grep -i "cpu" /var/log/syslog | tail -n 50
 
# Check for memory issues
grep -i "memory\|oom" /var/log/syslog
 
# View disk space warnings
grep -i "no space left" /var/log/syslog

Log File Permissions and Security

Log files contain sensitive information. Proper permissions are critical for security.

Checking Log File Permissions

# List permissions for log files
ls -l /var/log/syslog
 
# Check ownership
stat /var/log/syslog

Setting Proper Permissions

# Change ownership to root
sudo chown root:adm /var/log/syslog
 
# Set appropriate permissions
sudo chmod 640 /var/log/syslog
 
# Ensure the log directory has correct permissions
sudo chmod 755 /var/log

Restricting Access

# Add users to the adm group to read logs
sudo usermod -aG adm yourusername
 
# Verify group membership
groups yourusername

Best Practices for Log Management

1. Regular Log Review

Make it a habit to review logs regularly. Set up alerts for critical errors.

2. Centralized Logging

For distributed systems, consider centralized logging solutions like ELK Stack, Loki, or Graylog.

3. Log Level Management

Configure applications to use appropriate log levels (DEBUG, INFO, WARNING, ERROR, CRITICAL).

4. Log Retention Policies

Establish clear policies for how long logs should be retained based on compliance requirements and operational needs.

5. Log Encryption

For sensitive applications, consider encrypting log files at rest.

6. Structured Logging

Use structured logging formats (JSON, XML) for easier parsing and analysis.

Tools for Advanced Log Analysis

logwatch

logwatch is a Perl script that summarizes system logs.

# Install logwatch
sudo apt install logwatch
 
# Run logwatch
sudo logwatch

logcheck

logcheck automatically checks logs for suspicious activity.

# Install logcheck
sudo apt install logcheck
 
# Run logcheck
sudo logcheck

logtail

logtail follows a log file and outputs new lines.

# Install logtail
sudo apt install logtail
 
# Follow new log entries
sudo logtail -f /var/log/syslog

Troubleshooting Log Issues

Logs Not Being Written

If logs aren't being written, check:

# Verify the log file exists
ls -l /var/log/syslog
 
# Check file permissions
stat /var/log/syslog
 
# Check disk space
df -h
 
# Check if the service is running
systemctl status syslog
 
# Check for errors in the logrotate configuration
sudo logrotate -d /etc/logrotate.conf

Corrupted Log Files

If a log file is corrupted, you can create a new empty file:

# Backup the corrupted file
sudo cp /var/log/syslog /var/log/syslog.corrupted
 
# Create a new empty log file
sudo touch /var/log/syslog
 
# Restore appropriate permissions
sudo chmod 640 /var/log/syslog
sudo chown root:adm /var/log/syslog

Conclusion

The /var/log directory is the window into your Linux system's behavior. By understanding the different log files, how to read them, and how to analyze them, you can diagnose issues, monitor security, and maintain system health.

Remember these key points:

/var/log/syslog or /var/log/messages contains general system messages
/var/log/auth.log tracks authentication events
/var/log/dmesg contains kernel messages
Use tail, grep, and journalctl to analyze logs effectively
Implement proper log rotation to prevent disk space issues
Set appropriate permissions to protect sensitive log information

The next time your application misbehaves, don't panic. Start by checking /var/log. The answers are almost always there, waiting to be found.

Platforms like ServerlessBase simplify log management by providing centralized logging and monitoring dashboards, so you can focus on interpreting the data rather than hunting for it across multiple servers.