Syslog

And other logging programs like logrotate and LogWatch. Obviously there are much better and newer logging platform like ELK stack and graylog but this is good if you like a report each morning containing all the logs for specific machines.

Configuration

DNS

DNS should be configured to provide CNAME records for loghost or log that reference the collectors.

Collector

A collector must be configured to listen for incoming IP traffic on port 514.

Device

A device must be configured to send messages to loghost or log, as configured in DNS.

Setup on log001

A CentOS 7 server

vim /etc/rsyslog.conf
##added these lines
#apc monitoring
local3.*                                                /var/log/apc.log
# switch
local5.*                                                /var/log/switch
#IBM 10GB switch
local4.*                                                /var/log/10gig
if $hostname == 'web001' then /var/log/web001
if $hostname == 'web002' then /var/log/web002
if $hostname == 'mail001' then /var/log/mailserver
if $hostname == 'sql001' then /var/log/sql001

then I went to all of the apc devices, 10gig switch and switch and configured 
them to send logs to the local# at 10.0.0[log001]:514
#on remote machines, send logs to log001 

vim /etc/syslog

add the line
*.info;local5.none;mail.none;authpriv.none;cron.none    @log001.zedxinc.com

Logrotate

I also tweaked logrotate on log001 to compress logs and delete them after 30 days

Vim /etc/logrotate.d/syslog
/var/log/cron
/var/log/maillog
/var/log/messages
/var/log/secure
/var/log/spooler
/var/log/apc.log
/var/log/switch
/var/log/bladecenter
/var/log/10gig
/var/log/mailserver
/var/log/web001
/var/log/web001
/var/log/sql001
{
    sharedscripts
    postrotate
    compress
    maxage 15
        /bin/kill -HUP `cat /var/run/syslogd.pid 2> /dev/null` 2> /dev/null || true
    endscript
}

 

LogWatch

I needed LogWatch to be able to analyze to following log files. As logWatch would only report on standard cron, mailog, messages, secure, and spooler.

/var/log/cron
/var/log/maillog
/var/log/messages
/var/log/secure
/var/log/spooler
/var/log/apc.log
/var/log/switch
/var/log/bladecenter
/var/log/10gig
/var/log/mailserver
/var/log/web001
/var/log/web002
/var/log/sql002
create a configuration file for the specific log
cd /etc/logwatch
vim conf/logfiles/bladecenter.conf

##LogFile conf for bladecenter
LogFile = /var/log/bladecenter

Archive = bladecenter-*.gz

# Expand the repeats (actually just removes them now)
 *ExpandRepeats
############
configure a file for the service which will do the parsing of the logs.
cd /etc/logwatch
vim conf/services/bladecenter.conf

###LogWatch service for IBM Bladecenter

Title = "IBM Bladecenter logs"

LogFile = bladecenter

########
configure script of some sort to parse log file and only report on relevant information
cd /etc/logwatch
vim scripts/services/bladecenter

#!/usr/bin/env bash
# /etc/logwatch/scripts/services/bladecenter

# Change the line separator to split by new lines.
OLD_IFS=$IFS
IFS=$'\n'
LogFile=/var/log/bladecenter

# The contents of the log file are given in stdin.
for LINE in $( cat $LogFile ); do

    # Only lines matching this regexp will be included. can add debug, info, notice for testing
    if echo "$LINE" |egrep 'WARNING|ERROR|CRITICAL|ALERT|EMERGENCY' &> /dev/null; then

        # Every line we echo here will be included in the logwatch report.
        echo "$LINE"

    fi

done

IFS=$OLD_IFS
#################

repeat as necessary for each logfile you want to have LogWatch report on. If there are no relevant logs in the logfiles to be reported on than that section will be ignored and not included in the cron.daily email report.

 

Protocol

The BSD syslog protocol defines three types of hosts: collector, device, and relay. A device emits messages, a collector receives messages, and a relay receives messages and forwards them to a collector or relay.

Messages are transmitted via UDP; thus, syslog is an unreliable protocol, and may suffer from lost messages during high load on a collector or relay or network congestion.

A syslog message contains the following parts:

PRI
priority; decimal representation of facility << 3 | severity enclosed in angle brackets
HEADER
header

TIMESTAMP
the date and time when the message was created
HOSTNAME
the hostname of the machine that sent the message as determined by the machine that receives the message
MSG
message

TAG
the name of the process that created the message
CONTENT
message content

Facilities

Code Facility
0 kernel messages
1 user-level messages
2 mail system
3 system daemons
4 security/authorization messages1
5 messages generated internally by syslogd
6 line printer subsystem
7 network news subsystem
8 UUCP subsystem
9 clock daemon2
10 security/authorization messages1
11 FTP daemon
12 NTP subsystem
13 log audit1
14 log alert1
15 clock daemon2
16 local use 0 (local0)
17 local use 1 (local1)
18 local use 2 (local2)
19 local use 3 (local3)
20 local use 4 (local4)
21 local use 5 (local5)
22 local use 6 (local6)
23 local use 7 (local7)
  1. Various operating systems have been found to utilize Facilities 4, 10, 13 and 14 for security/authorization, audit, and alert messages which seem to be similar.
  2. Various operating systems have been found to utilize both Facilities 9 and 15 for clock (cron/at) messages.

Severities

Code Severity Description
0 Emergency system is unusable
1 Alert action must be taken immediately
2 Critical critical conditions
3 Error error conditions
4 Warning warning conditions
5 Notice normal but significant condition
6 Informational informational messages
7 Debug debug-level message