Saturday, November 5, 2016


 on  with No comments 
In , ,  
Syslog is an industry standard protocol for the purpose of message logging. It was developed by Eric Allman, originally as part of the Sendmail project.  Because of it's utility, it was adopted by other applications and eventually became the defacto default logging system on Unix like systems.  It was eventually documented by RFC 3164 (The BSD syslog Protocol) and standardized by RFC 5424 (The Syslog Protocol).  Many other RFCs exist extending the basic functionality such as RFC 5425 (TLS Transport Mapping for Syslog).

There are three main components to a syslog message.  First is the facility code.  This is used to specify the type of program that generated the syslog message. Examples are 0 for kernel messages (keyword kern), 1 for user level messages (keyword user) and 2 for mail system messages (keyword mail).  Next is the severity level.  The list of seventies ranges from 0/Emergency to 7/Debug, and should be immediately familiar to anyone with even a cursory interest in networking.  Finally, there is the Message, which is the meat of the entry.  The message has two fields: TAG, which is the name of the program or process that generated the entry, and CONTENT, which contains the specific details.

The standard severities are as follows:

Extremely critical “system unusable” messages. 
Messages that require the admin to take immediate action such as a 
network link going down.
A critical Condition such as an application not functioning properly.
An error messages such as an application’s storage limit has been 
exceeded and additional writes are failing.
May indicate that an error will occur if not addressed such as a 
previously set file system capacity warning.
Events that are unusual, but not error conditions
Normal operation such as an application starting, stopping or pausing 
Information useful to developers for debugging the application
Network devices use syslog to send or store messages about network activity or device events that appear to be abnormal in a logging subsystem.  This subsystem controls the delivery of logging messages to destinations such as the local logging buffer or to a central syslog server.  A central syslog server can be a Unix/Linux server whose sole purpose is the collection and storage of syslog messages from different devices, or a more complex system such as an Alienvault SEIM system, which will collect, store and correlate syslog messages in addition to data collected by other means such as netflow and the IDS analysis of packet capture.

For the record, as a security junkie, I'm using a single Alienvault 5.3.3 sensor/server for the collection of syslog events on my home network.  Every device capable of sending syslog messages sends them to it, and important servers have the OSSEC HIPS client.  For syslog collection and a weekly OpenVAS scan, it's quite comfortable with 4 vCPUs and 8GB of RAM.  It would probably run fine with less.  There is no noticeable impact on any of my other VMs.  If you're doing packet capture and analyzing it in real time with Snort, you'll obviously need more RAM at least.

Cisco devices support a number of logging destinations. Those destinations are:
  • The console.  This is where you'll see messages in real time in a local CLI session.
  • Telnet/SSH sessions.  This is functionally the same as the console, but must be explicitly enabled.
  • Internal buffer.  This is a section of memory in a device which stores the most recent messages generated by that device. It is not a long term snapshot, and it is not persistent across reboots.
  • Remote Syslog servers.  The syslog standard UDP port 514 is used to send events to a remote syslog server or servers. 
  • Cisco ASDM GUI.  The ASDM application can collect and display syslog messages from the ASA firewall.
  • Non-syslog servers. The network device can send syslog messages via SNMP traps or SNMP to servers configured to receive these messages.
Some documentation may state that storing more information than necessary is the best policy, but some research says otherwise.  For example, Chamberlain says that a single IDS or IPS unit can produce 20,000 events per day, and that a single firewall can produce tens of thousands of events (or in other words, syslog messages) per minute and that the accuracy of this data's correlation is the square of the number of sources.  Nguen et. al. found that security professionals are unlikely to even attempt to keep up with it all, instead ignoring most of this data until an intrusion has been detected by other means.  If a security professional is using this data to fill in the gaps after the fact, the attacker may have already accomplished their goals and sold what they stole.

Obviously, this line of reasoning isn't limited to just security related alerts.  You can also make the same argument for alerts relating to the day to day operation of devices.  And it's true for whatever methods are used for data collection, whether it's syslog, netflow, or propitiatory IDS platforms.  The more you log, the more likely that important things will get lost in the noise.  The best syslog analyzers aren't perfect.  And conversely, the less you log, the more likely you are to miss things.  You don't want to be awoken at 3am due to an outage when a log message a week ago would have alerted you that something was going wrong.

So there's no simple answer as to how much to log.  You'll have to work that out for yourself and your environment.  There's a ton of academic research out there on the topic of log retention that will give you a lot of guidance and advice.  At home, I'm logging everything possible as I want my Alienvault to have as much data as possible while I learn the platform.  Hopefully the tools will continue to get better, and it will someday become a simple matter of less is more.

Other best practices found in the documentation are to create a log retention policy.
  • Obviously, you want to be consistent, and have something in writing with teeth to back up what you're doing.  
  • Use multiple logging destinations.  You gain reliability and additional control at the expense of additional storage requirements.  And the price of storage continues to drop.
  • Protect your central logging servers, obviously.
  • Err on the side of caution.  Lean more towards to much rather than too little.
Now that we've gone over the basics, let's see some configuration.  For IOS devices, it's pretty straight forward and everything is done with just a handful of commands.

!  enable logging globally

logging on
!  enable the level of logging to the buffer

logging buffered informational 
!  enable the level of logging to the syslog server, and specify the server

logging trap debugging
logging host

For the ASA, its almost the same

!  enable logging globally and specify the syslog server
logging enable
logging host inside
!  specify the level of logging to individual destinations

logging buffered informational
logging trap debugging
logging asdm debugging

To verify, we have the single command, show logging.  This command is used on both IOS and the ASA.

Now that you have the basics of syslog down, you can see what interesting logs are on your server.



Post a Comment

Discuss this post!