If your organization manages Linux, AIX, HP-UX or Solaris servers in-house, chances are your system administrators at least occasionally need low-level access to those devices. Typically, administrators use some kind of serial console—for example, traditional serial port, Serial-over-LAN or Intelligent Platform Management Interface (IPMI). Managing and auditing console access is not trivial, so many organizations rely on the Conserver application to create session logs when accessing these servers via the serial console. These logs can be useful for various reasons—for example, maintenance or troubleshooting (to review why something crashed), security (to find out who did what—connecting user names to actual users) or compliance (to provide detailed session logs).

This article covers the following:

• How to parse and process serial console logs using syslog-ng Open Source Edition (Balabit).

• How to send the logs to Elasticsearch (Elastic), so you get a complete, searchable audit trail of the console access.

• How to integrate the console logs into a real-time monitoring system using Riemann.

Conserver

Conserver is a wonderful piece of software that lets you manage your infrastructure's serial consoles, whether they be old-style hardware serial ports or state-of-the-art Serial-over-LAN (SOL) baseboards. Its distributed design permits a decentralized user experience using a secure, TLS-encrypted protocol. The straightforward workflow consists of the user connecting to any conserver master node, which then forwards the traffic to the node that manages the console you want to access. As all masters share the same configuration file, it is very straightforward to redistribute consoles among servers automatically (provided they are virtual SOL devices, like IPMI) using configuration management (for example, using the Puppet module we developed at CC-IN2P3).

So where is the catch? As far as we are concerned at the Computing Centre of the National Institute of Nuclear Physics and Particle Physics (CC-IN2P3), the logging mechanism could be greatly improved, because conserver's design is quite ancient now. For example, it does not support logging to syslog. From a user perspective, logging is awesome, as you can use a keystroke to access the logs of the console, and the console logs contain the complete session. From an architecture perspective though, things are not so great, as every conserver master stores the logs of the consoles it manages locally in a file.

syslog-ng

This is where syslog-ng Open Source Edition comes into play. The idea is to transport the logfiles of the conserver masters to our favorite event store back ends, which are Riemann and Elasticsearch. They provide powerful real-time stream processing and long-term indexed storage capabilities, respectively. In addition, with syslog-ng, you simply can send the logs to Riemann and Elasticsearch directly; there is no need for any additional agents (like Logstash). To see how this system works before going into configuration details, watch this video.

The video shows what the user does in the console (in the top-right section of the screen), its effect on the real-time Riemann-dash dashboard (bottom-right) and the near-real-time Elasticsearch front end (Kibana|Elastic, on the left).

As you can see, the user activities and events of the session are transported to the back ends, including useful metadata like conserver.is_attached: true. This tells you whether or not someone was attached to the console (which is obviously the case in this example).

Requirements

To create the system shown in the demo video, you need the following software:

• syslog-ng Open Source Edition 3.7.2 or newer.

• conserver (tested with 8.2.1).

• Riemann (tested with 0.2.10).

• Elasticsearch (tested with 1.6.0).

Note that this article does not cover how to install, configure (in general) and get the above software working. You can find plenty of related tutorials on-line. If you need help with these tasks, check the documentation, mailing lists or on-line forums for the software you need help with. The following sections of this article explain how to configure the components of this infrastructure for the specific needs of our scenario as an example.

Configuring Conserver

The conserver configuration in our setup is nothing special. It creates a unified logfile, which will serve as the glue between conserver and syslog-ng. You can activate the unified logfile using either of the following methods:

1) Run the conserver executable with the -U /var/log/console.log flag.

2) Use the following configuration block in conserver.cf:

<config * { unifiedlog * /var/log/console.log; }

You also can set the server's general logfile (where conserver stores the global messages that are unrelated to individual consoles)—for example, to /var/log/conserver.log.

Both /var/log/conserver.log and /var/log/console.log will be inputs for syslog-ng. You might want to take special care of the log rotation of these files. As you are sending them to Elasticsearch, there is no need to keep them for too long.

Configuring syslog-ng Open Source Edition

You need to install syslog-ng locally on each conserver master and on a central host (that is, your logserver) that will gather all the events from the conserver hosts. The local instances will parse, process and enrich the console output, while the central host will collect them and send them over to the two back-end systems, Riemann and Elasticsearch. Note that you could get the same results using only local instances, but most people prefer to centralize first, for various reasons.

Configuring syslog-ng on the Conserver Hosts:

The following is an example configuration file for running syslog-ng on the conserver hosts. As you can see, it has three sources:

• s_internal tracks the internal messages of syslog-ng (very handy for troubleshooting, stored only locally).

• s_console reads the logs of the individual consoles.

• s_conserver reads the global messages of the conserver master.

The s_console and s_conserver sources process conserver's unified logfile. Since the format of the console and conserver messages is different, we have to configure syslog-ng to parse them differently, then forward them to the central syslog-ng server (you can add any other sources as needed for your environment):

@version: 3.7

@include scl.conf

options {
    threaded(yes);
};

source s_conserver {
  channel {
    source {
      file(
        '/var/log/conserver.log',
        flags(no-parse)
      );
    };
    parser {
      csv-parser(
        columns(tmp.date,PROGRAM,PID,MESSAGE)
        delimiters(' :')
        quote-pairs('[]()')
        flags(greedy)
      );
    };
    rewrite {
      set('$(strip $MESSAGE)', value(MESSAGE));
    };
  };
};

source s_console {
  channel {
    source {
      file('/var/log/console.log');
    };
    junction {
      channel {
        filter{
          program('\*/div>);
        };
        rewrite {
          subst('\*/div>, '', value(PROGRAM));
          set(
            'true',
            value('.SDATA.console.is_attached')
          );
        };
        flags(final);
      };
      channel {
        rewrite {
          set(
            'false',
            value('.SDATA.console.is_attached')
          );
        };
        flags(fallback);
      };
    };
    rewrite {
      set('$PROGRAM', value(HOST));
      set('console', value(PROGRAM));
    };
  };
};

source s_internal {
  internal();
};

destination d_remote {
  network(
      "",
      transport(tcp),
      port(514),
      flags(syslog-protocol)
  );
};

destination d_internal {
  file("/var/log/syslog-ng.log");
};

log {
  source(s_console);
  source(s_conserver);
  destination(d_remote);
};

log {
  source(s_internal);
  destination(d_internal);
};

Global Conserver Logs—the s_conserver Source:

If you are not familiar with syslog-ng, the s_conserver and s_console sections can be a bit intimidating. To better understand how they work, take a look at a sample message conserver produces:

[Thu Sep  3 22:29:52 2015] conserver (13550): 
 ↪[node42] automatic reinitialization

The related source definition contains three blocks:

1. source: the file path and the no-parse flag, which tells syslog-ng to read the logfile, but not to parse it, as a syslog message (because it is not exactly in syslog format).

2. parser: the csv-parser splits the message at the colons (:) and extracts the following fields: tmp.date, PROGRAM, PID and MESSAGE. The parser's other options ensure that the field values are parsed properly.

3. rewrite: defines a rewrite rule to remove leading and trailing spaces from the MESSAGE key. (If you find a way to omit this point using 2., please let me know.)

This configuration parses the above example message into the following structured data:

tmp.date: Thu Sep  3 22:29:52 2015
PROGRAM: conserver
PID: 13550
MESSAGE: [node42] automatic reinitialization

Console Logs—the s_console Source:

Here are two example messages from two different consoles:

node03: ACPI: No handler for Region [POWR] 
 ↪(ffff8808248bb150) [IPMI]

node66*: node66 login: root

The first one is an unattended message probably produced by an ACPI signal. The second one, as hinted by the * (asterisk) character appended to the name of the console, is a message produced while someone was attached to the console (using console node66). We will use this hint to produce additional metadata. The source consists again of three parts:

1. source: the file path, this time without flags. That way, syslog-ng will try to parse the message using the symbolic pattern %{PROGRAM}: %{MESSAGE}. As a result, node03 and node66* will be parsed into the PROGRAM key.

2. junction: a construct with two mutually exclusive (hence the final and fallback flags) channels (similar to a try: except structure in Python). The two channels correspond to the two cases in the example: the first one for messages when someone is attached to the console (thus, the PROGRAM field contains an asterisk character), and the second for messages without anybody attached. To tell one case from the other easily (for example, when reviewing the messages in Elasticsearch), this information is stored in the .SDATA.console.is_attached key.

3. rewrite: Rewrites the PROGRAM and HOST fields to their sane content: console and the name of the console, respectively.

So in the above examples, the messages are parsed into the following structured data:

PROGRAM: console
HOST: node03
MESSAGE: ACPI: No handler for Region [POWR] 
 ↪(ffff8808248bb150) [IPMI]
.SDATA.console.is_attached: false

PROGRAM: console
HOST: node66
MESSAGE: node66 login: root
.SDATA.console.is_attached: true

Forwarding the Logs to the Central syslog-ng server, d_remote:

The rest of the syslog-ng configuration is simple: we just send the structured payload using the syslog IETF RFC5424 protocol (hence the syslog-protocol flag) to the central syslog-ng server. All RFC5424 keys, including .SDATA.*, are sent over to the central syslog-ng server automatically. The only part that we parsed from the conserver logs that is not transferred to the central server is the tmp.date field. Instead, we will use the time when syslog-ng processes the message (which is a good approximation for current logs). If you absolutely want to use the value from tmp.date (because, for example, you want to send old conserver logs to the remote server), you can use the date parser from the syslog-ng-incubator project.

Configuring syslog-ng on the Central Logserver:

On the central syslog-ng server, we have to route the console and conserver messages received from the conserver hosts to the Riemann and Elasticsearch back ends. The following syslog-ng configuration does exactly that; the only adjustment is that it removes the .SDATA. prefix from the fields, so they are more readable:

@version: 3.7

@include scl.conf

options {
  threaded(yes);
};

block destination realtime (
  host()
  port(5555)
  type("udp")
  throttle(0)
  flush-lines(1)
)
{
  riemann(
    flush-lines(`flush-lines`)
    throttle(`throttle`)
    server(`host`)
    port(`port`)
    type(`type`)
    ttl("${ttl:-300}")
    host("$HOST")
    description("$MESSAGE")
    attributes(
      scope(all-nv-pairs)
      key(".SDATA.*"
        rekey( shift(7) )
      )
    )
  );
};

source s_remote_tcp {
  channel {
    source {
      network(
        transport(tcp)
        port(514)
        flags(syslog-protocol)
        tags("syslog")
        so-rcvbuf(8388608)
      );
    };
  };
};

source s_internal {
  internal();
};

destination d_elasticsearch {
  elasticsearch(
    index("syslog-${YEAR}.${MONTH}.${DAY}"),
    type("syslog"),
    flush-limit(1),
    template("$(format-json -s all-nv-pairs --rekey 
     ↪.SDATA.* --shift 7 --key ISODATE)")
    cluster("elasticsearch")
    port(9300)
    server("localhost")
    client_mode("transport")
    time-zone("UTC")
  );
};

destination d_internal {
  file("/var/log/syslog-ng.log");
};

destination d_riemann {
  realtime(
    host("riemann"),
  );
};

log {
  source(s_remote_tcp);
  destination(d_riemann);
  destination(d_elasticsearch);
};

log {
  source(s_internal);
  destination(d_internal);
};

Conclusion

From this article, you have learned how to create a system that allows you to review serial console logs in real time and make them accessible for free-text searching on a modern user interface. This is helpful for maintenance and troubleshooting purposes, and also for meeting auditing and compliance requirements. To achieve these goals, conserver can be integrated with Riemann and Elasticsearch. To integrate these services, you can use syslog-ng Open Source Edition, a flexible log collecting and processing application that can collect and parse the log messages and forward them to the Riemann Elasticsearch back end.

Improvements

The syslog-ng application is very flexible and has powerful message-processing capabilities. If you learn a bit about its possibilities, you can find several ways to improve the configuration described in the article. Here are some ideas that you can do with syslog-ng:

• Write a smarter parser to extract the name of the console from server messages (where available).

• Correlate the console and server messages, extract the name of the user name from the server messages and add them to the console messages. That way, console events contain the name of the attached user, which makes troubleshooting and auditing easier.

• Configure alerts for consoles that are attached for too long.

• Use the date parser from the syslog-ng incubator project to use the timestamp that conserver adds to the messages.

Useful syslog-ng Features

The syslog-ng Open Source Edition application is a syslog dæmon that allows you to collect your log data and much more—for example:

• Flexibly collect, parse, classify and correlate logs from various sources.

• Send log data to message queues, including AMQP, STOMP or Apache Kafka.

• Store your log data in plain files, HDFS, SQL databases or MongoDB.

• Forward your log data to monitoring tools like Riemann, Redis or Graphite.

• Process CSV, JSON or plain-text messages.

• Rewrite, reformat and transform your log messages.

• You can write your own modules in C, Java or Python.

Real-Time Monitoring with Riemann

Riemann helps you monitor distributed systems. It aggregates events from your servers and applications, and it allows you to combine and process these events with a powerful stream processing language. You can query the events and visualize the results of these queries on dashboards. To get notifications promptly, you also can trigger alerts (for example, in e-mail or SMS). Since the clients actively push the data into Riemann, your dashboards display up-to-date information (in contrast with other systems that only pull event data every few minutes). If you use the integrated WebSocket server, you even can have completely synchronous event handling, all the way from the event source to the browser.

Resources

Demo Video: https://webcast.in2p3.fr/videos-syslog_ng_conserver

Conserver: http://www.conserver.com

Serial-over-LAN (SOL): https://en.wikipedia.org/wiki/Serial_over_LAN

Intelligent Platform Management Interface (IPMI): https://en.wikipedia.org/wiki/Intelligent_Platform_Management_Interface

Puppet Module for Conserver, Developed by CC-IN2P3: http://github.com/ccin2p3/puppet-conserver

CC-IN2P3: http://cc.in2p3.fr

Accessing Conserver Logs with a Keystroke: http://conserver.com/docs/console.man.html

syslog-ng Open Source Edition: http://www.syslog-ng.org

Riemann: http://riemann.io

Elasticsearch: http://elastic.co/products/elasticsearch

Riemann-dash: http://riemann.io/dashboard.html

Kibana: http://elastic.co/products/kibana

syslog IETF RFC5424 Protocol: https://tools.ietf.org/html/rfc5424

Date parser from the syslog-ng-incubator project: https://github.com/balabit/syslog-ng-incubator/tree/master/modules/date

Using correlation in syslog-ng: https://www.balabit.com/sites/default/files/documents/syslog-ng-ose-latest-guides/en/syslog-ng-ose-guide-admin/html/patterndb-correlation.html

Alerting for consoles that are attached for too long: https://www.balabit.com/sites/default/files/documents/syslog-ng-ose-latest-guides/en/syslog-ng-ose-guide-admin/html/patterndb-actions-correlation.html