Loghost

Provides centralized logging infrastructure inside a project including remote rsyslog and ELK (Elasticsearch, Logstash, Kibana).

Please refer to chapter Logging for configuration examples and hints on using the ELK stack.

Components

  • rsyslog (srv interface, port UDP 514)

  • Elasticsearch (srv interface, ports TCP 9200 and 9300)

  • elasticsearch-curator

  • Logstash

  • logstash-forwarder

  • Kibana

Default setup

  • Elasticsearch, Logstash and Kibana are installed on the loghost.

  • logstash creates elasticsearch indices of the pattern logstash-YYYY.MM.DD.

  • elasticsearch-curator deletes indices older than 90 days.

  • logstash-forwarder ships locally generated logs from all machines to the loghost.

  • rsyslog forwards all syslog entries to the loghost.

Configuration

Below is a list of your configuration entry points for the involved components.

Warning

All configuration needs to be performed as a service user.

  • rsyslog: /etc/rsyslog.d/SNIPPET.conf

  • logstash-forwarder: /etc/logstash-forwarder/conf.d/SNIPPET.conf

  • logstash: /etc/logstash/conf.d/SNIPPET.conf

Interaction

  • rsyslog: sudo /etc/init.d/rsyslog restart for restarts after configuration changes

    Note

    rsyslog ignores invalid configuration statements, so be sure to check /var/log/messages for errors after a restart.

  • logstash-forwarder: sudo /etc/init.d/logstash-forwarder restart for restarts after configuration changes

  • Logstash: sudo /etc/init.d/logstash restart for restarts after configuration changes

  • Kibana: refer to our Logging section for how to interact with Kibana

Monitoring

We monitor for:

  • running processes

  • reachable ports

  • correctly written log files

  • correctly pruned Elasticsearch indices