How to: Setup Graylog2 and get logs into it

This post is about setting up a Graylog instance, getting information into it from a couple of different sources and searching the data. There's a lot I won't cover in this post (user management, dashboard, custom searches), but you will learn from using Graylog day to day.

What is Graylog?

Open source log management that actually works

Graylog can take all of your logs and store them in a central location, then allow you to search, process and manage them.

Technology behind Graylog

  • Elasticsearch - All logs
  • MongoDB - Metadata
  • graylog-server - Web interface, and server service

Why I prefer Graylog over an ELK stack

  • Graylog comes with user/role management, and LDAP integration out of the box
  • Graylog comes with alerts out of the box
  • Graylog is easier to manage and has more features

FYI: Graylog used to be in two parts. One for the server service (graylog-server) and one for the web UI (graylog-web). This is no longer the case.

Setup - let's do this

Using Fodor

I'm definitely biased as I made Fodor, but this is the easiest way to setup a brand new Graylog instance if you already have a DigitalOcean account. Fodor will create a Droplet for you, then provision everything needed whilst you read on or empty the dishwasher.

https://fodor.xyz/provision/fodorxyz/graylog2

Manual install

If you don’t want to use Fodor or DigitalOcean, you could:

Now you have an instance..

We need to setup an 'input' to send our logs to, then we'll take a look at how to use that data for the good of mankind.

graylog-system-inputs-menu

An input in Graylog is just a way for logs to get in - many types are supported out of the box; for this blog post I'll setup GELF UDP and Syslog UDP.

Adding the inputs

Go to the 'System' menu and 'Inputs'.

GELF UDP

gelf-input-setup

Syslog UDP

We use port 5140 (default is 514) as we can't listen on privileged ports (<1024) by default

syslog-input-setup

Using the inputs

Forwarding logs from another server with rsyslog

  • Provision a secondary server to send messages from rsyslog to your Graylog instance: https://fodor.xyz/provision/ashleyhindle/graylog2-example
  • Or, add *.* @{{ DOMAIN GOES HERE}}:5140;RSYSLOG_SyslogProtocol23Format to /etc/rsyslog.d/90-graylog.conf to forward all syslog messages to Graylog

You can even filter when forwarding from rsyslog:
if $msg contains 'cron' then @{{ DOMAIN GOES HERE}}:5140;RSYSLOG_SyslogProtocol23Format

Sending logs with PHP and Monolog

PHP example to send messages with Monolog in GELF format: https://github.com/ashleyhindle/graylog2-example

<?php  
/* Shortened */

$message = 'Everything is awesome when we\'re living our dream';
$graylogServer = 'dry-autumn-2579-194.fodor.xyz';

$gelfHandler = new GelfHandler(
    new Publisher(
        new UdpTransport($graylogServer, 5555)
    )
);
$log = new Logger('Lego');
$log->pushHandler($gelfHandler);

$log->addWarning('Warning: ' . $message);
$log->addError('Error: ' . $message);
$log->addInfo('Info: ' . $message);
$log->addDebug('Debug: ' . $message);

Searching the logs

http://docs.graylog.org/en/2.0/pages/queries.html lists all the great searches you can run, and the queries available.

Searching the log for my Macbook's name and an error level (source: ashleyhindle-siftware-macbook.local && level: 3) after running the above PHP script gives me this:
search-after-php-script

Searching for ssh gave me:
ssh-search

Live updating

Graylog's search page even has live updating and saved searches to make our lives easier:
live-updating-saved-searches

What's next?

  • The 'Streams' page will let you setup streams which you can attach alerts to
  • Add a dashboard
  • Add team members on the 'System' -> 'Users' screen