Manage System Logs with Graylog and rsyslogd

Graylog is widely used for log management and analysis.
There are 4 main components of Graylog:

• Graylog Server
• MongoDB
• ElasticSearch
• NGINX as ReverseProxy to secure Graylog API

Below steps will help, reader to basic installation of Single Node Graylog server and redirect system logs from rsyslogd to Graylog.
Do let me know your queries on ngurjar [at] neeleshgurjar [dot] co [dot] in

Check Graylog single-node installer for  AWS : https://github.com/neeleshg/aws-graylog-installer.git

 

 

 

 

 

 

 

 

 

 

 

 

 

– Install MongoDB

sudo su -
apt-get update && apt-get upgrade
apt-get install apt-transport-https openjdk-8-jre-headless uuid-runtime pwgen
apt-get install mongodb-server
systemctl restart mongodb

– Install & Configure ElasticSearch

wget -qO - https://packages.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
echo "deb https://packages.elastic.co/elasticsearch/2.x/debian stable main" | sudo tee -a /etc/apt/sources.list.d/elasticsearch-2.x.list
apt-get update && sudo apt-get install elasticsearch

– In /etc/elasticsearch/elasticsearch.yml provide cluster.name

cluster.name: graylog

– Restart elasticsearch service

systemctl daemon-reload
systemctl enable elasticsearch.service
systemctl restart elasticsearch.service

– Install & Configure Graylog-server:

wget https://packages.graylog2.org/repo/packages/graylog-2.2-repository_latest.deb
dpkg -i graylog-2.2-repository_latest.deb
apt-get update && apt-get install graylog-server

– Provide password_secret & root_password_sha2 in /etc/graylog/server/server.conf:

For password_secret, just copy the output of below command provide in config file:

pwgen -N 1 -s 96

For root_password_sha2 just copy the output of below command provide in config file:

echo -n yourpassword | sha256sum

– To Configure REST & Web interface provide below configuration in /etc/graylog/server/server.conf

rest_listen_uri = http://127.0.0.1:9000/api/
 

– Restart graylog service

systemctl daemon-reload
systemctl enable graylog-server.service
systemctl start graylog-server.service

– Install and Configure NGINX as ReverseProxy

apt-get install nginx
mv /etc/nginx/sites-available/default /etc/nginx/sites-available/default_orig

– Add below content to /etc/nginx/sites-available/default

server
{
   listen 80 default_server;
   listen [::]:80 default_server ipv6only=on;
   server_name <ELASTIC OR PUBLIC IP>;

   location /
    {
      proxy_set_header Host $http_host;
      proxy_set_header X-Forwarded-Host $host;
      proxy_set_header X-Forwarded-Server $host;
      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
      proxy_set_header X-Graylog-Server-URL http://<ELASTIC OR PUBLIC IP>/api;
      proxy_pass http://127.0.0.1:9000;
    }
}

– Restart Nginx

systemctl restart nginx

Now access http://<ElasticIP> in browser and login with admin / yourpassword

Integrating Graylog with Rsyslogd

– Create and Start Input in Graylog console:

• Go to Systems –> Input
• Select “Syslog UDP” and click Launch New Input
• Provide Node from dropbox
• Enter Title, Port (better to use higher than 1024)
• Select ‘Store full message’ if required
• Click ‘Save’

– On Linux Server create new file /etc/rsyslog.d/60-graylog.conf and add below line to it:

*.* @<graylog_server_ip>:<inputPort>;RSYSLOG_SyslogProtocol23Format

– Restart rsyslogd

systemctl restart rsyslogd

Check Sources & Streams –> All messages, you will see the logs