General Process:
- Cloudtrail will send all API logs to S3 bucket
- And will notify to SNS topic
- SNS will send log details to SQS queue
- Graylog will poll message from SQS and get the logs from that S3 file
- By default all logs will be sent to “All Messages” stream.
- Stream can be used to filter out the logs
- All Messages will be redirected to a Pipeline
- Pipeline Rule will search for pattern and if pattern matches then will redirect those logs to newly created stream.
Check Graylog single node installer for AWS: https://github.com/neeleshg/aws-graylog-installer.git
Steps:
– Create SNS Topic
– Create SQS and subscribe to that SNS topic
– Enable CloudTrail Logs and use the SNS topic
– Install aws plugin for graylog.
- Download jar file from HERE
- Copy it in plugin_dir which is normally: /usr/share/graylog-server/plugin
- Restart graylog server
– Create IAM User and attach below Policies to that user:
This user will be configured in graylog-aws plugin. Which will have access to SQS and S3 bucket.
– Graylog-S3 Policy:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "Stmt1411854479000",
"Effect": "Allow",
"Action": [
"s3:GetObject"
],
"Resource": [
"arn:aws:s3:::<BUCKET_NAME>/*"
]
}
]
}
– Graylog-SQS Policy:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "Stmt1411834838000",
"Effect": "Allow",
"Action": [
"sqs:DeleteMessage",
"sqs:ReceiveMessage"
],
"Resource": [
"<ARN of SQS Queue>"
]
}
]
}
Copy Access and Secret Keys of that user
– Configure Graylog plugin use above access and secret keys in it.
Go to System –> Configurations –> AWS Plugin Configuration –> Configure
– Create Stream for ec2 specific logs
Go to Streams –> Create Stream –> Enter Title and Description
Note the stream Name and ID.
To get stream ID, Just right click on Name of Stream and click “Copy Link Address”. Paste it somewhere. You will get id in it.
For eg. If URL is http://<graylog_ip>/streams/xxxyyyyz3ggg6666iiuop/search then Stream ID is “xxxyyyyz3ggg6666iiuop”
– Create & Configure Pipeline
– First need to create rule.
Go to System –> Pipelines –> Click “Manage Rules” –> Click “Create Rule”
Provide description of rule and define actual rule in “Rule Source”.
For eg. in our condition, rule will be similar like:
rule "filter ec2"
when
has_field("event_source") && to_string($message.event_source)=="ec2.amazonaws.com"
then
route_to_stream("Stream_Name","Stream ID")
end
- Click “Manage Pipelines” –> “Add new pipeline”
- Provide Title and Description of Pipeline and click “Save”
- Click “Edit Connections” and Select Stream “All messages” as Input connection to this pipeline.
- Click Edit of Stage 0
- Select created Rule in “Stage Rules” dropbox and click Save.
Now you will see all messages from ec2.amazonaws.com in newly created stream
Hi Neelesh,
Thanks for the write up. I don’t understand what is the need for SNS & SQS. Can’t graylog read the logs directly from S3 bucket?