Install filebeat

on RPM Based:

sudo rpm --import https://packages.elastic.co/GPG-KEY-elasticsearch

create /etc/yum.repos.d/elastic.repo

[elastic-6.x]
name=Elastic repository for 6.x packages
baseurl=https://artifacts.elastic.co/packages/6.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md
sudo yum install filebeat

on deb based:

wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
sudo apt-get install apt-transport-https
echo "deb https://artifacts.elastic.co/packages/6.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-6.x.list
sudo apt-get update && sudo apt-get install filebeat

Configure

/etc/filebeat/filebeat.yml:

#============================== Dashboards =====================================
# These settings control loading the sample dashboards to the Kibana index. Loading
# the dashboards is disabled by default and can be enabled either by setting the
# options here, or by using the `-setup` CLI flag or the `setup` command.
setup.dashboards.enabled: true
setup.kibana:

  # Kibana Host
  # Scheme and port can be left out and will be set to the default (http and 5601)
  # In case you specify and additional path, the scheme is required: http://localhost:5601/path
  # IPv6 addresses should always be defined as: https://[2001:db8::1]:5601
  host: "elkhost:5601"
#-------------------------- Elasticsearch output ------------------------------
output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["elkhost:9200"]

optionally, change the name

edit filebeat.yml, change “name:” to set the ‘beta.name’ on kibana

Modify pattern:

/usr/share/filebeat/module/nginx/access/ingest/default.json

      "patterns":[
        "\"?%{IP_LIST:nginx.access.remote_ip_list} - %{DATA:nginx.access.user_name} \\[%{HTTPDATE:nginx.access.time}\\] \"%{WORD:nginx.access.method} %{DATA:nginx.access.url} HTTP/%{NUMBER:nginx.access.http_version}\" %{NUMBER:nginx.access.response_code} %{NUMBER:nginx.access.body_sent.bytes} \"%{DATA:nginx.access.referrer}\" \"%{DATA:nginx.access.agent}\" \"%{DATA:nginx.access.http_x_forwarded_for}\" %{NUMBER:nginx.access.request_time}"
        ],

Create request_time Field

/etc/filebeat/fields.yml:

            - name: request_time
              type: float
              description: >
                processing time in seconds with a milliseconds resolution

Change the template

Add nginx.access.request_time field.

/usr/share/filebeat/kibana/default/index-pattern/filebeat.json

{\"count\": 0, \"analyzed\": false, \"aggregatable\": true, \"name\": \"nginx.access.request_time\", \"searchable\": true, \"indexed\": true, \"doc_values\": true, \"type\": \"number\", \"scripted\": false},

Update NGINX log format

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" $request_time';

     access_log /var/log/nginx/access.log main;

for EB use $http_x_forwarded_for instead of $remote_addr:

    log_format  main  '$http_x_forwarded_for - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" $request_time';

Enable nginx module on filebeat

filebeat modules enable nginx

Set for auto start on boot

rpm:

sudo update-rc.d filebeat defaults 95 10

or (deb)

sudo chkconfig --add filebeat

Sample Config Files

https://github.com/crochik/tldr

TL;DR


0 Comments

Leave a Reply

Your email address will not be published. Required fields are marked *