CentOS 6 安裝ELK(Elastic Search、Logstash、Kibana)

首先安裝 java環境
yum install java-1.7.0-openjdk httpd unzip

安裝Elastic Search
下載公鑰
rpm --import http://packages.elasticsearch.org/GPG-KEY-elasticsearch

在 /etc/yum.repos.d/ 目錄下新增檔案，例如 elasticsearch.repo，內容如下
vi /etc/yum.repos.d/elasticsearch.repo

[elasticsearch-1.3]
name=Elasticsearch repository for 1.3.x packages
baseurl=http://packages.elasticsearch.org/elasticsearch/1.3/centos
gpgcheck=1
gpgkey=http://packages.elasticsearch.org/GPG-KEY-elasticsearch
enabled=1

安裝Elastersearch
yum install elasticsearch

設定Elasticsearch
編輯 /etc/elasticsearch/elasticsearch.yml
vi /etc/elasticsearch/elasticsearch.yml
cluster.name: "LogCluster"
node.name: "LogMaster"
node.master: true
node.data: true

path.conf: /etc/elasticsearch
path.data: /datapool/data1
path.work: /datapool/work
path.logs: /datapool/log

mkdir /datapool
mkdir /datapool/data1
mkdir /datapool/work
mkdir /datapool/log

將服務加入系統
chkconfig --add elasticsearch

啟動服務
/etc/init.d/elasticsearch start

測試啟動是否成功
curl localhost:9200/_nodes/process?pretty

安裝Logstash
安裝金鑰
rpm --import http://packages.elasticsearch.org/GPG-KEY-elasticsearch

在 /etc/yum.repos.d/ 目錄下新增檔案，例如 logstash.repo，內容如下
vi /etc/yum.repos.d/logstash.repo
[logstash-1.4]
name=logstash repository for 1.4.x packages
baseurl=http://packages.elasticsearch.org/logstash/1.4/centos
gpgcheck=1
gpgkey=http://packages.elasticsearch.org/GPG-KEY-elasticsearch
enabled=1

安裝Logstash
yum install logstash

配置logstash，如下是Logstash的配置文件
vi /etc/logstash/conf.d/logstasg_access.conf
input {
file {
path => "/var/log/httpd/access_log"
type => "apache" # a type to identify those logs (will need this later)
}
}

filter {
if [type] == "apache" { # this is where we use the type from the input section
grok {
match => [ "message", "%{COMBINEDAPACHELOG}" ]
}
date {
# Try to pull the timestamp from the 'timestamp' field (parsed above with
# grok). The apache time format looks like: "18/Aug/2011:05:44:34 -0700"
match => [ "timestamp", "dd/MMM/yyyy:HH:mm:ss Z" ]
}
if [user-agent] != "-" and [user-agent] != "" {
useragent {
add_tag => [ "UA" ]
source => "user-agent"
}
}
if "UA" in [tags] {
if [device] == "Other" { mutate { remove_field => "device" } }
if [name] == "Other" { mutate { remove_field => "name" } }
if [os] == "Other" { mutate { remove_field => "os" } }
}
}
}


output {
elasticsearch {
host => "localhost"
cluster => "LogCluster"
node_name => "LogMaster"
}
}

啟動 logstash 服務
cd /opt/logstash/bin/
./logstash -f /etc/logstash/conf.d/logstasg_access.conf

安裝Kibana
wget  https://download.elasticsearch.org/kibana/kibana/kibana-3.1.2.zip &&  unzip kibana-3.1.2.zip &&  mv kibana-3.1.2  kibana && mv kibana  /var/www/html/

設定apache
vi /etc/httpd/conf/httpd.conf
<VirtualHost xxx.xxx.xxx.xxx:80>
ServerAdmin admin@opsnotes.com
DocumentRoot /var/www/html/kibana
ServerName kibana.opsnotes.net
ErrorLog logs/kibana.opsnotes.net-error_log
CustomLog logs/kibana.opsnotes.net-access_log common
</VirtualHost>

設定防火牆
#
# 開啟HTTP 80 port
#
iptables -A INPUT -p tcp --dport 80 -j ACCEPT
#
# 設定ElasticSearch使用的port 9200-9300
#
iptables -A INPUT -p tcp --dport 9200:9300 -j ACCEPT

重啟 apache 服務
service httpd restart

 

建立第一個Logstash設定檔

安裝完ELK之後，可以嘗試建立第一個設定檔。由於Kibana是設定在Apache服務中，我們就先拿apache access log來測試(位於 /var/log/httpd/access_log)
請到 /etc/logstash/conf.d 目錄下新增第一個設定檔，名稱可以任意定，例如 apache_access.conf。主要分成三個段落

    input : 設定輸入的類型、位置等資訊
        以檔案 file 的形式輸入
        指定檔案位置
    filter : 設定解析的方式、衍生欄位等
        logstash已經有內建解析access log的方式，請直接用 grok 的設定。
    output : 輸出的位置
        本範例直接輸出到 elasticsearch 中
        若您的elasticsearch有指定 cluster名稱及node名稱，請記得設定在這邊，以免找不到服務。
   
input {
  file {
    path => "/var/log/httpd/access_log"
    type => "apache"  # a type to identify those logs (will need this later)
  }
}

filter {
  if [type] == "apache" {   # this is where we use the type from the input section
    grok {
      match => [ "message", "%{COMBINEDAPACHELOG}" ]
    }
  }
}

output {
  elasticsearch {
    host => "localhost"
    cluster => "LogCluster"
    node_name => "LogMaster"
  }
}
設定完成後，還要修改 /etc/init.d/logstash ，將啟動的使用者及群組改為root，以免權限不足沒辦法讀取access_log。

LS_USER=root
LS_GROUP=root

最後重新啟動logstash服務
   
# service logstash restart
Killing logstash (pid 1993) with SIGTERM
Waiting logstash (pid 1993) to die...
Waiting logstash (pid 1993) to die...
logstash stopped.
logstash started.