ELK收集tomcat日志-linux-Yjc Blog

运行环境

测试环境只有一台服务器，本文的elk运行在同一台机器下。
- 操作系统：Centos 7.4
- elasticsearch：5.5.1
- logstash：5.5.1
- kibana：5.5.1

环境准备

1、关闭防火墙

systemctl disable firewalld
systemctl stop firewalld

2、关闭selinux

setenforce 0
sed -i "s/SELINUX=enforcing/SELINUX=disabled/g" /etc/selinux/config

3、修改limits.conf

vim /etc/security/limits.conf
    es soft memlock unlimited
    es hard memlock unlimitez
    es soft    nofile  65536
    es hard    nofile  65536

4、修改vm.max_map_count值
sysctl -w vm.max_map_count=262144

安装elasticsearch

1.解压安装包，并添加es用户：

mkdir -p /opt/elk/
tar zxf elasticsearch-5.5.1.tar.gz -C /opt/elk/

useradd es
echo "es" | passwd --stdin es
chown -R es:es /opt/elk/elasticsearch-5.5.1

2.修改elasticsearch.yml文件，添加以下参数：

cluster.name: es_cluster
node.name: es01
path.data: /opt/elk/elasticsearch-5.5.1/data
path.logs: /opt/elk/elasticsearch-5.5.1/logs
network.host: 192.168.30.52
discovery.zen.ping.unicast.hosts: ["192.168.30.52"]

node.master: true
node.data: true

3.启动elasticsearch：

su - es
/opt/elk/elasticsearch-5.5.1/bin/elasticsearch -d

4.验证elasticsearch：

curl -X GET http://192.168.30.52:9200/
{
  "name" : "es01",
  "cluster_name" : "es_cluster",
  "cluster_uuid" : "avD7bV4mQeqc7FZY0wHHsA",
  "version" : {
    "number" : "5.5.1",
    "build_hash" : "19c13d0",
    "build_date" : "2017-07-18T20:44:24.823Z",
    "build_snapshot" : false,
    "lucene_version" : "6.6.0"
  },
  "tagline" : "You Know, for Search"
}

安装logstash

1.解压安装包:
tar zxf logstash-5.5.1.tar.gz -C /opt/elk/

2.创建配置文件：

vim config/logstash_synframe.conf
input {
    file {
        codec => multiline {
                 pattern => "^\s"
                 what => "previous"
        }
        path => "/opt/tomcat/logs/catalina.out"
        start_position => "beginning"
    }
}

filter {
  grok {
        patterns_dir => "/opt/elk/logstash-5.5.1/logstash-patterns"
        match => {
            "message" => ["%{MYLOG}", "%{TOMCATLOG}"]
        }
        overwrite => ["message"]
  }
  mutate {
        add_field => { "log_ip" => "192.168.30.52" }
        remove_field => [ "tags", "@version", "path", "host" ]
  }

}


output {
  elasticsearch {
      index => "tomcat-synframe-%{+YYYY.MM.dd}"
      document_type => "tomcat_synframe"
      template => "/opt/elk/logstash-5.5.1/config/synframe_template.json"
      template_name => "synframe_template"
      template_overwrite => true
      hosts => ["192.168.30.52:9200"]
  }
}

配置文件调用了logstash-patterns目录的正则匹配规则，需要创建目录和正则文件：

mkdir /opt/elk/logstash-5.5.1/logstash-patterns
cd /opt/elk/logstash-5.5.1/logstash-patterns
vim java.pattern

JAVACLASS (?:[a-zA-Z$_][a-zA-Z$_0-9]*\.)*[a-zA-Z$_][a-zA-Z$_0-9]*
#Space is an allowed character to match special cases like 'Native Method' or 'Unknown Source'
JAVAFILE (?:[A-Za-z0-9_. -]+)
#Allow special <init> method
JAVAMETHOD (?:(<init>)|[a-zA-Z$_][a-zA-Z$_0-9]*)
#Line number is optional in special cases 'Native method' or 'Unknown source'
JAVASTACKTRACEPART %{SPACE}at %{JAVACLASS:class}\.%{JAVAMETHOD:method}\(%{JAVAFILE:file}(?::%{NUMBER:line})?\)
# Java Logs
JAVATHREAD (?:[A-Z]{2}-Processor[\d]+)
JAVACLASS (?:[a-zA-Z0-9-]+\.)+[A-Za-z0-9$]+
JAVAFILE (?:[A-Za-z0-9_.-]+)
JAVASTACKTRACEPART at %{JAVACLASS:class}\.%{WORD:method}\(%{JAVAFILE:file}:%{NUMBER:line}\)
JAVALOGMESSAGE (.*)
# MMM dd, yyyy HH:mm:ss eg: Jan 9, 2014 7:13:13 AM
CATALINA_DATESTAMP %{MONTH} %{MONTHDAY}, 20%{YEAR} %{HOUR}:?%{MINUTE}(?::?%{SECOND}) (?:AM|PM)
# yyyy-MM-dd HH:mm:ss,SSS ZZZ eg: 2014-01-09 17:32:25,527 -0800
TOMCAT_DATESTAMP 20%{YEAR}-%{MONTHNUM}-%{MONTHDAY} %{HOUR}:?%{MINUTE}(?::?%{SECOND}) %{ISO8601_TIMEZONE}
CATALINALOG %{CATALINA_DATESTAMP:timestamp} %{JAVACLASS:class} %{JAVALOGMESSAGE:logmessage}
# 2014-01-09 20:03:28,269 -0800 | ERROR | com.example.service.ExampleService - something compeletely unexpected happened...
TOMCATLOG %{TOMCAT_DATESTAMP:timestamp} \| %{LOGLEVEL:level} \| %{JAVACLASS:class} - %{JAVALOGMESSAGE:logmessage}

# 自定义
# 2020-07-01 15:36:24.583 INFO PID --- Root WebApplicationContext: initialization started
MYTIMESTAMP 20%{YEAR}-%{MONTHNUM}-%{MONTHDAY} %{HOUR}:?%{MINUTE}(?::?%{SECOND})
PID ([0-9]+)
MYLOG %{MYTIMESTAMP:mytimestamp} %{LOGLEVEL:level} %{PID:pid} --- %{JAVALOGMESSAGE:logmsg}

Logstash 官网提供了很多可以拿来即用的正则表达式，详细请参考 github 项目：https://github.com/logstash-plugins/logstash-patterns-core/tree/master/patterns
而具体的正则需要根据应用日志修改，如本文测试环境tomcat日志格式如：
2020-07-01 15:51:34.312 DEBUG 30600 --- [nio-8080-exec-1] c.f.s.c.u.clientinfo.ClientInfoFilter : 当前请求URL是xxx
那么日志分为时间、level、pid、详细信息：
%{MYTIMESTAMP:mytimestamp} %{LOGLEVEL:level} %{PID:pid} --- %{JAVALOGMESSAGE:logmsg}

另外，配置文件输入到es时指定了es索引的模板，需要创建模板：

vim config/synframe_template.json
{
"template": "tomcat-synframe-*",
"order": 1,
"settings": {
"number_of_shards": 1,
"index.refresh_interval": "60s",
"number_of_replicas": 0
},

"mappings": {
    "tomcat_synframe" : {
         "properties" : {
         "@timestamp" : { "type" : "date" }
        }
    }
}
}

因为本文测试环境只有一个es节点，故设置分片数为1，副本数为0。而生产环境应该设置至少三份副本，分片数看具体节点数配置。

3.启动logstash
nohup ./bin/logstash -w 1 -b 1000 -f /opt/elk/logstash-5.5.1/config/logstash_synframe.conf &> /opt/elk/logstash-5.5.1/logstash.log &

启动后，如logstash.log没异常信息，那么lostash已经将tomcat的日志收集到elasticsearch里，可通过curl http://192.168.30.52:9200/_cat/shards?v查看es的索引信息。

安装kibana

1.解压安装包
tar zxf kibana-5.5.1-linux-x86_64.tar.gz -C /opt/elk/

2.修改配置文件

vim config/kibana.yml
server.port: 5601
server.host: "192.168.30.52"
elasticsearch.url: "http://192.168.30.52:9200"

3.启动kibana
nohup bin/kibana &

4.添加索引
浏览器打开192.168.30.52:5601/app/kibana，如果第一次访问，那么需要添加索引索引名称是es创建的tomcat-synframe-*，添加后，可以在discover界面查看索引信息

目前kibana没有认证访问，后续需要添加nginx服务，通过nginx提供认证访问。