ELK 5.0.1+Filebeat5.0.1实时监控MongoDB日志并使用正则解析mongodb日志的示例分析

2024-04-02 19:04:59 583人浏览薄情痞子

摘要

elk 5.0.1+Filebeat5.0.1实时监控mongoDB日志并使用正则解析monGodb日志的示例分析，针对这个问题，这篇文章详细介绍了相对应的分析和解答，希望可以帮助更多想解决这个问题的小伙伴

elk 5.0.1+Filebeat5.0.1实时监控 mongoDB日志并使用正则解析monGodb日志的示例分析，针对这个问题，这篇文章详细介绍了相对应的分析和解答，希望可以帮助更多想解决这个问题的小伙伴找到更简单易行的方法。

关于ELK5.0.1的安装部署，请参考博文（ ELK 5.0.1+Filebeat5.0.1 for linux RHEL6.6 监控MongoDB日志），
重点说明如何适用filebeat实时监控mongodb数据库日志及在logstash正则解析mongodb日志。
部署完ELK5.0.1后，在需要监控mongodb日志的数据库服务器上部署filebeat来抓取日志，
首先需要修改filebeat配置文件：
[root@se122 filebeat-5.0.1]# pwd
/opt/filebeat-5.0.1
[root@se122 filebeat-5.0.1]#
[root@se122 filebeat-5.0.1]# ls
data filebeat filebeat.full.yml filebeat.template-es2x.JSON filebeat.template.json filebeat.yml scripts
[root@se122 filebeat-5.0.1]# cat filebeat.yml
filebeat :
prospectors :
-
paths :
- /root/rs0-0.log #filebeat负责实时监控的mongodb日志
document_type : mongodblog #指定filebeat发送到logstash的mongodb日志的文档类型为document_type，一定要指定（logstash接收解析匹配要使用）
input_type : log
reGIStry_file :
/opt/filebeat-5.0.1/data/registry
output.logstash:
hosts: ["10.117.194.228:5044"] #logstash服务部署的机器IP地址及运行的服务端口号
[root@se122 filebeat-5.0.1]#
其次修改logstash配置文件：
[root@rhel6 config]# pwd
/opt/logstash-5.0.1/config
[root@rhel6 config]# cat logstash_mongodb.conf
#input {
# stdin {}
#}
input{
beats {
host => "0.0.0.0"
port => 5044
type => mongodblog #指定filebeat输入的日志类型是mongodblog
}
}

filter {
if [type] == "mongodblog" { #过滤器，只处理filebeat发送过来的mogodblog日志数据
grok { #解析发送过来的mognodblog日志
match => ["message","%{TIMESTAMP_ISO8601:timestamp}\s+%{MONGO3_SEVERITY:severity}\s+%{MONGO3_COMPONENT:component}\s+(?:\[%{DATA:context}\])?\s+%{GREEDYDATA:body}"]
}
if [component] =~ "WRITE" {
grok { #第二层解析body部分，提取mongodblog中的command_type、db_name、command、spend_time字段
match => ["body","%{Word:command_type}\s+%{DATA:db_name}\s+\w+\:\s+%{GREEDYDATA:command}%{INT:spend_time}ms$"]
}
} else {
grok {
match => ["body","\s+%{DATA:db_name}\s+\w+\:\s+%{WORD:command_type}\s+%{GREEDYDATA:command}protocol.*%{INT:spend_time}ms$"]
}
}
date {
match => [ "timestamp", "UNIX", "YYYY-MM-dd HH:mm:ss", "ISO8601"]
remove_field => [ "timestamp" ]
}
}
}

output{
elasticsearch {
hosts => ["192.168.144.230:9200"]
index => "mongod_log-%{+YYYY.MM}"
}
stdout {
codec => rubydebug
}
}
[root@rhel6 config]#
然后，确保ELK服务端的服务进程都已经开启，启动命令：
[elasticsearch@rhel6 ]$ /home/elasticsearch/elasticsearch-5.0.1/bin/elasticsearch

[root@rhel6 ~]# /opt/logstash-5.0.1/bin/logstash -f /opt/logstash-5.0.1/config/logstash_mongodb.conf

[root@rhel6 ~]# /opt/kibana-5.0.1/bin/kibana
在远程端启动filebeat，开始监控mongodb日志：
[root@se122 filebeat-5.0.1]# /opt/filebeat-5.0.1/filebeat -e -c /opt/filebeat-5.0.1/filebeat.yml -d "Publish"
2017/02/16 05:50:40.931969 beat.go:264: INFO Home path: [/opt/filebeat-5.0.1] Config path: [/opt/filebeat-5.0.1] Data path: [/opt/filebeat-5.0.1/data] Logs path: [/opt/filebeat-5.0.1/logs]
2017/02/16 05:50:40.932036 beat.go:174: INFO Setup Beat: filebeat; Version: 5.0.1
2017/02/16 05:50:40.932167 logp.go:219: INFO Metrics logging every 30s
2017/02/16 05:50:40.932227 logstash.go:90: INFO Max Retries set to: 3
2017/02/16 05:50:40.932444 outputs.go:106: INFO Activated logstash as output plugin.
2017/02/16 05:50:40.932594 publish.go:291: INFO Publisher name: se122
2017/02/16 05:50:40.935437 async.go:63: INFO Flush Interval set to: 1s
2017/02/16 05:50:40.935473 async.go:64: INFO Max Bulk Size set to: 2048
2017/02/16 05:50:40.935745 beat.go:204: INFO filebeat start running.
2017/02/16 05:50:40.935836 registrar.go:66: INFO Registry file set to: /opt/filebeat-5.0.1/data/registry
2017/02/16 05:50:40.935905 registrar.go:99: INFO Loading registrar data from /opt/filebeat-5.0.1/data/registry
2017/02/16 05:50:40.936717 registrar.go:122: INFO States Loaded from registrar: 1
2017/02/16 05:50:40.936771 crawler.go:34: INFO Loading Prospectors: 1
2017/02/16 05:50:40.936860 prospector_log.go:40: INFO Load previous states from registry into memory
2017/02/16 05:50:40.936923 registrar.go:211: INFO Starting Registrar
2017/02/16 05:50:40.936939 sync.go:41: INFO Start sending events to output
2017/02/16 05:50:40.937148 spooler.go:64: INFO Starting spooler: spool_size: 2048; idle_timeout: 5s
2017/02/16 05:50:40.937286 prospector_log.go:67: INFO Previous states loaded: 1
2017/02/16 05:50:40.937404 crawler.go:46: INFO Loading Prospectors completed. Number of prospectors: 1
2017/02/16 05:50:40.937440 crawler.go:61: INFO All prospectors are initialised and running with 1 states to persist
2017/02/16 05:50:40.937478 prospector.go:106: INFO Starting prospector of type: log
2017/02/16 05:50:40.937745 log.go:84: INFO Harvester started for file: /root/rs0-0.log
我们看到，这里已经开始实时监控mongodb日志是/root/rs0-0.log；然后，我们去logstash开启的前台窗口，可以看到有如下信息：
{
"severity" => "I",
"offset" => 243843239,
"spend_time" => "0",
"input_type" => "log",
"source" => "/root/rs0-0.log",
"message" => "2017-02-04T14:03:30.025+0800 I COMMAND [conn272] command admin.$cmd command: replSetGetStatus { replSetGetStatus: 1 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:364 locks:{} protocol:op_query 0ms",
"type" => "mongodblog",
"body" => "command admin.$cmd command: replSetGetStatus { replSetGetStatus: 1 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:364 locks:{} protocol:op_query 0ms",
"command" => "{ replSetGetStatus: 1 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:364 locks:{} ",
"tags" => [
[0] "beats_input_codec_plain_applied"
],
"component" => "COMMAND",
"@timestamp" => 2017-02-04T06:03:30.025Z,
"db_name" => "admin.$cmd",
"command_type" => "replSetGetStatus",
"@version" => "1",
"beat" => {
"hostname" => "se122",
"name" => "se122",
"version" => "5.0.1"
},
"host" => "se122",
"context" => "conn272"
}
这说明logstash按照配置文件正常过滤并按照指定的正则解析了mongodblog日志，再到kibana创建索引：
ELK 5.0.1+Filebeat5.0.1实时监控MongoDB日志并使用正则解析mongodb日志的示例分析
然后，就能在kibana自定义视图查看到监控到的Mongodb日志了：