培训ELK日志监控报警实战课件.ppt

单击此处编辑母版标题样式,编辑母版文本样式,第二级,第三级,第四级,第五级,2,#,ELK,日志监控报警实战,磁云科技张人杰,2018.10.16,1,2,ELK日志监控报警实战磁云科技张人杰12,什么是,ELK,E:,ElasticSearch,L:,Logstash,K:,Kibana,2,8/7/2024,什么是ELKE:ElasticSearch28/20/20,运行效果,当服务器宕机时,立即发送邮件通知,3,8/7/2024,运行效果当服务器宕机时,立即发送邮件通知38/20/2023,环境搭建,(,一,)ElasticSearch,安装,1,、安装,elasticsearch,的,yum,源的密钥,rpm-import,https:/artifacts.elastic.co/GPG-KEY-elasticsearch,2,、配置,elasticsearch,的,yum,源,vim/etc/yum.repos.d/elasticsearch.repo,elasticsearch-6.x,name=Elasticsearch repository for 6.x packages,baseurl=https:/artifacts.elastic.co/packages/6.x/yum,gpgcheck=1,gpgkey=https:/artifacts.elastic.co/GPG-KEY-elasticsearch,enabled=1,autorefresh=1,type=rpm-md,3,、安装,elasticsearch,yum install-y elasticsearch,4,8/7/2024,环境搭建(一)ElasticSearch安装1、安装ela,环境搭建,(,一,)ElasticSearch,环境搭建,1,、需要安装,jdk1.8,版本以上的,java-version,2,、创建,elasticsearch data,的存放目录，并修改该目录的属主属组,mkdir-p/data/es-data,chown-R elasticsearch:elasticsearch/data/es-data,3,、修改,elasticsearch,的日志属主属组,chown-R elasticsearch:elasticsearch/var/log/elasticsearch/,4,、修改,elasticsearch,的配置文件,vim/etc/elasticsearch/elasticsearch.yml,5,8/7/2024,环境搭建(一)ElasticSearch环境搭建1、需要安,/etc/elasticsearch/elasticsearch.yml,编辑,找到配置文件中的,cluster.name,，打开该配置并设置集群名称,cluster.name:elk-tang,找到配置文件中的,node.name,，打开该配置并设置节点名称,node.name:elk-tang-1,修改,data,存放的路径,path.data:/data/es-data,修改,logs,日志的路径,path.logs:/var/log/elasticsearch/,注释配置内存使用用交换分区,#bootstrap.memory_lock:true,监听的网络地址,network.host:0.0.0.0,开启监听的端口,http.port:9200,增加新的参数，这样,head,插件可以访问,es(5.x,版本，如果没有可以自己手动加,),http.cors.enabled:true,http.cors.allow-origin:*,6,8/7/2024,/etc/elasticsearch/elasticsear,启动,ElasticSearch,/etc/init.d/elasticsearch start,7,8/7/2024,启动ElasticSearch/etc/init.d/ela,创建开机自启动服务,chkconfig elasticsearch on,8,8/7/2024,创建开机自启动服务chkconfig elasticsear,其他需要修改的参数,vim/etc/security/limits.conf,在末尾追加以下内容（,elk,为启动用户，当然也可以指定为*）,elk soft nofile 65536,elk hard nofile 65536,elk soft nproc 2048,elk hard nproc 2048,elk soft memlock unlimited,elk hard memlock unlimited,vim/etc/security/limits.d/XXX-nproc.conf,将里面的,1024,改为,2048,（,ES,最少要求为,2048,）*,soft nproc 2048,vim/etc/elasticsearch/elasticsearch.yml,加入以下内容,bootstrap.system,_call_,filter:false,9,8/7/2024,其他需要修改的参数vim/etc/security/lim,再次启动,/etc/init.d/elasticsearch restart,10,8/7/2024,再次启动/etc/init.d/elasticsearch,环境搭建（二）,安装,elasticsearch-head,插件,安装,node.js,sudo curl-sL-o/etc/yum.repos.d/khara-nodejs.repo,https:/copr.fedoraproject.org/coprs/khara/nodejs/repo/epel-7/khara-nodejs-epel-7.repo,sudo yum install-y nodejs nodejs-npm,安装,head,git clone git:/elasticsearch-head,npm install,npm run start,11,8/7/2024,环境搭建（二）安装elasticsearch-head插件,环境搭建（三）,安装,Logstash,环境,Logstash,需要安装到产生日志的服务器上,rpm-import,https:/artifacts.elastic.co/GPG-KEY-elasticsearch,yum install-y logstash,rpm-ql logstash,ln-s/usr/share/logstash/bin/logstash/bin/,12,8/7/2024,环境搭建（三）安装Logstash环境Logstash需要,Logstash,配置,input,file,path=,/var/log/nginx/access.log,start_position=beginning,ignore_older=0,filter,grok,patterns_dir=/opt/logstash/patterns,match=message=%NGINXACCESS,add_field=“resp_code”,“%response”,geoip,source=http_x_forwarded_for,target=geoip,database=/etc/logstash/GeoLite2-City.mmdb,add_field=geoipcoordinates,%geoiplongitude,add_field=geoipcoordinates,%geoiplatitude,mutate,convert=geoipcoordinates,float,convert=response,integer,convert=bytes,integer,replace=type=nginx_access,remove_field=message,date,match=timestamp,dd/MMM/yyyy:HH:mm:ss Z,mutate,remove_field=timestamp,output,elasticsearch,hosts=127.0.0.1:9200,index=logstash-nginx-access-%+YYYY.MM.dd,stdout codec=rubydebug,13,8/7/2024,Logstash配置input 138/20/2023,建立,grok,使用的表达式,mkdir-pv/opt/logstash/patterns,vi/opt/logstash/patterns/nginx,NGUSERNAME a-zA-Z.-+_%+,NGUSER%NGUSERNAME,NGINXACCESS%IPORHOST:clientip-%NOTSPACE:remote_user%HTTPDATE:timestamp(?:%WORD:verb%NOTSPACE:request HTTP/%NUMBER:httpversion|%DATA:rawrequest)%NUMBER:response(?:%NUMBER:bytes|-)%QS:referrer%QS:agent(?:%IPV4:http_x_forwarded_for|-),14,8/7/2024,建立grok使用的表达式mkdir-pv/opt/log,GeoIP,的数据库解析,ip,wget -xzvf,GeoLite2-City.tar.gz,mv GeoLite2-City_20181030/GeoLite2-City.mmdb/etc/logstash/.,15,8/7/2024,GeoIP的数据库解析ipwget geoli,测试配置文件并启动,Logstash,服务,logstash-t-f./elk.conf,nohup logstash f./elk.conf 2&1 /dev/null&,16,8/7/2024,测试配置文件并启动Logstash服务logstash-t,环境搭建（四）,Kibana,wget,https:/artifacts.elastic.co/downloads/kibana/kibana-,6,.4.,2,-linux-x86_64.tar.gz,#,注意需要与,ES,对应的版本,tar-xzf kibana-6.4.2-linux-x86_64.tar.gz,mv kibana-6.4.2-linux-x86_64/usr/local,ln-s/usr/local/kibana-,6,.4.,2,-linux-x86_64/usr/local/kibana,vim/usr/local/kibana/config/kibana.yml,17,8/7/2024,环境搭建（四）Kibanawget https:/art,/usr/local/kibana/config/kibana.yml,编辑,server.port:5601,server.host:0.0.0.0,elasticsearch.url:localhost:9200,kibana.index:.kibana,18,8/7/2024,/usr/local/kibana/config/kiban,安装,screen,以便于,kibana,在后台运行,yum-y install screen,screen,/usr/local/kibana/bin/kibana,19,8/7/2024,安装screen,以便于kibana在后台运行yum-y,Kibana,安装完成,打开浏览器并设置对应的,index,localhost:5601,20,8/7/2024,Kibana安装完成打开浏览器并设置对应的index htt,在,Kibana,上安装,sentinl,插件用于发送邮件提醒,1,、到,install,:1-03/Nov/2018:20:28:03+0800 GET/HTTP/1.1 200 0-Mozilla/5.0(X11;Linux x86_64;rv:52.0)Gecko/20100101 Firefox/52.0-/var/log/nginx/access.log,定期清理,ES,中的日志文件,curl-X DELETE ,localhost,:9200/,nginx,-*-date+%Y-%m-%d-d-$n days,22,8/7/2024,注意：需要定时清理日志文件定期清理nginx日志文件228/,防火墙配置,所有端口仅对内网开放,23,8/7/2024,防火墙配置所有端口仅对内网开放238/20/2023,结束,张人杰,2018.11.4,24,8/7/2024,248/20/2023,。