Step1. 修改Nginx的log格式（改為JSON格式）
把nginx的log_format改為以下的參數(shù)（修改/etc/nginx/nginx.conf）：
log_format main '{"@timestamp":"$time_iso8601","host":"$server_addr","clientip":"$remote_addr","size":$body_bytes_sent,"responsetime":$request_time,"upstreamtime":"$upstream_response_time","upstreamhost":"$upstream_addr","http_host":"$host","url":"$uri","xff":"$http_x_forwarded_for","referer":"$http_referer","agent":"$http_user_agent","status":"$status"}';
reload nginx后，看到access.log的格式如下：
{"@timestamp":"2017-12-13T17:29:49+08:00","host":"120.76.XX.XX","clientip":"120.76.XX.XXX","size":26963,"responsetime":0.000,"upstreamtime":"0.000","upstreamhost":"127.0.0.1:8080","http_host":"weixin.XXX.com","url":"/XXXXXXX/haowanyihao/thumb.png","xff":"111.22.65.171","referer":"-","agent":"WeChat/6.6.0.32 CFNetwork/811.4.18 Darwin/16.5.0","status":"200"}
Step2. 編寫(xiě)python程式

創(chuàng)新互聯(lián)公司主要從事成都做網(wǎng)站、成都網(wǎng)站制作、網(wǎng)頁(yè)設(shè)計(jì)、企業(yè)做網(wǎng)站、公司建網(wǎng)站等業(yè)務(wù)。立足成都服務(wù)井研,10多年網(wǎng)站建設(shè)經(jīng)驗(yàn),價(jià)格優(yōu)惠、服務(wù)專(zhuān)業(yè),歡迎來(lái)電咨詢(xún)建站服務(wù):18982081108

# -- coding: utf-8 --
'''
  By Willson Luo at 2017/11/23 v1.0
'''
import pandas as pd
import json,time,datetime,iso8601
from elasticsearch import Elasticsearch
from geoip import geolite2
# connect to elasticsearch database
es = Elasticsearch( "localhost:9200" )
es = Elasticsearch(hosts=[{'host': 'localhost', 'port': '9200'}],httpauth=('elastic', 'xxxxx'))
# nginx column name
#title    = ['@timestamp','host','clientip','size','responsetime','upstreamtime','upstreamhost','httphost','url','xff','referer','agent','status']
# nginx access log
ngxlog  = 'access.log'
ngxdata = open(ngxlog).readlines()
# nginx data(json format)
ngxjson = {}
for a1 in range(len(ngxdata)):
    step1 = ngxdata[a1].strip().split("\"")
    abc = iso8601.parsedate(step1[3])
    bcd = abc.strftime('%Y-%m-%dT%H:%M:%S%Z')
    cde = abc.strftime('%Y%m%d')
    ngxindex   = 'logstash-weixin-nginx-access-'+ cde
    ngxjson['@timestamp'] = bcd
    ngxjson['host'] = step1[7]
    ngxjson['size'] = step1[14].replace(":","").replace(",","")
    ngxjson['responsetime'] = step1[16].replace(":","").replace(",","")
    ngxjson['upstreamtime'] = step1[19]
    ngxjson['upstreamhost'] = step1[23]
    if step1[35] == "-":
        ngxjson['clientip']  = step1[11]
        ngxjson['httphost'] = step1[27]
        ipaddr = step1[11]
    else:
        ngxjson['clientip']  = step1[35].split(",")[0]
        ngxjson['httphost'] = step1[39]
        ipaddr = step1[35].split(",")[0]
    if "Apple" in step1[43]:
        ngxjson['agent']="Apple"
    elif "WeChat" in step1[43]:
        ngxjson['agent']="WeChat"
    elif "curl" in step1[43]:
        ngxjson['agent']="Linux"
    elif "Alibaba" in step1[43]:
        ngxjson['agent']="Aliyun"
    elif "Android" in step1[43]:
        ngxjson['agent']="Android"
    elif "MSIE" in step1[43]:
        ngxjson['agent']="IE"
    elif "Firefox" in step1[43]:
        ngxjson['agent']="Firefox"
    elif "Windows" in step1[43]:
        ngxjson['agent']="Windows"
    elif "Apache-Http" in step1[43]:
        ngxjson['agent']="Apache"
    else:
        ngxjson['agent']= step1[43]
    ngxjson['status']= step1[47]
    location = geolite2.lookup(ipaddr).location
    match = geolite2.lookup(ipaddr).getinfodict()
    location = []
    location.append(match['location']['longitude'])
    location.append(match['location']['latitude'])
    geoip = {}
    geoip['location'] = location
    if match.haskey('city'):
        city = match['city']['names']['en']
    else:
        city = "-"
    if match.haskey('country'):
        country = match['country']['names']['en']
    else:
        country = "-"
    if match.haskey('subdivisions'):
        subdivisions = match['subdivisions'][0]['names']['en']
    else:
        subdivisions = "-"
    ngxjson['geoip']        = geoip
    ngxjson['country']      = country
    ngxjson['subdivisions'] = subdivisions
    ngxjson['city']         = city
    ngxjson['possition']    = country+"-"+subdivisions+"-"+city
    print a1,ngxjson
    es.index( index=ngxindex, doctype="logs", body=ngxjson )

Step3. 通過(guò)Kibana把數(shù)據(jù)呈現(xiàn)處理
1> 先在Kibana把index匯入(一般第一步就讓你建立這個(gè)東西了）Kibana-->Management-->Kibana(Index Patterns)

2> 構(gòu)建可用視圖Kibana--> Visualize（這個(gè)東西比較見(jiàn)人見(jiàn)智）

Step4. 構(gòu)建Dashboard（就是把Visualize的內(nèi)容拖進(jìn)來(lái)）

第一次寫(xiě)B(tài)log，估計(jì)錯(cuò)漏不少，麻煩指正，謝謝

當(dāng)前文章：Python重寫(xiě)Logstash，把NginxAccessLog清洗后匯入ElasticDB
網(wǎng)站網(wǎng)址：http://chinadenli.net/article4/jpeooe.html

成都網(wǎng)站建設(shè)公司_創(chuàng)新互聯(lián)，為您提供企業(yè)網(wǎng)站制作、網(wǎng)站維護(hù)、微信小程序、營(yíng)銷(xiāo)型網(wǎng)站建設(shè)、動(dòng)態(tài)網(wǎng)站、關(guān)鍵詞優(yōu)化

廣告

聲明：本網(wǎng)站發(fā)布的內(nèi)容（圖片、視頻和文字）以用戶投稿、用戶轉(zhuǎn)載內(nèi)容為主，如果涉及侵權(quán)請(qǐng)盡快告知，我們將會(huì)在第一時(shí)間刪除。文章觀點(diǎn)不代表本網(wǎng)站立場(chǎng)，如需處理請(qǐng)聯(lián)系客服。電話：028-86922220；郵箱：631063699@qq.com。內(nèi)容未經(jīng)允許不得轉(zhuǎn)載，或轉(zhuǎn)載時(shí)需注明來(lái)源： 創(chuàng)新互聯(lián)