mongo日誌視覺化

原始mongod.log格式：

2020-11-26t21:33:45.896+0800 i network [listener] connection accepted from 10.10.6.192:52138 #1663452 (121 connections now open)

2020-11-26t21:33:45.896+0800 i network [conn1663452] received client metadata from 10.10.6.192:52138 conn1663452: , os: , platform: ".net core 3.1.4" }

2020-11-26t21:33:45.910+0800 i access [conn1663452] successfully authenticated as principal push on push from client 10.10.6.192:52138

2020-11-26t21:33:46.325+0800 i network [listener] connection accepted from 10.10.3.144:50170 #1663453 (122 connections now open)

2020-11-26t21:33:46.325+0800 i network [conn1663453] received client metadata from 10.10.3.144:50170 conn1663453: , os: , platform: "cpython 3.7.3.final.0" }

2020-11-26t21:33:46.326+0800 i network [listener] connection accepted from 10.10.3.144:50172 #1663454 (123 connections now open)

2020-11-26t21:33:46.326+0800 i network [conn1663454] received client metadata from 10.10.3.144:50172 conn1663454: , os: , platform: "cpython 3.7.3.final.0" }

2020-11-26t21:33:46.339+0800 i access [conn1663454] successfully authenticated as principal crawl on crawl from client 10.10.3.144:50172

2020-11-26t21:33:46.888+0800 i network [conn1663454] end connection 10.10.3.144:50172 (122 connections now open)

2020-11-26t21:33:46.888+0800 i network [conn1663453] end connection 10.10.3.144:50170 (121 connections now open

這種格式非常不便於閱讀，於是各種找尋mongo日誌視覺化工具，發現乙個mtools好工具，能夠格式化mongo日誌，並且可以直接輸出為json格式，於是學習了一下簡單的吧日誌傳到elk裡面展示。

(1)首先寫乙個簡單的指令碼：

#!/usr/bin/env bash

#取最近1min 執行耗時超過200ms慢日誌

start_time=`date -d "1 min ago" "+%b %e %r"`

然後加入crontab，每分鐘執行一次，取最近一分鐘的日誌新產生的日誌，這樣mongod.log就可以實時json化為mongo.json

# * * * * *

#min hour day month day_of_week

#* * * * * script -c "/backup_local/dba/scripts/mongo_mfilter.sh >>/tmp/crontab.log

檢查日誌輸出已經沒問題，然後開始接入elk,我這裡直接用阿里雲的日誌服務。

(2)安裝配置阿里雲的logstail，並且接入json檔案，接入以後的顯示效果為：

看起來還是不夠友好，然後再用json函式格式化一下，取關鍵資訊：

*|select replace(substr(datetime,1,19),'t',' ') as datetime,
duration,operation,thread,namespace,concat(json_format(json_array_get(split_tokens,5)),
json_format(json_array_get(split_tokens,6)),
json_format(json_array_get(split_tokens,7)),
json_format(json_array_get(split_tokens,8)),
json_format(json_array_get(split_tokens,9)),
json_format(json_array_get(split_tokens,10)),
json_format(json_array_get(split_tokens,11)),
json_format(json_array_get(split_tokens,12)),
json_format(json_array_get(split_tokens,13)),
json_format(json_array_get(split_tokens,14)),
json_format(json_array_get(split_tokens,15)),
json_format(json_array_get(split_tokens,16)),
json_format(json_array_get(split_tokens,17)),
json_format(json_array_get(split_tokens,18)),
json_format(json_array_get(split_tokens,19)),
json_format(json_array_get(split_tokens,20)),
json_format(json_array_get(split_tokens,21)),
json_format(json_array_get(split_tokens,22)),
json_format(json_array_get(split_tokens,23)),
json_format(json_array_get(split_tokens,24)),
json_format(json_array_get(split_tokens,25)),
json_format(json_array_get(split_tokens,26)),
json_format(json_array_get(split_tokens,27)),
json_format(json_array_get(split_tokens,28)),
json_format(json_array_get(split_tokens,29)),
json_format(json_array_get(split_tokens,30)),
json_format(json_array_get(split_tokens,31)),
json_format(json_array_get(split_tokens,32)),
json_format(json_array_get(split_tokens,33)),
json_format(json_array_get(split_tokens,34)),
json_format(json_array_get(split_tokens,35))) as query_sql where duration>200

這個是日誌服務sql語法，最終效果為：

最後再利用日誌服務自帶的告警功能配好webhook釘釘告警，簡單的mongo日誌視覺化就算是完成了。

mongo日誌視覺化

資料視覺化什麼是資料視覺化

GoAccess 視覺化 Web 日誌監控工具使用

視覺化建模

mongo日誌視覺化

資料視覺化 什麼是資料視覺化

GoAccess 視覺化 Web 日誌監控工具使用

視覺化建模

相關推薦

資料視覺化什麼是資料視覺化