基于Docker搭建监控系统

记录如何基于Docker搭建监控系统

StatsD

StatsD 用于统计和聚合系统的指标数据

1. StatsD client
   拥有多语言实现，用于将数据发送到StatsD Server
2. StatsD daemon
   监听StatsD client 发送的数据并整合数据发送(默认每10s)到后端
3. 监控后端 Graphite、influxDB

StatsD 协议

StatsD采用简单的行协议，StatsD Server默认采用UDP协议，可配置为 TCP协议

<bucket>:<value>|<type>[|@sample_rate]
* bucket: Metric 标识
* value: Metric 数值
* type: Metric 类型
* sample_rate: 采样率

StatsD Metric Types

1. Counting
   计数

people.number:10|c  -- people.number +10
people.number:2|c  -- people.number 10+2 = 12
people.number:-5|c  -- people.number 12-5 = 7
people.number:10|c|@0.1  -- people.number 7+10/0.1=107
flush 107
current people.number 107

2. Timing
   记录操作耗时，StatsD 会计算出平均值、标准差、最大值、最小值

people.number.api.mean_$PCT
people.number.api.sum_$PCT
people.number.api.upper_$PCT

• PCT表示百分比阀值，可配置

people.number.api:10|ms
people.number.api:101|ms
people.number.api:20|ms
people.number.api:15|ms
people.number.api:11|ms
flush前值列表为 [10,101,20,15,11]

假设 PCT为80 则刷新上报的值为: 
1. 首先剔除10%的峰值：5*0.8 = 4 (四舍五入) ，保留4个值，剔除剩余峰值101。当前[10,20,15,11]
2. 平均值: (10+20+15+11)/4  最大值: 20  最小值: 10

current people.number.api 0

3. Gauges
   在flush后值不会被清零

cat.food.amount:10|g  -- cat.food.amount +10
cat.food.amount:-2|g  -- cat.food.amount 10-2=8
cat.food.amount:+5|g  -- cat.food.amount 8+5=13
cat.food.amount:10|g  -- cat.food.amount 10
flush 10  
current cat.food.amount 10

4. Sets
   统计不重复的值有多少个

active.people.id:2|g  -- active.people.id +1
active.people.id:1|g  -- active.people.id 1+1 = 2
active.people.id:1|g  -- active.people.id 2
active.people.id:10|g  -- active.people.id 2+1 = 3
flush 3  
current active.people.id 0

Graphite

Graphite 由三部分构成

1. carbon
   监听时间序列数据
2. whisper
   存储时间序列数据
3. graphite-web
   展示时间序列数据

Graphite & StatsD 安装

docker run -it --name graphite --restart=always -p 8080:80 -p 2003-2004:2003-2004 -p 2023-2024:2023-2024 -p 8125:8125/udp -p 8126:8126 -d graphiteapp/graphite-statsd

访问Graphite

http://host:8080/dashboard
默认用户密码为：root/root
修改用户密码：http://host:8080/admin/auth/user/1/

Grafana

官方是这么说的：
No matter where your data is, or what kind of database it lives in, you can bring it together with Grafana. Beautifully

Grafana安装

docker run -it --name=grafana --restart=always -p 3000:3000 -d grafana/grafana

访问Grafana

http://host:3000
默认用户/密码：admin/admin

配置数据源

Metrics数据模拟

while true; do echo -n "people.number:$((RANDOM % 100))|c" | nc -w 1 -u 127.0.0.1 8125; done

设置Grafana图表

You don’t know about real loss, because it only occurs when you’ve loved something more than you love yourself. - Sean

Good Will Hunting