记录在Docker上创建单broker的Kafka过程
安装运行Kafka 创建bridge网络 1 docker network create -d bridge --subnet=10.0.0.0/24 kafka_net
启动zookeeper 1 docker run --restart=always -d -ti --name zookeeperService --network=kafka_net -p 2181:2181 -v /home/core/zookeeper/conf:/conf -v /home/core/zookeeper/data:/data -v /home/core/zookeeper/dataLog:/dataLog zookeeper
验证zookeeper是否启动正常 1 docker run --rm -it --network=host zookeeper zkCli.sh -server localhost:2181
启动Kafka 1 docker run -it --name kafkaService --network=kafka_net -p 9092:9092 -e KAFKA_ADVERTISED_HOST_NAME=localhost -e KAFKA_ZOOKEEPER_CONNECT=zookeeperService:2181 -d wurstmeister/kafka
运行上述命令后发现启动日志报错，错误信息显示系统内存不总
1 OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x00000000bad30000, 986513408, 0) failed; error='Cannot allocate memory' (errno=12)
查看kafka启动脚本，发现默认配置的JVM堆内存为1G，本机空闲内存只有67M，系统并未开启swap区
1 2 3 4 5 free -mh total used free shared buff/cache available Mem: 988Mi 682Mi 67Mi 200Mi 238Mi 17Mi Swap: 0B 0B 0B
开启swap区 使用文件添加swap区
dd if=/dev/zero of=/swap-file bs=1M count=1024
chmod 0600 /swap-file
mkswap /swap-file 格式化交换分区
swapon /swap-file 启动交换分区
swapon -s 查看交换分区
Swappiness
Swappiness的取值范围为0～100，通过修改/proc/sys/vm/swappiness文件的值来设置
再次启动Kafka，并覆盖JVM对内存参数
1 docker run -d -ti --name kafkaService --network=kafka_net -p 9092:9092 -e KAFKA_ADVERTISED_HOST_NAME=localhost -e KAFKA_ZOOKEEPER_CONNECT=zookeeperService:2181 -e KAFKA_HEAP_OPTS="-Xmx200M -Xms200M" wurstmeister/kafka
验证Kafka是否启动成功 1 2 3 4 5 6 7 8 9 Producer: docker run --rm -it --network=host ryane/kafkacat -b localhost:9092 -t myTopic -P Hello World ! Consumer: docker run --rm -it --network=host ryane/kafkacat -b localhost:9092 -t myTopic -C % Reached end of topic myTopicTest [0] at offset 0 Hello World ! % Reached end of topic myTopicTest [0] at offset 1
swap区的活动 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 vmstat 1 procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu----- r b swpd free buff cache si so bi bo in cs us sy id wa st 19 5 355328 61092 160 149916 95 35 482 54 29 45 1 1 97 1 0 9 10 355840 60640 176 148624 2784 1060 112192 1212 2420 5863 2 43 0 56 0 16 4 355328 59268 176 149840 2048 612 112268 680 2163 5319 2 32 0 66 0 14 20 355840 58216 168 151556 2876 1360 150376 1580 2960 6953 1 50 0 49 0 10 9 355584 57344 168 152156 1780 728 151144 772 2318 5504 1 42 0 57 0 19 0 355328 57852 200 150296 3528 676 114896 888 2900 5972 1 63 0 36 0 17 15 355584 54752 148 154576 2660 1216 165028 1528 2636 5672 1 55 0 44 0 11 4 355584 54008 188 155416 2896 1268 111956 1336 3175 6713 13 56 0 31 0 20 12 355840 58968 192 150796 2176 992 136236 1244 2453 6017 1 36 0 64 0 27 2 355328 55616 200 152832 3192 832 108232 992 2393 5416 2 39 0 60 0
虽然Kafka启动成功了，但是频繁的swap区交换导致系统响应变慢
查看Kafka对swap区的使用
获取容器应用在宿主机中的PID
获取进程地址信息
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 本例Kafka的PID为10250： sudo cat /proc/10250/smaps | head -n20 f3800000-100400000 rw-p 00000000 00:00 0 Size: 208896 kB KernelPageSize: 4 kB MMUPageSize: 4 kB Rss: 76048 kB Pss: 76048 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 68 kB Private_Dirty: 75980 kB Referenced: 56888 kB Anonymous: 76048 kB LazyFree: 0 kB AnonHugePages: 55296 kB ShmemPmdMapped: 0 kB Shared_Hugetlb: 0 kB Private_Hugetlb: 0 kB Swap: 129208 kB // 使用的swap区空间 SwapPss: 129208 kB Locked: 0 kB
启动Kafka Manager 1 2 3 docker run --restart=always -d -ti --name=kafkaManager --network=kafka_net -p 9000:9000 -e ZK_HOSTS=zookeeperService:2181 sheepkiller/kafka-manager
This is our time. This is our moment to define a generation. - Mark Zuckerberg
The Social Network
