虚拟机下搭建zookeeper + kafka集群（一）

1.主虚拟机：删掉网卡物理地址和UUID

[root@localhost network-scripts]# cat ifcfg-eth0
DEVICE=eth0
HWADDR=00:0C:29:D3:5C:EC
TYPE=Ethernet
UUID=616a16e9-9f95-4cca-8bfe-c1cf24750ebc
ONBOOT=yes
NM_CONTROLLED=yes
BOOTPROTO=static
IPADDR=192.168.57.22
NETMASK=255.255.255.0
GATEWAY=192.168.57.2
DNS1=8.8.8.8
DNS2=8.8.4.4
IPV6INIT=no

1.1.Linux虚拟机修改HOSTNAME

vi /etc/sysconfig/network

克隆的虚拟机网卡物理地址和被克隆的一样，需要删掉网卡，然后reboot重启虚拟机

rm -f /etc/udev/rules.d/70-persistent-net.rules

改ip重启网络

service network restart

2.安装JDK

rpm -ivh jdk-8u191-linux-x64.rpm

默认安装在 ls /usr/java/目录下

查看安装jdk安装包

rpm -qa| grep jdk

卸载安装的jdk

rpm -e `rpm -qa | grep jdk`

配置JAVA_HOME

JAVA_HOME=/usr/java/latest
PATH=$PATH:JAVA_HOME/bin
CLASSPATH=.
export JAVA_HOME
export PATH
export CLASSPATH

配完重新加载一下

source .bashrc

验证一下能不能取到

[root@note01 ~]# echo $JAVA_HOME
/usr/java/latest
[root@note01 ~]# echo $PATH
/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin:JAVA_HOME/bin
[root@note01 ~]# echo $CLASSPATH
.

3.主机名和ip映射

vi /etc/hosts

关闭防火墙

service iptables stop

查看防火墙状态

service iptables status

关掉开机启动防火墙

chkconfig iptables off

查看

chkconfig --list
chkconfig --list|grep iptables

4.安装zookeeper

解压zookeeper

tar -zxf zookeeper-3.4.6.tar.gz -C /usr/

cd到/usr/zookeeper-3.4.6/conf目录下copy

cd /usr/zookeeper-3.4.6/conf

cp zoo_sample.cfg zoo.cfg

修改配置zookeeper数据存放路径

dataDir=/root/zkdata

创建文件夹

mkdir /root/zkdata

各种命令

./bin/zkServer.sh

Usage: ./bin/zkServer.sh {start|start-foreground|stop|restart|status|upgrade|print-cmd}

启动:./bin/zkServer.sh start

5.安装单机kafka

tar -zxf kafka_2.11-2.2.0.tgz -C /usr/

kafka MQ服务修改config/server.properties

修改主机名：note01

listeners=PLAINTEXT://note01:9092

kafka日志

log.dirs=/usr/kafka-logs

zookeeper.connect=note01:2181

启动kafka

[root@note01 kafka_2.11-2.2.0]# ./bin/kafka-server-start.sh -daemon config/server.properties

关闭kafak

./bin/kafka-server-stop.sh

创建kafka topic

[root@note01 kafka_2.11-2.2.0]# ./bin/kafka-topics.sh --bootstrap-server note01:9092 --create --topic topic01 --partitions 3 --replication-factor 1

开启消费者，接受消息

./bin/kafka-console-consumer.sh --bootstrap-server note01:9092 --topic topic01 --group group1

开启生产者，发送消息

[root@note01 kafka_2.11-2.2.0]# ./bin/kafka-console-producer.sh --broker-list note01:9092 --topic topic01

6.kafka集群搭建

1.环境配置

hosts配置

[root@note03 etc]# vi hosts

192.168.57.23 note03 192.168.57.24 note04 192.168.57.25 note06

拷贝到note04， 05下

[root@note03 etc]# scp /etc/hosts note04:/etc/

[root@note03 etc]# scp /etc/hosts note05:/etc/

配置JAVA_HOME

[root@note03 ~]# vi .bashrc
[root@note03 ~]# scp .bashrc note04:~/
[root@note03 ~]# scp .bashrc note05:~/

重新加载环境变量

[root@note04 ~]# source .bashrc

2.同步时钟

首先安装 yum install ntp -y

如果失败

修改yum源站点配置：vi /etc/yum.repos.d/CentOS-Base.repo
对照清华大学镜像站中的目录修改后如下：

[base]
name=CentOS-$releasever - Base
#mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=os&infra=$infra
#baseurl=http://mirror.centos.org/centos/$releasever/os/$basearch/
baseurl=https://mirrors.tuna.tsinghua.edu.cn/centos-vault/6.8/os/x86_64/
gpgcheck=1
#gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-6
gpgkey=https://mirrors.tuna.tsinghua.edu.cn/centos-vault/6.8/os/x86_64/RPM-GPG-KEY-CentOS-6

#released updates
[updates]
name=CentOS-$releasever - Updates
#mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=updates&infra=$infra
#baseurl=http://mirror.centos.org/centos/$releasever/updates/$basearch/
baseurl=https://mirrors.tuna.tsinghua.edu.cn/centos-vault/6.8/updates/x86_64/
gpgcheck=1
#gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-6
gpgkey=https://mirrors.tuna.tsinghua.edu.cn/centos-vault/6.8/os/x86_64/RPM-GPG-KEY-CentOS-6

#additional packages that may be useful
[extras]
name=CentOS-$releasever - Extras
#mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=extras&infra=$infra
#baseurl=http://mirror.centos.org/centos/$releasever/extras/$basearch/
baseurl=https://mirrors.tuna.tsinghua.edu.cn/centos-vault/6.8/extras/x86_64/
gpgcheck=1
#gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-6
gpgkey=https://mirrors.tuna.tsinghua.edu.cn/centos-vault/6.8/os/x86_64/RPM-GPG-KEY-CentOS-6

#additional packages that extend functionality of existing packages
[centosplus]
name=CentOS-$releasever - Plus
#mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=centosplus&infra=$infra
#baseurl=http://mirror.centos.org/centos/$releasever/centosplus/$basearch/
baseurl=https://mirrors.tuna.tsinghua.edu.cn/centos-vault/6.8/centosplus/x86_64/
gpgcheck=1
enabled=0
#gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-6
gpgkey=https://mirrors.tuna.tsinghua.edu.cn/centos-vault/6.8/os/x86_64/RPM-GPG-KEY-CentOS-6

#contrib - packages by Centos Users
[contrib]
name=CentOS-$releasever - Contrib
#mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=contrib&infra=$infra
#baseurl=http://mirror.centos.org/centos/$releasever/contrib/$basearch/
baseurl=https://mirrors.tuna.tsinghua.edu.cn/centos-vault/6.8/contrib/x86_64/
gpgcheck=1
enabled=0
#gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-6
gpgkey=https://mirrors.tuna.tsinghua.edu.cn/centos-vault/6.8/os/x86_64/RPM-GPG-KEY-CentOS-6

和阿里云服务同步:ntp[1-7].aliyun.com

[root@note01 ~]# ntpdate ntp1.aliyun.com
[root@note01 ~]# clock -w

CMOS中的时间可能还没有改变，所以为了保险，需要使用 clock -w 把当前系统时间写入到CMOS中。

3.安装zookeeper

[root@note03 ~]# tar -zxf zookeeper-3.4.6.tar.gz -C /usr/
[root@note03 ~]# cd /usr/zookeeper-3.4.6/
[root@note03 zookeeper-3.4.6]# cp conf/zoo_sample.cfg  conf/zoo.cfg
[root@note03 zookeeper-3.4.6]# vi conf/zoo.cfg 

修改zoo.cfg配置

dataDir=/root/zkdata
和单机不一样的地方，需要配置
#在文件尾部追加服务器配置，集群服务器配置
server.1=note03:2888:3888
server.2=note04:2888:3888
server.3=note05:2888:3888

创建目录

[root@note03 zookeeper-3.4.6]# mkdir /root/zkdata

把zookeeper配置分别拷贝到另外两台机子上

[root@note03 zookeeper-3.4.6]# scp -r /usr/zookeeper-3.4.6 note05:/usr/

[root@note04 ~]# mkdir /root/zkdata
[root@note05 ~]# mkdir /root/zkdata

#在server1上修改服务器标识为1
[root@note03 zookeeper-3.4.6]# echo 1 > /root/zkdata/myid
#在server2上修改服务器标识为2
[root@note04 ~]# echo 2 > /root/zkdata/myid
#在server3上修改服务器标识为3
[root@note05 ~]# echo 3 > /root/zkdata/myid

启动zookeeper

[root@note03 zookeeper-3.4.6]# /usr/zookeeper-3.4.6/bin/zkServer.sh start zoo.cfg
[root@note04 ~]# /usr/zookeeper-3.4.6/bin/zkServer.sh start zoo.cfg
[root@note05 ~]# /usr/zookeeper-3.4.6/bin/zkServer.sh start zoo.cfg

安装配置kafka

[root@note03 ~]# tar -zxf kafka_2.11-2.2.0.tgz  -C /usr/
[root@note03 ~]# cd /usr/kafka_2.11-2.2.0/
[root@note03 kafka_2.11-2.2.0]# vi config/server.properties 

listeners=PLAINTEXT://note03:9092
log.dirs=/usr/kafka-logs
zookeeper.connect=note03:2181,note04:2181,note05:2181

[root@note03 kafka_2.11-2.2.0]# cd ..
[root@note03 usr]# scp -r kafka_2.11-2.2.0 note04:/usr/
[root@note03 kafka_2.11-2.2.0]# cd ..
[root@note03 usr]# scp -r kafka_2.11-2.2.0 note05:/usr/

note04,05拷贝后修改配置

[root@note04 ~]# vi /usr/kafka_2.11-2.2.0/config/server.properties 
broker.id=1
listeners=PLAINTEXT://note04:9092

broker.id=2
listeners=PLAINTEXT://note05:9092

启动kafka

[root@note03 ~]# cd /usr/kafka_2.11-2.2.0/
[root@note03 kafka_2.11-2.2.0]# ./bin/kafka-server-start.sh -daemon config/server.properties
[root@note03 kafka_2.11-2.2.0]# jps
1282 QuorumPeerMain
3433 Kafka
3482 Jps

[root@note03 kafka_2.11-2.2.0]# /usr/kafka_2.11-2.2.0/bin/kafka-topics.sh --bootstrap-server note03:9092,note04:9092,note05:9092 --create --topic topic01 --partitions 3 --replication-factor 2

查看topic详情

[root@note03 kafka_2.11-2.2.0]# /usr/kafka_2.11-2.2.0/bin/kafka-topics.sh --bootstrap-server note03:9092,note04:9092,note05:9092 --describe --topic topic01

删除topic

[root@note03 kafka_2.11-2.2.0]# /usr/kafka_2.11-2.2.0/bin/kafka-topics.sh --bootstrap-server note03:9092,note04:9092,note05:9092 --delete --topic topic01

修改topic下partitions，只能往大加，不能往小改

[root@note03 kafka_2.11-2.2.0]# /usr/kafka_2.11-2.2.0/bin/kafka-topics.sh --bootstrap-server note03:9092,note04:9092,note05:9092 --alter --topic topic01 --partitions 4

公众号【从零开始架构师】【wwg1234wwg1234】

虚拟机下搭建zookeeper + kafka集群（一）

原文：https://www.cnblogs.com/wwgtrendsetter/p/14586222.html