apache kafka_2.13-2.8.0 설치

1. host file 작성 (모든 노드에 배포)

172.30.1.101 kafka1.io
172.30.1.102 kafka2.io
172.30.1.103 kafka3.io

 

2. 키 교환

ssh-keygen
ssh-copy-id 172.30.1.101
ssh-copy-id 172.30.1.102
ssh-copy-id 172.30.1.103

 

3. jdk 설치

for i in {1..3} ; do ssh 172.30.1.10${i} "yum -y install java-1.8.0-openjdk.x86_64" ; done

 

4. zookeeper binary 파일 다운로드 및 압축해제

[root@kafka1 ~]# for i in {1..3} ; do ssh 172.30.1.10${i} "mkdir /zookeeper" ; done
[root@kafka1 ~]# wget https://archive.apache.org/dist/zookeeper/zookeeper-3.5.5/apache-zookeeper-3.5.5-bin.tar.gz
[root@kafka1 ~]# for i in {2..3} ; do scp apache-zookeeper-3.5.5-bin.tar.gz 172.30.1.10${i}:/root/ ; done
[root@kafka1 ~]# for i in {1..3} ; do ssh 172.30.1.10${i} "tar -zxf /root/apache-zookeeper-3.5.5-bin.tar.gz" ; done

 

5. kafka binary 파일 다운로드 및 압축해제

[root@kafka1 ~]# mkdir /kafka
[root@kafka1 ~]# cd /kafka/
[root@kafka1 kafka]# wget https://archive.apache.org/dist/kafka/2.8.0/kafka_2.13-2.8.0.tgz
[root@kafka1 kafka]# for i in {2..3} ; do ssh 172.30.1.10${i} "mkdir /kafka" ; done
[root@kafka1 kafka]# for i in {2..3} ; do scp kafka_2.13-2.8.0.tgz 172.30.1.10${i}:/kafka ; done
[root@kafka1 kafka]# for i in {1..3} ; do ssh 172.30.1.10${i} "cd /kafka ; tar -zxf /kafka/kafka_2.13-2.8.0.tgz" ; done

 

6. 방화벽 및 selinux 설정

[root@kafka1 bin]# for i in {1..3} ; do ssh 172.30.1.10${i} "systemctl stop firewalld ; systemctl disable firewalld" ; done
[root@kafka1 bin]# for i in {1..3} ; do ssh 172.30.1.10${i} "setenforce 0" ; done
[root@kafka1 bin]# vi /etc/selinux/config

/etc/selinux/config파일을 아래와 같이 변경 후 배포
SELINUX=disabled

[root@kafka1 bin]# for i in {2..3} ; do scp /etc/selinux/config 172.30.1.10${i}:/etc/selinux/config ; done

 

7. 주키퍼 설정

[root@kafka1 kafka]# cd /root/apache-zookeeper-3.5.5-bin/conf
[root@kafka1 conf]# mv zoo_sample.cfg zoo.cfg
[root@kafka1 conf]# vi zoo.cfg

zoo.cfg를 다음과 같이 변경
tickTime=2000
initLimit=10
syncLimit=5
dataDir=/zookeeper
dataLogDir=/var/lib/zookeeper
clientPort=2181
server.1=kafka1.io:2888:3888
server.2=kafka2.io:2888:3888
server.3=kafka3.io:2888:3888

[root@kafka1 conf]# for i in {2..3} ; do scp /root/apache-zookeeper-3.5.5-bin/conf/zoo.cfg 172.30.1.10${i}:/root/apache-zookeeper-3.5.5-bin/conf ; done

 

8. 주키퍼 myid 설정

myid는 1~255까지만 가능한거로 기억함

[root@kafka1 kafka]# echo 1 > /zookeeper/myid
[root@kafka1 kafka]# ssh 172.30.1.102 "echo 2 > /zookeeper/myid"
[root@kafka1 kafka]# ssh 172.30.1.103 "echo 3 > /zookeeper/myid"

 

9. zookeeper start

### zookeeper start
[root@kafka1 bin]# for i in {1..3} ; do ssh 172.30.1.10${i} "/root/apache-zookeeper-3.5.5-bin/bin/zkServer.sh start" ; done

### zookeeper status
[root@kafka1 bin]# for i in {1..3} ; do ssh 172.30.1.10${i} "/root/apache-zookeeper-3.5.5-bin/bin/zkServer.sh status" ; done

 

10. kafka 설정

[root@kafka1 bin]# cd /kafka/kafka_2.13-2.8.0/config
[root@kafka1 bin]# vi server.properties

broker.id=1
listeners=PLAINTEXT://172.30.1.101:9092
advertised.listeners=PLAINTEXT://172.30.1.101:9092
zookeeper.connect=172.30.1.101:2181,172.30.1.102:2181,172.30.1.103:2181/cpkafka
log.dirs=/kafka/kafka-logs

[root@kafka1 bin]# for i in {2..3} ; do scp /kafka/kafka_2.13-2.8.0/config/server.properties 172.30.1.10${i}:/kafka/kafka_2.13-2.8.0/config/server.properties ; done

 

11. 각 노드 별 설정 수정

kafka2.io에서 변경해야할 설정
broker.id=2
listeners=PLAINTEXT://172.30.1.102:9092
advertised.listeners=PLAINTEXT://172.30.1.102:9092

kafka3.io에서 변경해야할 설정
 broker.id=3
listeners=PLAINTEXT://172.30.1.103:9092
advertised.listeners=PLAINTEXT://172.30.1.103:9092

 

12. 카프카 실행

[root@kafka1 bin]# for i in {1..3} ; do ssh 172.30.1.10${i} "/kafka/kafka_2.13-2.8.0/bin/kafka-server-start.sh -daemon /kafka/kafka_2.13-2.8.0/config/server.properties" ; done