[Kafka] Docker에서 Kafka Multi Broker Cluster 구성하기

카프카의 분산처리 기능을 이용하려면 노드가 여러개여야한다.
노드 한 개로 kafka 통신해보기는 성공을 했으니 이제 브로커를 3개로 늘려봐야겠다.

이런 그림을 원한다!!

---

docker compose 수정

# kafka-cluster/docker-compose.yaml

version: "2"

services:
  zookeeper:
    ...

  broker-1:
    image: confluentinc/cp-kafka:7.3.0
    container_name: broker-1
    depends_on:
      - zookeeper
    ports:
      - 9092:9092
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://broker-1:29092,PLAINTEXT_HOST://localhost:9092
      ...

  broker-2:
    image: confluentinc/cp-kafka:7.3.0
    container_name: broker-2
    depends_on:
      - zookeeper
    ports:
      - 9093:9093
    environment:
      KAFKA_BROKER_ID: 2
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://broker-2:29093,PLAINTEXT_HOST://localhost:9093
      ...

  broker-3:
    image: confluentinc/cp-kafka:7.3.0
    container_name: broker-3
    depends_on:
      - zookeeper
    ports:
      - 9094:9094
    environment:
      KAFKA_BROKER_ID: 3
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://broker-3:29094,PLAINTEXT_HOST://localhost:9094
      ...

networks:
  default:
    name: my-network

• service에서 broker의 개수를 늘려주고
• 각각의 broker에 unique한 이름과 KAFKA_BROKER_ID, 포트번호를 배정해아한다.

Producer, Consumer 수정

# consumer.py
consumer = KafkaConsumer(
            'test-topic,
            bootstrap_servers=["broker-1:29092", "broker-2:29093", "broker-3:29094"],
            ...
            )

• producer, consumer의 bootstrap_servers 부분을 수정해준다.

kafka cluster 실행

$ docker compose -p kafka-cluster -f docker-compose.yaml up

$ docker ps -a

• zookeeper, 3개의 broker 총 4개의 컨테이너가 잘 실행되고 있다

consumer 실행

# terminal 1
$ docker compose -p consumer -f docker-compose.yaml up

producer 실행

# terminal 2
$ docker compose -p producer -f docker-compose.yaml up

• 잘 받아온다!

---

References

https://gist.github.com/rmoff/ba6c335367764953e390e273e019c083