Look In Java

Kafka at Docker

Lukasz D's photo
Lukasz D
·

2 min read

Apache Kafka can by set up an platform at your pc or dockized. For all testing and learning purposes I'll choose containers. For managing containers I prefer Rancher. You can use other tool, but at least one is required for further steps.

Kafka broker

For this setup I'm using Kafka broker provided by Confluent. Not all of images from Confluent are Community licensed, but this one is. For more details check: Docker Image Reference for Confluent Platform

For setup Kafka broker configuration use docker-compose.yml file.

---
version: '3'
services:
  zookeeper:
    image: confluentinc/cp-zookeeper:7.3.0
    container_name: zookeeper
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000
    volumes:
    - zoologs:/var/lib/zookeeper/log
    - zoodata:/var/lib/zookeeper/data

  broker:
    image: confluentinc/cp-kafka:7.3.0
    container_name: broker
    ports:
      - "9092:9092"
    depends_on:
      - zookeeper
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: 'zookeeper:2181'
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_INTERNAL:PLAINTEXT
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9092,PLAINTEXT_INTERNAL://broker:29092
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1      
    volumes:
    - kafkadata:/var/lib/kafka/data

volumes:
  zoologs:
  zoodata:
  kafkadata:

Run the broker

In a directory with docker-compose.yml file execute command:

docker-compose up

Testing the broker

Create a topic

Kafka relay on topics. So easiest topic definition is below. Execute command in new terminal:

docker exec broker kafka-topics --bootstrap-server broker:9092 --create --topic testtopic

Read message from the topic

I know that topic is empty and this command will not read anything from broker, but it will be useful after next step. So execute this command in new terminal: 

docker exec --interactive --tty broker kafka-console-consumer --bootstrap-server broker:9092 --topic testtopic --from-beginning

Write message to the topic

In new terminal execute command:

docker exec --interactive --tty broker kafka-console-producer --bootstrap-server broker:9092 --topic testtopic

Write some messages. The same messages should be shown in terminal where you are reading from topic.

Finish work

Now you know that your configuration is working and docker-compose.yml file can be used for other (more complicated) project. For turning off the broker, press Ctrl+C in console where docker compose were started or execute command: 

docker-compose down

Check message persistence

If you would like to check if messages that were send to the topic will be available after broker restart you need to do two steps. Start once more broker and execute command that will read messages from the topic. Because we set a flag --from-beginning all messages should be read once more.

DockerkafkaApache Kafka