instruction: Explain Docker containerization and best practices
output: Back
  Ask AI
  Start typing to search or try
  Contact support
  Guides
  Get started
  Manuals
  Developing event-driven applications with Kafka and Docker
  This guide explains how to run Apache Kafka in Docker containers.
  JavaScript
  Distributed systems
  20 minutes
  Â« Back to all guides
  Developing event-driven applications with Kafka and Docker
  Page options
  Copy page as Markdown for LLMs
  View page as plain text
  Ask questions with Docs AI
  Claude
  Open in Claude
  With the rise of microservices, event-driven architectures have become increasingly popular.
  Apache Kafka
  , a distributed event streaming platform, is often at the
  heart of these architectures. Unfortunately, setting up and deploying your own Kafka instance for development
  is often tricky. Fortunately, Docker and containers make this much easier.
  In this guide, you will learn how to:
  Use Docker to launch up a Kafka cluster
  Connect a non-containerized app to the cluster
  Connect a containerized app to the cluster
  Deploy Kafka-UI to help with troubleshooting and debugging
  Prerequisites
  The following prerequisites are required to follow along with this how-to guide:
  Docker Desktop
  Node.js
  yarn
  Basic knowledge of Kafka and Docker
  Launching Kafka
  Beginning with
  Kafka 3.3
  , the deployment of Kafka was greatly simplified by no longer requiring Zookeeper thanks to KRaft (Kafka Raft). With KRaft, setting up a Kafka instance for local development is much easier. Starting with the launch of
  Kafka 3.8
  , a new
  kafka-native
  Docker image is now available, providing a significantly faster startup and lower memory footprint.
  This guide will be using the apache/kafka image, as it includes many helpful scripts to manage and work with Kafka. However, you may want to use the apache/kafka-native image, as it starts more quickly and requires fewer resources.
  Starting Kafka
  Start a basic Kafka cluster by doing the following steps. This example will launch a cluster, exposing port 9092 onto the host to let a native-running application to connect to it.
  Start a Kafka container by running the following command:
  docker run -d --name
  kafka -p 9092:9092 apache/kafka
  Once the image pulls, youâll have a Kafka instance up and running within a second or two.
  The apache/kafka image ships with several helpful scripts in the
  /opt/kafka/bin
  directory. Run the following command to verify the cluster is up and running and get its cluster ID:
  docker
  exec
  -ti kafka /opt/kafka/bin/kafka-cluster.sh cluster-id --bootstrap-server :9092
  Doing so will produce output similar to the following:
  Cluster ID: 5L6g3nShT-eMCtK--X86sw
  Create a sample topic and produce (or publish) a few messages by running the following command:
  docker
  exec
  -ti kafka /opt/kafka/bin/kafka-console-producer.sh --bootstrap-server :9092 --topic demo
  After running, you can enter a message per line. For example, enter a few messages, one per line. A few examples might be:
  First message
  Second message
  Press
  enter
  to send the last message and then press ctrl+c when youâre done. The messages will be published to Kafka.
  Confirm the messages were published into the cluster by consuming the messages:
  docker
  exec
  -ti kafka /opt/kafka/bin/kafka-console-consumer.sh --bootstrap-server :9092 --topic demo --from-beginning
  You should then see your messages in the output:
  First message
  Second message
  If you want, you can open another terminal and publish more messages and see them appear in the consumer.
  When youâre done, hit ctrl+c to stop consuming messages.
  You have a locally running Kafka cluster and have validated you can connect to it.
  Connecting to Kafka from a non-containerized app
  Now that youâve shown you can connect to the Kafka instance from a command line, itâs time to connect to the cluster from an application. In this example, you will use a simple Node project that uses the
  KafkaJS
  library.
  Since the cluster is running locally and is exposed at port 9092, the app can connect to the cluster at localhost:9092 (since itâs running natively and not in a container right now). Once connected, this sample app will log messages it consumes from the
  demo
  topic. Additionally, when it runs in development mode, it will also create the topic if it isnât found.
  If you donât have the Kafka cluster running from the previous step, run the following command to start a Kafka instance:
  docker run -d --name
  kafka -p 9092:9092 apache/kafka
  Clone the
  GitHub repository
  locally.
  git clone https://github.com/dockersamples/kafka-development-node.git
  Navigate into the project.
  cd kafka-development-node/app
  Install the dependencies using yarn.
  yarn install
  Start the application using
  yarn dev
  . This will set the
  NODE_ENV
  environment variable to
  development
  and use
  nodemon
  to watch for file changes.
  yarn dev
  With the application now running, it will log received messages to the console. In a new terminal, publish a few messages using the following command:
  docker
  exec
  -ti kafka /opt/kafka/bin/kafka-console-producer.sh --bootstrap-server :9092 --topic demo
  And then send a message to the cluster:
  Test message
  Remember to press
  ctrl+c
  when youâre done to stop producing messages.
  Connecting to Kafka from both containers and native apps
  Now that you have an application connecting to Kafka through its exposed port, itâs time to explore what changes are needed to connect to Kafka from another container. To do so, you will now run the application out of a container instead of natively.
  But before you do that, itâs important to understand how Kafka listeners work and how those listeners help clients connect.
  Understanding Kafka listeners
  When a client connects to a Kafka cluster, it actually connects to a âbrokerâ. While brokers have many roles, one of them is to support load balancing of clients. When a client connects, the broker returns a set of connection URLs the client should then use for the client to connect for the producing or consuming of messages. How are these connection URLs configured?
  Each Kafka instance has a set of listeners and advertised listeners. The âlistenersâ are what Kafka binds to and the âadvertised listenersâ configure how clients should connect to the cluster. The connection URLs a client receives is based on which listener a client connects to.
  Defining the listeners
  To help this make sense, letâs look at how Kafka needs to be configured to support two connection opportunities:
  Host connections (those coming through the hostâs mapped port) - these will need to connect using localhost
  Docker connections (those coming from inside the Docker networks) - these can not connect using localhost, but the network alias (or DNS address) of the Kafka service
  Since there are two different methods clients need to connect, two different listeners are required -
  HOST
  DOCKER
  . The
  HOST
  listener will tell clients to connect using localhost:9092, while the
  DOCKER
  listener will inform clients to connect using
  kafka:9093
  . Notice this means Kafka is listening on both ports 9092 and 9093. But, only the host listener needs to be exposed to the host.
  In order to set this up, the
  compose.yaml
  for Kafka needs some additional configuration. Once you start overriding some of the defaults, you also need to specify a few other options in order for KRaft mode to work.
  services
  kafka
  image
  apache/kafka-native
  ports
  "9092:9092"
  environment
  # Configure listeners for both docker and host communication
  KAFKA_LISTENERS
  CONTROLLER://localhost:9091,HOST://0.0.0.0:9092,DOCKER://0.0.0.0:9093
  KAFKA_ADVERTISED_LISTENERS
  HOST://localhost:9092,DOCKER://kafka:9093
  KAFKA_LISTENER_SECURITY_PROTOCOL_MAP
  CONTROLLER:PLAINTEXT,DOCKER:PLAINTEXT,HOST:PLAINTEXT
  # Settings required for KRaft mode
  KAFKA_NODE_ID
  KAFKA_PROCESS_ROLES
  broker,controller
  KAFKA_CONTROLLER_LISTENER_NAMES
  CONTROLLER
  KAFKA_CONTROLLER_QUORUM_VOTERS
  localhost:9091
  # Listener to use for broker-to-broker communication
  KAFKA_INTER_BROKER_LISTENER_NAME
  DOCKER
  # Required for a single node cluster
  KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR
  Give it a try using the steps below.
  If you have the Node app running from the previous step, go ahead and stop it by pressing
  ctrl+c
  in the terminal.
  If you have the Kafka cluster running from the previous section, go ahead and stop that container using the following command:
  docker rm -f kafka
  Start the Compose stack by running the following command at the root of the cloned project directory:
  docker compose up
  After a moment, the application will be up and running.
  In the stack is another service that can be used to publish messages. Open it by going to
  http://localhost:3000
  . As you type in a message and submit the form, you should see the log message of the message being received by the app.
  This helps demonstrate how a containerized approach makes it easy to add additional services to help test and troubleshoot your application.
  Adding cluster visualization
  Once you start using containers in your development environment, you start to realize the ease of adding additional services that are solely focused on helping development, such as visualizers and other supporting services. Since you have Kafka running, it might be helpful to visualize whatâs going on in the Kafka cluster. To do so, you can run the
  Kafbat UI web application
  To add it to your own project (itâs already in the demo application), you only need to add the following configuration to your Compose file:
  services
  kafka-ui
  image
  kafbat/kafka-ui:main
  ports
  environment
  DYNAMIC_CONFIG_ENABLED
  "true"
  KAFKA_CLUSTERS_0_NAME
  local
  KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS
  kafka:9093
  depends_on
  kafka
  Then, once the Compose stack starts, you can open your browser to
  http://localhost:8080
  and navigate around to view additional details about the cluster, check on consumers, publish test messages, and more.
  Testing with Kafka
  If youâre interested in learning how you can integrate Kafka easily into your integration tests, check out the
  Testing Spring Boot Kafka Listener using Testcontainers guide
  . This guide will teach you how to use Testcontainers to manage the lifecycle of Kafka containers in your tests.
  Conclusion
  By using Docker, you can simplify the process of developing and testing event-driven applications with Kafka. Containers simplify the process of setting up and deploying the various services you need to develop. And once theyâre defined in Compose, everyone on the team can benefit from the ease of use.
  In case you missed it earlier, all of the sample app code can be found at dockersamples/kafka-development-node.