Kafka Console Consumer

In this section, the users will learn how a consumer consumes or reads the messages from the Kafka topics.

There are following steps taken by the consumer to consume the messages from the topic:

Step 1: Start the zookeeper as well as the kafka server initially.

Step2: Type the command: 'kafka-console-consumer' on the command line. This will help the user to read the data from the Kafka topic and output it to the standard outputs.

Note: Choose '.bat' or '.sh' as per the operating system.

The highlighted text represents that a 'bootstrap-server' is required for the consumer to get connected to the Kafka topics. Also, a 'topic_id' is required to know from which topic the consumer will read the messages.

Step3: After knowing all the requirements, try to consume a message from a topic using the command:

'kafka-console-consumer -bootstrap-server localhost:9092 -topic <topic_name>'. Press enter.

Note: Bootstrap server is the Kafka server, having port number=9092.

In the previous section, three messages were produced to this topic. But, in the above snapshot, 0 messages could be seen. It is because Apache Kafka does not read all the topics. A Kafka consumer will consume only those messages which are produced only when the consumer was in the active state. This can be categorized as a disadvantage of Apache Kafka.

Let's understand:

Open a new terminal. Launch the Kafka console producer. Keep both producer-consumer consoles together as seen below:

Now, produce some messages in the producer console. After doing so, press Ctrl+C and exit.

It is seen that all messages which are currently produced by the producer console are reflected in the consumer console. It is because the consumer is in an active state.

Reading whole messages

Apache Kafka allows to produce millions of messages. Sometimes, a consumer may require to read whole messages from a particular topic.

To do so, use '-from-beginning' command with the above kafka console consumer command as:

'kafka-console-consumer.bat -bootstrap-server 127.0.0.1:9092 -topic myfirst -from-beginning'. This command tells the Kafka topic to allow the consumer to read all the messages from the beginning(i.e., from the time when the consumer was inactive).

For example,

In the above snapshot, it is clear that all messages are displayed from the beginning.

Note: The order of the messages is not the 'total'. It is because the sequence is at the partition level only(as studied in the Kafka introduction section).

For this topic 'myfirst', we had three partitions. So, if a user wishes to see the order, create a topic with a single partition value. It will display whole messages in a sequence.

After completing the message exchange process, press 'Ctrl+C' and stop.

So, several messages can be consumed either from the beginning or from that state when the user wants the consumer to read.