Sending data to Kafka Topics
Kafka Console Producer
In order to send data to the Kafka topic, a producer is required. The role of the producer is to send or write data/messages to the Kafka topics.
In this section, we will learn how a producer sends messages to the Kafka topics.
There are following steps used to launch a producer:
Step1: Start the zookeeper as well as the kafka server.
Step2: Type the command: ‘kafka-console-producer‘ on the command line. This will help the user to read the data from the standard inputs and write it to the Kafka topic.
Note: Choose ‘.bat’ or ‘.sh’ as per the operating system.
The highlighted text represents that a ‘broker-list‘ and a ‘topic id‘ is required to produce a message. It is because a producer must know the id of the topic to which the data is to be written.
Step3: After knowing all the requirements, try to produce a message to a topic using the command:
‘kafka-console-producer -broker-list localhost:9092 -topic <topic_name>’. Press enter.
Note: Here, 9092 is the port number of the Kafka server.
Here, ‘myfirst’ topic is chosen to write messages to.
A ‘>’ will appear in the new line. Start producing some messages, as shown below:
Step4: Press ‘Ctrl+c’ and exist by pressing the ‘Y’ key.
So, in this way, a producer can produce/send several messages to the Kafka topics.
Producer with Keys
A Kafka producer can write data to the topic either with or without a key. If a producer does not specify a key, the data will be stored to any of the partitions with key=null, else the data will be stored to the specified partition only. A ‘parse.key‘ and a ‘key.seperator‘ is required to specify a key for the topic. The command used is:
Here, key is the specific partition, and value is the message to be written by the producer to the topic.
When a topic does not exist?
Suppose the producer wants to send messages to a new topic that does not exist yet. In such a situation, a warning will appear, as shown in the below snapshot, after producing a message. It is just a warning.
Why this warning?
The warning occurred because earlier the topic ‘demo’ didn’t exist. But, as soon the producer wrote a message, Kafka somehow created that topic. Although, no leader election held for this unexpected topic, ‘LEADER_NOT_AVAILABLE‘ error could be seen. But, for the next time, the producer can continue to write more messages as no warning will appear again. It is because the topic comes in the existing list now.
The users can check using the ‘-list‘ command, as shown below:
The topic ‘demo’ can be seen on the list.
Describing the new topic
As such topics which are created directly by the producer grabs the default number of partitions and its replication factor as 1.
For example,
The topic ‘demo’ when described using the ‘-describe‘ command, gives the value of ‘PartitionCount’ and ‘ReplicationFactor’ as 1(default value). Thus, it is always a better option to create a topic before producing messages to it.
Changing the Default Values
Follow the below steps to change the default values for the new topic:
- Open ‘server.properties‘ file using Notepad++, or any other text editor.
- Edit the value of num.partitions=1 to a new value. Let it be 3. So, whenever such new topics are introduced, the number of PartitionCount and ReplicationFactor will be 3(whatever the user has set).
- Save the file.
But, always create topics before.