Home » Kafka Real Time Example

Kafka Real Time Example

by Online Tutorials Library

Kafka Real Time Example

Till now, we learned how to read and write data to/from Apache Kafka. In this section, we will learn to put the real data source to the Kafka.

Here, we will discuss about a real-time application, i.e., Twitter. The users will get to know about creating twitter producers and how tweets are produced.

Twitter is a social networking service that allows users to interact and post the message. These messages are known as Tweets. The twitter users make interactions through posting and commenting on different posts through tweets.

To deal with Twitter, we need to get credentials for Twitter apps. It can be done by creating a Twitter developer account. To do so, follow the below steps:

Step1:Create a Twitter account, if it does not exist.

Step2: Open ‘developer.twitter.com’ on the web browser, as shown below:

Kafka Real Time Example

Click on the Apply option.

Step3: A new page will open. Click on “Apply for a developer account”

Kafka Real Time Example

Step4: A new page will open, asking the Intended use like, ‘How you will use Twitter data?”, and so on. A snapshot is shown below:

Kafka Real Time Example

After giving the appropriate answers, click on Next.

Step5: The next section is the Review section. Here, the user explanations will be reviewed by Twitter, as shown below:

Kafka Real Time Example

If twitter finds the answers appropriate, ‘Looks good’ option will get enabled. Then, move to the next section.

Step6: Finally, the user will be asked to review and accept the Developer Agreement. Accept the agreement by clicking on the checkbox. Submit the application by clicking on the ‘Submit Application’.

Kafka Real Time Example

Step7: After successful completion, an email confirmation page will open. Confirm with the provided email id and proceed further.

Step8: After confirmation, a new webpage will open. Click on ‘Create an app’ as shown below:

Kafka Real Time Example

Step9: Provide the app details, as shown in the below snapshot:

Kafka Real Time Example

Step10: After giving the app details, click on the ‘Create’ option. A dialog box will open “Review our Developer Terms”. Click on the ‘Create’ option. A snapshot is shown below:

Kafka Real Time Example

Finally, the app will be created in the following way:

Kafka Real Time Example

Note: When the app will be created. It will generate Keys and Tokens. Do not disclose them as these are the secret or sensitive information. If did so, the user can regenerate them for safety purposes.

Step11: After creating an app, we need to add the twitter dependency in the ‘pom.xml’ file. To do so, open ‘github twitter java’ on a web browser. A snapshot is shown below:

Kafka Real Time Example

Open the highlighted link or visit: ‘https://github.com/twitter/hbc’ to open directly.

Step12: There, the user will find the twitter dependency code. Copy the code and paste it in the ‘pom.xml’ file below the maven dependency code.

Kafka Real Time Example

A term ‘hbc’ is used in the dependency code. It stands for ‘Hosebird Client’ which is a java HTTP client. It is used for consuming Twitter’s standard streaming API. The Hosebird Client is divided into two modules:

  1. hbc-core: It uses a message queue. This message queue is further used by the consumer to poll for raw string messages.
  2. hbc-twitter4j: This is different from hbc-core as it uses the twitter4j listeners. Twitter4j is an unofficial java library through which we can easily integrate our java build application with various twitter services.

In the twitter dependency code, hbc-core is used. Users can also use twitter4j instead.

So, in this way, the first stage of the real-time example is completed.


You may also like