Kafka Real Time Example

Till now, we learned how to read and write data to/from Apache Kafka. In this section, we will learn to put the real data source to the Kafka.

Here, we will discuss about a real-time application, i.e., Twitter. The users will get to know about creating twitter producers and how tweets are produced.

Twitter is a social networking service that allows users to interact and post the message. These messages are known as Tweets. The twitter users make interactions through posting and commenting on different posts through tweets.

To deal with Twitter, we need to get credentials for Twitter apps. It can be done by creating a Twitter developer account. To do so, follow the below steps:

Step1:Create a Twitter account, if it does not exist.

Step2: Open 'developer.twitter.com' on the web browser, as shown below:

Click on the Apply option.

Step3: A new page will open. Click on "Apply for a developer account"

Step4: A new page will open, asking the Intended use like, 'How you will use Twitter data?", and so on. A snapshot is shown below:

After giving the appropriate answers, click on Next.

Step5: The next section is the Review section. Here, the user explanations will be reviewed by Twitter, as shown below:

If twitter finds the answers appropriate, 'Looks good' option will get enabled. Then, move to the next section.

Step6: Finally, the user will be asked to review and accept the Developer Agreement. Accept the agreement by clicking on the checkbox. Submit the application by clicking on the 'Submit Application'.

Step7: After successful completion, an email confirmation page will open. Confirm with the provided email id and proceed further.

Step8: After confirmation, a new webpage will open. Click on 'Create an app' as shown below:

Step9: Provide the app details, as shown in the below snapshot:

Step10: After giving the app details, click on the 'Create' option. A dialog box will open "Review our Developer Terms". Click on the 'Create' option. A snapshot is shown below:

Finally, the app will be created in the following way:

Note: When the app will be created. It will generate Keys and Tokens. Do not disclose them as these are the secret or sensitive information. If did so, the user can regenerate them for safety purposes.

Step11: After creating an app, we need to add the twitter dependency in the 'pom.xml' file. To do so, open 'github twitter java' on a web browser. A snapshot is shown below:

Open the highlighted link or visit: 'https://github.com/twitter/hbc' to open directly.

Step12: There, the user will find the twitter dependency code. Copy the code and paste it in the 'pom.xml' file below the maven dependency code.

A term 'hbc' is used in the dependency code. It stands for 'Hosebird Client' which is a java HTTP client. It is used for consuming Twitter's standard streaming API. The Hosebird Client is divided into two modules:

1. hbc-core: It uses a message queue. This message queue is further used by the consumer to poll for raw string messages.
2. hbc-twitter4j: This is different from hbc-core as it uses the twitter4j listeners. Twitter4j is an unofficial java library through which we can easily integrate our java build application with various twitter services.

In the twitter dependency code, hbc-core is used. Users can also use twitter4j instead.

So, in this way, the first stage of the real-time example is completed.