Spark Word Count Example

In Spark word count example, we find out the frequency of each word exists in a particular file. Here, we use Scala language to perform Spark operations.

Steps to execute Spark word count example

In this example, we find and display the number of occurrences of each word.

• Create a text file in your local machine and write some text into it.

$ nano sparkdata.txt 

• Check the text written in the sparkdata.txt file.

$ cat sparkdata.txt

• Create a directory in HDFS, where to kept text file.

$ hdfs dfs -mkdir /spark

• Upload the sparkdata.txt file on HDFS in the specific directory.

$ hdfs dfs -put /home/codegyani/sparkdata.txt /spark

• Now, follow the below command to open the spark in Scala mode.

$ spark-shell

• Let's create an RDD by using the following command.

scala> val data=sc.textFile("sparkdata.txt")

Here, pass any file name that contains the data.

• Now, we can read the generated result by using the following command.

scala> data.collect;

• Here, we split the existing data in the form of individual words by using the following command.

scala> val splitdata = data.flatMap(line => line.split(" "));

• Now, we can read the generated result by using the following command.

scala> splitdata.collect;

• Now, perform the map operation.

scala> val mapdata = splitdata.map(word => (word,1));

Here, we are assigning a value 1 to each word.

• Now, we can read the generated result by using the following command.

scala> mapdata.collect;

• Now, perform the reduce operation

scala> val reducedata = mapdata.reduceByKey(_+_);

Here, we are summarizing the generated data.

• Now, we can read the generated result by using the following command.

scala> reducedata.collect;

Here, we got the desired output.