TheDeveloperBlog.com
C-Sharp | Java | Python | Swift | GO | WPF | Ruby | Scala | F# | JavaScript | SQL | PHP | Angular | HTML
Data Flow in MapReduce
Data Flow in MapReduce with tutorial, introduction, environment setup, first app hello world, state, props, flexbox, height and width, listview, scrollview, images, buttons, router, etc.
Data Flow In MapReduceMapReduce is used to compute the huge amount of data . To handle the upcoming data in a parallel and distributed form, the data has to flow from various phases. 
Phases of MapReduce data flow
Input readerThe input reader reads the upcoming data and splits it into the data blocks of the appropriate size (64 MB to 128 MB). Each data block is associated with a Map function. Once input reads the data, it generates the corresponding key-value pairs. The input files reside in HDFS. Note - The input data can be in any form.Map functionThe map function process the upcoming key-value pairs and generated the corresponding output key-value pairs. The map input and output type may be different from each other. Partition functionThe partition function assigns the output of each Map function to the appropriate reducer. The available key and value provide this function. It returns the index of reducers. Shuffling and SortingThe data are shuffled between/within nodes so that it moves out from the map and get ready to process for reduce function. Sometimes, the shuffling of data can take much computation time. The sorting operation is performed on input data for Reduce function. Here, the data is compared using comparison function and arranged in a sorted form. Reduce functionThe Reduce function is assigned to each unique key. These keys are already arranged in sorted order. The values associated with the keys can iterate the Reduce and generates the corresponding output. Output writerOnce the data flow from all the above phases, Output writer executes. The role of Output writer is to write the Reduce output to the stable storage.
Next TopicMapReduce API
|
Related Links:
- Data Mining Architecture
- Data Structures | DS Tutorial
- Computer Network | Data Link Controls
- DS for symbols tables
- Data Preprocessing in Machine learning
- Data Mining Tools
- Data Mining vs Machine Learning
- What is Data Science: Tutorial, Components, Tools, Life Cycle, Applications
- DS Algorithm
- DS Asymptotic Analysis
- Data Flow in Angular 7 Forms
- Data Mining Tutorial
- Data Mining Cluster Analysis
- Data Mining vs Data Warehousing
- DS Structure
- DS Array
- DS 2D Array
- DS Quene
- Different types of Clustering Algorithm
- Data Mining vs Big Data
- Data Mining Bayesian Classification
- Data Mining World Wide Web
- Data Types in C
- Verbal Reasoning | Data Sufficiency 2
- Data Mining Techniques
- DS Introduction
- Data Structure Interview Questions (2021)
- Verbal Reasoning | Data Sufficiency 1
- Top 25 Data Science Interview Questions (2021)
- Data Warehouse Tutorial
- Data Flow in MapReduce
- DS Stack
- Data Abstraction in C++
- Top 25 Data Warehouse Interview Questions (2021)
- Data Link layer
- Data Security Considerations
- DS Pointer