Hadoop MapReduce in HGrid247-3.5

Hadoop MapReduce is a distributed data processing framework used to process very large amounts of data (Big Data) on computer clusters.

MapReduce works by dividing the work into two main stages:

• Map Phase
• Reduce Phase

This model allows parallel data processing on multiple nodes in a cluster.

MapReduce Processing Phases

1. Map Phase

At this stage, the following is done:

• reading data from HDFS
• splitting data into several parts (splits)
• carry out the transformation or mapping process

The output of a map is usually in the form of a pair:

(key, value)

Example:
Input data:

    Jakarta, 
    Bandung, 
    Jakarta, 
    Surabaya

Output from Map:

    (Jakarta,1) 
    (Bandung,1) 
    (Jakarta,1) 
    (Surabaya,1)

2. Shuffle Phase

Once the Map is complete, Hadoop will perform the following processes:

• grouping the same keys
• send data to the appropriate reducer

Example:

    Jakarta → [1,1] 
    Bandung → [1] 
    Surabaya → [1]

This process is called Shuffle and Sort .

3. Reduce Phase

Reduce is responsible for:

• perform data aggregation
• produce final output

Example:

    Jakarta → 2 
    Bandung → 1 
    Surabaya → 1

MapReduce Workflow Architecture

Simple MapReduce architecture:

    Input Data (HDFS)
            ↓
         Map Phase
            ↓
       Shuffle & Sort
            ↓
        Reduce Phase
            ↓
     Output Data (HDFS)

MapReduce Job Generated by HGrid247

HGrid247 can automatically generate Java-based MapReduce code from workflows created in the GUI.
HGrid247 Workflow → compiled into:

    JAR Application

The JAR file contains:

• mapper
• reducer
• driver program

Running MapReduce Job on Hadoop Cluster

To run a MapReduce job on a Hadoop cluster use the following command:

    hadoop jar <jar-app> <class-to-run> <parameter-list>
            

Parameter explanation:

    Parameter           Information
    ----------------------------------------------------------------
    jar-app             JAR file the result of compiling the workflow
    class-to-run	the main class that will be run
    parameter-list	parameter input and output

Example Execution

Example of running a MapReduce job generated by HGrid247:

    hadoop jar workshop_mr.jar workshop.Lab01 /user/hadoop/input /user/hadoop/output

Explanation:

    Parameter               Meaning
    workshop_mr.jar         compiled MapReduce application
    workshop.Lab01          main class of the workflow
    /user/hadoop/input      input folder in HDFS
    /user/hadoop/output     output folder in HDFS