Apache Tez in HGrid247-3.5

Apache Tez is a data processing framework designed toefficientlycomplex DAG (Directed Acyclic Graph)-based computational workflows

Tez was developed to improve data processing performance in the Apache Hadoop ecosystem , especially compared to traditional MapReduce .

Tez runs on top of Apache Hadoop YARN as a resource manager.

DirectedAcyclic Graph (DAG)

The main concept of Apache Tez is DAG (Directed Acyclic Graph) .

DAGs allow for a more flexible data processing pipeline than the MapReduce model, which only has two stages.

Tez vs MapReduce

The main differences between Tez and MapReduce:

    Feature             MapReduce	Tez
    ------------------------------------------
    ProcessingModel	Map → Reduce	DAG
    Pipeline            Fixed           Flexible
    Performance         Slower          Faster
    Optimization	Limited         More optimal

Tez is often used by tools such as:

• Apache Hive
• Apache Pig

to improve query performance.

Apache Tez in HGrid247

As of HGrid247-3.5 , workflows created in the GUI can be compiled into Apache Tez programs .

The output is usually a file:

    workshop_tez.jar

Running Tez Job

To run a Tez job use the following command:

    hadoop jar <jar-app> <class-to-run> -Dmapreduce.framework.name=yarn-tez <parameter-list>
            

Parameter:

    Parameter                               Meaning
    -----------------------------------------------------------
    jar-app                                 application JAR file
    class-to-run                            main class
    -Dmapreduce.framework.name=yarn-tez     running the Tez framework
    parameter-list                          parameter workflow

Example Execution

Example of running a Tez job generated by HGrid247:

    hadoop jar workshop_tez.jar workshop.Lab01 -Dmapreduce.framework.name=yarn-tez /user/hadoop/input /user/hadoop/output

Parameter explanation:

    Parameter               Description
    workshop_tez.jar        Tez app
    workshop.Lab01          main class
    /user/hadoop/input      folder input pada HDFS
    /user/hadoop/output     folder output pada HDFS