Menu Tools

Menu Tools in HGrid247-3.5

The Tools menu in HGrid247-3.5 is used to configure the system, cluster, data processing engine, and workflow execution environment .

This menu is very important because it is related to the integration of HGrid247 with Big Data platforms such as:

• Apache Hadoop
• Apache Spark
• Apache Flink
• Apache Atlas

1. Hadoop Cluster Client Setting

Function:

    Hadoop Cluster Client Setting

This menu is used to configure the HGrid247 connection with the Hadoop Cluster via the edge node .

Parameters that are usually configured:

    Parameter               Description
    -------------------------------------------------
    Edge Node IP            IP/Host-name of server edge node
    SSH Port                SSH port
    Username                user of server edge node
    Passoword               password of of server edge node
    Hadoop Home             Hadoop installation location
    Spark Home              Spark installation location
    Working Directory       Working Directory in the server edge node

Once the configuration is complete, HGrid247 can run workflows directly on the cluster.

2. File System & HDFS Explorer

Function:

    File System & HDFS Explorer

This menu is used to explore the local file system and HDFS on the cluster .

Through this feature, users can:

• view directory structure
• create directory
• upload file ke HDFS
• download files from HDFS
• delete files
• check input and output data

HDFS is part of:

• Apache Hadoop

3. Run Workflow on Cluster

Function:

    Run Workflow on Cluster

This menu is used for:

• Compiling workflow
• Build applications according to the selected engine
• Running workflows directly on the cluster

4. Convert Jar to HGrid247-3.5 Project

Function:

    Convert Jar to HGrid247-3.5 Project

This menu is used to convert the JAR file generated by HGrid247 back into an HGrid247 project .

The purpose of this feature:

• reverse engineering the workflow
• modify old workflow
• perform application debugging

Example:

    workshop_spark2.jar

converted back to HGrid247 project.

5. Processing Target Engine

Function:

    Processing Target Engine

This menu is used to select the data processing engine that will be used by the workflow .

Available engines are usually:

    Engine          Description
    MapReduce       Batch processing
    Spark           Fast in-memory processing
    Flink           Stream/Batch processing
    Beam            Unified pipeline for stream/batch processing
    Tez             DAG (Directed Acyclic Graph) tasks, for improving Map Reduce
    PySpark         Python API for Apache Spark

6. Built Jar Directory

Function:

    Built Jar Directory

This menu is used to determine the storage location for the JAR file resulting from the build workflow . Example folder output:

    /home/hgrid/build/

The resulting file is for example:

    /home/hgrid/build/workshop_spark2.jar
    /home/hgrid/build/workshop_flink.jar
    /home/hgrid/build/workshop_mr.jar

7. Building Jar Library Option

Function:

    Building Jar Library Option

This menu is used to set how main and additional libraries are included into the JAR file during the build process .

8. Atlas Client Configuration

Function:

    Atlas Client Configuration

This menu is used to configure the connection to Apache Atlas .

Apache Atlas is a data governance and metadata management system .

9. Local Computer Configuration

Function:

    Local Computer Configuration

This menu is used to set up a local environment to run workflows without a cluster .

Usually used for:

• testing workflow
• debugging pipeline
• local development

10. PySpark Code Configuration

Function:

    PySpark Code Configuration

This menu is used to configure the PySpark code generation of the HGrid247 workflow .