Point of Optimization in HGrid247

Optimization in HGrid247 aims to improve data processing performance , especially in distributed computing environments like Hadoop, Spark, and Flink.
This optimization is achieved by reducing unnecessary resource usage and increasing the efficiency of the data distribution process.

Some optimization strategies that can be applied in HGrid247 are as follows.

1. Chaining Transformation Functions

One optimization technique is to combine several transformation functions sequentially (chaining) .
If possible, several transformation operations can be combined in one process so that:

• reduce execution overhead
• reduce the number of processing stages
• improve pipeline efficiency

2. Avoiding or Minimizing Shuffle

Shuffle is the process of redistributing data between nodes in a distributed computing system.
This process is among the most expensive operations in Big Data processing because it involves:

• data transfer over a network
• sorting data
• auxiliary I/O disk

In HGrid247, the shuffle process usually occurs at:

• Group By Assembly
• Join Assembly

Therefore, the number of shuffles should be minimized as much as possible.

1. Minimize the Number of Shuffled Records
   One way to reduce shuffle is to reduce the number of records processed in the shuffle stage .
   The techniques used are:
   
   Using Combiner before Group By
   Combiner Assembly performs local aggregation before the data is sent to the Group By process.
   Examples of aggregations that can use Combiner:
   • MIN
   • MAX
   • COUNT
   • SUM
   Pipeline without combiner:

       Source → Group By → Aggregator
   

   Optimized pipeline:

       Source → Combiner → Group By → Aggregator
   

   Notes:
   Combiner is only used in batch processing
   
   
2. Minimize the Amount of Columns in Shuffle
   Besides the number of records, the number of fields or columns in the shuffled records also affects performance.
   
   To optimize:
   • delete unused fields
   • perform column selection before the Group By or Join process
   
   
   
3. Avoid Shuffle Using Reference Join (if possible)
   Shuffle can also be avoided by using Reference Join or Lookup Transformation techniques .
   This technique is suitable for use when:
   • the dataset on the right side of the join is relatively small
   • the dataset can be loaded into memory
   Standard pipeline:

       Dataset A → Join → Dataset B
   

   Optimized pipeline:

       Dataset A → Reference Join → Small Dataset B
   

   Profit:
   • no need to shuffle
   • faster in data processing
   
   
4. Using Appropriate Number of Parallelization
   
   1. Optimization in MapReduce and Tez Engine In MapReduce or Tez engines , the number of reducers can be controlled with the parameter:

          mapred.reduce.tasks
      

      Example configuration:

          mapred.reduce.tasks = 10
      

      This setting determines the number of reducers used in the data distribution process.
      
      
   2. Optimization in Spark Engine In the Spark engine , parallelism optimization can be done through several configuration parameters.
      
      1. spark.default.parallelism
      
      Specifies the default number of parallel tasks in Spark.

          
          spark.default.parallelism
      

      2. spark.sql.shuffle.partitions
      
      Determines the number of partitions used in the shuffle operation.

           spark.sql.shuffle.partitions
      

      3. Distribution
      
      The number of partitions can also be set directly in the Group By or Join Assembly.
      This function helps distribute data more evenly across the cluster.
   3. Optimization in Flink Engine
      In Flink Engine , the amount of parallelism can be set while running a job.
      Contoh command:

          flink run -p 8 job.jar
      

      or use the options:

          --parallelism
      

      This parameter determines the number of tasks to run in parallel in a Flink cluster.