Number of input and output link

Number of input link	:	One
Number of output link	:	One

Global Combiner Assembly() or (Local Combiner , Group By and Aggregator in one assembly) performs end-to-end aggregation by combining:

1. Local Combiner (Mapper-side aggregation)
2. Group By (Shuffle & grouping)
3. Final Aggregator (Reducer-side aggregation)

into a single logical assembly.

This assembly is designed to minimize shuffle data, optimize network usage, and simplify pipeline design while ensuring correct global aggregation results.

How it works

1. Mapper Phase Processes input records Emits key–value pairs
2. Local Combiner Phase
   • Aggregates data locally on the same node
   • Produces partial aggregation results
3. Shuffle Phase
   • Only pre-aggregated results are transferred across the network
   • Data is partitioned and grouped by key
4. Reducer Phase
   • Performs final aggregation
   • Produces globally correct aggregated output

Aggregate Types

1. Basic Aggregator
   Provides standard aggregation functions:
   1. Count
      Calculates the number of records
   2. Max
      Calculates the maximum value
   3. Min
      Calculates the minimum value
   4. Sum
      Calculates the sum of values
2. Condition Aggregator
   Performs aggregation only when a specified condition is satisfied:
   1. ConditionCount
      Calculates the count based on a condition
   2. ConditionMax
      Calculates the maximum value based on a condition
   3. ConditionMin
      Calculates the minimum value based on a condition
   4. ConditionSum
      Calculates the sum of values based on a condition
3. Calculation Aggregator
   Performs aggregation based on mathematical expressions applied to fields:
   1. CalculationMax
      Calculates the maximum value based on a math calculation
   2. CalculationMin
      Calculates the minimum value based on a math calculation
   3. CalculationSum
      Calculates the sum of values based on a math calculation
   The mathematical calculation is applied before aggregation, both in the local combiner and in the final reducer, ensuring correctness.

Important Constraints

• Global Combiner works only with associative and commutative operations, such as:
  • Sum
  • Count
  • Min
  • Max
• Not suitable for:
  • Average (directly)
  • Median
  • Percentile
  • Standard Deviation

Calculating Average (Best Practice)

To compute Average, use the following approach:

1. Use CalculationSum or Sum → compute total value
2. Use Count → compute number of records
3. After Global Combiner Assembly, apply the formula:

         average = sum / count
           

This guarantees correctness across distributed partitions.

Advantages

• Reduces shuffle data significantly
• Improves performance for large-scale aggregations
• Simplifies pipeline (single assembly instead of three)
• Ensures correct global aggregation

Supported Engines

Map Reduce	Spark 1.x RDD	Spark 2.x RDD	Spark 2.x Dataset	Apache Beam 2.29.0 Batch	Apache Beam 2.29.0 Streaming	Apache Flink 1.12.x Batch	Apache Flink 1.12.x Streaming	Apache Tez 0.9.1	PySpark