Number of input and output link

Number of input link	:	One
Number of output link	:	One

Aggregator Assembly ()performs aggregation operations on grouped data.
It is typically used after a Group By Assembly, where records are grouped based on one or more keys.
The aggregation functions are then applied to each group to produce summarized results.

Prerequisites

• Group By Assembly is mandatory before using Aggregator Assembly.
• Aggregation is performed per group, not on raw records.
• The shuffle and grouping logic is handled by the Group By Assembly.

Aggregate Function Types

Aggregator Assembly supports the following categories of aggregation functions:

1. Basic Aggregate Functions
   1. Average
      Calculates the average (mean) of values in a group.
   2. Count
      Calculates the number of records in a group.
   3. CountDistinct
      Calculates the number of distinct values in a group.
   4. Max
      Calculates the maximum value in a group.
   5. Median
      Calculates the median value in a group.
   6. Min
      Calculates the minimum value in a group.
   7. Sum
      Calculates the sum value in a group.
   8. StandardDeviation
      Calculates the sample standard deviation.
      Using the formula :
   9. StdDevPop
      Calculates the population standard deviation.
      Using the formula :
   10. Percentile
       In statistics, a percentile divides a dataset into 100 equal parts.
       • Example:
         • The 20th percentile is the value below which 20% of the observations fall.
       • If the parameter contains multiple values (separated by commas ,):
         • Multiple output values are produced
         • Output values are separated by |
   11. Decile
       A decile divides sorted data into 10 equal parts.
       • Each decile represents 10% of the dataset.
       • If multiple decile values are provided (comma-separated):
         • Multiple outputs are generated
         • Output values are separated by |
   12. Quartile
       A quartile divides the dataset into four equal parts.
       The main quartiles are:
       • Quartile 1 (Q1)
         • 25th percentile
         • Middle value between the minimum and the median
       • Quartile 2 (Q2)
         • Median (50th percentile)
       • Quartile 3 (Q3)
         • 75th percentile
         • Middle value between the median and the maximum
       If multiple quartile values are specified (comma-separated), multiple output values are produced and separated by |.
   13. UDFAggregation247
       Custom aggregation implemented using a Java-based UDF.
   14. UDFPythonAggregation247
       Custom aggregation implemented using a Python-based UDF.
2. Condition Aggregator
   Condition-based aggregators apply aggregation only when a condition is met.
   1. ConditionAverage
      Calculates the average based on a condition.
   2. ConditionCount
      Calculates the count based on a condition.
   3. ConditionMax
      Calculates the maximum based on a condition.
   4. ConditionMin
      Calculates the minimum based on a condition.
   5. ConditionSum
      Calculates the sum based on a condition.
3. Calculation Aggregator
   Calculation aggregators perform aggregation based on computed expressions.
   1. CalculationAverage
      Calculates the average based on a mathematical calculation.
   2. CalculationMax
      Calculates the maximum based on a mathematical calculation.
   3. CalculationMin
      Calculates the minimum based on a mathematical calculation.
   4. CalculationSum
      Calculates the sum based on a mathematical calculation.

Behavior

• Aggregation is applied per group
• One output record is produced for each group
• Supports numeric and statistical aggregations
• Multiple aggregation results can be generated per group

Performance Notes

• Aggregation performance depends on:
  • Number of groups
  • Volume of data per group
• Heavy aggregations benefit from proper partitioning in Group By Assembly
• Percentile, decile, and quartile operations are more computationally expensive than basic aggregates

Supported Engines

Map Reduce	Spark 1.x RDD	Spark 2.x RDD	Spark 2.x Dataset	Apache Beam 2.29.0 Batch	Apache Beam 2.29.0 Streaming	Apache Flink 1.12.0 Batch	Apache Flink 1.12.0 Streaming	Apache Tez 0.9.1	PySpark