Group By & Assembly

Number of input and output link

Number of input link	:	One
Number of output link	:	One

Group By & Aggregator Assembly (

) or (GroupBy

and Aggregator

in one assembly) groups and aggregates data based on particular field(s). Group By & Aggregator is used to group and aggregate data in a single assembly. The shuffle process also occurs in this assembly.

There are two tabs in this assembly parameter:

Grouping Tab

Consists of three parameters for the Group By process:

Group Fields : indicates the field(s) for secondary sort (optional).
Partitioning Class : indicates the partitioning class used to shuffle data (applicable for Map Reduce and Spark RDD engines).
There are some options for Partitioning Class:
- Default Partitioning : Default Hash partitioning
- cascading.solusi247.partition.AlphabetPartitioning : partitioning based on the first character (read as alphabet) of the group value
- cascading.solusi247.partition.AlphaNumPartitioning : partitioning based on the first character (read as alpha-num) of the group value
- cascading.solusi247.partition.RangeTextPartitioning : partitioning based on a range of text (string). This range is defined in the Partition Parameter. For example, if we want all text beginning with a capital letter to be sent to partition 0, and all text beginning with a lowercase letter to be sent to partition 1, then the content of the Partition Parameter is A-Z,a-z.
  Any text outside of the range will be sent to partition 0
- cascading.solusi247.partition.RangeNumberPartitioning : partitioning based on a range of numbers. This range is defined in the Partition Parameter. For example, if we want all values between 0 and 1000 to be sent to partition 0, and all values between 1001 and 10000 to be sent to partition 1, then the content of the Partition Parameter is 0-100,1001-1000.
  Any value outside of the range will be sent to partition 0
Partition Parameter : indicates the parameter of the partitioning class, when not using the Default Partitioner

Aggregation Tab

Consists of three aggregate function types:

Basic : Basic Aggregate Functions
1. Average : Calculates the Average
2. Count : Calculates the Count
3. CountDistinct : Calculates the Count Distinct
4. Max : Calculates the Maximum Value
5. Median : Calculates the Median Value
6. Min : Calculates the Minimum Value
7. Sum : Calculates the Sum Value
8. StandardDeviation : Calculates the Sample Standard Deviation
  Using the formula :
9. StdDevPop : Calculates the Population Standard Deviation
  Using the formula :
10. Percentile : In statistics, a percentile (or a centile) is a type of quantile which divides the given probability distribution, or sample, into 100 equal-sized intervals; this allows the data to be analyzed in terms of percentages. For example, the 20th percentile is the value (or score) below which 20% of the observations are found, and above which 80% are found.
  When the parameter value is more than 1 (separated by ','), multiple output values will be produced (separated by '|').
11. Decile : In descriptive statistics, a decile is any of the nine values that divide the sorted data into ten equal parts, so that each part represents 1/10 of the sample or population.
  When the parameter value is more than 1 (separated by ','), multiple output values will be produced (separated by '|').
12. Quartile : In statistics, a quartile is a type of quantile which divides the number of data points into four parts, or quarters, of more-or-less equal size. The data must be ordered from smallest to largest to compute quartiles; as such, quartiles are a form of order statistic.
  The three main quartiles are as follows: .
  - The first quartile (Quartile 1) is defined as the middle number between the smallest number (minimum) and the median of the data set. It is also known as the lower or 25th empirical quartile, as 25% of the data is below this point.
  - The second quartile (Quartile 2) is the median of a data set; thus 50% of the data lies below this point.
  - The third quartile (Quartile 3) is the middle value between the median and the highest value (maximum) of the data set. It is known as the upper or 75th empirical quartile, as 75% of the data lies below this point.
  When parameter value more then 1 (separated by ','), multiple output values will be produced (separated by '|').
13. UDFAggregation247
14. UDFPythonAggregation247
Condition Aggregator
1. ConditionAverage : Calculates the Average based on a condition
2. ConditionCount : Calculates the Count based on a condition
3. ConditionMax : Calculates the Maximum Value based on a condition
4. ConditionMin : Calculates the Minimum Value based on a condition
5. ConditionSum : Calculates the Sum Value based on a condition
Calculation Aggregator
1. CalculationAverage : Calculates the Average based on a math calculation
2. CalculationMax : Calculates the Maximum Value based on a math calculation
3. CalculationMin : Calculates the Minimum Value based on a math calculation
4. CalculationSum : Calculates the Sum Value based on a math calculation

Supported Engines

Map Reduce	Spark 1.x RDD	Spark 2.x RDD	Spark 2.x Dataset	Apache Beam 2.29.0 Batch	Apache Beam 2.29.0 Streaming	Apache Flink 1.12.x Batch	Apache Flink 1.12.x Streaming	Apache Tez 0.9.1	PySpark