Number of input and output link

Number of input link	:	One
Number of output link	:	One

Group By Assembly () groups records based on one or more specified fields. It also supports secondary sorting, allowing records within each group to be sorted by one or more additional fields.

Group By Assembly is required when performing:

• Aggregation
• Buffer
• Buffer Transformation

This assembly is responsible for executing the shuffle process in distributed processing engines.

Shuffle Process Overview

During a shuffle, all records with the same group key are sent to the same reducer or task. The shuffle process includes:

• Partitioning
  Determines which reducer or task receives each group key.
• Data Transfer
  Moves data across the network between distributed nodes.
• Sorting and Grouping
  Organizes incoming records by group key at the destination node.

Note:
The shuffle process is often one of the most expensive operations in distributed computing due to heavy network I/O and disk I/O.

Parameters of Group By

1. Reverse Order
   Indicates whether the group sorting order should be reversed.
   Default value: false
2. Group Fields
   Specifies the field(s) used to define the grouping key.
   Mandatory for grouping
   Multiple fields can be specified
3. Sort Field
   Specifies one or more fields for secondary sorting within each group.
   Optional
4. Partitioning Class
   Specifies the partitioning strategy used during the shuffle process.
   Applicable for Hadoop MapReduce and Spark RDD engines
   
   Available Partitioning Classes
   
   • Default Partitioning
     Uses the default hash-based partitioning.
   • cascading.solusi247.partition.AlphabetPartitioning
     Partitions data based on the first character of the group value, interpreted as an alphabet.
   • cascading.solusi247.partition.AlphaNumPartitioning
     Partitions data based on the first character of the group value, interpreted as alphanumeric.
   • cascading.solusi247.partition.RangeTextPartitioning
     Partitions data based on defined beginning text ranges.
     Example:

     A-Z,a-z
     

     • Uppercase values → partition 0
     • Lowercase values → partition 1
     • Values outside the range → partition 0
   • cascading.solusi247.partition.RangeNumberPartitioning
     Partitions data based on numeric ranges.
     Example:

     0-1000,1001-10000
     

     • 0–1000 → partition 0
     • 1001–10000 → partition 1
     • Values outside the range → partition 0
   • cascading.solusi247.partition.RoundRobinPartitioning
     Distributes records evenly across partitions for load balancing.
5. Partition Parameter
   Specifies additional parameters for the selected partitioning class.
   Required when using:
   • RangeTextPartitioning
   • RangeNumberPartitioning
   Not required for:
   • Default Partitioning
   • RoundRobinPartitioning

Behavior

• Records are grouped based on the specified Group Fields
• Optional secondary sorting is applied using Sort Field
• All records with the same group key are processed together
• Shuffle is executed automatically by the engine
• Enables downstream operations such as aggregation and buffering

Typical Use Cases

• Aggregating data by key (SUM, COUNT, AVG, etc.)
• Preparing data for Buffer or Buffer Transformation
• Load balancing output data
• Sorting records within groups

Notes

• Group By Assembly can significantly impact performance due to shuffle overhead
• Use efficient partitioning strategies to minimize data skew
• RoundRobinPartitioning is recommended when balanced output is required

Supported Engines

Map Reduce	Spark 1.x RDD	Spark 2.x RDD	Spark 2.x Dataset	Apache Beam 2.29.0 Batch	Apache Beam 2.29.0 Streaming	Apache Flink 1.12.0 Batch	Apache Flink 1.12.0 Streaming	Apache Tez 0.9.1	PySpark