Number of input and output link

Number of input link	:	One
Number of output link	:	One

Group By & Buffer Assembly () or (GroupBy and Buffer in one assembly) combines grouping, sorting, shuffling, and buffer-based processing into a single assembly.
It groups data based on one or more fields, optionally sorts the data, and then applies a Buffer Function to process all records within each group.

Key Characteristics

• Performs grouping + buffer processing in one assembly
• Shuffle process occurs in this assembly
• Requires partitioning for distributed processing
• Processes all records of a group at once
• Can produce one or multiple output records per group
• Suitable for ranking, top/bottom N, running totals, SCD, and routing

How It Works

• Input records are grouped by Group Fields
• Data is shuffled so that records with the same group key go to the same reducer/partition
• Records within each group can be sorted
• The Buffer Function receives the full group
• Buffer emits one or more output records

Assembly Parameters

This assembly has two parameter tabs:

1. Grouping & Sorting Tab

   • Reverse Order
     Indicates whether the group sorting order should be reversed.
     Default value: false
   • Group Fields
     Specifies the field(s) used to define the grouping key.
     Mandatory for grouping
     Multiple fields can be specified
   • Sort Field
     Specifies one or more fields for secondary sorting within each group.
     Optional
   • Partitioning Class
     Specifies the partitioning strategy used during the shuffle process.
     Applicable for Hadoop MapReduce and Spark RDD engines
     
     Available Partitioning Classes
     
     • Default Partitioning
       Uses the default hash-based partitioning.
     • cascading.solusi247.partition.AlphabetPartitioning
       Partitions data based on the first character of the group value, interpreted as an alphabet.
     • cascading.solusi247.partition.AlphaNumPartitioning
       Partitions data based on the first character of the group value, interpreted as alphanumeric.
     • cascading.solusi247.partition.RangeTextPartitioning
       Partitions data based on defined beginning text ranges.
       Example:

       A-Z,a-z
       

       • Uppercase values → partition 0
       • Lowercase values → partition 1
       • Values outside the range → partition 0
     • cascading.solusi247.partition.RangeNumberPartitioning
       Partitions data based on numeric ranges.
       Example:

       0-1000,1001-10000
       

       • 0–1000 → partition 0
       • 1001–10000 → partition 1
       • Values outside the range → partition 0
   • Partition Parameter
     Specifies additional parameters for the selected partitioning class.
     Required when using:
     • RangeTextPartitioning
     • RangeNumberPartitioning
     Not required for:
     • Default Partitioning

2. Buffer Function Tab

   Defines the Buffer Function applied to each group.
   • Ranking
   • Top N Sum
   • Top N Count
   • Top N Average
   • Top N Max
   • Top N Min
   • Bottom N Sum
   • Bottom N Count
   • Bottom N average
   • Bottom N Max
   • Bottom N Min
   • Accumulation
   • Accumulations
   • Route
   • Running
   • N-Tile
   • NTileAggBucket
   • WindowPartition

Performance Considerations

• Buffer functions are memory-intensive
• Avoid very large groups without proper partitioning
• Always:
  • Define Group Fields carefully
  • Use Sort Field only when needed
  • Choose the correct Partitioning Class

Typical Use Cases

• Ranking and leaderboards
• Top/Bottom N per category
• Slowly Changing Dimension (SCD)
• Running totals and cumulative metrics
• Group-based routing logic

Supported Engines

Map Reduce	Spark 1.x RDD	Spark 2.x RDD	Spark 2.x Dataset	Apache Beam 2.29.0 Batch	Apache Beam 2.29.0 Streaming	Apache Flink 1.12.x Batch	Apache Flink 1.12.x Streaming	Apache Tez 0.9.1	PySpark