Number of input and output link

Number of input link	:	One
Number of output link	:	One

Group By & Buffer Transformation Assembly () or (GroupBy and Buffer-Transformation in one assembly) groups data based on one or more specified fields and applies one or more Buffer Transformation Functions to each group.
This assembly may also perform sorting on one or more fields.

Important:
The shuffle process occurs in this assembly. During shuffle, records with the same group key are redistributed across nodes to ensure that all records in the same group are processed together.

Assembly Parameters

This assembly consists of two configuration tabs:

1. Grouping & Sorting Tab

   This tab defines how data is grouped, sorted, and partitioned during the shuffle process.
   • Reverse Order
     Indicates whether the group sorting order should be reversed.
     Default value: false
   • Group Fields
     Specifies the field(s) used to define the grouping key.
     Mandatory for grouping
     Multiple fields can be specified
   • Sort Field
     Specifies one or more fields for secondary sorting within each group.
     Optional
   • Partitioning Class
     Specifies the partitioning strategy used during the shuffle process.
     Applicable for Hadoop MapReduce and Spark RDD engines
     
     Available Partitioning Classes
     
     • Default Partitioning
       Uses the default hash-based partitioning.
     • cascading.solusi247.partition.AlphabetPartitioning
       Partitions data based on the first character of the group value, interpreted as an alphabet.
     • cascading.solusi247.partition.AlphaNumPartitioning
       Partitions data based on the first character of the group value, interpreted as alphanumeric.
     • cascading.solusi247.partition.RangeTextPartitioning
       Partitions data based on defined beginning text ranges.
       Example:

       A-Z,a-z
       

       • Uppercase values → partition 0
       • Lowercase values → partition 1
       • Values outside the range → partition 0
     • cascading.solusi247.partition.RangeNumberPartitioning
       Partitions data based on numeric ranges.
       Example:

       0-1000,1001-10000
       

       • 0–1000 → partition 0
       • 1001–10000 → partition 1
       • Values outside the range → partition 0
   • Partition Parameter
     Specifies additional parameters for the selected partitioning class.
     Required when using:
     • RangeTextPartitioning
     • RangeNumberPartitioning
     Not required for:
     • Default Partitioning

2. Buffer Transformation Function Tab

   This tab defines the buffer-based transformations applied to each group.
   • Bult-in
     • Sum
       Calculates the sum of values in a field within a group.
     • Max
       Calculates the minimum of values in a field within a group.
     • Min
       Calculate the minimum of value of a field
     • Average
       Calculates the average value of a field in a group.
     • Count
       Calculates the number of records in a group.
     • Rank
       Calculates rank ordering within a group.
       
       Rank Parameter Options
       

       Parameter	Description	Example
       0 (default)	Multi-rank without gap	1, 2, 2, 3, 3, 4
       1	Multi-rank with gap	1, 2, 2, 4, 4, 6
       2	Single rank (strict ordering)	1, 2, 3, 4, 5, 6

     • StandardDeviation
       Calculates the standard deviation of values in a field.
     • First
       Returns the first value of a field in a group or sub-group.
     • Last
       Finds the last value of a group or sub-group
     • FirstNextChange
       Finds the first value of the next group or sub-group after a change occurs.
     • DistanceFromPrevious
       Retrieves the previous record’s value for a field.
     • ValueFromPrevious
       Get the previous value
     • Sequence
       Generates a sequential number for records in a group.
     • Route
       Identifies the route or path of values of a field within a group or sub-group.
     • Concate
       Concatenates values of a field within a group or sub-group.
     • SimpleCalculation
       Performs a simple arithmetic calculation using field values in the same record.
     • Pareto
       Calculates Pareto (80/20) distribution values, commonly used in business analytics.
   • Custom
     Allows user-defined buffer transformation logic.f

   Emit Modes (Output Behavior)

   Buffer Transformation supports three emit strategies:
   1. EMIT FOR EVERY RECORD
      • Output is emitted for each input record
      • Suitable for:
        • Running totals
        • Sequence numbers
        • ValueFromPrevious
   2. EMIT ON CHANGE
      • Output is emitted only when a field value (or combination of fields) changes
      • Suitable for:
        • Change detection
        • State transition analysis
        • Slowly changing data
   3. EMIT FOR EVERY GROUP
      • Output is emitted once per group
      • Suitable for:
        • Group summaries
        • Final transformation results

   Buffer Transformation vs Aggregator

   Aspect	Aggregator	Buffer Transformation
   Input	Grouped data	Grouped data
   Output	One row per group	Flexible
   Access to full group	❌ Limited	✅ Full
   Running calculations	❌	✅
   Ranking & sequencing	❌	✅
   Memory usage	Low	Higher

   Performance Considerations

   • Buffer transformations are memory-intensive
   • Large groups can increase memory usage
   • Always:
     • Use proper partitioning
     • Sort data only when needed
     • Avoid unnecessary buffer logic

   Typical Use Cases

   • Ranking and leaderboard calculation
   • Running totals and deltas
   • Sequence numbering
   • Pareto analysis
   • Change detection
   • Slowly change data

Supported Engines

Map Reduce	Spark 1.x RDD	Spark 2.x RDD	Spark 2.x Dataset	Apache Beam 2.29.0 Batch	Apache Beam 2.29.0 Streaming	Apache Flink 1.12.x Batch	Apache Flink 1.12.x Streaming	Apache Tez 0.9.1	PySpark