Buffer Transformation Assembly (
) is used to transform grouped data or grouped streams by applying buffer-based transformation functions.
Unlike Aggregator Assembly (which reduces data into aggregated results), Buffer Transformation can produce transformed records per row, per change, or per group, depending on the emit mode.
This assembly must operate on grouped data, meaning it logically follows a Group By process.
Key Characteristics
- Works on grouped data
- Has access to all records in a group
- Can emit:
- One output per record
- One output per change
- One output per group
- Supports running, ranking, sequencing, and historical transformations
- Suitable for analytical transformations, not just aggregation
Built-in Buffer Transformation Functions
- Bult-in
- Sum
Calculates the sum of values in a field within a group.
- Max
Calculates the minimum of values in a field within a group.
- Min
Calculate the minimum of value of a field
- Average
Calculates the average value of a field in a group.
- Count
Calculates the number of records in a group.
- Rank
Calculates rank ordering within a group.
Rank Parameter Options
| Parameter | Description | Example |
|---|
| 0 (default) | Multi-rank without gap | 1, 2, 2, 3, 3, 4 |
| 1 | Multi-rank with gap | 1, 2, 2, 4, 4, 6 |
| 2 | Single rank (strict ordering) | 1, 2, 3, 4, 5, 6 |
- StandardDeviation
Calculates the standard deviation of values in a field.
- First
Returns the first value of a field in a group or sub-group.
- Last
Finds the last value of a group or sub-group
- FirstNextChange
Finds the first value of the next group or sub-group after a change occurs.
- DistanceFromPrevious
Retrieves the previous record’s value for a field.
- ValueFromPrevious
Get the previous value
- Sequence
Generates a sequential number for records in a group.
- Route
Identifies the route or path of values of a field within a group or sub-group.
- Concate
Concatenates values of a field within a group or sub-group.
- SimpleCalculation
Performs a simple arithmetic calculation using field values in the same record.
- Pareto
Calculates Pareto (80/20) distribution values, commonly used in business analytics.
- Custom
Allows user-defined buffer transformation logic function.
Emit Modes (Output Behavior)
Buffer Transformation supports three emit strategies:
- EMIT FOR EVERY RECORD
- Output is emitted for each input record
- Suitable for:
- Running totals
- Sequence numbers
- ValueFromPrevious
- EMIT ON CHANGE
- Output is emitted only when a field value (or combination of fields) changes
- Suitable for:
- Change detection
- State transition analysis
- Slowly changing data
- EMIT FOR EVERY GROUP
- Output is emitted once per group
- Suitable for:
- Group summaries
- Final transformation results
Buffer Transformation vs Aggregator
| Aspect | Aggregator | Buffer Transformation |
|---|
| Input | Grouped data | Grouped data |
| Output | One row per group | Flexible |
| Access to full group | ❌ Limited | ✅ Full |
| Running calculations | ❌ | ✅ |
| Ranking & sequencing | ❌ | ✅ |
| Memory usage | Low | Higher |
Performance Considerations
- Buffer transformations are memory-intensive
- Large groups can increase memory usage
- Always:
- Use proper partitioning
- Sort data only when needed
- Avoid unnecessary buffer logic
Typical Use Cases
- Ranking and leaderboard calculation
- Running totals and deltas
- Sequence numbering
- Pareto analysis
- Change detection
- Slowly change data
A short tutorial on
Using Buffer Transformation
Supported Engines
Map
Reduce |
Spark 1.x
RDD |
Spark 2.x
RDD |
Spark 2.x
Dataset |
Apache Beam
2.29.0 Batch |
Apache Beam
2.29.0 Streaming |
Apache Flink
1.12.0 Batch |
Apache Flink
1.12.0 Streaming |
Apache Tez
0.9.1 |
PySpark |
 |
|
|
|
|
|
|
|
|
|
