Number of input and output link

Number of input link	:	One
Number of output link	:	One

Global TopN Assembly () or (Local TopN , Group By and Buffer/Ranking in one assembly) is used to select the top N records across all partitions, producing a globally ranked Top N result rather than a partition-local one.

This assembly internally combines:

• Local Top N (partition-level filtering),
• Group By (to bring all candidates together),
• Buffer / Ranking (to compute the final global ranking).

How It Works

1. Local Top N phase
   • Each partition selects its own Top N records based on the defined sort field(s).
   • This step significantly reduces the amount of data that must be shuffled.
2. Shuffle & Group By phase
   • The locally selected Top N records from all partitions are shuffled and grouped.
   • All candidate records are brought together for global evaluation.
3. Buffer / Ranking phase
   • A buffer or ranking function is applied to sort records globally.
   • Only the final Top N records across the entire dataset are emitted.

Key Characteristics

• Produces globally correct Top N results
• Optimized for distributed systems by minimizing shuffle volume
• Internally performs local pruning before global aggregation

Example

Input Data (3 Partitions)
Partition 1

    ID	Score
    ---------
    A	95
    B	80
    C	70

Partition 2

    ID	Score
    ---------
    D	92
    E	85
    F	60

Partition 3

    ID	Score
    ---------
    G	98
    H	75
    I	65

Configuration

• Sort Field: Score (Descending)
• N = 3

Output (Global Top 3)

    Rank    ID      Score
    ----------------------
    1       G       98
    2       A       95
    3       D       92

Common Use Cases

• Top N customers by revenue
• Top N products by sales
• Highest scores, rankings, or metrics across large datasets
• Global leaderboards and performance dashboards

Performance Considerations

• More expensive than local Top N due to shuffle
• Much more efficient than doing a full global sort on all records
• Best practice: always prune locally first, then rank globally

Summary

The Global Top N Assembly efficiently computes the top N records across all partitions by combining local Top N filtering with a global Group By and Buffer/Ranking process. This approach ensures correctness while maintaining high performance in distributed data processing pipelines.

Supported Engines

Map Reduce	Spark 1.x RDD	Spark 2.x RDD	Spark 2.x Dataset	Apache Beam 2.29.0 Batch	Apache Beam 2.29.0 Streaming	Apache Flink 1.12.x Batch	Apache Flink 1.12.x Streaming	Apache Tez 0.9.1	PySpark