Avoiding shuffle

1. Joining two sources data using Join Assembly

On the flow above, shuffled is needed to send records with the same key to the same partition/machine.

2. Joining two sources data using Reference-Join Assembly

In the reference-join right-side data is broadcasted to all partitions, so that each partition can lookup a record that match to left site join. Thus no shuffle process is needed.

3. Joining two sources data using Reference-Lookup Transformation-Function

In the reference-lookup, right-side data is treated as distributedcache and loaded to all partitions, so that each partition can lookup a record in the distributedcashe that match to left site join. Thus no shuffle process is needed.