Number of input and output link

Number of input link	:	Two
Number of output link	:	One

Reference Join Assembly () (Map-Side Join) performs a map-side join by joining two datasets based on a common join key using an in-memory hash reference.

When one dataset is small enough to fit into memory, it is loaded into each mapper (or partition). Records from the larger dataset are then joined directly with the in-memory dataset during the map phase.

Because the join is executed entirely on the map side, no reduce phase or shuffle process is required, making this join significantly faster than a regular join.

Key Characteristics

• The right-side dataset is treated as the reference dataset
• The reference dataset is loaded into memory for each mapper/partition
• The left-side dataset is streamed and joined using hash lookup
• No shuffle phase
• High performance, but requires careful dataset sizing

Supported Join Methods

Reference Join Assembly supports two (2) join types:

• Inner Join
• Left Join

How It Works

• The right-side (reference) dataset is read entirely and stored in memory as a hash table.
• Each mapper processes a partition of the left-side dataset.
• For every record in the left-side dataset, a hash lookup is performed against the in-memory reference data.
• Matching records are emitted immediately as output.

Supported Engines

Map Reduce	Spark 1.x RDD	Spark 2.x RDD	Spark 2.x Dataset	Apache Beam 2.29.0 Batch	Apache Beam 2.29.0 Streaming	Apache Flink 1.12.0 Batch	Apache Flink 1.12.0 Streaming	Apache Tez 0.9.1	PySpark