Number of input and output link

Number of input link	:	One
Number of output link	:	One

Hash Member Check Assembly () is used to detect whether a record has already appeared within a partition by using an exact hash-based membership check. It appends a membership flag to each input record to indicate whether the record is new or already exists in the collection.

Unlike Bloom Filter–based checks, this assembly provides exact results with no false positives or false negatives, at the cost of higher memory usage.

Functionality

For each input record processed within a partition:

• Flag = 0 → the record is not yet a member of the collection
• Flag = 1 → the record already exists in the collection

The membership information is maintained using an in-memory hash structure.

How It Works

• Each partition maintains a hash set / hash map
• A hash key is generated from the selected field(s)
• The hash structure is checked for key existence
• If the key does not exist:
  • The record is marked with flag 0
  • The key is inserted into the hash structure
• If the key already exists:
  • The record is marked with flag 1
• The record is emitted together with the membership flag

Input and Output Structure

Input Record

[field1, field2, field3, ...]

Output Record

[field1, field2, field3, ..., hash_member_flag]

Where:

• hash_member_flag = 0 → first occurrence
• hash_member_flag = 1 → duplicate record

Example

Input Records (single partition)

OrderID	Customer
O1001	Alice
O1002	Bob
O1001	Alice
O1003	John
O1002	Bob

Output Records

OrderID	Customer	Hash Member Flag
O1001	Alice           0
O1002	Bob             0
O1001	Alice           1
O1003	John            0
O1002	Bob             1

Key Characteristics

• Operates within each partition
• Uses exact hashing
• No shuffle phase
• Deterministic results
• Memory usage grows with the number of unique keys

Performance Characteristics

Aspect              Description
Accuracy            Exact (100%)
False Positives     No
False Negatives     No
Time Complexity     O(1) average per record
Memory Usage        Proportional to unique records

Common Use Cases

• Exact duplicate detection
• Data quality validation
• Enforcing uniqueness constraints
• Pre-processing before joins or aggregations
• Small to medium datasets where accuracy is critical

Limitations

• Memory usage increases with data size
• Partition-scoped only (not global)
• Not suitable for extremely large cardinality datasets without sufficient memory

Comparison with Bloom Filter Member Check Assembly

Feature             Hash Member Check	Bloom Filter Member Check
Accuracy            Exact               Probabilistic
False Positives     No                  Yes
False Negatives     No                  No
Memory Usage        Higher              Very Low
Scalability         Moderate            Very High

Summary

The Hash Member Check Assembly performs exact membership detection using hashing and adds a binary flag to each record indicating whether it has appeared before within a partition. It is ideal when accuracy is mandatory and the dataset size is manageable in memory.

Supported Engines

Map Reduce	Spark 1.x RDD	Spark 2.x RDD	Spark 2.x Dataset	Apache Beam 2.29.0 Batch	Apache Beam 2.29.0 Streaming	Apache Flink 1.12.0 Batch	Apache Flink 1.12.0 Streaming	Apache Tez 0.9.1	PySpark