Number of input and output link

Number of input link	:	One
Number of output link	:	One

Bloom Filter Member Check Assembly () is used to identify whether a record has already appeared within a partition by using a Bloom Filter–based membership test. It appends a membership flag to each input record, indicating whether the record is new or already a member of the collection.

This assembly is optimized for high-performance duplicate detection with very low memory usage.

Functionality

For each record processed in a partition:

• Flag = 0 → the record is not yet a member of the Bloom Filter (first occurrence)
• Flag = 1 → the record is already a member of the Bloom Filter (possible duplicate)

The Bloom Filter is updated as records are processed.

How It Works

• Each partition maintains its own Bloom Filter
• A hash is generated from the selected key field(s)
• The Bloom Filter is queried for membership
• If the key is not present, the record is marked as 0 and added to the filter
• If the key appears to be present, the record is marked as 1
• The record is emitted together with the membership flag

Input and Output Structure

Input Record

[field1, field2, field3, ...]

Output Record

[field1, field2, field3, ..., bloom_member_flag]

Where:

• bloom_member_flag = 0 → not previously seen
• bloom_member_flag = 1 → already seen (probabilistic)

Example

Input Records (single partition)

TransactionID	User
T001            Alice
T002            Bob
T001            Alice
T003            John
T002            Bob

Output Records

TransactionID	User	Bloom Member Flag
T001            Alice	0
T002            Bob	0
T001            Alice	1
T003            John	0
T002            Bob	1

Important Characteristics

• Operates locally within each partition
• No shuffle or global aggregation
• Extremely memory-efficient
• Designed for large-scale streaming or batch data
• May produce false positives, but never false negatives

Bloom Filter Properties

Property	Description
False Positive	Possible (flag = 1 even if record is new)
False Negative	Impossible
Memory Usage	Fixed and very small
Performance	O(1) per record

Common Use Cases

• Fast duplicate detection in massive datasets
• Pre-filtering before expensive joins
• Streaming data de-duplication
• Data quality validation at scale

Limitations

• Duplicate detection is approximate
• Results are partition-scoped
• Not suitable when exact duplicate detection is required

Comparison with Duplicate Check Assembly

Feature             Bloom Filter Member Check	Duplicate Check
Accuracy            Probabilistic               Exact
Memory Usage        Very low                    Higher
Speed               Very fast                   Fast
False Positives     Yes                         No
False Negatives     No                          No

Summary

The Bloom Filter Member Check Assembly efficiently identifies whether a record has already appeared within a partition by using a probabilistic Bloom Filter. It is ideal for high-volume data pipelines where memory efficiency and speed are more important than perfect accuracy.

Supported Engines

Map Reduce	Spark 1.x RDD	Spark 2.x RDD	Spark 2.x Dataset	Apache Beam 2.29.0 Batch	Apache Beam 2.29.0 Streaming	Apache Flink 1.12.0 Batch	Apache Flink 1.12.0 Streaming	Apache Tez 0.9.1	PySpark