Duplicate Check Assembly (
) is used to detect duplicate records within a partition and append a duplicate indicator flag to each record.
This assembly does not remove records. Instead, it marks whether a record is a duplicate or not, allowing downstream processes to decide how to handle duplicates.
Functionality
For each record in a partition:
- 0 → The record is not a duplicate (first occurrence)
- 1 → The record is a duplicate (repeated occurrence)
Duplicate detection is performed locally within each partition, based on the configured key fields.
How It Works
- Records are processed partition by partition
- A key is generated from the selected field(s)
- The first occurrence of a key is marked as non-duplicate (0)
- Any subsequent record with the same key is marked as duplicate (1)
- The original record is emitted together with the duplicate flag
Input and Output Structure
Input Record
[field1, field2, field3, ...]
Output Record
[field1, field2, field3, ..., duplicate_flag]
Where:
- duplicate_flag = 0 → unique record
- duplicate_flag = 1 → duplicate record
Example
Input Records (within one partition)
ID Name Amount
1 John 100
2 Alice 200
1 John 100
3 Bob 150
2 Alice 200
Output Records
ID Name Amount Duplicate Flag
1 John 100 0
2 Alice 200 0
1 John 100 1
3 Bob 150 0
2 Alice 200 1
Important Characteristics
- Duplicate detection is partition-local
- No shuffle operation is performed
- Fast and memory-efficient
- Suitable for pre-cleaning or data quality checks
Common Use Cases
- Identifying duplicate transactions
- Flagging duplicate log records
- Preparing data before deduplication or aggregation
- Data quality validation pipelines
Limitations
- Does not detect duplicates across partitions
- To detect global duplicates, a Group By or shuffle-based assembly must be used
Summary
The Duplicate Check Assembly identifies duplicate records within each partition and appends a flag indicating duplication status. It is a lightweight and efficient way to mark duplicates without modifying or removing any records, enabling flexible downstream processing.
Supported Engines
Map
Reduce |
Spark 1.x
RDD |
Spark 2.x
RDD |
Spark 2.x
Dataset |
Apache Beam
2.29.0 Batch |
Apache Beam
2.29.0 Streaming |
Apache Flink
1.12.0 Batch |
Apache Flink
1.12.0 Streaming |
Apache Tez
0.9.1 |
PySpark |
 |
|
|
|
|
|
|
|
|
 |
