Number of input and output link

Number of input link	:	One
Number of output link	:	Two

Split Filter Assembly () is used to evaluate records using a filter expression and split the data into two output branches:

1. Matched Output – records that satisfy the filter condition
2. Unmatched Output – records that do not satisfy the filter condition

This assembly is useful when both valid and invalid (or accepted and rejected) records need to be processed separately.

Purpose

• Apply conditional filtering while preserving rejected records
• Enable parallel handling of matched and unmatched data
• Support validation, exception handling, and auditing workflows

Output Branches

• Success / Match Branch
  Contains records that match the filter expression
• Failure / No-Match Branch
  Contains records that do not match the filter expression
  (No data is discarded)

Filter Expression Structure

A Filter Expression is composed of the following elements:

• Open bracket and close bracket
  Used to group expressions and define evaluation order
• Negation Operation (NOT)
  Reverses the result of a condition
• Field
  The field whose value will be evaluated
• Operation (Comparison Operator)
  Defines how the field value is evaluated
• Value
  The value used for comparison (if required)
• Logic
  Boolean logic to combine multiple conditions

Supported Comparison Operations

• Equality and Comparison
  • Equal
    Checks if the field value is equal to the specified value
    Value required
  • Not Equal
    Checks if the field value is not equal to the specified value
    Value required
  • Greater Than ( > )
    Checks if the field value is greater than the specified value
    Value required
  • Smaller Than ( < )
    Checks if the field value is smaller than the specified value
    Value required
  • Greater or Equal ( >= )
    Checks if the field value is greater than or equal to the specified value
    Value required
  • Smaller or Equal ( <= )
    Checks if the field value is smaller than or equal to the specified value
    Value required
• Null and Type Checks
  • Is Null
    Checks whether the field value is null or empty
    No value required
  • Is Not Null
    Checks whether the field value is not null or empty
    No value required
  • Is Number
    Checks whether the field value is numeric
    No value required
  • Is Not Number
    Checks whether the field value is not numeric
    No value required
• String Operations
  • Start With
    Checks if the field value starts with a specified string
    Value required
  • End With
    Checks if the field value ends with a specified string
    Value required
  • Like
    Pattern-based string matching
    Value required
  • Match Pattern
    Matches the field value against a regular expression
    Value required (regex)
• List and Date Operations
  • In List
    Checks whether the field value exists within a specified list
    List value required
  • Is Date
    Validates whether the field value matches a given date format
    Date format value required
• Logical Operators
  Multiple filter conditions can be combined using:
  • AND
    All conditions must be true
  • OR
    At least one condition must be true
  • Brackets can be used to control evaluation precedence.

Behavior

• Each incoming record is evaluated independently
• Records matching the expression go to the Match branch
• Records not matching the expression go to the No-Match branch
• No records are discarded

Typical Use Cases

• Data validation with error handling
• Separating clean and dirty data
• Conditional routing in ETL pipelines
• Auditing and monitoring rejected records
• Implementing business rule branching

Performance Notes

• Similar performance characteristics to Filter Assembly
• Allows downstream processing without losing rejected data
• Complex expressions may impact evaluation speed

Supported Engines

Map Reduce	Spark 1.x RDD	Spark 2.x RDD	Spark 2.x Dataset	Apache Beam 2.29.0 Batch	Apache Beam 2.29.0 Streaming	Apache Flink 1.12.0 Batch	Apache Flink 1.12.0 Streaming	Apache Tez 0.9.1	PySpark