Number of input and output link

Number of input link	:	One
Number of output link	:	One

Filter Assembly () is used to include or exclude records based on a defined filter expression.
Only records that satisfy the filter condition are passed to the next assembly in the flow.

Purpose

• Control data flow by filtering records
• Remove invalid, unwanted, or irrelevant data
• Apply conditional logic before transformations, joins, or aggregations

Filter Expression Structure

A Filter Expression is composed of the following elements:

• Open bracket and close bracket
  Used to group expressions and define evaluation order
• Negation Operation (NOT)
  Reverses the result of a condition
• Field
  The field whose value will be evaluated
• Operation (Comparison Operator)
  Defines how the field value is evaluated
• Value
  The value used for comparison (if required)
• Logic
  Boolean logic to combine multiple conditions

Supported Comparison Operations

• Equality and Comparison
  • Equal
    Checks if the field value is equal to the specified value
    Value required
  • Not Equal
    Checks if the field value is not equal to the specified value
    Value required
  • Greater Than ( > )
    Checks if the field value is greater than the specified value
    Value required
  • Smaller Than ( < )
    Checks if the field value is smaller than the specified value
    Value required
  • Greater or Equal ( >= )
    Checks if the field value is greater than or equal to the specified value
    Value required
  • Smaller or Equal ( <= )
    Checks if the field value is smaller than or equal to the specified value
    Value required
• Null and Type Checks
  • Is Null
    Checks whether the field value is null or empty
    No value required
  • Is Not Null
    Checks whether the field value is not null or empty
    No value required
  • Is Number
    Checks whether the field value is numeric
    No value required
  • Is Not Number
    Checks whether the field value is not numeric
    No value required
• String Operations
  • Start With
    Checks if the field value starts with a specified string
    Value required
  • End With
    Checks if the field value ends with a specified string
    Value required
  • Like
    Pattern-based string matching
    Value required
  • Match Pattern
    Matches the field value against a regular expression
    Value required (regex)
• List and Date Operations
  • In List
    Checks whether the field value exists within a specified list
    List value required
  • Is Date
    Validates whether the field value matches a given date format
    Date format value required
• Logical Operators
  Multiple filter conditions can be combined using:
  • AND
    All conditions must be true
  • OR
    At least one condition must be true
  • Brackets can be used to control evaluation precedence.

Behavior

• Each record is evaluated independently
• Records that satisfy the filter expression are passed through
• Records that do not satisfy the expression are discarded
• Supports complex nested logical expressions

Typical Use Cases

• Removing null or invalid records
• Filtering data by thresholds or ranges
• Applying business rules
• Pre-filtering data before joins or aggregations
• Data cleansing and validation

Performance Notes

• Filtering is typically lightweight and efficient
• Complex expressions with regex or multiple conditions may increase processing cost
• Early filtering reduces data volume for downstream assemblies, improving overall performance

Supported Engines

Map Reduce	Spark 1.x RDD	Spark 2.x RDD	Spark 2.x Dataset	Apache Beam 2.29.0 Batch	Apache Beam 2.29.0 Streaming	Apache Flink 1.12.0 Batch	Apache Flink 1.12.0 Streaming	Apache Tez 0.9.1	PySpark