Number of input and output link

Number of input link	:	One
Number of output link	:	One or Two

Expression Filter Assembly () is used to include or exclude records from a data stream based on a logical expression condition.

If a record matches the condition, it passes through the assembly. If it does not match, the record is filtered out or is sent to no-match branch (if exist).

This component is commonly used to select relevant data before further processing.

Output Branches

• Success / Match Branch
  Contains records that match the filter expression
• Failure / No-Match Branch
  Contains records that do not match the filter expression
  (No data is discarded)

1. Basic Concept

The Expression Filter evaluates conditions based on fields in the data stream.

Example record:

    id,name,city,price
    1,Arif,Jakarta,2500
    2,John,Bandung,1200
    3,Siti,Jakarta,3000

Example filter condition:

    city=Jakarta AND price>2000

Result:

1,Arif,Jakarta,2500
3,Siti,Jakarta,3000

Record 2 is removed or sent to no-match branch because it does not meet the condition.

2. Valid Comparison Operators

    No	Operator                Description                 Example
    ---------------------------------------------------------------------------
    1	=                       Equal                       city=Jakarta
    2	!=                      Not equal                   name!=John
    3	<                       Less than                   salary<3000
    4	<=                      Less than or equal          salary<=300
    5	>                       Greater than                price>1500
    6	>=                      Greater than or equal       price>=1500
    7	Is Not Null             Field has value             department Is Not Null
    8	Is Null                 Field has no value          hired_date Is Null
    9	In List                 Value exists in list        city In List Jakarta,Hanoi,Kuala Lumpur
    10	Starts With             String begins with          city Starts With Jak
    11	Ends With               String ends with            phone Ends With 0122
    12	Like                    String similarity           name Like Arif
    13	Matchs Pattern          Regex matching              name Matchs Pattern .+\s+.+
    14	Equals Ignore Case	Case-insensitive equal      name Equals Ignore Case john
    15	Is Number               Field is numeric            id Is Number
    16	Is Not Number           Field not numeric           id Is Not Number
    17	Is Letter               Field contains letters      id Is Letter
    18	Is Date                 Field is date format        hired_date Is Date yyyy/MM/dd

3. Logical Operators

Expression Filter also supports logical operators.

    Operator        Meaning
    ----------------------------------
    NOT             Negation
    AND             Both conditions must be true
    OR              At least one condition must be true

Example:

    city=Jakarta AND price>2000

Example with OR:

    city=Jakarta OR city=Bandung

Example with NOT:

    NOT city=Jakarta

4. Priority Evaluation Using Brackets

Parentheses determine priority of evaluation.

Example:

        (city=Jakarta OR price>2500) AND item Ends With 4

Evaluation order:

• (city=Jakarta OR price>2500)
• AND item Ends With 4

5. Comparison Rules

1. Field Compared With Constant Value
   Format:

       field_name operator constant_value
   

   Examples:

       price>30
       name Equals Ignore Case John
       city=Jakarta
   

2. Field Compared With Another Field
   Format:

       field_name operator [other_field_name]
   

   Examples:

   price>[average]
   name Equals Ignore Case [supervisor]
   

   Here, the value of another column is used.

6. Example Filter Expression

Example condition:

    (city=Alberta OR price>2.5) AND item Ends With 4

Meaning:
A record will pass if:

    city = Alberta OR price > 2.5
    AND
    item ends with 4

7. Using FROM_ARGUMENT

FROM_ARGUMENT() allows filter values to come from runtime parameters.

Example:

    (city=FROM_ARGUMENT(city_name) OR price>2.5) AND item Ends With 4

Meaning:

The city value is taken from program arguments.

Example runtime value:

    city_name = Jakarta

Final evaluation becomes:

     (city=Jakarta OR price>2.5) AND item Ends With 4

Another example:

        FROM_ARGUMENT(filter_expression)

Here the entire filter expression is supplied externally.

8. Using FROM_CONFIG

FROM_CONFIG() retrieves values from a workflow configuration file, where the file name is :

       <packages-name>.<workflow_name>.conf

Example:

    (city=FROM_CONFIG(city_name) OR price>2.5) AND item Ends With 4

Example configuration file content:

    city_name=Jakarta

Another usage:

    FROM_CONFIG(filter_expression)

This loads the entire filter expression from configuration.

9. Typical Use Cases

Expression Filter is commonly used for:

• Fraud detection
• Data cleaning
• Stream filtering
• Event selection in CEP
• Pre-processing before aggregation

Supported Engines

Map Reduce	Spark 1.x RDD	Spark 2.x RDD	Spark 2.x Dataset	Apache Beam 2.29.0 Batch	Apache Beam 2.29.0 Streaming	Apache Flink 1.12.0 Batch	Apache Flink 1.12.0 Streaming	Apache Tez 0.9.1	PySpark