Number of input and output link

Number of input link	:	One
Number of output link	:	One

Profiling Assembly () is used to generate statistical profiling information from the first input field of incoming records.
This assembly is typically used for data exploration, data quality analysis, and schema understanding.

The first input field is treated as a delimited text column, which can be split into multiple logical columns and profiled individually.

How It Works

• The first field of each record is read as a string
• The field is split into columns using a specified delimiter
• Selected columns are analyzed based on their data types
• Profiling aggregations are applied
• Profiling results are emitted in a structured format
• A Combiner is applied when the aggregations are associative and commutative operations, such as Min, Max, Sum, and Count

Parameters of Profiling assembly

• Number of Columns
  Indicates the total number of columns after splitting the first field
• Column Delimiter
  Specifies the delimiter used to split the first field
  Examples:
  • , (comma)
  • | (pipe)
  • \t (tab)
  • ; (semicolon)
• Excluded Column Index
  Specifies column index(es) to exclude from profiling
  Indexing is 1-based
  Example:
  • 3 → exclude column 3
  • 6-9 → exclude columns 6,7,8 and 9
  • 2,5 → exclude columns 2 and 5
• Group By Column Index
  Specifies a column index used to group profiling results
  Useful for profiling data per category, region, or type
• Column Type
  Defines how each column is interpreted.
  Column Type Attributes
  • Index
    • Column index (1-based)
    • Can be a single index or range
    • Example: 1, 2, 1-5
  • Type
    • Data type of the column
    • Default type: double
    • Other examples:
      • String
      • Integer
      • Long
      • Double
• Aggregations
  Specifies which statistical aggregations are calculated for profiling
  Common Aggregations
  • Count
  • Min
  • Max
  • Sum
  • Average
  • Standard Deviation
  • Distinct Count

Conditional Aggregation

The Profiling Assembly also allows you to apply aggregation only when certain conditions are satisfied.

This means aggregation results can be calculated based on specific filtering conditions.

Example conditions:

• Only values greater than 0
• Only records from a specific group
• Only numeric values

This allows more flexible and targeted profiling analysis.

Output Format

Each output row follows this format:

• Without Group

      F<field-index<> | agg1 | agg2 | ... | aggN
  

  Example

      f1|1000|0.5|98.7
      f2|1000|1.2|120.3
  

  Where:
  • F<field-index> (f1,f2) → column index being profiled
  • agg1...aggN → aggregation results
• With Group

      group_field_value|F<field-index<> | agg1 | agg2 | ... | aggN
  

  Example

      group_A|F1|1000|0.5|98.7
      group_B|F2|1000|1.2|120.3
      group_A|F1|100|0.8|98.7
      group_B|F2|100|1.5|120.3
  

  Where:
  • group-value (group_A,group_B) → Group Field value
  • F<field-index> (F1,F2) → column index being profiled
  • agg1...aggN → aggregation results

Example Use Case

Input Record

1001,Jakarta,4500.75,2024-01-01
1002,Jakarta,4505.00,2024-01-02
1001,Bandung,4506.75,2024-01-03
1002,Jakarta,4510.00,2024-01-04

1. Case 1
   Parameters
   • Column Delimiter: ,
   • Number of Columns: 4
   • Excluded Column Index: 2,4
   • Column Types:
     • Column 1: Double
     • Column 3: Double
   • Aggregations:
     • Count, Min, Max, Average
   Output

       f1|4|1001.0|1002.0|1001.5
       f3|4|4500.75|4510.0|4505.625
   

2. Case 2
   Parameters
   • Column Delimiter: ,
   • Number of Columns: 4
   • Excluded Column Index: 2,4
   • Group By Column Index: 2
   • Column Types:
     • Column 1: Double
     • Column 3: Double
   • Aggregations:
     • Count, Min, Max, Average
   Output

       Bandung|f1|1|1001.0|1001.0|1001.0            
       Bandung|f3|1|4506.75|4506.75|4506.75         
       Jakarta|f1|3|1001.0|1002.0|1001.6666666666666
       Jakarta|f3|3|4500.75|4510.0|4505.25          
   

3. Case 3
   Parameters
   • Column Delimiter: ,
   • Number of Columns: 4
   • Excluded Column Index: 2,4
   • Group By Column Index: 2
   • Column Types:
     • Column 1: Integer
     • Column 3: Double
   • Aggregations:
     • Count, Min, Max, Average
   Output

       Bandung|f1|1|1001|1001|1001         
       Bandung|f3|1|4506.75|4506.75|4506.75
       Jakarta|f1|3|1001|1002|1001         
       Jakarta|f3|3|4500.75|4510.0|4505.25      
   

Key Characteristics

    Aspect              Description
    -----------------------------------
    Input Scope         First input field only
    Indexing            1-based
    Shuffle Phase       Yes
    Data Modification   No
    Purpose             Profiling & data quality analysis

Common Use Cases

• Dataset profiling before transformation
• Detecting outliers and anomalies
• Data quality assessment
• Schema inference
• Pre-processing validation in ETL pipelines

Summary

The Profiling Assembly provides a structured and configurable way to analyze and summarize data statistically by splitting and profiling the first input field.
It is especially useful during early pipeline design, data validation, and analytics preparation.

Supported Engines

Map Reduce	Spark 1.x RDD	Spark 2.x RDD	Spark 2.x Dataset	Apache Beam 2.29.0 Batch	Apache Beam 2.29.0 Streaming	Apache Flink 1.12.0 Batch	Apache Flink 1.12.0 Streaming	Apache Tez 0.9.1	PySpark