Window Partition

🔷 Window Partition (Windowing / Partition-Level Aggregation)

📌 What is Window Partition?

Window Partition is a function that performs aggregation calculations on each partition , then the results are assigned to each row in that partition.

➡️ Similar to the WINDOW function in SQL:

    AVG(sales) OVER (PARTITION BY dealer_id)

It means:

Data is divided per group (partition)
Aggregation is done per group
The aggregate value is assigned to all rows in the group.

⚙️ Parameter

Format:

    Field,Operations,Values Separator

1️⃣ Field

The number field to be calculated.
If you want more than one → separate them with :
Example:

    sales:profit

2️⃣ Operations

The aggregation operations to be computed on each partition.

Supported operations:

    Operation       Function
    --------------------------
    Average         Rate-rate
    Sum             Total
    Count           Number of rows
    Min             Minimum value
    Max             Maximum value
    Median          Median
    StdDev          Sample standard deviation
    StdDevPop       Population standard deviation

Can multi-operate → separate with :
Example:

    average:sum:count

3️⃣ Values Separator

Separator for output values.

Default: , Can be changed, for example:

📤 Input & Output Field

📥 Input:

All previous fields

📤 Output:

All input fields
All grouped-fields with postfix_new
A new field named value contains the result of the window operation

📘 Sample Dataset

    emp_name            dealer_id   sales
    -------------------------------------
    Kameko French       2           16233
    Ursa George         3           15427
    Ferris Brown        1           19745
    Noel Meyer          1           19745
    Abel Kim            3           12369
    Raphael Hull        1           8227
    Jack Salazar        1           9710
    May Stout           3           9308
    Haviva Montoya      2           9308

🧪 Example 1 — Parameters:

Group Field :

    dealer_id

Buffer Parameter :

   sales,average

Output :

    emp_name_new        dealer_id_new   sales_new       value
    -----------------------------------------------------------
    Ferris Brown 	1               19745.0         14356.75
    Noel Meyer   	1               19745.0         14356.75
    Raphael Hull 	1               8227.0          14356.75
    Jack Salazar 	1               9710.0          14356.75
    Kameko French	2               16233.0         12770.5
    Haviva Montoya	2               9308.0          12770.5
    Ursa George  	3               15427.0         12368.0
    Abel Kim     	3               12369.0         12368.0
    May Stout    	3               9308.0          12368.0

🧪 Example 2 — Parameters:

Group Field :

    dealer_id

Buffer Parameter :

   sales,average:sum

Output :

    emp_name_new        dealer_id_new   sales_new       value
    --------------------------------------------------------------------
    Ferris Brown 	1               19745.0         14356.75,57427.0
    Noel Meyer   	1               19745.0         14356.75,57427.0
    Raphael Hull 	1               8227.0          14356.75,57427.0
    Jack Salazar 	1               9710.0          14356.75,57427.0
    Kameko French	2               16233.0         12770.5,25541.0
    Haviva Montoya	2               9308.0          12770.5,25541.0
    Ursa George  	3               15427.0         12368.0,37104.0
    Abel Kim     	3               12369.0         12368.0,37104.0
    May Stout    	3               9308.0          12368.0,37104.0

🚀 When is Window Partition used?

Very useful for:

✔ Segmentation per group
✔ Comparing row values with group averages
✔ Calculating percentage contributions
✔ Feature engineering in data mining / ML
✔ Statistics per category

Below is a sample workflow demonstrating the WindowPartition Process:

Flow of Data Processing :

Hfs_source_1 :
HFS Source Hfs_source_1 reads flat file HDFS with properties :

Input Path	:	Arg[0]
File Type	:	Hfs
Combine Input	:	false
Fields in Lower Case	:	false
Delimiter	:

Transformator_1 :
Transformator Transformator_1 transforms the output of 'Hfs_source_1' using the following transfromation function(s)

Input Field	Operation	Output Field	Type
line	RegexSplitter(Splitter=\|)	emp_name dealer_id sales	String String String

GroupBy_1 :
Group By GroupBy_1 groups the output of 'Transformator_1'

Grouped Field(s) :

dealer_id

Buffer_1 :
Buffer Buffer_1 processes the output of 'GroupBy_1'

Using buffer operation pocesss :

WindowPartition

With paramter :

sales,average:sum

Input Field	Buffer Operation	Output Field	Type
emp_name dealer_id sales	WindowPartition(sales,average:sum)	emp_name dealer_id sales emp_name_new dealer_id_new sales_new VALUE	String Integer Double String Integer Double String

Transformator_2 :
Transformator Transformator_2 transforms the output of 'Buffer_1' using the following transfromation function(s)

Input Field	Operation	Output Field	Type
emp_name_new	No Operation	emp_name_new	String
dealer_id_new	No Operation	dealer_id_new	Integer
sales_new	No Operation	sales_new	Double
VALUE	No Operation	VALUE	String

Hfs_sink_1 :
HFS Sink Hfs_sink_1 writes HDFS the output of 'Transformator_2' with properties :

Fields Name	:	emp_name_new,dealer_id_new,sales_new,VALUE
Output Path	:	Arg[1]
Output Name	:	hgrid247-#####
Sink Mode	:	SinkMode.REPLACE
Partition Field(s)	:
Splitter	:
Index	: