NTileAggBucket

🔷 NTileAggBucket

🔍 Definition

NTileAggBucket is a combination of:

➡️ NTile (data bucketization)
➕ Aggregation per bucket

👉 So:

Data is divided into several buckets, then statistics are calculated for each bucket.

⚙️ Parameter

Format: 
    N,Field,Aggregation Operations,Values Separator

1. 🎯 N (Bucket)

Number of buckets
Example:

2. 📊 Field

Fields for which aggregates will be calculated 
    Example: sales, views,amount

3. 🔢 Aggregation Operations

Can be more than one, separated by: 
    Operation       Function
    -----------------------
    Max             maximum value
    Min             minimum value
    Sum             total value
    Average         average value
    Count           amount of data
Example: 
    Max:Min:Sum:Average:Count

4. 🔤 Values Separator

Aggregation output separator 
    Default: ,
Can be changed, for example: 
    |

📥 Input & 📤 Output

Input Output

🧠 How it works

  1. Data is partitioned (for example dealer_id)
  2. Data is sorted (for example sales)
  3. Divided into N buckets
  4. Each bucket is calculated aggregation

📊 Example

The following sample data contains three fields: emp_name, dealer_id, and sales.

    emp_name        dealer_id   sales
    ----------------------------------
    Kameko French   2       16233
    Ursa George     3       15427
    Ferris Brown    1       19745
    Noel Meyer      1       19745
    Abel Kim        3       12369
    Raphael Hull    1       8227
    Jack Salazar    1       9710
    May Stout       3       9308
    Haviva Montoya  2       9308
Group And Sort 
    Grouped Field       :dealer_id
    Sorted Field        :sales (ascending)
Processed with NTileAggBucket, partitioned by dealer_id, ordered by sales, and using parameter :


Using parameter 1:
   3,sales,Max
The output of NTileAggBucket Buffer (field dealer_id, bucket, values) :
    dealer_id   bucket  values
    ---------------------------
    1	1           9710.0
    1	2           19745.0
    1	3           19745.0
    2	1           9308.0
    2	2           16233.0
    3	1           9308.0
    3	2           12369.0
    3	3           15427.0

Using parameter 2:
   3,sales,Max:Min:Sum:Average:Count
The output of NTileAggBucket Buffer (field dealer_id, bucket, values) :
    dealer_id   bucket  values
    -----------------------------------------------------
    1           1	9710.0,8227.0,17937.0,8968.5,2
    1           2	19745.0,19745.0,19745.0,19745.0,1
    1           3	19745.0,19745.0,19745.0,19745.0,1
    2           1	9308.0,9308.0,9308.0,9308.0,1
    2           2	16233.0,16233.0,16233.0,16233.0,1
    3           1	9308.0,9308.0,9308.0,9308.0,1
    3           2	12369.0,12369.0,12369.0,12369.0,1
    3           3	15427.0,15427.0,15427.0,15427.0,1

Using parameter 3:
   3,sales,Max:Min:Sum:Average:Count,|
The output of NTileAggBucket Buffer (field dealer_id, bucket, values) :
    dealer_id   bucket  values
    -----------------------------------------------------
    1           1	9710.0|8227.0|17937.0|8968.5|2
    1           2	19745.0|19745.0|19745.0|19745.0|1
    1           3	19745.0|19745.0|19745.0|19745.0|1
    2           1	9308.0|9308.0|9308.0|9308.0|1
    2           2	16233.0|16233.0|16233.0|16233.0|1
    3           1	9308.0|9308.0|9308.0|9308.0|1
    3           2	12369.0|12369.0|12369.0|12369.0|1
    3           3	15427.0|15427.0|15427.0|15427.0|1

💡 Important Insights

1. Suitable for Data Segmentation

Example: 
    Bucket	Meaning
    --------------------
    1           Low
    2           Medium
    3           High

2. Deeper Analysis

You can get it right away:

⚠️ Important Things

❗ Data must be sorted
❗ Use numeric fields
❗ Pay attention to the order of aggregation output

Below is a sample workflow demonstrating the NTileAggBucket Process :


Flow of Data Processing :

Hfs_source_1 :
HFS Source Hfs_source_1 reads flat file HDFS with properties :
Input Path : Arg[0]
File Type : Hfs
Combine Input : false
Fields in Lower Case : false
Delimiter :


Transformator_1 :
Transformator Transformator_1 transforms the output of 'Hfs_source_1' using the following transfromation function(s)
Input Field Operation Output Field Type
line 
RegexSplitter(Splitter=|)
 emp_name
dealer_id
sales 		String
String
String


GroupBy_1 :
Group By GroupBy_1 groups the output of 'Transformator_1'

Grouped Field(s) :

  dealer_id

Sorted Field(s) :

  sales  Ascending

Buffer_2 :
Buffer Buffer_2 processes the output of 'GroupBy_1'

Using buffer operation pocesss :

  NTileAggBucket

With paramter :

  3,sales,Max:Min:Sum:Average:Count,|

Input Field Buffer Operation Output Field Type
emp_name
dealer_id
sales 		NTileAggBucket(3,sales,Max:Min:Sum:Average:Count,|) emp_name
dealer_id
sales
BUCKET
Value 		String
Integer
Double
Integer
String


Transformator_3 :
Transformator Transformator_3 transforms the output of 'Buffer_2' using the following transfromation function(s)
Input Field Operation Output Field Type
dealer_id No Operation dealer_id Integer
BUCKET No Operation BUCKET String
Value No Operation Value String


Hfs_sink_1 :
HFS Sink Hfs_sink_1 writes HDFS the output of 'Transformator_3' with properties :
Fields Name : dealer_id, BUCKET,Value
Output Path : Arg[1]
Output Name : hgrid247-#####
Sink Mode : SinkMode.REPLACE
Partition Field(s) :
Splitter :
Index :