Using Buffer Transformation

Buffer Transformation is used to process data that already grouped and sorted.

Suppose we have a part of csv-data like below :


    62yyxxxxxxxxx,41126,7427,51001,msc,20151206035741
    62yyxxxxxxxxx,1126,42313,51001,ggsn,20151206080941
    62yyxxxxxxxxx,1126,42313,51001,ggsn,20151206135355
    62yyxxxxxxxxx,1118,72963,51001,ggsn,20151207000101
    62yyxxxxxxxxx,1118,73474,51001,ggsn,20151207085428
    62yyxxxxxxxxx,1118,73730,51001,ggsn,20151207085805
    62yyxxxxxxxxx,1126,42313,51001,mSTerminating,20151206135410
    62yyxxxxxxxxx,1118,73731,51001,ggsn,20151207090057
    62yyxxxxxxxxx,1118,65795,51001,ggsn,20151207090416
    62yyxxxxxxxxx,1126,42313,51001,ggsn,20151206135418
    62yyxxxxxxxxx,1119,18025,51001,ggsn,20151207101242
    62yyxxxxxxxxx,1119,18025,51001,mSTerminating,20151207101259
    62yyxxxxxxxxx,1126,42322,51001,ggsn,20151206100440
    62yyxxxxxxxxx,1119,18025,51001,ggsn,20151207101311
    62yyxxxxxxxxx,1118,3998981,51001,ggsn,20151207095648
    62yyxxxxxxxxx,1118,3998978,51001,ggsn,20151207101327

Above csv-data has 6 fields, recall the field's nameS are :MSISDN, LAC, CI, MCCMNC, DIRECTION and EVENT_DATE

The purpose of this guidance is to find EVENT_DATE a msisdn enter a LAC-CI and EVENT_DATE a msisdn left a LAC-CI.


Steps creating Workflow

Step 1 : Create a Workflow

Create a workflow named TestBufferTrans.
Drag drag and drop Hfs (), from Assembly Pallete drag and drop Transformator () into Workflow canvas.
Draw link from Hfs to Transformator, so that like figure below:




Fill the Transformator property so that like figure bellow :




Step 2 : Group Data by MSISDN and sort by EVENT_DATA.

Drag GroupBy Assembly() and drop on canvas.
Draw link from transformator to GroupBy, so that like figure bellow :




Fill the GroupBy property so that like figure bellow :




The expected outcome after GroupBy process will be :

    62yyxxxxxxxxx	41126	7427	51001	msc	20151206035741
    62yyxxxxxxxxx	1126	42313	51001	ggsn	20151206080941
    62yyxxxxxxxxx	1126	42322	51001	ggsn	20151206100440
    62yyxxxxxxxxx	1126	42313	51001	ggsn	20151206135355
    62yyxxxxxxxxx	1126	42313	51001	mSTerminating	20151206135410
    62yyxxxxxxxxx	1126	42313	51001	ggsn	20151206135418
    62yyxxxxxxxxx	1118	72963	51001	ggsn	20151207000101
    62yyxxxxxxxxx	1118	73474	51001	ggsn	20151207085428
    62yyxxxxxxxxx	1118	73730	51001	ggsn	20151207085805
    62yyxxxxxxxxx	1118	73731	51001	ggsn	20151207090057
    62yyxxxxxxxxx	1118	65795	51001	ggsn	20151207090416
    62yyxxxxxxxxx	1118	3998981	51001	ggsn	20151207095648
    62yyxxxxxxxxx	1119	18025	51001	ggsn	20151207101242
    62yyxxxxxxxxx	1119	18025	51001	mSTerminating	20151207101259
    62yyxxxxxxxxx	1119	18025	51001	ggsn	20151207101311
    62yyxxxxxxxxx	1118	3998978	51001	ggsn	20151207101327


Step 3 : Create Buffer Transformation

Drag Buffer Transformator Assembly() and drop on canvas.
Draw link from GroupBy to Buffer Transformator, so that like figure bellow :






Fill the Buffer Transformator property so that like figure bellow :




Step 4 : Select the interest fields

Drag Transformator Assembly() and drop on canvas.
Draw link from Buffer-Transformator to Transformator, so that like figure bellow :




Fill the Transformator property so that like figure bellow :




Step 5 : Complete the Workflow

Drag Hfs () from Sink Palette and drop into canvas. Create link from transformator_2 to Hfs_sink_1.



Step 6 : Compile and Generate TestBuffTrans Workflow to map-reduce application

Click button generate () on tool bar.

If we run this job TestBuffTrans, the result will be :

    62yyxxxxxxxxx	41126	7427	51001	msc	20151206035741	20151206080941
    62yyxxxxxxxxx	1126	42313	51001	ggsn	20151206080941	20151206100440
    62yyxxxxxxxxx	1126	42322	51001	ggsn	20151206100440	20151206135355
    62yyxxxxxxxxx	1126	42313	51001	ggsn	20151206135355	20151207000101
    62yyxxxxxxxxx	1118	72963	51001	ggsn	20151207000101	20151207085428
    62yyxxxxxxxxx	1118	73474	51001	ggsn	20151207085428	20151207085805
    62yyxxxxxxxxx	1118	73730	51001	ggsn	20151207085805	20151207090057
    62yyxxxxxxxxx	1118	73731	51001	ggsn	20151207090057	20151207090416
    62yyxxxxxxxxx	1118	65795	51001	ggsn	20151207090416	20151207095648
    62yyxxxxxxxxx	1118	3998981	51001	ggsn	20151207095648	20151207101242
    62yyxxxxxxxxx	1119	18025	51001	ggsn	20151207101242	20151207101327
    62yyxxxxxxxxx	1118	3998978	51001	ggsn	20151207101327