Sample of Flow Documentation

Documentation of Lab03

Description :

Join between youtube_video and category

Job Configuration :
Configuration Name Configuratition Value
mapreduce.job.user.classpath.first true
mapreduce.map.memory.mb 2000

Flow of Data Processing :

input video :
HFS Source input video reads flat file HDFS with properties :
Input Path :
File Type : Hfs
Combine Input : false
Fields in Lower Case : false
Delimiter :


input category :
HFS Source input category reads flat file HDFS with properties :
Input Path :
File Type : Hfs
Combine Input : false
Fields in Lower Case : false
Delimiter :


Transformator_1 :
Transformator Transformator_1 tranforms the output of 'input video' as follow
Input Field Operation Output Field Type
line 
RegexSplitter(Splitter=,)
 video_id
title
channel_title
category_id
tags
views
likes
dislikes
comment_total
thumbnail_link
date 		String
String
String
String
String
String
String
String
String
String
String


Filter_1 :
Filter Filter_1 filters the output of 'Transformator_1' with the criterium as follow
  video_id Is Not Null


Transformator_2 :
Transformator Transformator_2 tranforms the output of 'Filter_1' as follow
Input Field Operation Output Field Type
video_id No Operation video_id String
title No Operation title String
channel_title No Operation channel_title String
category_id No Operation category_id String
tags No Operation tags String
views 
NVL(0)
 views String
likes 
NVL(0)
 likes String
dislikes 
NVL(0)
 dislikes String
comment_total 
NVL(0)
 comment_total String
thumbnail_link No Operation thumbnail_link String
date 
AddPostfix(.2017) -> StringToTimestamp(dd.MM.yyyy) -> TimestampToString(yyyyMMdd)
 date String


GroupBy_1 :
Group By GroupBy_1 groups the output of 'Transformator_2' with group field category_id.

Aggregator_1 :
Aggregator Aggregator_1 Aggregates the output of 'GroupBy_1' with the following aggregation
Input Field Aggregator Output Field Type
views 
Sum
 sum_view String
views 
Max
 max_view String
likes 
Sum
 sum_like String
likes 
Max
 max_like String
dislikes 
Sum
 sum_dislike String
dislikes 
Max
 max_dislike String


split category :
Transformator split category tranforms the output of 'input category' as follow
Input Field Operation Output Field Type
line 
RegexSplitter(Splitter=|)
 category_id
category_name
valid 		String
String
String


Join_1 :
Joiner Join_1 joins (RightJoin) the outputs of 'Aggregator_1' and 'split category' with join field (category_id->category_id).

Transformator_4 :
Transformator Transformator_4 tranforms the output of 'Join_1' as follow
Input Field Operation Output Field Type
category_name No Operation categoryname String
sum_view No Operation sum_view String
max_view No Operation max_view String
sum_like No Operation sum_like String
max_like No Operation max_like String
sum_dislike No Operation sum_dislike String
max_dislike No Operation max_dislike String


Hfs_sink_2 :
HFS Sink Hfs_sink_2 writes HDFS the output of 'Transformator_4' with properties :
Fields Name : categoryname,sum_view,max_view,sum_like,max_like,sum_dislike,max_dislike
Output Path :
Output Name : hgrid247-#####
Sink Mode : SinkMode.REPLACE
Partition Field(s) :
Splitter :
Index :