Data Processing Guidance

To study the data processing process on HGrid247-3.5 , two types of datasets were used:

Dataset YouTube Video
Dataset Apache Web Log

The dataset is used to practice some key concepts in data pipelines such as:

data cleansing
data aggregation
data joining
JSON parsing
log processing

I. Dataset YouTube

The YouTube dataset consists of the following files.

youtube_videos.csv
This file contains YouTube video information such as:

    video id
    video title
    channel
    category
    number of views
    likes
    dislikes
    number of comments

Sample Data

    I7TSzpDetTY,"Gal Gadot Monologue - SNL","Saturday Night Live",24,SNL|Saturday Night Live|SNL Season 43|Episode 1727|Leslie Jones|s43|s43e2|episode 2|live|new york|comedy|sketch|funny|hilarious|late night|host|music|guest|laugh|impersonation|actor|improv|musician|gal gadot|wonder woman|justice league|fast and furious 6|israeli|model|sam smith|too good at goodbyes|in the lonely hour|grammy|monologue,27721,2050,146,356,https://i.ytimg.com/vi/I7TSzpDetTY/default.jpg,08.10
    fwDZ8uhqJg0,"Eric Church - Why Not Me | October 4, 2017","Grand Ole Opry",10,Grand Ole Opry|Opry|live|concert|performance|Nashville|Tennessee|Country|Music|Grand Opry|The Grand Opry|Opry Live|Country Music|Live Performance|Opryland|Carrie Underwood|Alan Jackson|Alison Krauss|Ashley Monroe|Barbara Mandrell|Big & Rich|American Country|Trace Adkins|#TheCircle|Performance|Historic|Music Venue|Eric Church|New Song|Why You Why Not Me,5389625,31170,2547,2651,https://i.ytimg.com/vi/fwDZ8uhqJg0/default.jpg,08.10
    D1IBXE2G6zw,"Lin-Manuel Miranda - Almost Like Praying feat Artists for Puerto Rico [Music Video]","Atlantic Records",10,music|official|music video|lin-manual miranda|almost like praying|Marc Anthony|atlantic records|warner music group|Alex Lacamoire|Ana VillafaÃ±e|Anthony Ramos|Camila Cabello|Dessa|Ednita Nazario|Fat Joe|Gilberto|Santa Rosa|Gina Rodriguez|Gloria Estefan|Jennifer Lopez|Joell Ortiz|Juan Luis Guerra|Luis Fonsi|Pedro CapÃ³|PJ Sin Suela|Rita Moreno|RubÃ©n Blades|Tommy Torres|Hispanic Federation UNIDOS Fund|puerto rico,1802878,71780,2124,5772,https://i.ytimg.com/vi/D1IBXE2G6zw/default.jpg,08.10
    fUjicxMPDzs,"Pacific Rim Uprising - Official Trailer (HD)","Legendary",1,[none],2839775,37111,5271,11416,https://i.ytimg.com/vi/fUjicxMPDzs/default.jpg,08.10
    . . .
    . . .
    . . .

Data Structure

    No	Field           Description
    ------------------------------------
    1	video_id	Unique video ID
    2	title           Video title
    3	channel_title	Nama channel
    4	category_id	Video category ID
    5	tags            Video tags are separated by `
    6	views           Number of views
    7	likes           Number of likes
    8	dislikes	Number of dislikes
    9	comment_total	Number of comments
    10	thumbnail_link	URL thumbnail
    11	date            Upload date

category.dat
This file contains references to YouTube video categories .
Sample Data

    1|"Film & Animation"|1
    2|"Autos & Vehicles"|1
    10|"Music"|1
    15|"Pets & Animals"|1
    17|"Sports"|1
    18|"Short Movies"|1
    19|"Travel & Events"|1
    20|"Gaming"|1
    21|"Videoblogging"|1
    22|"People & Blogs"|1
    23|"Comedy"|1
    24|"Entertainment"|1
    . . .
    . . .
    . . .

Data Structure

    No	Field           Description
    ------------------------------------
    1	category_id	Category ID
    2	category_name	Video category name
    3	valid           Category status (0 or 1)

category_id.json
This JSON file contains YouTube category metadata from the API .
JSON Structure Example

    {
     "kind": "youtube#videoCategoryListResponse",
     "etag": "\"m2yskBQFythfE4irbTIeOgYYfBU/S730Ilt-Fi-emsQJvJAAShlR6hM\"",
     "items": [
      {
       "kind": "youtube#videoCategory",
       "etag": "\"m2yskBQFythfE4irbTIeOgYYfBU/Xy1mB4_yLrHy_BmKmPBggty2mZQ\"",
       "id": "1",
       "snippet": {
        "channelId": "UCBR8-60-B28hp2BmDPdntcQ",
        "title": "Film & Animation",
        "assignable": true
       }
      },
      {
       "kind": "youtube#videoCategory",
       "etag": "\"m2yskBQFythfE4irbTIeOgYYfBU/UZ1oLIIz2dxIhO45ZTFR3a3NyTA\"",
       "id": "2",
       "snippet": {
        "channelId": "UCBR8-60-B28hp2BmDPdntcQ",
        "title": "Autos & Vehicles",
        "assignable": true
       }
      },
      . . .

This data is used to learn JSON parsing using JSONPathParser .

II. Data Processing Guide in HGrid247-3.5

This data processing guide presents several case studies to help users understand how to perform data processing using HGrid247-3.5. The guide demonstrates how to design workflows, process datasets, perform data transformations, and analyze log data.

The following case studies are discussed in this guide.

Creating a Project in HGrid247-3.5
In this section, we learn how to create an HGrid247 project and understand the structure of an HGrid247 project.
Topics covered include:
- Creating a new project in HGrid247-3.5
- Understanding the project structure
- Creating packages and workflows
- Organizing project components
An HGrid247 project typically contains several components such as:
- Project
- Package
- Workflow
- Script
These components are used to organize and manage data processing workflows.
Data Cleansing (youtube_video.csv)
In this case study, the youtube_video.csv dataset is used to demonstrate the process of data cleansing.
The main objectives are:
- Creating a workflow
- Reading data using HFS Source
- Processing data using Assembly operators
- Writing results using HFS Sink
- Defining data transformation
- Defining filter conditions
Typical operations performed during data cleansing include:
- Removing invalid records
- Filtering unnecessary data
- Transforming fields into the required format
This process ensures that the dataset is clean and ready for further analysis.
Data Aggregation (youtube_video.csv)
In this section, we learn how to perform data aggregation using the YouTube dataset.
The topics covered include:
- Copying an existing workflow
- Performing data grouping
- Applying aggregation functions
- Grouping data by specific fields such as category or channel
Common aggregation operations include:
- calculating sum
- calculating max
Aggregation allows users to obtain summarized insights from large datasets.
Join Between youtube_video.csv and category.dat
This case study demonstrates how to join two datasets.
The datasets involved are:
- youtube_video.csv (video information)
- category.dat (category reference data)
Through this join operation, the category_id in the video dataset can be matched with the corresponding category_name from the reference dataset.

This process enriches the dataset by adding meaningful information from another source.
Join Between youtube_video.csv and category_id.json
This section demonstrates how to work with JSON data files in HGrid247.
The learning objectives include:
- Reading JSON files
- Extracting data using JSONPathParser transformation function
- Performing join operations between datasets
The JSON file contains metadata about YouTube video categories. The JSONPathParser function is used to extract relevant fields from the JSON structure before joining the data with the YouTube dataset.
Apache Log Processing
The final case study focuses on processing Apache web server logs.
In this section, users will learn how to analyze log data using HGrid247.
Topics covered include:
- Parsing log data using Grok
- Performing log analysis using Condition Aggregator
Grok is used to extract structured fields from unstructured log text, such as:
- IP address
- request method
- URL
- response status code
- timestamp
After extracting these fields, the Condition Aggregator can be used to perform statistical analysis, such as:
- counting requests per status code
- identifying error responses
- analyzing traffic patterns

HGrid247-3.5 'Data-Processing' Guidance

I. Dataset YouTube

II. Data Processing Guide in HGrid247-3.5