Assemblies of HGrid247-3.5 DE
In HGrid247-3.5 Data Engineering (DE) , Assembly is an operator used to perform data processing in a workflow . Assembly sits between Source and Sink , and is used to perform various operations such as: Assembly is a core component of the ETL/ELT process in the data pipeline.

Following are the different types of Assembly operators provided by HGrid247-3.5.

1. Basic Data Flow Assemblies

IconAssemblyDescription
Pipe Simple operator to pass data without change
Tranformator Changing/Transform the structure or value of data
Split Transform Transformator with additional output for data that fails to be transformed

2. Grouping and Aggregation Assemblies

Used to group and summarize data .
IconAssemblyDescription
Group By Group data based on specific fields
Aggregator Calculate aggregate values such as SUM, COUNT, AVG
Group And Aggregator Aggregation that starts with group-by
Combiner Local aggregation on data partitions
Global Combiner Global aggregation after local combiner process

3. Join Assemblies

Used to combine data from two or more datasets .
IconAssemblyDescription
Join (Reduce-Side Join) Joins performed at the reduce level
Join By Reference (Map-Side Join) Joins performed at the map stage use dataset references.
Pre Join Preparation process before joining
Post Join Process after joining is done
Merge Merge multiple datasets into one

4. Filtering Assemblies

Used to filter data based on certain conditions .
IconAssemblyDescription
Filter Filter data based on expression
Split Filter Filter with additional output for data that does not meet the conditions
SQL-like Filer Filter using SQL-like expressions

5. Buffer and Sorted Processing

Used to process data that has been grouped and sorted .
IconAssemblyDescription
Buffer Processing data that has been grouped and sorted
Group By and Buffer Buffer operation preceded by Group By
Buffer Transformation Data transformation in buffer
Group By And Buffer Transformation Buffer Transformation preceded by Group By

6. User Defined Function (UDF)

Used to run custom functions created by users .
IconAssemblyDescription
UDF Function Custom functions using Java
UDF Function Python Custom functions using Python
UDF Buffer Custom buffer using Java
UDF Buffer Pyton Custom buffer using Python

7. Data Validation and Checking

Used to check data quality .
IconAssemblyDescription
Duplicate Check Checking for duplicate data
Bloom Filter Check Custom functions using Python
Hash Member Check Checking data membership based on hash

8. Data Analysis and Ranking

Used for statistical analysis or data ranking .
IconAssemblyDescription
Top N Retrieving the N best data points in a local partition
Global Top N Retrieving the best N data globally
Data Profiling Statistical analysis of data structures

9. Data Processing Utilities

Several additional operators to help the workflow process.
IconAssemblyDescription
Data Duplication Duplicating data flow
Debug Saving debugging data
Event Time Assigner Determine the event timestamp on streaming data
Alert Sends notifications when certain conditions are met