Sources of HGrid247-3.5 DE

In HGrid247-3.5 Data Engineering (DE) , Source is the initial operator in the workflow that functions to read or import data from various external sources . The read data is then processed by Assembly before finally being stored in Sink .

HGrid247 provides various types of Source operators to support various data formats and storage systems.

1. File-Based Sources

This source is used to read data from local file systems or distributed storage such as HDFS .

Icon	Source	Description
	Flat File Source / HFS Source	Reading data from simple text files such as CSV or TXT
	XML File Format Source	Reading data in XML format
	Parquet Source	Reading files in Apache Parquet format
	Avro Source	Reading files in Apache Parquet format
	HTML Source	Reading and extracting data from HTML files
	Row & Column Custom Separator Source	Read files with customizable row and column separators
	ORC Source	Reading files in Optimized Row Columnar (ORC) format
	Custom Source	Reads various other file formats such as JSON, PDF, Excel, MS Word, image, pattern-delimiter, geospatial, ASN.1, and other
	Sequence Name-Content File	Reads Sequence Name Content file built using the Forklift application

2. Database Sources

This source is used to read data from relational and non-relational databases .

Icon	Source	Description
	Hive Source Single Output Field	Reading data from Hive with one output field
	Hive Source Multiple Output Fields	Reading data from Hive with multiple output fields
	RDBMS Source Single Output Field	Reading data from a relational database with a single output field
	RDBMS Source Multiple Output Fields	Reading data from a relational database with multiple output fields
	JDBC Source	Reads data from RDBMS using JDBC drivers, serving as an alternative to Sqoop.
	MongoDB Source	Reading data from a MongoDB database
	HBase Source	Retrieving data from Apache HBase database
	ElasticSearch Source	Retrieving data from Elasticsearch index
	Solr Source	Fetching data from Apache Solr search platform

3. Streaming and Messaging Sources

This source is used to read streaming data or message queues .

Icon	Source	Description
	Kafka Source / Consumer	Consuming streaming data from Apache Kafka topics
	MQTT Source	Retrieving data from MQTT message broker
	Socket Source	Reading streaming data via socket connection
	UDF Stream Source	Reading streaming data from a User Defined Function (UDF)

4. Web and Social Media Sources

This source is used to retrieve data from the internet or online platforms .

Icon	Source	Description
	Web Crawler Source	Retrieving data from web pages through the crawling process
	Twitter Source	Fetch data from the Twitter API such as tweets or metadata

5. Utility Sources

Additional sources used to create or control data flow in a workflow .

Icon	Source	Description
	Data Feeder	Used to generate or send data to a workflow for testing or simulation purposes.