Hive Sink

Hive Sink (

) is used to store processed data into a Hive database running on the Hadoop ecosystem. Hive is typically used as a data warehouse for big data analysis using SQL queries (HiveQL) .
For Hive Sink to function properly, some Hadoop and Hive configuration files must be available in the Spark configuration directory.
The configuration file must be placed in the following directory:

    SPARK_HOME/conf

Required files include:

hive-site.xml
core-site.xml
hdfs-site.xml

If the hive-site.xml file is not available, the system will automatically:

1. Create a database metastore named:

    metastore_db

2. Creates a warehouse directory specified by the parameters:

    spark.sql.warehouse.dir

By default the warehouse directory is located at:

    spark-warehouse

This directory is used to store Hive tables created by Spark.

Parameter Hive Sink

Database Name
This parameter specifies the name of the Hive database where the table will be created or used.
Table Name
Specifies the name of the Hive table that will be used to store the processed data.
Partition Fields
Used to define partition columns in Hive tables .
Partition is used to improve query performance on big data .
If more than one field is used, they are separated by a comma.
Example:
```
    Partition Fields : year,month
```
This means that the data will be stored in a directory structure like:
```
    /warehouse/user_activity/year=2025/month=02/    
```

Save Mode
This parameter determines how data is written to the Hive table .
Mode options:

    Mode            Description
    -------------------------------------------------
    append          Add new data without deleting old data
    overwrite       Deleting old data and replacing it with new data
    ignore          Do nothing
    ErrorIfExist    Generates an error if the table already exists

File Format
Determines the file storage format in Hive .
Format options:

    Format          Description
    ---------------------------------------
    csv             Text file with comma separator
    parquet         Highly efficient columnar format
    orc             Optimized Row Columnar for Hive
    json            JSON-based data format

Session Connection
Determines the type of connection used to interact with Hive .
There are two options:

    Connection              Description
    -----------------------------------------------
    SparkSession            Using Spark SQL to interact with Hive
    HiveWarehouseSession    Using Hive Warehouse Connector

Important Notes

The user running the Spark Job must have write access to the Hive Warehouse Directory .
If using HiveWarehouseSession , it is only supported on:

    Spark 2.x

The following libraries must be included when running a Spark job:

    hive-warehouse-connector-assembly-1.0.0.3.1.0.0-78.jar

This library is used to enable communication between Spark and Hive LLAP .

Supported Engines

Map Reduce	Spark 1.x RDD	Spark 2.x RDD	Spark 2.x Dataset	Apache Beam 2.29.0 Batch	Apache Beam 2.29.0 Streaming	Apache Flink 1.12.0 Batch	Apache Flink 1.12.0 Streaming	Apache Tez 0.9.1	PySpark