Hive Source () is used to read data from a Hive database.

To enable Hive access, place the following configuration files in the $SPARK_HOME/conf directory:

• hive-site.xml
• core-site.xml
• hdfs-site.xml

If hive-site.xml is not configured, Spark will automatically:

• Create a local metastore_db directory in the current working directory
• Create a warehouse directory defined by spark.sql.warehouse.dir
  Default value: spark-warehouse

Configuration Parameters

• Database Name The name of the Hive database to read from Session Connection
• Connection type used to access Hive
  Supported values:
  • SparkSession
  • HiveWarehouseSession
• Hive Server Parameter
  Used for retrieving Hive metadata.
  • JDBC Driver Class
    Hive JDBC driver class name.
  • URL
    Hive URL JDBC Connection
  • Hive User
    Hive User Name
  • Hive Password
    Hive Password
• Query Statement
  SQL query used to retrieve data from the Hive database
• Fields Contents Separator
  Delimiter used to separate field values in the query result

Important Notes

The user running the Spark Job must have write access to the Hive Warehouse Directory .
If using HiveWarehouseSession , it is only supported on:

    Spark 2.x

The following libraries must be included when running a Spark job:

    hive-warehouse-connector-assembly-1.0.0.3.1.0.0-78.jar

This library is used to enable communication between Spark and Hive LLAP .

If using SparkSession , add the following configuration when submitting job.

    --conf spark.sql.catalogImplementation=hive

Supported Engines

Map Reduce	Spark 1.x RDD	Spark 2.x RDD	Spark 2.x Dataset	Apache Beam 2.29.0 Batch	Apache Beam 2.29.0 Streaming	Apache Flink 1.12.0 Batch	Apache Flink 1.12.0 Streaming	Apache Tez 0.9.1	PySpark