RDMBS Source () is used to read data from an RDBMS using JDBC.

The output of this source is a single field.
When multiple columns are queried, their values are concatenated by default using the pipe character (|).

Required Parameters

• Connection parameter for RDBMS connection and metadata retrieval
  
  • Driver Class Name (*)
    JDBC driver class name
  • URL Connection (*)
    JDBC connection URL for the target RDBMS
  • Username (*)
    RDBMS user name
  • Password (*)
    Password for the specified user
• Query Statement (*)
  SQL query used to retrieve data
  Multiple Queries
  • If more than one query is specified:
  • Each query is assigned to one map/partition
  • Supported in MapReduce, Spark, and Flink Batch
  • Parallelism equals the number of queries
• Fetch Size
  the number of rows that should be retrieved from the database in a single network trip when more data is needed.
  Default value: 500
• Field Value's Character Separator
  Indicates the separator between field values.
  If not specified, the default is |.

Batch Processing Parameters

(Applicable to batch engines and single-query execution)

• Partitioning Parameter
  Number of partitions (parallel processes)
  Default value: 1
• Partition Field
  Field used as the partition base
  Default: the first field found in the query statement
  The partition range is calculated automatically using the minimum and maximum values of this field
• Incremental
  Indicates whether incremental querying is enabled
  When enabled:
  • The checkpoint value is searched in a checkpoint table first
  • If not found, it is searched in a checkpoint file
  • If neither exists, a checkpoint file is created
• Checkpoint Table Rules
  
  • Must have the same structure as the queried table
  • Table name format:

        <queried-table-name>_checkpoint
        

• Checkpoint File Rules
  File name format:

       <queried-table-name>_<partition-field-name>.checkpoint
  

• Incremental Query Behavior
  Only records where the partition field value is greater than the checkpoint value are queried

Streaming Parameters

(Applicable to streaming engines)

• Checkpoint Sequence Parameters
  • Sequence Field
    Sequence field included in the query
  • Sequence Field-2
    Optional second sequence field
  • Field Type Data type of the sequence field
    Supported types:
    
    • Long
    • Integer
    • String
    • Timestamp
    • Double
• Checkpoint Table Reference & Sleep Time
  • Table
    Table name containing the sequence reference
  • Field
    Field storing the lower bound of the sequence value
    Updated after each query
  • Field-2
    Optional second field storing the lower bound
  • Sleep Time (ms)
    Time to wait between consecutive queries (in milliseconds)

  External Parameter Sources (*)

  Parameter values can be retrieved from local files, HDFS, or database tables.
  • From a Local File
    Use the prefix file:
    Syntax

        file:<file-location>
    

    Example

        file:/user/root/data.txt
    

  • From an HDFS File
    Use the prefix hdfs:
    Syntax

       hdfs:<hdfs-file-location>
    

    Example

       hdfs:/user/hdfs/input/param.txt
    

  • From a Database Table
    Use the prefix table:
    Syntax

        table:<jdbc-driver#jdbc-url#table-name#field-name#user-name#password>
    

    Example

       table:com.mysql.jdbc.Driver#jdbc:mysql://localhost:3306/test#table1#field1#abc#pwdabc
    

  Supported Engines

  Map Reduce	Spark 1.x RDD	Spark 2.x RDD	Spark 2.x Dataset	Apache Beam 2.29.0 Batch	Apache Beam 2.29.0 Streaming	Apache Flink 1.12.x Batch	Apache Flink 1.12.x Streaming	Apache Tez 0.9.1	PySpark