RDMBS2 Source () is used to read data from an RDBMS using JDBC.

Unlike RDBMS Source, the number of output fields is equal to the number of queried fields. Each column in the query result is mapped to a separate output field.

Required Parameters

• Connection parameter for RDBMS connection and metadata retrieval.
  
  • Driver Class Name (*)
    JDBC driver class name
  • URL Connection (*)
    JDBC connection URL for the target RDBMS
  • Username (*)
    RDBMS user name
  • Password (*)
    Password for the specified user
• Query Statement (*)
  SQL query used to retrieve data
  Multiple Queries
  • If more than one query is specified:
  • Each query is assigned to one map/partition
  • Supported in MapReduce, Spark, and Flink Batch
  • Parallelism equals the number of queries
• Fields Declaration
  Defines the output schema
  Specifies field names and field types corresponding to the query result columns

Batch Processing Parameters

(Applicable to batch engines and single-query execution)

• Partitioning Parameter
  Number of partitions (parallel processes)
  Default value: 1
• Partition Field
  Field used as the partition base
  Default: the first field found in the query statement
  The partition range is calculated automatically using the minimum and maximum values of this field
• Incremental
  Indicates whether incremental querying is enabled
  When enabled:
  • The checkpoint value is searched in a checkpoint table first
  • If not found, it is searched in a checkpoint file
  • If neither exists, a checkpoint file is created
• Checkpoint Table Rules
  
  • Must have the same structure as the queried table
  • Table name format:

        <queried-table-name>_checkpoint
        

• Checkpoint File Rules
  File name format:

       <queried-table-name>_<partition-field-name>.checkpoint
  

• Incremental Query Behavior
  Only records where the partition field value is greater than the checkpoint value are queried

Streaming Parameters

(Applicable to streaming engines)

• Checkpoint Sequence Parameters
  • Sequence Field
    Sequence field included in the query
  • Sequence Field-2
    Optional second sequence field
  • Field Type Data type of the sequence field
    Supported types:
    
    • Long
    • Integer
    • String
    • Timestamp
    • Double
• Checkpoint Table Reference & Sleep Time
  • Table
    Table name containing the sequence reference
  • Field
    Field storing the lower bound of the sequence value
    Updated after each query
  • Field-2
    Optional second field storing the lower bound
  • Sleep Time (ms)
    Time to wait between consecutive queries (in milliseconds)

  External Parameter Sources (*)

  Parameter values can be retrieved from local files, HDFS, or database tables.
  • From a Local File
    Use the prefix file:
    Syntax

        file:<file-location>
    

    Example

        file:/user/root/data.txt
    

  • From an HDFS File
    Use the prefix hdfs:
    Syntax

       hdfs:<hdfs-file-location>
    

    Example

       hdfs:/user/hdfs/input/param.txt
    

  • From a Database Table
    Use the prefix table:
    Syntax

        table:<jdbc-driver#jdbc-url#table-name#field-name#user-name#password>
    

    Example

       table:com.mysql.jdbc.Driver#jdbc:mysql://localhost:3306/test#table1#field1#abc#pwdabc
    

  Supported Engines

  Map Reduce	Spark 1.x RDD	Spark 2.x RDD	Spark 2.x Dataset	Apache Beam 2.29.0 Batch	Apache Beam 2.29.0 Streaming	Apache Flink 1.12.x Batch	Apache Flink 1.12.x Streaming	Apache Tez 0.9.1	PySpark