Kafka Consumer () reads streaming data from an Apache Kafka Messaging Service.

The parameters of Kafka Consumer

• Servers
  Specifies the Kafka bootstrap.servers value
  Example: host1:9092,host2:9092
• Topic
  Kafka topic name to consume messages from
• Group ID
  Specifies the Kafka group.id for consumer group management
• Value Deserializer
  Deserializer class used to read Kafka message values
  Supported deserializers:
  • StringDeserializer
  • KafkaAvroDeserializer
• Start Reading
  Specifies the starting position for reading Kafka messages
  Supported options:
  • StartFromEarliest
    Reads from the beginning of the Kafka messages.
  • StartFromLatest
    Reads from the end of the Kafka messages.
  • StartFromGroupOffsets
    Reads from the group offset of the Kafka messages.
  • StartFromTimestamp
    Reads from a particular timestamp of the Kafka messages.
• Offset Timestamp
  Required only when Start Reading is set to StartFromTimestamp
  Timestamp format:

  yyyy-MM-dd HH:mm:ss
                      

• Fields
  Names of the fields read from Kafka messages
• Fields Type
  Data types corresponding to each field
• Other Parameter
  Additional Kafka consumer properties
  Format:

  param-1=value-1,param-2=value-2,...,param-N=value-N
  

  Example:

  enable.auto.commit=false,auto.offset.reset=earliest
  

Supported Engines

Map Reduce	Spark 1.x RDD	Spark 2.x RDD	Spark 2.x Dataset	Apache Beam 2.29.0 Batch	Apache Beam 2.29.0 Streaming	Apache Flink 1.12.0 Batch	Apache Flink 1.12.0 Streaming	Apache Tez 0.9.1	PySpark