Kafka Producer () is used to send data from the pipeline to the Kafka messaging service .

Apache Kafka is a distributed messaging system and streaming platform used for:

• real-time data streaming
• event processing
• data pipeline between systems
• microservices communication

Kafka Producer functions as a message sender (message publisher) to Kafka.

Parameters of Kafka Producer

1. Servers
   Specifies the address of the Kafka broker server .
   This parameter is the same as the configuration:

       bootstrap.servers
   

   Example:

       Servers = localhost:9092
   

   If the Kafka cluster:

       Servers = kafka1:9092,kafka2:9092,kafka3:9092
   

   Kafka will use this server as a bootstrap for cluster connections .
2. Topic
   Specifies the name of the Kafka topic to which the message is sent.
   Topic is a message storage channel in Kafka .
   Example:

       Topic = telecom-events
   

   This means all data will be sent to the topic:

       telecom-events   
   

3. Sending Method
   Determines how Kafka Producer sends messages .
   
   There are 3 shipping methods .
   • Fire And Forget
     The producer sends the message without waiting for confirmation from Kafka.
     Characteristics:
     • very fast
     • there is no guarantee that messages will be saved
     Example:

         Sending Method = Fire And Forget
     

   • Synchronously
     The producer waits for confirmation from Kafka before continuing .
     Characteristics:
     • safer
     • slower
     Example:

         Sending Method = Synchronously
     

   • Asynchronously
     Producer sends messages asynchronously .
     Characteristics:
     • fast
     • still receive callbacks from Kafka
     Example:

         Sending Method = Asynchronously
     

     This method is often used in real-time streaming systems .
4. Value Serializer
   Specifies the message serialization method before sending it to Kafka .
   Serialization is necessary because Kafka only stores byte arrays .
   There are 3 serializers available .
   • StringSerializer
     Convert data to string before sending.
     Example:

         Value Serializer = StringSerializer
     

     Example message:

         "TX001,62812345678,5000"
     

   • KafkaAvroSerializer
     Using Avro format for data serialization.
     Apache Avro is an efficient data serialization format. Example:

         Value Serializer = KafkaAvroSerializer
     

     Profit:
     • schema-based
     • efficient
     • suitable for big data pipeline
   • TupleToJSonSerializer
     Convert data into JSON format .
     Example:

         Value Serializer = TupleToJSonSerializer
     

     Example JSON message:

         {
         "id": "TX001",
         "msisdn": "62812345678",
         "amount": 5000
         }
     

     JSON is often used for:
     • microservices
     • REST API integration
     • event streaming
5. Fields
   Specifies the fields to be sent as Kafka messages .
   Example:

       id, msisdn, lac, ci, event_date
   

   This field comes from the previous assembly in the pipeline .
6. Fields Type
   Determine the data type of each field .
   Example:

       String, String, Integer, Integer, Timestamp
   

   Mapping example:

       id           String
       msisdn       String
       lac          Integer
       ci           Integer
       event_date   Timestamp   
   

7. Other Parameter
   Used to add additional parameters to Kafka Producer .
   Format:

       param-1=value-1,param-2=value-2,...,param-N=value-N
   

   Example configuration:

       acks=all,retries=3,batch.size=16384  
   

Supported Engines

Map Reduce	Spark 1.x RDD	Spark 2.x RDD	Spark 2.x Dataset	Apache Beam 2.29.0 Batch	Apache Beam 2.29.0 Streaming	Apache Flink 1.12.0 Batch	Apache Flink 1.12.0 Streaming	Apache Tez 0.9.1	PySpark