Avro Sink () is used to write output data into an Avro format file .

Apache Avro is a data serialization format that is widely used in Big Data ecosystems such as:

• Apache Hadoop
• Apache Kafka
• Apache Spark
• Apache Flink

Avro advantages:

• Schema is stored with data
• Smaller file size
• Supports compression
• Perfect for data pipelines and streaming

Parameters of Avro Sink

1. Output Fields Specifies the field names to be written to the Avro file .
   Example:
   Output Fields:

       MSISDN, LAC, CI, MCCMNC, DIRECTION, EVENT_DATE
   

   This field will be the column schema in the Avro file .
2. Output Data Type
   Determine the data type for each field .
   Example:

       String, Integer, Integer ,String ,String ,Timestamp 
   

   This field will be the column type in the Avro file .
3. Compression
   Used to compress Avro files to make the file size smaller.
   Available codec options:

       Codec                                           Description
       --------------------------------------------------------------------
       org.apache.hadoop.io.compress.DeflateCodec      Standard compression
       org.apache.hadoop.io.compress.SnappyCodec       Fast compression (often used in Big Data)
       org.apache.hadoop.io.compress.BZip2Codec        High compression
       org.apache.hadoop.io.compress.GZipCodec         High and stable compression
   

   Example configuration:

       Compression = org.apache.hadoop.io.compress.SnappyCodec
   

   Snappy is usually used most often because:
   • fast
   • efficient for Big Data processing
4. Output Path
   Specifies the location where the Avro file will be saved.
   • If omitted, the input path must be supplied as a command-line argument when running the job.
   • For dynamic parameters, FROM_ARGUMENT(arg_name) can be used to pass the input path at runtime.
5. Default File System
   Specifies the default file system used for the output path.
   Examples:
   • hdfs://
   • file://
   • s3a://
   • s3n://
6. Sink Mode
   Defines the behavior when the output directory already exists.
   Options:
   • KEEP
     Keeps the existing file
   • REPLACE
     Replaces existing files
     Old output files will be overwritten
7. Output Partition Parameter
   
   • Partition Fields
     Used to split the output folder based on a specific field value .
     Example:

         Partition Fields = EVENT_DATE
     

     Folder results:

         /output/
             EVENT_DATE=20251201/part-0000.avro
         /output/
             EVENT_DATE=20251202/part-0001.avro
     

   • Splitter
     Used to separate field values if the field has a specific delimiter.
     Example:

         EVENT_DATE = 2025-12-06
         Splitter = "-"
     

   • Index
     Determines which part of the split result to use .
     The index uses 0-based indexing .
     Example:

     2025-12-06
     

     Split:

         0 = 2025
         1 = 12
         2 = 06
     

     If:

         Index = 0
     

     For partitions:

         year=2025
     

Supported Engines

Map Reduce	Spark 1.x RDD	Spark 2.x RDD	Spark 2.x Dataset	Apache Beam 2.29.0 Batch	Apache Beam 2.29.0 Streaming	Apache Flink 1.12.x Batch	Apache Flink 1.12.x Streaming	Apache Tez 0.9.1	PySpark