Parquet Sink () is used to write output data into a Parquet format file .

Parquet is a highly efficient columnar data storage format for:

• Big Data Processing
• Data Warehouse
• Apache Spark
• Apache Hive
• Apache Hadoop

Parquet Advantages:

• smaller file size
• faster query process
• supports compression
• suitable for large-scale data analysis

Parameters of Parquet Sink

1. Write Format
   Determines the format used to write the output file.
   Options:
   • HadoopFormat
   • DataFrame (usually used for Spark Engine).
2. Output Fields
   Specifies the field names to be written to the Parquet file .
   Example:
   Output Fields:

       MSISDN, LAC, CI, MCCMNC, DIRECTION, EVENT_DATE
   

   This field will be the column schema in the Parquet file .
3. Output Field Type
   Determine the data type for each field .
   Example:

       String, Integer, Integer ,String ,String ,Timestamp 
   

   This field will be the column type in the Parquet file .
4. Fields in lower case
   Determines whether field names will be converted to lowercase .
   Options:
   • True → all fields become lowercase
   • False → remains the same as the original name
5. Compression
   Used to compress Parquet files to make the file size smaller.
   Codec options:

       Compression                         Information
       CompressionCodecName.GZIP           High compression, small files
       CompressionCodecName.LZO            Fast compression
       CompressionCodecName.SNAPPY         Fast compression, common in Big Data
       CompressionCodecName.UNCOMPRESSED	without compression
   

   Default:

       CompressionCodecName.UNCOMPRESSED
   

6. Block Size
   Specifies the HDFS block size for Parquet files.
   Default:

       following HDFS configuration
   

   Example:

       Block Size = 128MB
   

7. Page Size
   Determines the internal page size of Parquet .
   A page is a small unit within a block.
   Example:

       Page Size = 1MB
   

   Smaller pages → queries can be faster.
8. Output Path
   Parquet file storage location.
   • If omitted, the input path must be supplied as a command-line argument when running the job.
   • For dynamic parameters, FROM_ARGUMENT(arg_name) can be used to pass the input path at runtime.
9. Default File System
   Specifies the default file system used for the output path.
   Examples:
   • hdfs://
   • file://
   • s3a://
   • s3n://
10. Sink Mode
    Defines the behavior when the output directory already exists.
    Options:
    • KEEP
      Keeps the existing file
    • REPLACE
      Replaces existing files
      Old output files will be overwritten
11. Output Partition Parameter
    
    • Partition Fields
      Used to split the output folder based on a specific field value .
      Example:

          Partition Fields = EVENT_DATE
      

      Folder results:

          /output/
              EVENT_DATE=20251201/
          /output/
              EVENT_DATE=20251202/
      

    • Splitter
      Used to separate field values if the field has a specific delimiter.
      Example:

          EVENT_DATE = 2025-12-06
          Splitter = "-"
      

    • Index
      Determines which part of the split result to use .
      The index uses 0-based indexing .
      Example:

      2025-12-06
      

      Split:

          0 = 2025
          1 = 12
          2 = 06
      

      If:

          Index = 0
      

      For partitions:

          year=2025
      

    Partition is very important for:
    Spark query
    Hive table
    Big Data analytics
    

Supported Engines

Map Reduce	Spark 1.x RDD	Spark 2.x RDD	Spark 2.x Dataset	Apache Beam 2.29.0 Batch	Apache Beam 2.29.0 Streaming	Apache Flink 1.12.x Batch	Apache Flink 1.12.x Streaming	Apache Tez 0.9.1	PySpark