ORC Sink () is used to write output data to an ORC (Optimized Row Columnar) format file. .

ORC is a columnar storage file format optimized for Big Data analytics , especially on systems such as:

• Apache Hive
• Apache Spark
• Apache Flink
• Apache Hadoop

ORC Advantages:

• Very efficient compression
• Supports indexing
• Queries are much faster than CSV files
• Optimized for data warehouse

Parameters of ORC Sink

1. Output Path
   Specify the folder where the ORC files will be saved.
   • If omitted, the input path must be supplied as a command-line argument when running the job.
   • For dynamic parameters, FROM_ARGUMENT(arg_name) can be used to pass the input path at runtime.
2. Field In Lower Case
   Determines whether output field names are converted to lowercase.
   Example:

       TRUE
   

   If the initial field:

       MSISDN
       EVENT_DATE
   

   So it becomes:

       msisdn
       event_date
   

   This is often used because many Big Data systems use lowercase schema .
3. Create Index
   Determines whether the ORC file creates an internal index.
   Mark:

       TRUE 
       FALSE
   

   If TRUE , then:
   • Faster queries
   • Filter data more efficiently
4. Compressed
   Determines whether the ORC file is compressed or not .
   Mark:

       TRUE 
       FALSE
   

   If TRUE , the file size will be smaller.
   
   ORC typically uses automatic compression such as:
   • ZLIB
   • SNAPPY
   • LZO
5. Sink Mode
   Defines the behavior when the output directory already exists.
   Options:
   • KEEP
     Keeps the existing file
   • REPLACE
     Replaces existing files
     Old output files will be overwritten
6. Using Vectorization Batch
   Determines whether the ORC file is written using Vectorization Batch
   Mark:

       TRUE 
       FALSE
   

7. Output Partition Parameter
   
   • Partition Fields
     Used to split the output folder based on a specific field value .
     Example:

         Partition Fields = EVENT_DATE
     

     Folder results:

         /output/
             EVENT_DATE=20251201/part-0000.orc
         /output/
             EVENT_DATE=20251202/part-0001.orc
     

   • Splitter
     Used to separate field values if the field has a specific delimiter.
     Example:

         EVENT_DATE = 2025-12-06
         Splitter = "-"
     

   • Index
     Determines which part of the split result to use .
     The index uses 0-based indexing .
     Example:

     2025-12-06
     

     Split:

         0 = 2025
         1 = 12
         2 = 06
     

     If:

         Index = 0
     

     For partitions:

         year=2025
     

8. Output Fields
   Defines the ORC field structure .
   Containing:
   • Field name
   • Field type
   Example:

       MSISDN       String
       LAC          Integer
       CI           Integer
       MCCMNC       String
       DIRECTION    String
       EVENT_DATE   Timestamp
   

   If the ORC Schema is empty , then Output Fields is used as the ORC schema .
9. ORC Schema
   ORC schema is a complete structure definition of ORC data .
   Containing:
   • Field name
   • Field type
   • Metadata
   Example ORC Schema:

       struct<
       msisdn:string,
       lac:int,
       ci:int,
       mccmnc:string,
       direction:string,
       event_date:timestamp
       >
   

   If the ORC Schema has been filled in, then this schema is used .
10. Streaming Parameters
    These parameters are used when running stream processing engines.
    • Beam Streaming Parameter
      Window Length (seconds)
      Determines the length of the output window . Example:

          60
      

      This means that every 60 seconds data will be written to the output file.
    • Flink Streaming Parameter
      
      1. Rollover Interval (seconds)
         Time interval to create new output files .
         Example:

             300
         

         This means that every 5 minutes a new file is created.
      2. Inactivity Interval (seconds)
         If there is no data within a certain interval, the file will be closed.
         Example:

             120
         

         This means that if there is no data for 2 minutes , the file is considered complete.
      3. Max Part Size (bytes)
         Maximum output file size.
         Example:

             134217728
         

             This is the same as:
         

             128 MB
         

         If the size is reached, the system will create a new file.

ORC Data Type :

ORC Type	Hadoop Writable Type
binary	org.apache.hadoop.io.BytesWritable
bigint	org.apache.hadoop.io.LongWritable
boolean	org.apache.hadoop.io.BooleanWritable
char	org.apache.hadoop.io.Text
date	org.apache.hadoop.hive.serde2.io.DateWritable
decimal	org.apache.hadoop.hive.serde2.io.HiveDecimalWritable
double	org.apache.hadoop.io.DoubleWritable
float	org.apache.hadoop.io.FloatWritable
int	org.apache.hadoop.io.IntWritable
smallint	org.apache.hadoop.io.ShortWritable
string	org.apache.hadoop.io.Text
timestamp	org.apache.orc.mapred.OrcTimestamp
tinyint	org.apache.hadoop.io.ByteWritable
varchar	org.apache.hadoop.io.Text

Supported Engines

Map Reduce	Spark 1.x RDD	Spark 2.x RDD	Spark 2.x Dataset	Apache Beam 2.29.0 Batch	Apache Beam 2.29.0 Streaming	Apache Flink 1.12.0 Batch	Apache Flink 1.12.0 Streaming	Apache Tez 0.9.1	PySpark