Row & Column Custom Delimiter Sink () is used to write output files with custom delimiter characters for rows and columns .

This assembly allows users to define their own delimiters , making it very flexible for producing files with specific formats.

Usually used when:

• The file format does not use standard newlines.
• The destination system requires a special delimiter.
• Integration with legacy systems
• Create custom file formats for data exchange

Parameter Row & Column Custom Delimiter Sink

• Writer Format
  Determines the format used to write the output file.
  Options:
  • HadoopFormat
  • DataFrame (usually used for Spark Engine).
• Char Row Delimiter
  Determines the separator character between rows (records) .
  Examples of rows delimiter:

      Delimiter       Information
      ---------------------------------------
      ;               semicolon
      \001            ASCII control character
      \u0001          Unicode control character
      \n              newline
      \t              tab
  

  Example:

      Char Row Delimiter = ;
  

  Output:

      1001|John|5000;1002|Mary|7000;1003|David|3000 
  

• Char Column Delimiter
  Determines the separator character between columns (fields) .
  Example of delimiter:

      Delimiter       Format
      ----------------------------------
      ,               CSV    `	`
      ;               Semicolon separated
      \t              Tab separated
      \001            ASCII control character
      \u0001          Unicode control character
  

  Example:

      Char Column Delimiter = |
  

  Output:

      1001|John|5000
      1002|Mary|7000
      1003|David|3000    
  

• Output Path
  Specifies the location where the output file will be saved .
  • If omitted, the input path must be supplied as a command-line argument when running the job.
  • For dynamic parameters, FROM_ARGUMENT(arg_name) can be used to pass the input path at runtime.
• Default File System
  Specifies the default file system used for the output path.
  Examples:
  • hdfs://
  • file://
  • s3a://
  • s3n://
• Compression
  Used to compress Parquet files to make the file size smaller.
  Codec options:

      Compression                         Information
      CompressionCodecName.GZIP           High compression, small files
      CompressionCodecName.LZO            Fast compression
      CompressionCodecName.SNAPPY         Fast compression, common in Big Data
      CompressionCodecName.UNCOMPRESSED	without compression
  

  Default:

      CompressionCodecName.UNCOMPRESSED
  

• Sink Mode
  Defines the behavior when the output directory already exists.
  Options:
  • KEEP
    Keeps the existing file
  • REPLACE
    Replaces existing files
    Old output files will be overwritten
• Output Partition Parameter
  
  • Partition Fields
    Used to split the output folder based on a specific field value .
    Example:

        Partition Fields = EVENT_DATE
    

    Folder results:

        /output/20251201/
        /output/20251202/
    

  • Splitter
    Used to separate field values if the field has a specific delimiter.
    Example:

        EVENT_DATE = 2025-12-06
        Splitter = "-"
    

  • Index
    Determines which part of the split result to use .
    The index uses 0-based indexing .
    Example:

    2025-12-06
    

    Split:

        0 = 2025
        1 = 12
        2 = 06
    

    If:

        Index = 0
    

    For partitions:

        year=2025
    

  Partition is very important for:
  Spark query
  Hive table
  Big Data analytics
  

Supported Engines

Map Reduce	Spark 1.x RDD	Spark 2.x RDD	Spark 2.x Dataset	Apache Beam 2.29.0 Batch	Apache Beam 2.29.0 Streaming	Apache Flink 1.12.x Batch	Apache Flink 1.12.x Streaming	Apache Tez 0.9.1	PySpark