HFS Sink

Flat File Sink (

) is used to write output records into a flat file sequentially.
Each record is written line by line and separated by a newline character (\n).

This assembly is commonly used to store processed stream data into files such as logs, reports, or exported datasets.

Parameters of Flat File Sink

Writer Format
Determines the format used to write the output file.
Options:
- HadoopFormat
- DataFrame (usually used for Spark Engine).
Field Delimiter
A character used to separate fields in each record (only for Spark Engine).

Example delimiters:
- , (CSV format)
- |
- ;
- \t (tab)
Example output:
```
1001,John,5000
1002,Mary,7000
```
Output Path
The path to the output directory.
- If omitted, the input path must be supplied as a command-line argument when running the job.
- For dynamic parameters, FROM_ARGUMENT(arg_name) can be used to pass the input path at runtime.
Default File System
Specifies the default file system used for the output path.
Examples:
- hdfs://
- file://
- s3a://
- s3n://
Compression
Specifies the compression codec used for the output file.
Examples:
- gzip
- snappy
- lzo
- lz4
Purpose:
- Reduce storage size
- Improve transfer efficiency
Sink Mode
Defines the behavior when the output directory already exists.
Options:
- KEEP
  Keeps the existing file
- REPLACE
  Replaces existing files
  Old output files will be overwritten

Output Name
Defines the naming pattern of generated output files.
There are six available patterns

Basic Pattern

    hgrid247-#####

Example:

    hgrid247-00000
    hgrid247-00001

With Date

    hgrid247-#####-YYMMDD

Example:

    hgrid247-00000-180503
    hgrid247-00001-180503

With Date and Hour

    hgrid247-#####-YYMMDDHH

Example:

    hgrid247-00000-18050309
    hgrid247-00001-18050311

With Date, Hour, and Minute

    hgrid247-#####-YYMMDDHHMI

Example:

    hgrid247-00000-1805030950
    hgrid247-00001-1805031150

With Date, Hour, Minute, and Second

    hgrid247-#####-YYMMDDHHMISS

Example:

    hgrid247-00000-180503095020
    hgrid247-00001-180503115030

Output-Partitioning
Used to partition the output folder based on field content.

Partition Fields
This feature allows the system to separate output data into different folders based on field values if the parameter 'Output Name' is set to 'Output-Partitioning'.
- Partition Fields
  Fields used to determine how output data is partitioned.
  Example field:
```
    country
```
  Result:
```
    /output/USA/
    /output/JAPAN/
    /output/INDONESIA/
```
- Splitter
  A string delimiter used to split a field value before partitioning.
  Example:
  Field value:
```
    2025-03-06
```
  Splitter:
```
    -
```
  Result:
```
    2025 | 03 | 06
```
- Index
  Defines which split part will be used for partitioning.
  Example:
  Field value:
```
    2025-03-06
```
  Splitter:
```
    -
```
  Split result:
```
    [0] 2025
    [1] 03
    [2] 06
```
  If Index = 0, the partition will be:
```
    /output/2025/
```
Header Included
Determines whether field names are written as the first row.

Example output:
```
    id,name,amount
    1001,John,5000
    1002,Mary,7000
```
Header Delimited
Defines the separator used between header fields if the parameter 'Header Included' is selected.

Example:
```
    id|name|amount
```
Streaming Parameters These parameters are used when running stream processing engines.
- Beam Streaming Parameter
  Window Length (seconds)
  Defines the time window length for writing output.
  
  Example:
```
    Window Length = 60 seconds
```
  Output file will be generated every 1 minute.
- Flink Streaming Parameter
  1. Rollover Interval (seconds)
    Defines how often a new output file is created.
    
    Example:
```
    Rollover Interval = 300 seconds
```
    A new file is created every 5 minutes.
  2. Inactivity Interval (seconds)
    Defines the maximum idle time before closing the file.
    
    If no data arrives during this period, the file will be finalized.
  3. Max Part Size (bytes)
    Defines the maximum size allowed per output file.
    
    Example:
```
    Max Part Size = 134217728 bytes (128 MB)
```
    If the file reaches this size, a new file will be created automatically.

Supported Engines

Map Reduce	Spark 1.x RDD	Spark 2.x RDD	Spark 2.x Dataset	Apache Beam 2.29.0 Batch	Apache Beam 2.29.0 Streaming	Apache Flink 1.12.0 Batch	Apache Flink 1.12.0 Streaming	Apache Tez 0.9.1	PySpark