StartsWithPatternInputFormat is used to parse a text file in which the beginning of each record is marked by a specific pattern defined using a
regular expression.
Configuration
The configuration names consist of the following parameters:
- hgrid247.record.startPattern
Indicates the pattern that marks the start of a record.
- hgrid247.record.patternFlag
Indicates the pattern flag.
Default value: MULTILINE
Multiple flags can be specified by separating them with |.
- hgrid247.record.splitable
Indicates whether the input file is splittable or not.
Default value: not splittable.
- hgrid247.split.size
Indicates the size of a split.
Default value: the HDFS block size.
Example
Given the following content in a text file:
2017-01-18;15:08:15.477 \B1.WWW01 IND.000.1000 6700
FR: LABS247 R: SOLUSI247
Uns:recpt prntr: Tran status 0, Paper ok c_00001
2017-01-18;15:08:15.485 \B1.S247 R: 3000 2017
Master socket caller 111117:192.168.1.1 detected c_00002
To read the above text, use the following configuration:
| Configuration Name | Configuration Value |
|---|
| hgrid247.record.startPattern | \\d\\d\\d\\d-\\d\\d-\\d\\d; |
Output Result
The input will be parsed into the following two records:
Record 1
2017-01-18;15:08:15.477 \B1.WWW01 IND.000.1000 6700
FR: LABS247 R: SOLUSI247
Uns:recpt prntr: Tran status 0, Paper ok c_00001
Record 2
2017-01-18;15:08:15.485 \B1.S247 R: 3000 2017
Master socket caller 111117:192.168.1.1 detected c_00002
Behavior
- Detects the start of each record using the configured start pattern
- Groups all lines following the start pattern until the next start pattern is found
- Outputs each detected block as a single record
- Preserves the original formatting of the input text
Typical Use Cases
- Parsing log files where records star with a known marker
- Processing transaction or audit logs with header-based record boundaries
- Extracting multi-line records from unstructured text files
