Apache Beam in HGrid247-3.5

Apache Beam is a unified pipelineprogramming modelfor processing data inbatch and streaming.

Beam provides: Runners allow the same pipeline to run on multiple distributed processing engines.

Apache Beam Architecture

The Apache Beam architecture consists of three main components. 
    Beam Pipeline
          ↓
    Beam SDK
          ↓
    Runner
          ↓
    Distributed Engine
Runners can run pipelines on various platforms such as:

Apache Beam Pipeline Model

Pipeline Beam consists of several components: 
    Component       Description
    ---------------------------------------------
    Pipeline        definition of data processing flow
    PCollection     distributed dataset
    Transform       data transformation operations
    Runner          pipeline execution engine
Pipeline example: 
    Input Source
          ↓
    Transform
          ↓
    Aggregation
          ↓
    Output Sink

Apache Beam in HGrid247

As of HGrid247-3.5 , workflows can be compiled into Apache Beam programs .
br> The output is usually: 
    workshop_beam.jar

Running Apache Beam Job

Beam Pipelines can be run using a variety of runners .
  1. Direct Runner
    Direct Runner is used to run pipelines locally (usually for testing).

    Command: 
        java -cp workshop_beam.jar;BeamRunner\direct-runner\* workshop.Lab01 --input=file:///home/input/* --output=file:///home/output
    Explanation: 
        Parameter                   Meaning
    workshop_beam.jar           Beam app
    BeamRunner\direct-runner\*  Direct runner Beam libraries
    workshop.Lab01              main class
    --input                     data input location
    --output                    output location
  2. Spark Runner
    Beam Pipelines can also be run using Apache Spark as the execution engine.

    Command: 
        spark-submit --class workshop.Lab01 --master yarn \
    --jars BeamRunner/spark-runner/beam-runners-spark-2.29.0.jar,\
    BeamRunner/spark-runner/beam-sdks-java-io-hadoop-file-system-2.29.0.jar \
    workshop_beam.jar \
    --runner=SparkRunner \
    --input=hdfs:///user/user01/input/* \
    --output=hdfs:///user/user01/output
    Runners used: 
        SparkRunner
    Input and output using HDFS .
  3. Flink Runner
    Beam Pipelines can also be run using Apache Flink .

    Command: 
        flink run -c workshop.Lab01 \
    -j /user01/workshop_beam.jar;/user01/BeamRunner/flink-runer/beam-runners-flink-1.9-2.29.0.jar \
    --runner=FlinkRunner \
    --input=hdfs:///user/user01/data/input \
    --output=hdfs:///user/user01/output
    Runners used: 
        FlinkRunner
  4. Google Dataflow Runner
    Beam Pipelines can also be run on the cloud using Google Cloud Dataflow .

    Command: 
    java -jar workshop_beam.jar \
--runner=DataflowRunner \
--project=<YOUR_GCP_PROJECT_ID> \
--region=<GCP_REGION> \
--tempLocation=gs://<YOUR_GCS_BUCKET>/temp/
    Important parameters: 
        Parameter           Description
    ------------------------------------------
    project             ID project Google Cloud
    region              region cloud
    tempLocation	temporary file location in GCS
  5. Samza Runner
    Pipeline Beam can also be run using Apache Samza .

    Command: 
        mvn exec:java -Dexec.mainClass=wokshop.Lab01 \
    -Psamza-runner \
    -Dexec.args="--runner=SamzaRunner \
    --inputFile=/path/to/input \
    --output=/path/to/counts"
    Runners used: 
        SamzaRunner

Apache Beam Runner Comparison

    Runner              Use Case
    DirectRunner	Local testing
    SparkRunner         Batch processing pada Spark cluster
    FlinkRunner         Streaming/Batch processing
    DataflowRunner	Cloud-native pipeline
    SamzaRunner         Event-driven streaming

Apache Beam Advantages

Using Apache Beam provides several advantages.

Unified Batch and Streaming

The same pipeline can be used for:

Portable Pipelines

Pipelines can be run on multiple runners.

Scalable

Can be run on large clusters.

Cloud Integration

Beam can run on: