UDF Buffer Assembly () grouped and sorted data using a Java-based buffer function defined by the user.
It processes a group of records at a time and may emit zero or more output records.


Unlike record-based transformations, this assembly processes one group of records at a time and may emit zero or more output records for each group.

This assembly must be preceded by a Group By (or Group By–based) assembly.

Parameters of the UDF Buffer

• UDF Buffer-class
  A Java class that extends

        com.labs247.buffer.BufferFunction247.
              

• Parameter
  A parameter passed to the UDF Buffer class.
• New Output Field(s)
  A list of new output field names along with their corresponding data types.
  Defines the schema of records emitted by the buffer.

Creating a User Defined Buffer

A User Defined Buffer is created by implementing a Java class that:

• Extends <com.labs247.buffer.BufferFunction247>
• Implements the abstract method:
  

  void operate(Iterator<List<Comparable>> records)

  This method is invoked once for each group of records.

Methods Provided by BufferFunction247

The class com.labs247.buffer.BufferFunction247 provides the following methods:

1. public abstract void operate(Iterator<List<Comparable>> records)
   Called once for each group.
   Receives an iterator over all records in the group.
2. public String getParameter()
   Returns the parameter value passed to the buffer.
3. public void prepare()
   Called immediately after the buffer object is created. Used for initialization logic.
4. public void cleanup()
   Called after all groups have been processed.
   Used for resource cleanup.
5. public final void addToOutputCollector(List<Comparable> record)
   Emits a complete output record (multiple fields).
6. public final void addToOutputCollector(Comparable value)
   Emits a single-field output record.
7. public List<String> getIncomingFields()
   Returns the list of field names from the incoming records.
8. public long getRecordNumber()
   Returns the number of records in the current group.

Example: Java-Based User Defined Buffer

The following example calculates minimum, maximum, and average values for a grouped dataset.

package hgrid.plugins.buffer;

import java.util.Iterator;
import java.util.List;

public class MinMaxAvg extends com.labs247.buffer.BufferFunction247 {
    Comparable min;
    Comparable max;
    double sum=0.0;
    
    @Override
    public void operate(Iterator<List<Comparable>> records) {
        while (records.hasNext()){
            List<Comparable> record=records.next();
            Comparable data=record.get(1);
            if (min==null){
                min=data;
            }else{
                min= min.compareTo(data)<0? min:data;
            }
            if (max==null){
                max=data;
            }else{
                max= max.compareTo(data)>0? max:data;
            }
            try{
                sum=sum+Double.parseDouble(data.toString());
            }catch(Exception e){}
        }
        sum=sum/getRecordNumber();
        String result=min.toString()+this.getParameter()+
                      max.toString()+this.getParameter()+
                      sum;
        addToOutputCollector(result);
    }  
}

Deploying the UDF Buffer

To make the UDF Buffer available in the HGrid247-3.5 GUI:

• Package the class into a JAR archive
• Place the JAR file into the plugins directory

      HGrid247-3.5/
      ├── plugins/
      │     └── MinMaxAvg.jar
  

• Restart the HGrid247 application (if required)

Required Dependency

Java-based plug-ins require the following library:

    libCommon/HGrid247CommonLib-3.5.jar

Summary

The UDF Buffer Assembly:

• Processes data group by group
• Enables custom aggregation and analytics logic
• Eliminates the need for predefined buffer functions
• Is ideal for advanced calculations, custom statistics, and complex group-level processing

Supported Engines

Map Reduce	Spark 1.x RDD	Spark 2.x RDD	Spark 2.x Dataset	Apache Beam 2.29.0 Batch	Apache Beam 2.29.0 Streaming	Apache Flink 1.12.0 Batch	Apache Flink 1.12.0 Streaming	Apache Tez 0.9.1	PySpark