ElasticSearch Sink () is used to store processed data into the ElasticSearch system . ElasticSearch is a distributed search and analytics engine often used for real-time search, log analytics, system monitoring, and big data indexing .

In the HGrid247 pipeline , ElasticSearch Sink is usually placed at the end of the workflow to store the transformed data so that it can be quickly queried using the REST API.

Parameters of ElasticSearch Sink

1. Protocol
   This parameter specifies the type of communication protocol used to access ElasticSearch.
   The default protocol is:

       http
   

   However, if you use a secure connection you can use:

       https
   

2. Host/IP
   This parameter contains the host address or IP of the ElasticSearch server that will receive the data.
   Example:

       Host/IP : 192.168.1.50
   

   or

       Host/IP : elasticsearch.server.local  
   

3. Port Number
   This parameter specifies the port used by the ElasticSearch server .
   The default ports for ElasticSearch are:

       9200    
   

4. Index Name
   This parameter specifies the name of the ElasticSearch index that will store the data.
   Indexes in ElasticSearch are similar to tables in a relational database .
5. Type Name
   This parameter specifies the data type in index .
   In older versions of ElasticSearch (before 7.x), a single index could have multiple types.
   Example:

       Type Name : transaction
   

   However, the latest version of ElasticSearch usually only uses one type called:

       _doc
   

6. Basic Authentication :
   This parameter is used if ElasticSearch requires login authentication .
   • User Name
     Specify a username for authentication.
   • Password
     Password of the user.
7. SSL Authentication :
   Used if the ElasticSearch connection uses HTTPS or Secure Socket Layer (SSL) .
   • JKS Path
     The location of the Java Key Store (JKS) file that contains the security certificate.
     Example:

         JKS Path : /opt/security/elasticsearch.jks   
     

   • SSL Password
     Password to open JKS file .

Supported Engines

Map Reduce	Spark 1.x RDD	Spark 2.x RDD	Spark 2.x Dataset	Apache Beam 2.29.0 Batch	Apache Beam 2.29.0 Streaming	Apache Flink 1.12.0 Batch	Apache Flink 1.12.0 Streaming	Apache Tez 0.9.1	PySpark