MongoDB SOURCE () reads data from a MongoDB database and supports both batch and streaming processing modes.
It allows filtering, field selection, and checkpoint-based incremental reading for streaming jobs.

Parameters of MongoDB Source

• Host Name (IP)
  Host name or IP address of the MongoDB server
• Port Number
  Port number on which the MongoDB service runs
• Data Base Name
  Name of the MongoDB database to read from
• Collection Name
  Name of the MongoDB collection to be queried
• User Name
  MongoDB user name (if authentication is enabled)
• Password
  Password for the MongoDB user
• Connection Option
  Additional MongoDB connection options
  Example:

      authSource=admin&readPreference=primary
  

• SSL Enabled
  Enables or disables SSL/TLS connection to MongoDB
• Check Point File
  File name used to store the last processed document ID
  Enables incremental/continuous reading in streaming mode
  If the file does not exist, it will be created automatically
• Field(s) Declaration br List of MongoDB fields to be read
  Each field includes:
  • Field name
  • Field data type
  Only declared fields will be included in the output
• Filter
  MongoDB filter expression used to restrict queried documents
  Equivalent to a MongoDB find() query condition

Comparison operators in the MongoDB filter.

Symbol	Description
$eq	Matches values that are equal to a specified value Syntax: { 'field': { '$eq': value } }
$gt	Matches values that are greater than a specified value Syntax: { 'field': { '$gt': value } }
$gte	Matches values that are greater than or equal to a specified value Syntax: { 'field': { '$gte': value } }
$in	Matches any of the values specified in an array Syntax: { 'field': { '$in': [ value1, value2 ... valueN ] } }
$lt	Matches values that are less than a specified value Syntax: { 'field': { '$lt': value } }
$lte	Matches values that are less than or equal to a specified value Syntax: { 'field': { '$lte': value } }
$ne	Matches all values that are not equal to a specified value Syntax: { 'field': { '$ne': value } }
$nin	Matches none of the values specified in an array Syntax: { 'field': { '$nin': [ value1, value2 ... valueN ] } }

Logical operators in the MongoDB filter.

Symbol	Description
$and	Joins query clauses with a logical AND returns all documents that match the conditions of both clauses Syntax: { '$and': [ { expression1 }, { expression2 } , ... , { expressionN } ] }
$or	Joins query clauses with a logical OR returns all documents that match the conditions of either clause Syntax: { '$or': [ { expression1 }, { expression2 } , ... , { expressionN } ] }
$not	Inverts the effect of a query expression and returns documents that do not match the query expression Syntax: { '$not': [ { expression1 }, { expression2 } , ... , { expressionN } ] }
$nor	Joins query clauses with a logical NOR returns all documents that fail to match both clauses Syntax: { '$nor': [ { expression1 }, { expression2 } , ... , { expressionN } ] }

Example of a MongoDB Filter Expression

1. To get records/data where the content of the field likes is greater than 100.
   

   {'likes' : { '$gt' : 100 }}

2. To get record/data where the content of the field title is equal to 'HGrid247 DE'
   

   {'title' : 'HGrid247 DE' }}

3. To get record/data where content of the field likes is greater than 100 and the content of the field title is equal to 'HGrid247 DE'
   

   {'$and' : [{'likes' : { '$gt' : 100 }},{'title' : 'HGrid247 DE'}]}

4. To get record/data where the content of the field likes is greater than 100 or the content of the field title is equal to 'HGrid247 DE'
   

   {'$or' : [{'likes' : { '$gt' : 100 }},{'title' : 'HGrid247 DE'}]}

5. More Complex Filter
   

   {
    '$and' : 
       [
         { '$or' : [ { 'price' : 0.99 }, { 'price' : 1.99 } ] },
         { '$or' : [ { 'sale' : true }, { 'qty' : { '$lt' : 20 } } ] }
       ]
   }

---

Supported Engines

Map Reduce (Tested on hadoop 2.7.3 MongoDB 6.0)	Spark 1.x RDD	Spark 2.x RDD (Tested on spark 3.0.1 MongoDB 6.0)	Spark 2.x Dataset (Tested on spark 3.0.1 MongoDB 6.0)	Apache Beam 2.29.0 Batch (Tested on MongoDB 6.0)	Apache Beam 2.29.0 Streaming (Tested on MongoDB 6.0)	Apache Flink 1.12.x Batch (Tested on MongoDB 6.0)	Apache Flink 1.12.x Streaming (Tested on MongoDB 6.0)	Apache Tez 0.9.1	PySpark

---