Web Crawler Source () rreads (crawls) streaming data from websites.
It can crawl web content starting from a fixed URL or dynamically from URLs stored in a database.
The crawler follows links up to a configurable depth and streams the fetched content into the data pipeline.

Parameters of Web Crawler Source

• Source of Web-site
  Indicates the type of source website, either Constant or FromDB
  • Constant
    Url Source
    The starting URL of the website to be crawled
    Example:

        https://example.com
    

  • FromDB
    • Jdbc Driver
      JDBC driver class name for database connectivity
    • Url Connection
      JDBC connection URL of the database
    • User Name
      Database user name
    • Password
      Database password
    • Table Name
      Name of the table containing URLs to be crawled
    • Reference Field Field/column that contains the URL value
    • Flag Field
      Field/column containing an integer flag
      Only records where the flag value is 0 will be crawled
      This field can be updated externally to control crawling behavior
• Max Depth
  Maximum link depth to crawl from the starting URL
  Depth 0 means only the source page is crawled
• Exclusive Pattern
  Regular expression pattern for URLs to be excluded from crawling
  Useful for skipping unwanted pages (e.g., logout, admin, media files)
• Crawl External site
  Indicates whether links pointing to external domains should be crawled
• Number of Mapper
  Number of parallel crawler workers
  Controls crawling parallelism and performance

Supported Engines

Map Reduce	Spark 1.x RDD	Spark 2.x RDD	Spark 2.x Dataset	Apache Beam 2.29.0 Batch	Apache Beam 2.29.0 Streaming	Apache Flink 1.12.0 Batch	Apache Flink 1.12.0 Streaming	Apache Tez 0.9.1	PySpark