⚠️ Error: Kryo Reference Table Overflow (Spark)

📌 Causes

This error occurs because:

➡️ Reference table in Kryo serializer exceeds integer limit (overflow)

Usually appears in the form of:

NegativeArraySizeException

🧠 Simple Explanation

In Apache Spark :

• Kryo is used as a serializer (for high performance)
• Kryo stores object references in an internal table.

Problem:

• Number of references too large → overflow → error

🎯 When Did It Happen?

Often occurs when:

• Very big data
• Many complex objects
• Large join/aggregation process
• Long pipeline in workflow

✅ Ultimate Solution

🔧 Disable Reference Tracking

Add the following configuration to the workflow:

    spark.kryo.referenceTracking=false

📍 How to Set Up on HGrid247

1. Open Workflow
2. Click on an empty area (canvas)
3. Select Workflow Property
4. Add configuration:

       Configuration Name                  Configuration Value
       -------------------------------------------------------
       spark.kryo.referenceTracking	false
       

   see :https://stackoverflow.com/questions/44414429/spark-negativearraysizeexception

🧩 Impact of Solutions

✔ Avoid overflow
✔ Reduce memory usage for tracking

⚠ Consequences:

• Object reference not tracked
• Can increase data size (because no reference reuse)