[BUG] a CollectionAccumulator has null _list, causing the executor's heartbeat thread to die

[abellina]

I am seeing an issue that I can't readily explain so far. At this stage I am just filing the bug. What we see is a CollectionAccumulator that appears to have a null _list member (according to the exception). More research needs to be done to find the cause. I should mention that our tests will pass, since Spark will add a new Executor, invalidate map output, and go ahead, even passing a test it is in the middle of running.

21/01/14 19:37:26 ERROR Utils: Uncaught exception in thread executor-heartbeater
java.lang.NullPointerException
	at org.apache.spark.util.CollectionAccumulator.isZero(AccumulatorV2.scala:457)
	at org.apache.spark.executor.Executor.$anonfun$reportHeartBeat$2(Executor.scala:902)
	at org.apache.spark.executor.Executor.$anonfun$reportHeartBeat$2$adapted(Executor.scala:902)
	at scala.collection.TraversableLike.$anonfun$filterImpl$1(TraversableLike.scala:256)
	at scala.collection.Iterator.foreach(Iterator.scala:941)
	at scala.collection.Iterator.foreach$(Iterator.scala:941)
	at scala.collection.AbstractIterator.foreach(Iterator.scala:1429)
	at scala.collection.IterableLike.foreach(IterableLike.scala:74)
	at scala.collection.IterableLike.foreach$(IterableLike.scala:73)
	at scala.collection.AbstractIterable.foreach(Iterable.scala:56)
	at scala.collection.TraversableLike.filterImpl(TraversableLike.scala:255)
	at scala.collection.TraversableLike.filterImpl$(TraversableLike.scala:249)
	at scala.collection.AbstractTraversable.filterImpl(Traversable.scala:108)
	at scala.collection.TraversableLike.filterNot(TraversableLike.scala:355)
	at scala.collection.TraversableLike.filterNot$(TraversableLike.scala:355)
	at scala.collection.AbstractTraversable.filterNot(Traversable.scala:108)
	at org.apache.spark.executor.Executor.$anonfun$reportHeartBeat$1(Executor.scala:902)
	at scala.collection.Iterator.foreach(Iterator.scala:941)
	at scala.collection.Iterator.foreach$(Iterator.scala:941)
	at scala.collection.AbstractIterator.foreach(Iterator.scala:1429)
	at scala.collection.IterableLike.foreach(IterableLike.scala:74)
	at scala.collection.IterableLike.foreach$(IterableLike.scala:73)
	at scala.collection.AbstractIterable.foreach(Iterable.scala:56)
	at org.apache.spark.executor.Executor.reportHeartBeat(Executor.scala:896)
	at org.apache.spark.executor.Executor.$anonfun$heartbeater$1(Executor.scala:200)
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
	at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1932)
	at org.apache.spark.Heartbeater$$anon$1.run(Heartbeater.scala:46)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

[revans2]

Just FYI: I ran into a similar issue with some of the TPC-DS tests, but it showed up only on the CPU run, not on the GPU run. I suspected that it was a race condition in spark itself, but I have not really pushed much on it.

[gerashegalov]

@abellina what Spark version was this issue encountered with?

[abellina]

@gerashegalov it was using spark 3.0.1.

[gerashegalov]

This issue is explained in SPARK-20977 where an incompletely initialized accumulator is registered violation JLS Section 17.5.3