[BUG] integration_tests seeing NPE in the executor-heartbeat thread

[abellina]

I am seeing an issue that I can't readily explain so far. At this stage I am just filing the bug. What we see is a CollectionAccumulator that appears to have a null _list member (according to the exception). More research needs to be done to find the cause. I should mention that our tests will pass, since Spark will add a new Executor, invalidate map output, and go ahead, even passing a test it is in the middle of running.

21/01/14 19:37:26 ERROR Utils: Uncaught exception in thread executor-heartbeater
java.lang.NullPointerException
	at org.apache.spark.util.CollectionAccumulator.isZero(AccumulatorV2.scala:457)
	at org.apache.spark.executor.Executor.$anonfun$reportHeartBeat$2(Executor.scala:902)
	at org.apache.spark.executor.Executor.$anonfun$reportHeartBeat$2$adapted(Executor.scala:902)
	at scala.collection.TraversableLike.$anonfun$filterImpl$1(TraversableLike.scala:256)
	at scala.collection.Iterator.foreach(Iterator.scala:941)
	at scala.collection.Iterator.foreach$(Iterator.scala:941)
	at scala.collection.AbstractIterator.foreach(Iterator.scala:1429)
	at scala.collection.IterableLike.foreach(IterableLike.scala:74)
	at scala.collection.IterableLike.foreach$(IterableLike.scala:73)
	at scala.collection.AbstractIterable.foreach(Iterable.scala:56)
	at scala.collection.TraversableLike.filterImpl(TraversableLike.scala:255)
	at scala.collection.TraversableLike.filterImpl$(TraversableLike.scala:249)
	at scala.collection.AbstractTraversable.filterImpl(Traversable.scala:108)
	at scala.collection.TraversableLike.filterNot(TraversableLike.scala:355)
	at scala.collection.TraversableLike.filterNot$(TraversableLike.scala:355)
	at scala.collection.AbstractTraversable.filterNot(Traversable.scala:108)
	at org.apache.spark.executor.Executor.$anonfun$reportHeartBeat$1(Executor.scala:902)
	at scala.collection.Iterator.foreach(Iterator.scala:941)
	at scala.collection.Iterator.foreach$(Iterator.scala:941)
	at scala.collection.AbstractIterator.foreach(Iterator.scala:1429)
	at scala.collection.IterableLike.foreach(IterableLike.scala:74)
	at scala.collection.IterableLike.foreach$(IterableLike.scala:73)
	at scala.collection.AbstractIterable.foreach(Iterable.scala:56)
	at org.apache.spark.executor.Executor.reportHeartBeat(Executor.scala:896)
	at org.apache.spark.executor.Executor.$anonfun$heartbeater$1(Executor.scala:200)
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
	at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1932)
	at org.apache.spark.Heartbeater$$anon$1.run(Heartbeater.scala:46)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

[abellina]

I must have saved this issue twice. Closing in favor of #1522, which is an exact duplicate.