前些天发现了一个巨牛的人工智能学习网站，通俗易懂，风趣幽默，忍不住给大家分享一下。点击跳转到网站：https://www.captainai.net/dongkelun
前言
总结Spark开发中遇到的异常及解决办法，之前也写过几篇，之所以不再一个异常写一篇博客，是因为现在Spark用的比较熟悉了一些，觉得没必要把异常信息写那么详细了，所以就把异常总结在一篇博客里了，这样既能备忘也方便查找。
1、之前的几篇
2、 spark.executor.memoryOverhead
堆外内存（默认是executor内存的10%），当数据量比较大的时候，如果按默认的就会有下面的异常，导致程序崩溃
异常
1 | Container killed by YARN for exceeding memory limits. 1.8 GB of 1.8 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. |
解决
具体值根据实际情况配置
新版1
--conf spark.executor.memoryOverhead=2048
旧版1
--conf spark.yarn.executor.memoryOverhead=2048
新版如果用旧版，会:
1 | WARN SparkConf: The configuration key 'spark.yarn.executor.memoryOverhead' has been deprecated as of Spark 2.3 and may be removed in the future. Please use the new key 'spark.executor.memoryOverhead' instead. |
3、No more replicas available for rdd_
异常
1 | 19/01/08 12:36:46 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_3250_73 ! |
解决
增大executor的内存1
--executor-memory 4G
4、Failed to allocate a page
异常
1 | 19/01/09 09:12:39 WARN TaskMemoryManager: Failed to allocate a page (1048576 bytes), try again. |
解决
增大driver的内存1
--driver-memory 6G
参考
TaskMemoryManager: Failed to allocate a page, try again
5、Uncaught exception in thread task-result-getter-3
异常
1 | 19/01/10 09:31:50 ERROR Utils: Uncaught exception in thread task-result-getter-3 |
解决
增大driver的内存1
--driver-memory 6G
具体看参考
参考
6、spark.driver.maxResultSize
异常
1 | ERROR TaskSetManager: Total size of serialized results of 30 tasks (1108.5 MB) is bigger than spark.driver.maxResultSize (1024.0 MB) |
解决
增大spark.driver.maxResultSize1
--conf spark.driver.maxResultSize=2G
7、Dropping event from queue eventLog
异常
1 | 19/05/20 11:49:54 ERROR AsyncEventQueue: Dropping event from queue eventLog. This likely means one of the listeners is too slow and cannot keep up with the rate at which tasks are being started by the scheduler. |
解决
增大spark.scheduler.listenerbus.eventqueue.capacity(默认为10000)1
--conf spark.scheduler.listenerbus.eventqueue.capacity=100000
- 旧版用spark.scheduler.listenerbus.eventqueue.size
1
19/05/21 14:38:15 WARN SparkConf: The configuration key 'spark.scheduler.listenerbus.eventqueue.size' has been deprecated as of Spark 2.3 and may be removed in the future. Please use the new key 'spark.scheduler.listenerbus.eventqueue.capacity' instead.
具体看参考
参考
Spark里Histroy Server丢task，job和Stage问题调研
本文由 董可伦 发表于 伦少的博客 ,采用署名-非商业性使用-禁止演绎 3.0进行许可。
非商业转载请注明作者及出处。商业转载请联系作者本人。
本文标题：Spark 异常总结及解决办法
本文链接：https://dongkelun.com/2019/01/09/sparkExceptions/
欢迎关注我的公众号