首页 > 其他 > 详细

Spark以yarn方式运行时抛出异常

时间:2019-06-22 21:29:15      阅读:402      评论:0      收藏:0      [点我收藏+]

Spark以yarn方式运行时抛出异常:

cluster.YarnClientSchedulerBackend: Yarn application has already exited with state FINISHED!

解决方案:

主要是给节点分配的内存少,yarn kill了spark application。

1.停止hadoop集群

[root@node06 bin]# stop-all.sh

2.给yarn-site.xml增加配置:

<property>
    <name>yarn.nodemanager.pmem-check-enabled</name>
    <value>false</value>
</property>
<property>
    <name>yarn.nodemanager.vmem-check-enabled</name>
    <value>false</value>
    <description>Whether virtual memory limits will be enforced for containers</description>
</property>
<property>
    <name>yarn.nodemanager.vmem-pmem-ratio</name>
    <value>4</value>
    <description>Ratio between virtual memory to physical memory when setting memory limits for containers</description>
</property>

3.将master上将l修改后的yarn-site.xm文件覆盖到各个slaves节点

[root@node06 hadoop]# scp yarn-site.xml node07:`pwd`
yarn-site.xml                                                                                                              100% 1565     1.5KB/s   00:00    
[root@node06 hadoop]# scp yarn-site.xml node08:`pwd`
yarn-site.xml                                                                                                              100% 1565     1.5KB/s   00:00    
[root@node06 hadoop]# scp yarn-site.xml node09:`pwd`

 

4.重新启动hadoop集群

[root@node06 hadoop]# start-all.sh

5.重新向yarn提交spark任务

[root@node06 bin]# ./spark-submit --master yarn --class org.apache.spark.examples.SparkPi ../lib/spark-examples-1.6.0-hadoop2.6.0.jar 100

 

Spark以yarn方式运行时抛出异常

原文:https://www.cnblogs.com/liuys635/p/11070168.html

(0)
(0)
   
举报
评论 一句话评论(0
关于我们 - 联系我们 - 留言反馈 - 联系我们:wmxa8@hotmail.com
© 2014 bubuko.com 版权所有
打开技术之扣,分享程序人生!