首页 > 其他 > 详细

【hadoop】 running beyond virtual memory错误原因及解决办法

时间:2015-03-12 11:03:48      阅读:400      评论:0      收藏:0      [点我收藏+]

问题描述:

   在hadoop中运行应用,出现了running beyond virtual memory错误。提示如下:

Container [pid=28920,containerID=container_1389136889967_0001_01_000121] is running beyond virtual memory limits. Current usage: 1.2 GB of 1 GB physical memory used; 2.2 GB of 2.1 GB virtual memory used. Killing container.

原因:从机上运行的
Container试图使用过多的内存,而被NodeManager kill掉了。
[摘录] The NodeManager is killing your container. It sounds like you are trying to use hadoop streaming which is running as a child process of the map-reduce task. The NodeManager monitors the entire process tree of the task and if it eats up more memory than the maximum set in mapreduce.map.memory.mb or mapreduce.reduce.memory.mb respectively, we would expect the Nodemanager to kill the task, otherwise your task is stealing memory belonging to other containers, which you don‘t want.
解决方法:
mapred-site.xml中设置map和reduce任务的内存配置如下:

<property>
  <name>mapreduce.map.memory.mb</name>
  <value>1536</value>
</property>
<property>
  <name>mapreduce.map.java.opts</name>
  <value>-Xmx1024M</value>
</property>
<property>
  <name>mapreduce.reduce.memory.mb</name>
  <value>3072</value>
</property>
<property>
  <name>mapreduce.reduce.java.opts</name>
  <value>-Xmx2560M</value>
</property>


附录:

Container is running beyond memory limits

http://stackoverflow.com/questions/21005643/container-is-running-beyond-memory-limits

【hadoop】 running beyond virtual memory错误原因及解决办法

原文:http://www.cnblogs.com/scw2901/p/4331682.html

(0)
(0)
   
举报
评论 一句话评论(0
关于我们 - 联系我们 - 留言反馈 - 联系我们:wmxa8@hotmail.com
© 2014 bubuko.com 版权所有
打开技术之扣,分享程序人生!