首页 > 其他 > 详细

一个MapReduce 程序示例 细节决定成败

时间：2016-05-28 02:04:16      阅读：325      评论：0      收藏：0      [点我收藏+]
最近在看MapReduce，想起一直都是Copy 然后修改的方法来写。突然想试试自己动手写一个及其简单的mr程序。
细节决定成败啊，不试不知道，一试才能发现平时注意不到的细节。
下面是我用了很快时间写好的一个程序，注意，这份是有问题的！

点击(此处)折叠或打开

  1. package wordcount;

  3. import java.io.IOException;

  5. import org.apache.commons.lang.StringUtils;
  6. import org.apache.hadoop.conf.Configuration;
  7. import org.apache.hadoop.conf.Configured;
  8. import org.apache.hadoop.fs.Path;
  9. import org.apache.hadoop.io.LongWritable;
  10. import org.apache.hadoop.io.Text;
  11. import org.apache.hadoop.mapreduce.Job;
  12. import org.apache.hadoop.mapreduce.Mapper;
  13. import org.apache.hadoop.mapreduce.Reducer;
  14. import org.apache.hadoop.mapreduce.lib.input.TextInputFormat;
  15. import org.apache.hadoop.mapreduce.lib.output.TextOutputFormat;
  16. import org.apache.hadoop.util.Tool;
  17. import org.apache.hadoop.util.ToolRunner;
  18. import org.apache.log4j.Logger;

  20. public class MyWordCountJob extends Configured implements Tool {
  21.         Logger log = Logger.getLogger(MyWordCountJob.class);

  23.         public class MyWordCountMapper extends
  24.                         Mapper<LongWritable, Text, LongWritable, Text> {
  25.                 Logger log = Logger.getLogger(MyWordCountJob.class);

  27.                 LongWritable mapKey = new LongWritable();
  28.                 Text mapValue = new Text();
  29.                 @Override
  30.                 protected void map(LongWritable key, Text value, Context context)
  31.                                 throws IOException, InterruptedException {
  32.                         mapKey.set(key.get());
  33.                         mapValue.set(value.toString());
  34.                         log.info("Mapper: mapKey--" + mapKey.get() + "mapValue --"+ mapValue.toString());
  35.                         context.write(mapKey, mapValue);
  36.                 }

  38.         }

  40.         public class MyWordCountReducer extends Reducer<LongWritable, Text, LongWritable, Text> {

  42.                 @Override
  43.                 protected void reduce(LongWritable key, Iterable<Text> values,Context context)
  44.                                 throws IOException, InterruptedException {
  45.                         for(Text value :values)
  46.                                 context.write(key, value);
  47.                 }
  48.         }

  50.         @Override
  51.         public int run(String[] args) throws Exception {
  52.                 log.info("begin to run");
  53.                 Job job = Job.getInstance(getConf(), "MyWordCountJob");
  54.                 job.setJarByClass(MyWordCountJob.class);

  56.                 Path inPath = new Path("demos/pigdemo.txt");
  57.                 Path outPath = new Path("demos/pigdemoOut.txt");

  59.                 outPath.getFileSystem(getConf()).delete(outPath,true);
  60.                 TextInputFormat.setInputPaths(job, inPath);
  61.                 TextOutputFormat.setOutputPath(job, outPath);


  64.                 job.setMapperClass(MyWordCountJob.MyWordCountMapper.class);
  65.                 job.setReducerClass(MyWordCountJob.MyWordCountReducer.class);
  66.                 job.setInputFormatClass(TextInputFormat.class);
  67.                 job.setOutputFormatClass(TextOutputFormat.class);

  69.                 job.setMapOutputKeyClass(LongWritable.class);
  70.                 job.setMapOutputValueClass(Text.class);
  71.                 job.setOutputKeyClass(LongWritable.class);
  72.                 job.setOutputValueClass(Text.class);
  73.                 return job.waitForCompletion(true)?0:1;
  74.         }
  75.         public static void main(String [] args){
  76.                 int result = 0;
  77.                 try {
  78.                         result = ToolRunner.run(new Configuration(), new MyWordCountJob(), args);
  79.                 } catch (Exception e) {
  80.                         e.printStackTrace();
  81.                 }
  82.                 System.exit(result);
  83.         }

  85. }
写完成编译，打包然后运行。

点击(此处)折叠或打开

  1. 16/05/10 22:43:46 INFO mapreduce.Job: Running job: job_1462517728035_0033
  2. 16/05/10 22:43:54 INFO mapreduce.Job: Job job_1462517728035_0033 running in uber mode : false
  3. 16/05/10 22:43:54 INFO mapreduce.Job: map 0% reduce 0%
  4. 16/05/10 22:43:58 INFO mapreduce.Job: Task Id : attempt_1462517728035_0033_m_000000_0, Status : FAILED
  5. Error: java.lang.RuntimeException: java.lang.NoSuchMethodException: wordcount.MyWordCountJob$MyWordCountMapper.<init>()
  6.         at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:131)
  7.         at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:721)
  8.         at org.apache.hadoop.mapred.MapTask.run(MapTask.java:339)
  9.         at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:162)
  10.         at java.security.AccessController.doPrivileged(Native Method)
  11.         at javax.security.auth.Subject.doAs(Subject.java:396)
  12.         at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
  13.         at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:157)
  14. Caused by: java.lang.NoSuchMethodException: wordcount.MyWordCountJob$MyWordCountMapper.<init>()
  15.         at java.lang.Class.getConstructor0(Class.java:2706)
  16.         at java.lang.Class.getDeclaredConstructor(Class.java:1985)
  17.         at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:125)
  18.         ... 7 more

  20. 16/05/10 22:44:02 INFO mapreduce.Job: Task Id : attempt_1462517728035_0033_m_000000_1, Status : FAILED
  21. Error: java.lang.RuntimeException: java.lang.NoSuchMethodException: wordcount.MyWordCountJob$MyWordCountMapper.<init>()
  22.         at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:131)
  23.         at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:721)
  24.         at org.apache.hadoop.mapred.MapTask.run(MapTask.java:339)
  25.         at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:162)
  26.         at java.security.AccessController.doPrivileged(Native Method)
  27.         at javax.security.auth.Subject.doAs(Subject.java:396)
  28.         at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
  29.         at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:157)
  30. Caused by: java.lang.NoSuchMethodException: wordcount.MyWordCountJob$MyWordCountMapper.<init>()
  31.         at java.lang.Class.getConstructor0(Class.java:2706)
  32.         at java.lang.Class.getDeclaredConstructor(Class.java:1985)
  33.         at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:125)
  34.         ... 7 more

  36. 16/05/10 22:44:07 INFO mapreduce.Job: Task Id : attempt_1462517728035_0033_m_000000_2, Status : FAILED
  37. Error: java.lang.RuntimeException: java.lang.NoSuchMethodException: wordcount.MyWordCountJob$MyWordCountMapper.<init>()
  38.         at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:131)
  39.         at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:721)
  40.         at org.apache.hadoop.mapred.MapTask.run(MapTask.java:339)
  41.         at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:162)
  42.         at java.security.AccessController.doPrivileged(Native Method)
  43.         at javax.security.auth.Subject.doAs(Subject.java:396)
  44.         at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
  45.         at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:157)
  46. Caused by: java.lang.NoSuchMethodException: wordcount.MyWordCountJob$MyWordCountMapper.<init>()
  47.         at java.lang.Class.getConstructor0(Class.java:2706)
  48.         at java.lang.Class.getDeclaredConstructor(Class.java:1985)
  49.         at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:125)
  50.         ... 7 more

  52. 16/05/10 22:44:14 INFO mapreduce.Job: map 100% reduce 100%
  53. 16/05/10 22:44:14 INFO mapreduce.Job: Job job_1462517728035_0033 failed with state FAILED due to: Task failed task_1462517728035_0033_m_000000
  54. Job failed as tasks failed. failedMaps:1 failedReduces:0

  56. 16/05/10 22:44:15 INFO mapreduce.Job: Counters: 6
  57.         Job Counters
  58.                 Failed map tasks=4
  59.                 Launched map tasks=4
  60.                 Other local map tasks=3
  61.                 Data-local map tasks=1
  62.                 Total time spent by all maps in occupied slots (ms)=99584
  63.                 Total time spent by all reduces in occupied slots (ms)=0

上面的问题百思不得甚解，完全不知道什么地方错了。
然后跟之前copy的代码进行比对。终于找出了问题所在！
注意Mapper 与 Reducer 类写成内部类，一定要加static ！！！！
留个小任务，查看一下生成的结果文件可以发现什么？
使用TextInputFormat时，进入map 函数中的LongWritable类型的key 代表什么？
经实验确认这个key 其实是本行的首字符在整个文件中的偏移量。

一个MapReduce 程序示例 细节决定成败

原文：http://blog.itpub.net/30066956/viewspace-2107549/
踩
(0)
赞
(0)
   
举报
评论 一句话评论（0）
分享档案
更多>
2021年09月23日 (328)
2021年09月24日 (313)
2021年09月17日 (191)
2021年09月15日 (369)
2021年09月16日 (411)
2021年09月13日 (439)
2021年09月11日 (398)
2021年09月12日 (393)
2021年09月10日 (160)
2021年09月08日 (222)
最新文章
更多>
教程昨日排行
更多>
友情链接
汇智网   PHP教程   插件网  
关于我们 - 联系我们 - 留言反馈 - 联系我们:wmxa8@hotmail.com
© 2014 bubuko.com 版权所有
打开技术之扣，分享程序人生！