首页 > 其他 > 详细

spark开发程序编写注意点

时间:2015-03-19 20:21:02      阅读:395      评论:0      收藏:0      [点我收藏+]


看书时发现书中写的不一定是正确的。

eg:

scala> val line = sc.textFile("hdfs://Spark:9000/user/root/README.md")
15/03/19 20:03:04 INFO MemoryStore: ensureFreeSpace(202004) called with curMem=744765, maxMem=280248975
15/03/19 20:03:04 INFO MemoryStore: Block broadcast_4 stored as values in memory (estimated size 197.3 KB, free 266.4 MB)
15/03/19 20:03:04 INFO MemoryStore: ensureFreeSpace(16322) called with curMem=946769, maxMem=280248975
15/03/19 20:03:04 INFO MemoryStore: Block broadcast_4_piece0 stored as bytes in memory (estimated size 15.9 KB, free 266.3 MB)
15/03/19 20:03:04 INFO BlockManagerInfo: Added broadcast_4_piece0 in memory on a261.datanode.hadoop.qingdao.youku:46352 (size: 15.9 KB, free: 267.2 MB)
15/03/19 20:03:04 INFO BlockManagerMaster: Updated info of block broadcast_4_piece0
line: org.apache.spark.rdd.RDD[String] = hdfs://Spark:9000/user/root/README.md MappedRDD[11] at textFile at <console>:16

scala> val linenum = line.filter(x=>x.contains("spark"))
linenum: org.apache.spark.rdd.RDD[String] = FilteredRDD[12] at filter at <console>:18

最后在执行linenum.count  得出的是什么?

       其实求出的是“spark”这个字符串出现在多少行当中;一行中可能有多个“spark”字符串,切不可认为是“spark”字符串在文章中一共出现了几次。











spark开发程序编写注意点

原文:http://blog.csdn.net/u013361361/article/details/44463563

(0)
(0)
   
举报
评论 一句话评论(0
关于我们 - 联系我们 - 留言反馈 - 联系我们:wmxa8@hotmail.com
© 2014 bubuko.com 版权所有
打开技术之扣,分享程序人生!