首页 > 其他 > 详细

Spark Transformations

时间:2016-10-23 20:14:36      阅读:219      评论:0      收藏:0      [点我收藏+]

 

We all know the following fact:

1, RDD are immutable

2, Never modify RDD in place

3, Transform RDD to another RDD

There are 2 different transformations for RDD, one is narrow transformation:

技术分享

transformations like map, flatMap, filter all are narrow transformation, which means shuffle won‘t happen, so it‘s fast, it‘s speed just depends on:

1, availability of local memory

2, CPU speed

another is wide transfomration:

技术分享

transformations like groupByKey, reduceByKey, repartition all are wide tranformation, the network speed in shuffle is the key to it‘s speed, so it‘s slower

the final comparison:

技术分享

Spark Transformations

原文:http://www.cnblogs.com/dingjunnan/p/5990431.html

(0)
(0)
   
举报
评论 一句话评论(0
关于我们 - 联系我们 - 留言反馈 - 联系我们:wmxa8@hotmail.com
© 2014 bubuko.com 版权所有
打开技术之扣,分享程序人生!