首页 > 其他 > 详细

spark coalesce和repartition的区别和使用场景

时间:2019-08-20 23:58:52      阅读:567      评论:0      收藏:0      [点我收藏+]

区别:

repartition底层调用的是coalesce方法,默认shuffle

def repartition(numPartitions: Int)(implicit ord: Ordering[T] = null): RDD[T] = withScope {
coalesce(numPartitions, shuffle = true)
}

 

coalesce方法的shuffle参数默认为false,默认不shuffle

def coalesce(numPartitions: Int, shuffle: Boolean = false)(implicit ord: Ordering[T] = null)
    : RDD[T] = withScope {
  if (shuffle) {
    /** Distributes elements evenly across output partitions, starting from a random partition. */
    val distributePartition = (index: Int, items: Iterator[T]) => {
      var position = (new Random(index)).nextInt(numPartitions)
      items.map { t =>
        // Note that the hash code of the key will just be the key itself. The HashPartitioner
        // will mod it with the number of total partitions.
        position = position + 1
        (position, t)
      }
    } : Iterator[(Int, T)]
 
    // include a shuffle step so that our upstream tasks are still distributed
    new CoalescedRDD(
      new ShuffledRDD[Int, T, T](mapPartitionsWithIndex(distributePartition),
      new HashPartitioner(numPartitions)),
      numPartitions).values
  } else {
    new CoalescedRDD(this, numPartitions)
  }
}

 

使用场景:

如果你减少分区数,考虑使用coalesce,这样可以避免执行shuffle。但是假如内存不够用,可能会引起内存溢出。

spark coalesce和repartition的区别和使用场景

原文:https://www.cnblogs.com/Alcesttt/p/11386049.html

(0)
(0)
   
举报
评论 一句话评论(0
关于我们 - 联系我们 - 留言反馈 - 联系我们:wmxa8@hotmail.com
© 2014 bubuko.com 版权所有
打开技术之扣,分享程序人生!