首页 > 其他 > 详细

spark MLlib的 pipeline方式

时间:2015-07-09 17:51:05      阅读:155      评论:0      收藏:0      [点我收藏+]
spark mllib的pipeline,是指将多个机器学习的算法串联到一个工作链中,依次执行各种算法。

在Pipeline中的每个算法被称为“PipelineStage”,表示其中的一个算法。PipelineStage分为两种类型,Estimator和Transformer,其中
  • Transformer将数据转换为两一种形式(例如修改格式),以供后续的Estimator使用,统一的转换函数transform;
  • Estimator是由数据得到一个Mode(Mode也是继承于Transformer),有统一触发的函数fit。

然后一个“综合”的算法就可以通过pipeline封装起来。这样做的好处是可以很方便的替换算法。例如,我们在应用中往往只是笼统的期望一个“分类”、”拟合“这样的功能,但不知道是用分类或拟合的那个算法效果是最好的,有了这种pipeline机制后,很方便替换各种分类和拟合算法,从而得到最好的效果。

/**
* :: Experimental ::
* A simple pipeline, which acts as an estimator. A Pipeline consists of a sequence of stages, each
* of which is either an [[Estimator]] or a [[Transformer]]. When [[Pipeline#fit]] is called, the
* stages are executed in order. If a stage is an [[Estimator]], its [[Estimator#fit]] method will
* be called on the input dataset to fit a model. Then the model, which is a transformer, will be
* used to transform the dataset as the input to the next stage. If a stage is a [[Transformer]],
* its [[Transformer#transform]] method will be called to produce the dataset for the next stage.
* The fitted model from a [[Pipeline]] is an [[PipelineModel]], which consists of fitted models and
* transformers, corresponding to the pipeline stages. If there are no stages, the pipeline acts as
* an identity transformer.
*/
@Experimental
class Pipeline(override val uid: String) extends Estimator[PipelineModel] {





spark MLlib的 pipeline方式

原文:http://www.cnblogs.com/zwCHAN/p/4633753.html

(0)
(0)
   
举报
评论 一句话评论(0
关于我们 - 联系我们 - 留言反馈 - 联系我们:wmxa8@hotmail.com
© 2014 bubuko.com 版权所有
打开技术之扣,分享程序人生!