首页 > 其他 > 详细

使用kafka作为生产者生产数据到hdfs(单节点)

时间:2019-11-13 13:20:19      阅读:76      评论:0      收藏:0      [点我收藏+]

查看kafka官网的userguide

agent.sources = kafkaSource

agent.channels = memoryChannel

agent.sinks = hdfsSink

agent.sources.kafkaSource.type = org.apache.flume.source.kafka.KafkaSource

agent.sources.kafksSource.zookeeperConnect = 192.168.57.11:2181

agent.sources.kafkaSource.topic = tets_zhang

agent.sources.kafkaSource.groupld = test-consumer-group

agent.sources.kafkaSource.kafka.consumer.timeout.ms = 100

 

agent.channels.memoryChannel.type = memory

agent.channels.memoryChannel.capacity = 100

agent.channels.memoryChannel.transactionCapacity = 100

 

agent.sinks.hdfsSink.type = hdfs

agent.sinks..hdfsSink.hdfs.path = hdfs://beicai/test/zhang

agent.sinks.hdfsSink.hdfs.writeFormat = Text

agent.sinks.hdfsSink.hdfs.fileType = DataStream

 

agent.sinks.hdfsSink.hdfs..rollSize = 1024

agent.sinks.hdfsSink.hdfs.rollCount = 0

agent.sinks.hdfsSink.hdfs.rollinterval = 60

agent.sinks.hdfsSink.hdfs.filePrefix = test

agent.sinks.hdfsSink.hdfs.fileSuffix = .data

 

agent.sinks.hdfsSink.hdfs.inUserPrefix = _

agent.sinks.hdfsSink.hdfs.inUserSuffix=

agent.sinks.hdfsSink.hdfs.fileType = DataStream

agent.sinks.hdfsSink.hdfs.writeFormat = TEXT

agent.sinks.hdfsSink.hdfs.rollnterval = 1

agent.sinks.sink1.hdfs.filePrefix = A

 

agent.sources.kafkaSource.channels = memoryChannel

agent.sinks.hdfsSink.channel = memoryChanne

使用kafka作为生产者生产数据到hdfs(单节点)

原文:https://www.cnblogs.com/zdy0521/p/11847948.html

(0)
(0)
   
举报
评论 一句话评论(0
关于我们 - 联系我们 - 留言反馈 - 联系我们:wmxa8@hotmail.com
© 2014 bubuko.com 版权所有
打开技术之扣,分享程序人生!