首页 > 其他 > 详细

[AWS - DA] Messaging - Kinesis

时间:2021-06-03 10:32:34      阅读:27      评论:0      收藏:0      [点我收藏+]

Overview

技术分享图片

 

技术分享图片

  • Different from SQS, once data into Kinesis, it cannot be deleted. SQS will be deleted after processed
  • Kinesis is regional

技术分享图片

 

Kinesis Producers

  • Data record consists of:
    • Sequence number (unique per partition key within shard)
    • Partition key ?? 
    • Data Blocl (up to 1 MB)
  • Write throughput: 1 MB/sec or 1000 records/sec per shard
  • API: PutRecord

技术分享图片

 

技术分享图片

The same idea as DynamoDB for hot key issue.

 

技术分享图片

There are three ways to handle peak time:

1. Increase shards, nothing wrong with that, just pay more

2. Exponential Backoff

3. Using highly distributed partiton key

Uing SQS to buffer is not a good idea, Kinesis is designed for real time processing... what‘s the point if you add buffer? Kinesis doesn‘t polling the queue as well. So this is not a valid option.

 

Consumers

技术分享图片

Two modes:

  • Shared (Classic) Fan-out Consumer:
    • pull mode
    • number of Consumers will affect the speed
    • API: GetRecords
  • Enhanced Fan-out Consumer
    • push mode
    • each consumer is 2MB/Sec, number of consumer will not affect speed
    • API: SubscribeToShard

技术分享图片

 

技术分享图片

  • One shard can only be read by one KCL
  • One KCL can read multi shards
  • So max number of KCL = No. Shards

技术分享图片

技术分享图片

技术分享图片

 

Kinesis Firehose

技术分享图片

  • Real time write data
  • To Redshift, Elasticsearch, S3
  • Save failed data to S3 bucket
  • Batch write

技术分享图片

技术分享图片

 

Kinesis Data Analytics

  • Wirte SQL
  • Time-series analytics
  • Real-time dashboards

技术分享图片

 

[AWS - DA] Messaging - Kinesis

原文:https://www.cnblogs.com/Answer1215/p/14842758.html

(0)
(0)
   
举报
评论 一句话评论(0
关于我们 - 联系我们 - 留言反馈 - 联系我们:wmxa8@hotmail.com
© 2014 bubuko.com 版权所有
打开技术之扣,分享程序人生!