Overview
- Different from SQS, once data into Kinesis, it cannot be deleted. SQS will be deleted after processed
- Kinesis is regional
Kinesis Producers
- Data record consists of:
- Sequence number (unique per partition key within shard)
- Partition key ??
- Data Blocl (up to 1 MB)
- Write throughput: 1 MB/sec or 1000 records/sec per shard
- API: PutRecord
The same idea as DynamoDB for hot key issue.
There are three ways to handle peak time:
1. Increase shards, nothing wrong with that, just pay more
2. Exponential Backoff
3. Using highly distributed partiton key
Uing SQS to buffer is not a good idea, Kinesis is designed for real time processing... what‘s the point if you add buffer? Kinesis doesn‘t polling the queue as well. So this is not a valid option.
Consumers
Two modes:
- Shared (Classic) Fan-out Consumer:
- pull mode
- number of Consumers will affect the speed
- API: GetRecords
- Enhanced Fan-out Consumer
- push mode
- each consumer is 2MB/Sec, number of consumer will not affect speed
- API: SubscribeToShard
- One shard can only be read by one KCL
- One KCL can read multi shards
- So max number of KCL = No. Shards
Kinesis Firehose
- Real time write data
- To Redshift, Elasticsearch, S3
- Save failed data to S3 bucket
- Batch write
Kinesis Data Analytics
- Wirte SQL
- Time-series analytics
- Real-time dashboards
[AWS - DA] Messaging - Kinesis
原文：https://www.cnblogs.com/Answer1215/p/14842758.html
