@rob_rich
Planning for the varied levels of once-data-processing with Kafka
by Rob Richardson
About Me
Rob Richardson is a software craftsman building web properties in ASP.NET and Node, React and Vue. He's a frequent speaker at conferences, user groups, and community events, and a diligent teacher and student of high quality software development. You can find this and other talks on https://robrich.org/presentations and follow him on twitter at @rob_rich.
Spoiler:
We're compensating for failure
There are only two hard problems in distributed systems: 2. Exactly-once delivery 1. Guaranteed order of messages 2. Exactly-once delivery
source: twitter.com/mathiasverraes/status/632260618599403520B.K.
(Before Kafka)
A.K
(After Kafka)
Let's zoom in
to one of these messages
As a system,
I need to do these things:
| Read message from Kafka topic | Add data to database | Tell Kafka I'm done with the message | Business logic |
What order do we put them in?
What if there's a failure on this step?
That was hard
MemSQL:
Exactly Once from Kafka
The easy button
MemSQL
Very fast, Distributed SQL Database
Download free:
https://msql.co/kafka-meetup
- On-prem, in containers, or in the cloud
- row-store tables in memory
persisted to disk - column-store tables on disk
heavily cached - ACID compliant transactions
- data partitioned across commodity hardware
- free to 128 gig ram across 4 leaf nodes
System Architecture
| stuff | > |
|
> |
|
> | stuff |