Planning for the varied levels of once-data-processing with Kafka

by Rob Richardson



About Me

Rob Richardson

Rob Richardson is a software craftsman building web properties in ASP.NET and Node, React and Vue. He's a frequent speaker at conferences, user groups, and community events, and a diligent teacher and student of high quality software development. You can find this and other talks on https://robrich.org/presentations and follow him on twitter at @rob_rich.


We're compensating for failure

There are only two hard problems in distributed systems: 2. Exactly-once delivery 1. Guaranteed order of messages 2. Exactly-once delivery

source: twitter.com/mathiasverraes/status/632260618599403520

(Before Kafka)

(After Kafka)

Let's zoom in

to one of these messages

As a system,
I need to do these things:

Read message from Kafka topic Add data to database Tell Kafka I'm done with the message Business logic

What order do we put them in?

What if there's a failure on this step?

That was hard

Exactly Once from Kafka

The easy button



Very fast, Distributed SQL Database

Download free:

  • On-prem, in containers, or in the cloud
  • row-store tables in memory
    persisted to disk
  • column-store tables on disk
    heavily cached
  • ACID compliant transactions
  • data partitioned across commodity hardware
  • free to 128 gig ram across 4 leaf nodes

System Architecture

stuff > Kafka > MemSQL > stuff

Kafka and *-once processing