@rob_rich

Planning for the varied levels of once-data-processing with Kafka

by Rob Richardson

@rob_rich

https://robrich.org/

About Me

Rob Richardson

Rob Richardson is a software craftsman building web properties in ASP.NET and Node, React and Vue. He's a frequent speaker at conferences, user groups, and community events, and a diligent teacher and student of high quality software development. You can find this and other talks on his blog at https://robrich.org/presentations and follow him on twitter at @rob_rich.

Spoiler:

We're compensating for failure

There are only two hard problems in distributed systems: 2. Exactly-once delivery 1. Guaranteed order of messages 2. Exactly-once delivery

source: twitter.com/mathiasverraes/status/632260618599403520

B.K.
(Before Kafka)

A.K
(After Kafka)

Let's zoom in

to one of these messages

As a system,
I need to do these things:

Read message from Kafka topic Add data to database Tell Kafka I'm done with the message Business logic

What order do we put them in?

What if there's a failure on this step?

That was hard

MemSQL:
Exactly Once from Kafka

The easy button

MemSQL

MemSQL

Very fast, Distributed SQL Database

Download free:
https://msql.co/kafka-meetup

  • On-prem, in containers, or in the cloud
  • row-store tables in memory
    persisted to disk
  • column-store tables on disk
    heavily cached
  • ACID compliant transactions
  • data partitioned across commodity hardware
  • free to 128 gig ram across 4 leaf nodes

System Architecture

stuff > Kafka > MemSQL > stuff

Kafka and *-once processing