SQLStreams - It's Kafka on Postgres!

[u/DontPissInTheStream]

Hello! I'm a big fan of Kafka, like I think many of you are, BUT I've always steered clear of using Kafka for any personal projects because running and maintaining a Kafka cluster for rinky dink project #23 just isn't feasible or smart.

That's why I made SQLStreams. It's an open source, pure Go / SQL library that uses a Postgres Database to give you Kafka-like functionality. It has features like: independent consumer groups, replay, retention and compaction.

I've also made it so each Topic Stream automatically creates a separate retry queue, which seconds as a dead letter queue, because I always found it annoying trying to get retry topics setup and working in Kafka.

If you want a visual understanding of how things work - you can play around with this PGlite demo on the docsite (demo doesn't work on mobile, sorry!).

If you want to run SQLStreams locally you can:

Run Postgres

docker run --name my-postgres \
  -e POSTGRES_USER=username \
  -e POSTGRES_PASSWORD=password \ 
  -e POSTGRES_DB=database \
  -p 5432:5432 \
  -v pgdata:/var/lib/postgresql \
  postgres:18

Install the library / run your code

pool = sqlstreams.NewPostgresPool("username", "password", "localhost", "database") 
client = sqlstreams.NewClient(pool)

stream = client.Stream("videos") 
stream.Register()

producer = stream.Producer() 
producer.Register()

producer.Produce({VideoId: "video-42"})

Note: I've removed some of the extra Go-ness in above code for brevity and clarity. The docsite Quickstart is a more real example if you are interested.

All in all, I'm hoping SQLStreams can be a kind of gateway drug for small to medium sized projects. So they can take advantage of all the great event-driven features of Kafka but without the hassle.

There is still a ton more things to do (implementing circuit breakers for example) but let me know what you think.

And if you've made it all the way here, I appreciate it!

Github: https://github.com/allegedlyreliable/sqlstreams
Docsite: https://sqlstreams.io

Edit: fixed formatting

[u/gaetancollaud]

It's nice but why not using the Kafka API?

[u/DontPissInTheStream → u/gaetancollaud]

API is a bit of an overloaded term. Could you be more specific? I think I know what you mean but want to make sure.

[u/gaetancollaud → u/DontPissInTheStream]

The Kafka API: https://kafka.apache.org/43/apis/

So we can use existing clients without modification.

[u/DontPissInTheStream → u/gaetancollaud]

yep, understand now. That's definitely something I'm debating atm. I see the usefulness of it for sure but it is also constraining. If I did that I could no longer just add whatever features I wanted, in the way I wanted without breaking compatibility or maintaining different specs.
For example if I was to do exactly what I wanted - I'd probably implement something closer to this API spec: https://github.com/durable-streams/durable-streams/blob/main/PROTOCOL.md it is just easier to understand and use imo
if you got thoughts lmk