Posted by b-man 2 hours ago
I'm thinking of Redis in particular. If you're using it as incredibly fast but not critical storage, it's trivial to set up and it ~never crashes or requires maintenance. It creates no headaches, and in exchange gives me a k/v store that I can thrash without worrying about performance (I know it's fast), downstream impact (am I slowing down critical-path SQL queries), etc. Especially in the age of LLMs, which I've found to be great at devops-type tasks, I feel slightly less compelled to simplify my stack.
I'm far more hesitant to throw a more formal MQ in the mix though... mostly from experience in that a lot of mid level ("senior") developers don't really understand queues very well at all. Even if conceptually, using tables for queues is more complex.
That said, I will use PG for workflows similar to K/V, Document (JSONB) and other structures over reaching for say MongoDB, etc.
It's not that it's necessarily complex. But if you don't need it, don't use it. The business could use your time elsewhere
For a persistent store.. sure... but that's true for PostgreSQL as well, which has some pretty painful major version migrations by comparison to other options.
A better response to my comment could have been "just let it go" or something. ₍₍(˶>ᗜ<˶)⁾⁾
An example of a table that would benefit from this would be rate-limits / concurrency-limits, which are commonly implemented using Redis instead of Postgres.
What I don't want is long running unrelated transactions keeping old tuples alive just it case they're needed. My third suggestion where transactions attempting to read old versions fail is probably better than the first two where isolation gets silently downgraded, since it avoid that problem.
This has two implications:
1) make sure if you use Postgres for anything beyond core rdbms functionality that there is no dependency between the two, so you can rip out the additional functionality and move to a different platform when you end up needing to reduce the load on your db server
2) if using Postgres for non-essentials complicates your db backup workflow, risks the data integrity, makes it difficult to maintain or upgrade your Postgres instance (eg you have to wait months or years for compatibility with newer Postgres versions), or loads relatively shoddy or unstable code into the beating heart of your application, then you should either use a different Postgres server/install/container for these ancillary services or bite the bullet and introduce an alternative dependency, depending on which makes more sense.
> only after pushing Postgres to its limits, documenting why it was insufficient, and accepting the operational cost of the alternative
I love Postgres as a DB but, really, this is ridiculous. No doubt these extensions can do the job well-enough but you might as well invest in learning the right tool for the problem from the start, when the stakes are still pretty low. Why wait until, ahem, Postgres is pushed to the limit before you spin up a Redis cluster?
You don't get free opcost by using Postgres for everything. Arguably if you end up with a monolith of a database, you are paying a higher opcost (imagine if too much caching can affect all CRUD ops in your platform). Or you can manage a cluster of PG instances but that's no less complex---each plugin still comes with its own opcost!
No Silver Bullet, No Free Lunch, and all that. If your problem domain really warrants something outside of relational storage, you're gonna pay that complexity cost one way or another. You can't escape it by shoehorning everything in Postgres, fantastic as a DB as it is.
If you have bunch of files and don't have any structure, yes filesystem is great but the moment when you need consistency & performance (which you need sooner rather than later) use databases.
Investing early doesn't hurt when you build a product that you know will have many writers.
I guess its more the rapid start-up mindset to get it up and running fast to sell the company, and leave the problem for someone else which is why a lot of our world is falling apart...
For example: I worked with Rails from 2009-2024 and I haven't come across a single Rails project in the wild that didn't have the queue du jour installed: sideqik+redis, delayed-job before, etc. And then since it's there, people just end up using.
Like right now I am thinking about a system that has a password reset process and you need to keep track of a user id and a reset token, one or two timestamps, maybe a state variable and a flag or two. That's well under 100 bytes and the cluster size for a typical fs is 4kb or more plus there is the cost of the directory entry. If the OS is Windows it has to ask the Security Manager when ever you open it or delete it which is even more heavyweight.
It's common now for applications that handle lots of little "files" to store them as blobs in SQLlite! See https://sqlite.org/fasterthanfs.html
I have started just using Postgres to back queues. It is simpler (although I have spun up new apps with Redis so many times it is only a small improvement) but more importantly it is cheaper. I really do try a lot of stuff out, so completely removing a infrastructure piece is a nice money saver. Again, not massive but it's cheaper.
The downsides of doing this in the prototype/MVP context are minimal. At the scale of prototype and MVPs, I certainly don't see any difference in performance.
I did note in the graphic it listed Kafka but in the lower table graphic I did not see any Postgres replacement for Kafka. If I am at the scale where I really want Kafka, it is probably for performance and I just can't believe there's anyway Postgres could provide that.
So, I love the flexibility and all-in-one abilities of Postgres for prototyping and making MVPs. But at my job, nobody is proposing exclusively using Postgres for persistence.
SELECT FOR UPDATE SKIP LOCKED works great for turning Postgres into a job queue. It does not work the same way on CRDB and you will likely have to rewrite those queries or use an external job queue.
Unless of course, you already know how to operate rabbitmq.