Top
Best
New

Posted by ajayvk 10/27/2024

Using SQLite as storage for web server static content(clace.io)
273 points | 126 commentspage 3
ajayvk 10/29/2024|
Wanted to add some context about why this approach made sense for Clace.

Clace is built for use cases where a team wants to deploy web tools for internal use. Tens or hundreds of (small) apps can be hosted on one machine. Clace provides blue-green staged deployment, preview env, OAuth access control, secrets management etc for the apps. Apps can be updated atomically, i.e. a reload command can update multiple apps in one go. If any of them fail to update, the whole operation is aborted.

Clace apps can be of three types 1) Backend actions (defined in Starlark) which have an auto generated form UI 2) Hypermedia based apps where Go Templates and HTMX are used to create a custom UI 3) Containerized apps, where an container image is created and started. Clace can be used to just proxy the container APIs or again build a hypermedia based UI for those APIs.

A SQLite database is used for all app metadata (version info, config etc). The same database is used to store all the files for the app. This would be all the files for the hypermedia app, templates and static files, or the files to build the container image. Basically the source code is uploaded from GitHub or local disk to the database.

So a command like

     clace app create --approve https://github.com/claceio/apps/utils/bookmarks  /bookmarks
will get the source code for the app from GitHub and write it into the SQLite database. Later, running

     clace app reload --promote /bookmarks
will fetch the latest source code from GitHub and update the app to use that. The older version is retained in the database. A command like

     clace version switch previous /bookmarks
will change back to the previous version. No call to GitHub is required for this switch, since the versioning is done in the database. This works even if git is not being used (local disk was used for initial app creation).

The versioning, de-duplication and other file metadata handling are much easier because of using the database. It would have been possible to do it on the file system, but I believe it would have required more effort.

account42 10/28/2024||
> For Clace, the decision was made to use SQLite for app files storage instead of using the file system. The reasoning was mainly to be able to do atomic version changes.

Isn't this pointless when those files are transferred via multiple requests? What does it matter if you atomically update files A and B together when the browser requests A before your update and B after it.

And if you don't care about that then rsync with --delay-updates is just as good.

101008 10/28/2024||
Can someone do a WordPress plugin that stores all the media in a sqlite? Or S3? I'm tired of downloading and uploading gigas from wp-uploads everytime I need to migrate my WPs from one hosting to another.

There is a plugin for that but it's super expensive.

dbrueck 10/28/2024||
It's fun to experiment and try out ideas - kudos to the author for doing this and sharing the results with the community, I love that.

As far as the results go, though, I don't see any realistic scenario where this is a net win vs a symlink. :)

ipsum2 10/28/2024||
I've also considered SQLite but decided against it. Using a custom compression means that you can't do a full text search while compressed on the file like you would in btrfs or zfs.
slavboj 10/27/2024||
"We put a filesystem in your database on your filesystem which is a database, so you can serve while you serve while you serve." [inception theme plays]
hinkley 10/27/2024||
I want an nginx plugin that does this. And for a cache.
osigurdson 10/27/2024||
While I like the idea of using sqlite, how to avoid downtime? Do you host the file on NFS or something?
ajayvk 10/28/2024|
If you mean how do backups, there are tools like https://litestream.io/ which help with SQLite backup to S3 (or other cloud storage)
osigurdson 10/28/2024||
I was thinking more of the situation where you have two services running in an HA configuration. It seems that the only way to make this work would be to share the database file on NFS (but I can see downsides with that as well since this only really approximates a file system). I think a single replica service backed by cluster attached storage would generally work ok. I guess that is what most people are doing.
gwbas1c 10/28/2024||
TLDR: I thought it would be for sites that are heavy on charts, searches, or other forms of data analysis, but it isn't. Clance also is planning on moving to Postgres.

----

I don't really "get" this. A lot of the comments here are focused on SQLite, but the page clearly says:

> Clace currently runs on a single node. When multi-node support is added later, the plan is to use a shared Postgres database instead of using local SQLite for metadata and file data storage.

Let's start with some basics:

1: What problem is Clance trying to solve? (IE, did the authors encounter problems hosting / generating static sites, and Clance is designed to solve a specific problem they had?)

2: Why can't Clance be multi-node today? (What potential problems would happen if I had multiple Clance nodes pointed to the same database, or github repo?)

Naively, I thought Clance wasn't serving truly static sites; IE, I thought Clance would be serving sites that allow querying static / infrequently changing data. I thought it would be for sites that are heavy on charts, searches, or other forms of data analysis.

pkphilip 10/28/2024|
If it is static content why have a database at all?
marc136 10/29/2024|
In general, e.g. to place the files and their metadata like checksum, access rights, image previews, foreign keys right next to each other.

In the authors use case, it might also be easier to drop the executable and one big database file on a random shared file store in the company network instead of many small files and a database file for metadata.

And it might be faster to read files from db than from fs https://sqlite.org/fasterthanfs.html

More comments...