Top
Best
New

Posted by birdculture 1 day ago

Package managers keep using Git as a database, it never works out(nesbitt.io)
752 points | 433 commentspage 5
mikepurvis 1 day ago|
The nix cli almost exclusively pulls GitHub as zipballs. Not perfect but certainly far faster than a real git clone.
pxc 1 day ago|
That it supports fetching via Git as well as various via forge-specific tarballs, even for flakes, is pretty nice. It means that if your org uses Nix, you can fall back to distribution via Git as a solution that doesn't require you to stand up any new infra or tie you to any particular vendor, but once you get rolling it's an easy optimization to switch to downloading snapshots.

The most pain probably just becomes from the hugeness of Nixpkgs, but I remain an advocate for the huge monorepo of build recipes.

mikepurvis 1 day ago||
Yes agreed. It’s possible to imagine some kind of cached-deltas scheme to get faster/smaller updates, but I suspect the folks who would have to build and maintain that are all on gigabit internet connections and don’t feel the complexity is worth it.
pxc 1 day ago||
> It’s possible to imagine some kind of cached-deltas scheme to get faster/smaller updates

I think the snix¹ folks are working on something like this for the binary caches— the greater granularity of the content-addressing offers morally the same kind of optimization as delta RPMs: you can download less of what you don't need to re-download.

But I'm not aware of any current efforts to let people download the Nixpkgs tree itself more efficiently. Somehow caching Git deltas would be cool. But I'd expect that kind of optimization to come from a company that runs a Git forge, if it's generally viable, and to benefit many projects other than Nix and Nixpkgs.

--

1: https://snix.dev/

mikepurvis 1 day ago||
Yes indeed. That said nix typically throws away the .git dir so it would require some work to adapt a solution to nix that operates at the git repo level.

The ideal for nix would be “I have all content at commit X and need the deltas for content at commit Y” and i suspect nix would be fairly unique in being able to benefit from that. To the point that it might actually make sense to just implement the fact git repo syncs and have a local client serving those tarballs to the nix daemon.

drzaiusx11 1 day ago||
I'd add git gemfile dependencies to the list of languages called out here as well. It supports git repos, but in general it's a bad idea unless you are diligent with git tag use and disallow git tag mutability, which also assumes you have complete control of your git dependencies...
drzaiusx11 1 day ago||
One of the first things I did at my current place of employment was to detangle the mess of gemfile git dependencies and get them to adopt semver and an actual package repo. There were so many footguns with git dependencies in ruby we were getting taken down by friendly fire on the daily...
pizlonator 1 day ago||
What is the alternative?

"Use a database" isn't actionable advice because it's not specific enough

yawaramin 21 hours ago|
Use an SQLite database file, stream out delta updates to clients using zipped plaintext or Litestream or something.
dromologist 1 day ago||
We wanted to pull updated code in our undockerized instances when they were instantiated, so we decided to pull the code from GitHub. Worked out pretty well though after a thousand trials we got a 502 and now we're one step closer to being forced into a CD pipeline.
xpressvideoz 1 day ago||
The article lists Git-based wiki engines as a bad usage of Git. Can anybody recommend alternatives? I want something that can be self-hosted, is easily modified by text editors, and has individual page history, preferably with Markdown.
krbaccord941x 23 hours ago||
I understand article is concerning RFC2789, in cloning whole indexes for lang indexes, but /cargo/src shallow-clones need another layer, where tertiary compilation or decompression takes place in mutex libraries, whether its SSL certificate is dependent on HTTP fetch.
wg0 1 day ago||
Why not use SQLite then as database for package managers? A local copy could be replicated easily with delta fetch.
skylurk 1 day ago||
Sounds like it worked pretty well several times? But yeah it does not scale forever.
didip 1 day ago|
So… What we need is a globally distributed git seeders of all open source github content, then?

Seems possible if every git client is also a torrent client.

More comments...