llama.cpp has router mode these days, swapping out models for you, you no longer need llama-swap.

AI doesn't necessarily know what feels like a good tradeoff to you. I'm sure it could help guide you though.

Wow it’s aggressively vibe coded. Nothing inherently wrong with that, but it looks a bit amateurish which is funny.

I’m still waiting on 98.css to become the standard for vibe coded sites. You don’t have to read docs anyway if you’re just using LLMs! All you have to do is say “use 98.css” and you have a 10/10 site

https://jdan.github.io/98.css/

> At this point the options are llama-server or vLLM

Which last time I checked, both use different formats of the weights, the former GGUF while the latter .safetensors. I mostly end up using vLLM these days and I'm a bit more performance sensitive than what I used to be. Just a shame it's a hassle to share the weights between them with conversion and what not, either batched or on-startup.

The gap was MUCH larger in the past, but in my tests, oMLX and llama.cpp are now very similar (within 10%) in both prompt processing and generation speed. GGUF ecosystem provides a better selection of quants, in my experience Unsloth ones are excellent.

You probably can just use Vulkan. Why don't you think Vulkan is an alternative? Can it not handle BLAS and memory access? What is a graphics card supposed to be doing that isn't efficient in Vulkan?

I did report my test results on the first one. :)

The second I didn't say it's any of llama.cpp's "fault", but it is _related_ to llama.cpp since it's being shipped in another system, aye?

Can't stick to the old hash either, because older version have different bugs. E.g. on older versions the same Qwen3.6 model reliably fails to call specific tools due to template issues, while just having the newer llama.cpp version has that fixed. So different versions - different bugs, rather than no bugs.

Why the beating you are trying to gimme, mate? :)

I just switched to Vulkan, and be done with it. :)

As much as I can tell, the ROCm version of llama.cpp would be a bit faster on prompt processing, but about the same on the token generation as Vulkan. Real life benchmarks don't seem to give any "ROCm or nothing" sort of vibes. And the difference between the performance of different models are way bigger than the difference between the llama.cpp versions (and versus different runtimes like the llama.cpp/GGUF and the MLX runtimes on Mac for the same models)...

I've tinkered enough with the serving, that I'd rather do something with them with, say 10% slower speed, than spending hours on seting things up again... YMMV

Hipfire works on my 7900xtx best of all. Biggest surprise - Qwen3.6-27B dense does not grind to a halt with context depths all the way up to 250K! Measured at 1K, 8K, 32K, 64K, 128K, 192K, 250K - Hipfire speed holds close to 40 tok/s. Finished 5 days run of gpu 100% inferencing, Hipfire did not crash even once afaics. I assumed speed dropping like a stone on Qwen models was a feature/bug of the model, and "nothing can be done about it". Hipfire showed me wrong - pleasantly surprised there.

ty for digging this up!

Package management provides cryptographic attestation over the entire process, including the scripts run. Nothing is arbitrary nor mutable in the default state. The files touched will be in a predictable place and the scope of privilege escalation is clearly defined.

It's more about installation location for me. For these types of projects, I don't even install it. I just clone, build and run from there. I have no idea where the bash script is going to put things.

I mean it's probably not, I just haven't got used to it yet. It's about the same level of security as installing a windows app on win2000 25 years ago and blindly downloading a .exe off the internet and running it to get into the install wizard. But indeed I also kind of blindly trust that whatever I'm getting from the debian trixie officially gpg-signed packages isn't backdoored.

One thing I do not do as a matter of practice is install things with a ridiculous number of recursive npm dependencies.

Rendered text cannot be assumed to equal the underlying text, unfortunately

This. I use mise's github backend `mise use --global --pin github:ggml-org/llama.cpp` to grab the release binaries for Linux, Windows and macOS.

Exactly. That way of assuming everyone is a seasoned dev or sysadmin always baffles me. There are a ton of tinkerers out there, we are legion, and we are often reluctant to ask, especially here (as suggested by another parallel comment) because we don't want to receive an answer like this. Full circle.

If you’re unsure how to do it, you can ask others for help or Claude will tell you.

Point Claude Code at a repository and ask how to install it safely. You don’t have to know about make or cryptography of HTTPS or anything, really. It will walk you through the options and risk.

If you have questions about any part of it—i.e. you don’t recognize an acronym or deeply understand why something works—you can ask.

Or ask here! HN is filled with smart humans.

GP is saying "it's not hard", yet you seem to be hearing "you should already know how to do this without ever looking it up".

Those aren't the same assertions.

It's not a sensible assumption that a process must necessarily be difficult or complex just because you don't already know how to do it. There are an unenumerable number of tasks each of us don't know how to do and have never done before which are not difficult at all.

It's literally three steps, assuming you have the equivalent of the debian "build-essential" dependencies installed on your system for cmake, compiler. It's the exact three rows of my post, pasted one at a time into the CLI. The llama-server build guide page is actually pretty good.

I mean, sure, if there's people who can't figure that out, they're probably better off using a GUI that is a wrapper on top of somebody else's precompiled llama-server, like unsloth studio or lm studio. There's a good sized market for that and I wish them well.

One way I prevent possible catastrophic fuckups is that the 'doing code work' box that runs opencode or pi or whatever, is its entirely own separate VM and desktop environment (running as a xen or kvm guest and with its own LVM logical volume as boot/root and /home disk), than the machine running llama-server itself.

The harness gets the openai-compatible endpoint fed into it to talk to llama-server across the network, but the VM has no access whatsoever to my personal files, mail, backups/deep storage, fileserver, Documents folder, etc.

> security concerns

Yeah I recently tried the coding harness that's recommended here, Pi, in a bubble wrap sandbox and was horrified to learn that it spams multiple warnings at you if you don't give it write access to its own config/extension folder... Everyone else is rawdogging it I guess.

It's okay-ish for Rust because you only need the curl|bash for Rustup, which is meant as a development environment. On production you can, for example, use APT to download it from Debian.

Mostly agree but at least with git clone you have a hash and the malicious code has to be served to everyone, and GitHub is less likely to get hijacked by a malicious third party.

But yes, still trusting the project with arbitrary code execution on your machine, including build formulas that pull stuff from the internet and suffer from all the above anyways

I was able to fit only small models Qwen3.5-4B in RTX3070 which is not very useful for Python and SQL generation thought. When I wan to test larger open LLM models I often just use cloud resources.

The dev behind ollama is adamant that ollama doesn't use llama.cpp (based on a technicality -- it uses ggml, which is created by the same people behind llama.cpp and is the backend of llama.cpp)

He made such a big fuss about ollama implementing their own kernels and felt slighted about the online comments saying ollama didn't properly credit llama.cpp and it kind of left a bad taste in the mouth among the local inference community.

For me personally, it was this that made me avoid them at all costs: https://github.com/ollama/ollama/issues/11714#issuecomment-3...