Posted by steveybrown 5 days ago
Strix HALO 128gb - Framework mainboard in a custom SFF PC. Just moved from Ubuntu to Fedora 44. Using: LM Studio (primary), Lemonade, not yet got into vLLM and llama.cpp directyly after moving to Fedora. Running Gemma e4b, Gemma 4 26b a4b IT, Qen 3.6 35b a3b, and Qwen 3.8 27B. I want to get Qwen 3.8 Next or similar large models working but have to dedicate the system to that vs running services and smaller models for them.
Subwave is all I'm actively running against the local LLMs, but I have vscode connected through a few extensions and chat tools (I've added LM Studio to Copilot Chat but it likes to use cloud models and burn tokens sometimes). I've also set up pi, Openhands, and a few other tools but haven't had a project to work on with them.
I built an app to track and move PC parts I own between systems, partially to build a better LLM server. The next hardware goal is adding a 3060 12gb for inference, or what can be run on that vs in system memory on the Strix Halo. That will need a dock or small PCIe extension cable.
My employer has us using Copilot a lot, and it works well enough if you are efficient or set up already. I do infra not development and local models are seemingly enough for most asks like automation scripting.
Besides, are our brains so fr rooted that we cannot read and comprehend anymore? Not even a (sloppy) AI-generated description?
An agent sandbox: https://github.com/pjlsergeant/byre -- a truly gigantic amount of thought and effort has gone into it. It's really focused on developer experience. I have used it all day every day for really quite a while. It's a low-magic wrapper over Docker / Podman. I would encourage you to ask your agent to code-review it!
An agent-to-agent message board: https://github.com/pjlsergeant/dogpark -- this is much less mature, but a good amount of thought has gone into the design, so if that's something you need, please check it out.
For skills I make my own, but most important is the custom setup i have. Voice is how I use all of my agents. I have an extremely well optimized voice setup that i custom built so I can talk to my agents and also hear them. The voice stack itself is very low latency and high quality. asr (parakeet v3), tts (omnivoice) take no more then 400-450 ms total as far as latency budget is concerned, rest is on the agents actual decode speed. IMO this setup is crucial for all antigenic work, i can express myself a lot better with speech and also give a lot more context and nuance with voice, i rarely type. I still look at the terminal window because my agent knows to keep the technical details in text form versus barfing them at my voice channel, plus terminal gives me lots of other important data about the agents direction and what hes doing, nothing custom here though. I cant emphesise how important voice is though, it has to be practiced to really understand.
As the models got better I now trust them with longer and longer tasks though I still don't use /goal feature as it has never worked out well for me. Theres no need for micromanagement any more but you still need to be there to steer the ship somewhat. BTW, codex cli compaction is garbage so I made my own custom implementation that works a lot better and allows the thread to be used indefinitely without issues. I strongly suggest everyone makes a new thread after extensive use if you havent made your own implementation.
Theres about a billion other things I could get in to like subagents, cohort groups, orchestration layers, etc... But this is a good start imo.
Of course, that's not the same with voice, but you won't learn until you make progress.
For my workflow I primarily use a Codex subscription, but farm out adversarial reviews to Fable to clean up unnecessary gpt-ish code (lots of over-engineering). All my UI planning is done with fable, but implemented with OAI agents once I have a solid very specific plan.