Posted by deepakjois 6 days ago
Now if only someone could do the same for shell scripts. Packaging, dependency management, and reproducibility in shell land are still stuck in the Stone Ages. Right now it’s still curl | bash and hope for the best, or a README with 12 manual steps and three missing dependencies.
Sure, there’s Nix... if you’ve already transcended time, space, and the Nix manual. Docker? Great, if downloading a Linux distro to run sed sounds reasonable.
There’s got to be a middle ground simple, declarative, and built for humans.
But: it’s the same skill set for every one of those things. This is why it’s an investment worth making IMO. If you’re only going to ever use it for one single thing, it’s not worth it. But once you’ve learned it you’ll be able to leverage it everywhere.
Python scripts with or without dependencies, uv or no uv (through the excellent uv2nix which I can’t plug enough, no affiliation), bash scripts with any dependencies you want, etc. suddenly it’s your choice and you can actually choose the right tool for the job.
Not trying to derail the thread but it feels germane in this context. All these little packaging problems go away with Nix, and are replaced by one single giant problem XD
ChatGPT writes pretty good nix now. You can simply paste any errors in and it will fix it.
#! /usr/bin/env nix-shell
#! nix-shell -i bash -p imagemagick cowsay
# scale image by 50%
convert "$1" -scale 50% "$1.s50.jpg" &&
cowsay "done $1.q50.jpg"
If you do care, then welcome to learning about niv, flakes, etc.
[0]: admittedly 3 years ago or so.
For my use case, most things I don't mind tracking mainline, but some things I want to fix (chromium is very large, python changes a lot, or some version broke things)
``` #! nix-shell -i bash -p "cowsay" '(import (fetchTarball { url="https://github.com/NixOS/nixpkgs/archive/eb090f7b923b1226e8b... sha256 = "15iglsr7h3s435a04313xddah8vds815i9lajcc923s4yl54aj4j";}) {}).python3' ```
[1] flakes really aren't bad either, especially if you think about it as just doing above, but automatically
The most shocking part about nix is the nix-shell (I know I can use it in other distros but hear me out once), its literally so cool to install projects for one off.
Want to record a desktop? Its one of those tasks that for me I do just quite infrequently and I don't like how in arch, I had to update my system with obs as a dependency always or I had to uninstall it. Ephemerality was a concept that I was looking for before nix since I always like to try out new software/keep my home system kind of minimalist-ish Cool. nix-shell -p obs-studio & obs and you got this.
honestly, I like a lot of things about nix tbh. I still haven't gone too much into the flake sides of things and just use it imperatively sadly but I found out that nix builds are sandboxed so I found a unique idea of using it as a sandbox to run code on reddit and I think I am going to do something cool with it. (building something like codapi , codapi's creator is kinda cool if you are reading this mate, I'd love talking to ya)
And I personally almost feel as if some software could truly be made plug n play (like imagine hetzner having nix os machines (currently I have heard that its support is finnicky) but then somehow a way to get hetzner nix os machines and then I almost feel as if we can get something really really close to digital ocean droplets/ plug n play without any isolation that docker provides because I guess docker has its own usecases but I almost feel as if managing docker stuff is kinda harder than nix stuff but feel free to correct me as I am just saying what I am feelin using nix.
I also wish if something like functional lua (does fxn lua exist??) -> nix transpiler because I'd like to write lua instead of nix to manage my system but I guess nix is fine too!
I guess I will try out nix on hetzner for sure one day. This is really cool!!! Thanks! I didn't expect you to respond. This is really really cool. You made my day to whoever responded with this. THANKS A LOT KATIE. LOTS OF LOVE TO HETZNER. MAY YOU BE THE WAY YOU ARE, SINCE Y'ALL ARE PERFECT.
When there's some random little utility I need I don't always bother to install it. It's just `, weirdlittleutil`.
IMO it should stay that way, because any script that needs those things is way past the point where shell is a reasonable choice. Shell scripts should be small, 20 lines or so. The language just plain sucks too much to make it worth using for anything bigger.
An if statement in, for instance bash, just runs any command and then runs one of two blocks of code based on the exit status of that command. If the exit status is truthy, it runs what follows the `then`. If it's falsey, it rhns what follows the `else`. (`elsif` is admittedly gratuitous syntax— it would be better if it were just implemented as an if inside an else statement.) This seems quite similar to other programming languages and like not very much to remember.
I'll admit that one thing I do in my shell scripts is avoid "fake syntax"— I never use `[` or `[[` because these obscure the real structure of the statements for the sake of cuteness. I just write `test`, which makes clear that it's just an ordinary command, ans also signals to someone who isn't sure what it's doing that they can find out just by running `man test`, `help test`, `info test`, etc., from the same shell.
I also agree that if statements and if expressions should be kept few and simple. But in some ways it's actually easier to do this in shell languages than in many others! Chaining && and/or || can often get you through a substantial script without any if statements at all, let alone nested ones.
What I would worry about more is that it breaks `sh` compatibility.
When Bash sees it, it treats it as something that needs to be matched, like a string, and won't stop taking user input in an interactive session until it sees the `]]` or something else that causes a syntax error. If I write `[` and just hit enter, I get an error from the `[` command, same as if I ran an external one. But if I use a `[[`, I get an error message back from Bash itself about a malformed conditional (and/or it will wait for input before trying to execute the command):
2 bash which -a [
/Users/pxc/.nix-profile/bin/[
/bin/[
2 ~
bash [
bash: [: missing `]'
2 ~
2 bash /bin/[
[: missing ]
2 ~
2 bash [[
∙ no prompt
bash: conditional binary operator expected
bash: syntax error near `prompt'
2 ~
2 bash [[
∙ a =
bash: unexpected argument `newline' to conditional binary operator
bash: syntax error near `='
This same kind of special handling by the parser probably allows for other differences in error messages, but I don't write Bash that produces such errors, so I couldn't tell you. ;)
> more certain to be a bash built-in
If you want to be sure that you're using a built-in rather than a command, you can use the `builtin` command. But because `[[` is actually special syntax, it's technically not a builtin, so you can't use it this way! Check it out:
~ took 34m8s
2 bash
2 ~
bash builtin [[
bash: builtin: [[: not a shell builtin
2 ~
1 bash builtin [
bash: [: missing `]'
Imo, that's the real reason to avoid it. I'm all for embracing Bashisms when it makes code more legible. For instance, I think it's great to lean on associative arrays, `shopt -s lastpipe`, and `mapfile`. They're innovations (or deviations, depending on how you look at it ;), like `[[`, but I feel like they make the language clearer and more elegant while `[[` actually obfuscates the beauty of `if` statements in shell languages, including Bash.
It's all reasonable enough if you go and look it up, but the script immediately becomes harder to reason about. Conditionals shouldn't be this hard.
[0]: https://tldp.org/LDP/Bash-Beginners-Guide/html/sect_07_01.ht...
if ! grep -qF something /etc/some/config/file 2>/dev/null; then
do_something
fi
In the case of Bash, `test` is a built-in command rather than an external program, and it also has two other names, `[` and `[[`. I don't like the latter two because they look, to a naive reader, like special syntax built into the shell— like something the parser sees as unique and different and bear a special relationship to if-statements— but they aren't and they don't. And in fact you can use them in other shells that don't have them as built-ins, if you implement them as external commands. (You can probably find a binary called `[` on your system right now.)
(Actually, it looks like `[[` is even worse than "fake syntax"... it's real special syntax. It changes how Bash interprets `&&` and `||`. Yikes.)
But if you don't like `test`, you don't have to use it; you can use any command you like!
For instance, you might use `expr`:
if expr "1 > 0"; then
echo this will always run
else
echo this will never run
fi
If you really don't like `test`, don't even need to use it for checking the existence or type (dir, symlink, socket, etc.) of files! You can use GNU `find` for that, or even sharkdp's `fd` if you ache for something new and shiny.
Fish actually has something really nice here in the `path` built-in, which includes long options like you and I both wish `test` had. You can write:
if path -q --type=dir a/b/c
touch a/b/c/some-file
end
grep -qxF "$A" <<< "$B"
test "A" = "$B"
string match --entire $A $B
Finally, there's no need to use the likes of `test` for combining conditions. I certainly never do. You can just write
test "$A" = "$B" && test "$C" = "$D"
[ "$A" = "$B" -a "$C" = "$D" ]
As for readability: any program in any language is difficult to understand if you don't know the interfaces or behaviors of the functions it invokes. `[`/`test` is no different from any such function, although it appears that `[[` is something weirder and, imo, worse.
Last year I wrote (really, grew like a tumor) a 2000 line Fish script to do some Podman magic. The first few hundred lines were great, since it was "just" piping data around - shell is great at that!
It then proceeded to go completely off the rails when I went full sunk cost fallacy and started abusing /dev/shm to emulate hash tables.
E: just looked at the source code. My "build system" was another Fish script that concatenated several script files together. Jeez. Never again.
> any script that needs those things
It's not really a matter of needing those things, necessarily. Once you have them, you're welcome to write scripts in a cleaner, more convenient way. For instance, all of my shell scripts used by colleagues at work just use GNU coreutils regardless of what platform they're on. Instead of worrying about differences in how sed behaves with certain flags, on different platforms, I simply write everything for GNU sed and it Just Works™. Do those scripts need such a thing? Not necessarily. Is it nicer to write free of constraints like that? Yes!
Same thing for just choosing commands with nicer interfaces, or more unified syntax... Use p7zip for handling all your archives so there's only one interface to think about. Make heavy use of `jq` (a great language) for dealing with structured data. Don't worry about reading input from a file and then writing back to it in the same pipeline; just throw in `sponge` from moreutils.
> The language just plain sucks too much
There really isn't anything better for invoking external programs. Everything else is way clunkier. Maybe that's okay, but when I've rewritten large-ish shell scripts in other languages, I often found myself annoyed with the new language. What used to be a 20-line shell script can easily end up being 400 lines in a "real" language.
I kind of agree with you, of course. POSIX-ish shells have too much syntax and at the same time not enough power. But what I really want is a better shell language, not to use some interpreted non-shell language in their place.
Been there, tried to, got a huge slap in the face.
I will never voluntarily run a bunch of non-Linux/BSD servers again.
I haven't touched AIX or HPUX in probably a decade and I thought they were a weird idea back then: proprietary UNIX? Is it still 1993?
I hope so, for their sake. shudder
Woke: Dependency management used for installing an interpreter for a better programming language to write your script in it
Bespoke: Dependency management used for installing your script
#!/bin/bash
make $1
There is no package manager that is going to make a shell script I write for macOS work on Linux if that script uses commands that only exist on macOS.
We use it at $work to manage dev envs and its much easier than Docker and Nix.
It also installs things in parallel, which is a huge bonus over plain Dockerfiles
If you're allowed to install any deps go with uv, it'll do the rest.
I'm also kinda in love with https://babashka.org/ check it out if you like Clojure.
Knowing the Python packaging ecosystem, uv could very well be replaced by something else. It feels different this time, but we won't know for a while yet.
I guess the issue is just that nobody has documented how to do that one specific thing so you can only learn this technique by trying to learn Nix as a whole.
So perhaps the thing you're envisaging could just be a wrapper for this Nix logic.
There's a reason why distroless images exist. :)
Hmm, last time I checked, uv installs into ~/.local/share/uv/python/cpython-3.xx and can not be installed globally e.g. inside a minimal docker without any other python.
So basically it still runs in a venv.
No matter what I tried it's always a symlink into ~/.local
>You can still use uv run or create and activate a virtual environment to use python directly.
You still need some kind of venv, even with the power of uv.
https://simonwillison.net/2024/Dec/19/one-shot-python-tools/
And there was a March discussion of a different blog post:
https://news.ycombinator.com/item?id=43500124
I hope this stays on the front page for a while to help publicize it.
at the end of article it shows an mcp to fetch youtube subs. I've made a similar one using simonw's llm as a fragment, if you find it useful.
llm -f youtube:<id>
llm -f yt:<lang>:<id>
I'm also looking forward to the maturity of Rust-based type checkers, but solely because one can almost always benefit from an order of magnitude improvement in speed of type checking, not because Python type-checking is a "shit show".
I do grant you that for outsiders, the fact that the type checker from the Python organization itself is actually a second rate type checker (except for when one uses Django, etc, and then it becomes first-rate) is confusing.
Strangely, I've found myself building personal cross platform apps in game engines because of that.
And uv is fast — I mean REALLY fast. Fast to the point of suspecting something went wrong and silently errored, when it fact it did just what I wanted but 10x faster than pip.
It (and especially its docs) are a little rough around the edges, but it's bold enough and good enough I'm willing to use it nonetheless.
Example:
if __name__ == "__main__":
from myapp.cli import parse_args
args = parse_args()
# The program will exit here if `-h` is given
# Now do the heavy imports
from myapp.lib import run_app
run_app(args)An extreme version of this is the colcon build tool for ROS 2 workspaces:
https://github.com/colcon/colcon-core/blob/master/setup.cfg#...
Unsurprisingly, startup time for this is not great.
It's a tool for installing different versions of Python, it would be weird for it to assume it already had one available.
It feels like like it's doing heavy work between each line printed! I don't know any other cli tool doing that either.
About 95% of my Python utilities are now Go binaries cross-compiled to whatever env they're running in. The few remaining ones use (API) libraries that aren't available for Go or aren't mature enough for me to trust them yet.
But that ship has sailed for me and I'm a uv convert.
Thought I was the only one thinking this. Got to open an issue, I think it would be nice to have some more examples showcasing different use cases.
uvx --with mkdocs-material --with mkdocs-material-extensions --with mkdocs-nav-weight mkdocs serve -a localhost:1337
Hopefully more languages follow suit on this pattern as it can be extremely useful for many cases, such as passing gists around, writing small programs which might otherwise be written in shell scripts, etc.
I still do because:
- Go gives me a single binary
- Dependencies are statically linked
- I don’t need any third-party libs in most scenarios
- Many of my scripts make network calls, and Go has a better stdlib for HTTP/RPC/Socket work
- Better tooling (built-in formatter, no need for pytest, go vet is handy)
- Easy concurrency. Most of my scripts don’t need it, but when they do, it’s easier since I don’t have to fiddle with colored functions, external libs, or, worse, threads.
That said, uv is a great improvement over the previous status quo. But I don’t write Python scripts for reasons that go beyond just tooling. And since it’s not a standard tool, I worry that more things like this will come along and try to “improve” everything. Already scarred and tired in that area thanks to the JS ecosystem. So I tend to prefer stable, reliable, and boring tools over everything else. Right now, Go does that well enough for my scripting needs.
I still use both Go and Python. But Python gives me access to a lot more libraries that do useful stuff. For example the YouTube transcript example I wrote about in the article was only possible in Python because afaik Go doesn't have a decent library for transcript extraction.
Here's how it's relevant :)
If I’ve ever had to run a “script” in any type of deployed ENV it’s always been done in that ENVs python shell .
So I still don’t see what the fuss is about?
I work on a massive python code base and the only benefit I’ve seen from moving to UV is it has sped up dep installation which has had positive impact on local and CI setup times.
"missing x..."
pip install x
run script
"missing y..."
pip install y
> y not found
google y to find package name
pip install ypackage
> conflict with other package
realize I forgot a venv and have contaminated my system python
check pip help output to remember how to uninstall a package
clean up system python
create venv at cwd
start over
...
</end of time>
>check pip help output to remember how to uninstall a package
>clean up system python
>create venv at cwd
>start over
This hits disturbingly close to home.
People complain because the experience is less confusing in many other languages. Think Go, Rust, or even JS. All the tooling chaos and virtual environment jujitsu are real deterrents for newcomers. And it’s not just beginners complaining about Python tooling. Industry veterans like Armin Ronacher do that all the time.
uv is a great step in the right direction, but the issue is that as long as the basic tooling isn’t built into the language binary, like Go’s tools or Rust’s Cargo, more tools will pop up and fragment the space even further.
Tools like uv solve the "it works on my machine" problem. And it's also incredibly fast.
https://packaging.python.org/en/latest/specifications/pylock...
Issue is since there are no standardized build tool (pip, uv both are third party), there are a zillion ways of generating this lockfile unlike go.mod or cargo.toml. So it doesn't work in many scenarios and it's confusing as hell.
People have suggested using other languages that might be faster but the business always choices what’s best for everyone to work with.
What I don't like about the "Python + Framework + Postgres" argument is that it often lacks context. This is a formidable combination for starting a business and finding PMF. But unless you've seen Python and Postgres completely break under 100k RPS and petabyte-scale data, it's hard to understand the limitations just from words. Python is fantastic, but it has its limits and there are cases where it absolutely doesn't work. This “single language everywhere” mindset is how we ended up with JavaScript on the backend and desktop.
Anyone can write Python, and with LLMs, there's not much of a moat around knowing a single language. There's also no reason not to familiarize yourself with others, since it broadens your perspective. Of course, some businesses scale quite well with Python or JavaScript. But my point isn't to abandon Python. It's to gain experience in other languages so that when people criticize Python’s build tools, you can genuinely empathize with those concerns. Otherwise, comments like “Python tooling is fine” from people who have mostly worked with only Python are hard to take seriously.
What if you don't have an environment set up? I'm admittedly not a python expert by any means but that's always been a pain point for me. uvx makes that so much easier.
Nowadays I normally use `python venv/bin/<some-executable>`, or `conda run -n <some-env> <some-executable>`, or packaged it in a Singularity container. And even though I hear a lot of good things about uv, given that my job uses public money for research, we try to use open source and standards as much as possible. My understanding is that uv is still backed by a company, and at least when I checked it some time ago (in peps discussions & GH issues) they were no implementing the PEPs that I needed -- even if they did, we would probably still stay with simple pip/setuptools to avoid having to use research budget to update our build if the company ever changed its business model (e.g. what anaconda did some months/year? ago).
Digressing: the Singularity container is useful for research & HPC too, as it creates a single archive, which is faster to load on distributed filesystems like the two I work on (GPFS & LustreFS) instead of loading many small files over network.
python3 -m venv venv
Activate the virtual environment:
source venv/bin/activate
Deactivate the virtual environment:
deactivate
Which one is easier to run, especially for someone who doesn't use python everyday?
If you don't see what the fuss is about, I'm happy for you. Sounds like you're living in a fairly isolated environment. But I can assure you that uv is worth a lot of fussing about, it's making a lot of our lives a lot easier.
$ uv run --python=3.13 --with-requirements <(uv export --script script.py) -- python
>>> from script import X
$ uv run --with-script script.py python
$ "$(uv python find --script script.py)"
>>> from script import X
You can make `--interactive` or whatever you want a CLI flag from the script. I often make these small Typer CLIs with something like that (or in this case, in another dev script like this, I have `--sql` for entering a DuckDB SQL repl)
cat ~/.local/bin/uve
#!/bin/bash
temp=$(mktemp)
uv export --script $1 --no-hashes > $temp
uv run --with-requirements $temp vim $1
unlink $tempI think their docs could use a little bit of work, especially there should be a defined path to switch from a requirements.txt based workflow to uv. Also I felt like it's a little confusing how to define a python version for a specific project (it's defined in both .python-version and pyproject.toml)
How to migrate from requirements.txt: https://pydevtools.com/handbook/how-to/migrate-requirements.... How to change the Python version of a uv project: https://pydevtools.com/handbook/how-to/how-to-change-the-pyt...
Let me know if there are other topics I can hit that would be helpful!
Actually, I’ve thought of something! Migrating from poetry! It’s something I’ve been meaning to look at automating for a while now (I really don’t like poetry).
The `requires-version` field in `pyproject.toml` defines a range of compatible versions, while `.python-version` defines the specific version you want to use for development. If you create a new project with uv init, they'll look similar (>=3.13 and 3.13 today), but over time `requires-version` usually lags behind `.python-version` and defines the minimum supported Python version for the project. `requires-version` also winds up in your package metadata and can affect your callers' dependency resolution, for example if your published v1 supports Python 3.[old] but your v2 does not.
pyproject.toml is about allowing other developers, and end users, to use your code. When you share your code by packaging it for PyPI, a build backend (uv is not one, but they seem to be working on providing one - see https://github.com/astral-sh/uv/issues/3957 ) creates a distributable package, and pyproject.toml specifies what environment the user needs to have set up (dependencies and python version). It has nothing to do with uv in itself, and is an interoperable Python ecosystem standard. A range of versions is specified here, because other people should be able to use your code on multiple Python versions.
The .python-version file is used to tell uv specifically (i.e. nobody else) specifically (i.e., exact version) what to do when setting up your development environment.
(It's perfectly possible, of course, to just use an already-set-up environment.)
Try `uvx migrate-to-uv` (see https://pypi.org/project/migrate-to-uv/)
(If you want to write TOML, or do other advanced things such as preserving comments and exact structure from the original file, you'll want tomlkit instead. Note that it's much less performant.)
Some stuff like npm or dotnet do need an npm update / dotnet restore when I switch platforms. At first attempt uv seems like it just doesn't really like this and takes a fair bit of work to clean it up when switching platforms, while using venvs was fine.