It could go and do the task perfectly as instructed, or it could do something completely different that you haven't asked for and destroy everything in its path in the process.
I personally found that if you don't give it write access to anything that you can't easily restore and you review and commit code often it saves me a lot of time. It also makes the whole process more enjoyable, since it takes care of a lot of boilerplate for me.
It's definitely NOT intelligent, it's more like a glorified autocomplete but it CAN save a huge amount of time if used correctly.
> (The horse) is way slower and less reliable than a train but can go more places
What does the 'train' represent here?
A guess: perhaps off-the-shelf software? - rigid, but much faster if it goes where (/ does what) you want it to.
Maybe the train is software that's built by SWEs (w/ or w/o AI help). Specifically built for going from A to B very fast. But not flexible, and takes a lot of effort to build and maintain.
Another one I like is "Hungry ghosts in jars."
https://bsky.app/profile/hikikomorphism.bsky.social/post/3lw...
Requires ammo (tokens) which can be expensive.
Requires good aim to hit the target.
Requires practice to get good aim.
Dangerous in the hands of the unskilled (like most instruments or tools).
^^ We are skeptical of AIs (and people) that claim they have consciousness ;-)
