And structured APIs are about 1e9x more expensive than not invoking an LLM in the first place compared to using deterministic code to do something ... it's not like any of this is rational based on compute.

I have only found Computer Use useful for GUI app local debugging. Presumably it will also be useful for getting around protections for external apps that don't want AI to interact with them, or for interfacing with legacy apps or those built without AI in mind.

I don't think any new app should ever be specifically designed for AI to interact with them through computer use

My "best practice" is to use as little "visual" (computer use) tooling and as much api + cli tooling as possible specifically to save on tokens.

Tokens a resource and should be managed as such.

It would be great if institutions like banks provided proper APIs.

Metadata and structure beats AI every time.

The hard part about the web is that API's aren't just available even if the website owner wants them exposed (big if).

I embedded a Google Calendar widget on my Book a demo page, I don't know the API and Google doesn't expose/maintain one either.

What we are doing at Retriever AI is to instead reverse engineer the website APIs on the fly and call them directly from within the webpage so that auth/session tokens propoagate for free: https://www.rtrvr.ai/blog/ai-subroutines-zero-token-determin...

by design: https://en.wikipedia.org/wiki/Desire_path

IMO, this is the argument for doing work in the first place.

Just wondering: RPA companies like UiPath ard dead in the water, right?

Blackhat SEO spamming knew this 20 years ago

In what world would a vision agent be the default, when whatever HTTP-based mechanism a site uses to communicate with the server can usually be reverse-engineered and easily emulated with widely available HTTP request libraries, HTML parsers, and JavaScript engines, and at worst you can use something like Puppeteer to navigate and control applications at a significantly higher level than image scraping and simulating user input?

It seems like you'd need a deliberately hostile app before a vision agent would even be considered as an option.