Top
Best
New

Posted by franze 12 hours ago

Show HN: Let your AI agents paint big arrows, boxes and text on your screen(github.com)
360 points | 157 commentspage 3
dr_kiszonka 7 hours ago|
I take this opportunity to shame GitHub for completely ignoring the mobile experience in their own Android app. The project's README is pages upon pages of largely blank space. In general, the app does not render mermaid diagrams and does not allow for zooming in, so smaller pictures are unusable. Even if you access pictures directly in a repo's source, you still can't zoom in. iPython Notebooks, which are extremely common in data science, are an "unsupported file type" and are not rendered.

(OP, nice project! Sorry for my rant.)

bel8 7 hours ago|
I'm forced to use GH mobile app because of 2FA.

But it's a subpar experience compared to just opening github on the browser.

githubnuoo 6 hours ago||
it was driving me mad too. you can disable github.com opening in the github app: app-info/set-as-default/open-supported-links
m-s-y 10 hours ago||
Genuine question…

Why does this need to be a skill? Literally just ask Claude to “call out actions in these screenshots with numbered stylized arrows” and you get the same thing without burning tens of thousands of tokens in context.

IanCal 10 hours ago||
There is a skill, but you don't have to use it, though you'd have to explain how to use the app.

> burning tens of thousands of tokens in context.

~1400?

> Literally just ask Claude to “call out actions in these screenshots with numbered stylized arrows”

Is there a macos native thing it uses? Or are you talking about screenshots? This draws on your screen instead.

raviinits 6 hours ago||
[flagged]
ballofrubber1 10 hours ago||
With a skill claude will know when to use it without you specifically prompting it.
ElijahLynn 3 hours ago||
This is going to be really useful actually, nice work! Can't wait for it to be on Linux too!
TekMol 12 hours ago||
Swift, Shell, Python and Objective-C

Does one need 4 programming languages to draw something on a mac?

sitzkrieg 12 hours ago|
welcome to zombocom. err i mean modern HN :-(
iandanforth 7 hours ago||
Pointing is surprisingly overlooked in a lot of tools and I look forward to using this. It reminds me of what I thought was a killer feature on AnyBots telepresense robots, they had a laser you could use to point at things while piloting the robot.
ghm2180 10 hours ago||
Man, Ive lost count of How many times have I had to repeat this over the phone to my parents; The repo has all the punch lines in the README

> "It's this window, not that one." You have 14 Chrome windows. The agent knows which one it means: --window "Google Chrome:Pull request". It even picks the right tab: --app "Google Chrome:Pull request".

> Remote help. "No, the other gear icon." Point at it instead of describing it.

xyzsparetimexyz 11 hours ago||
Seems like a pretyu useful way to help infants use desktop computers
ipsod 11 hours ago|
Have you met users?
ceroxylon 6 hours ago||
This is like the alerts in new cars that remind you to check for other passengers when you exit... helpful, but a worrying sign of the times.
LoneRanger1024 8 hours ago||
I think this makes a lot of sense for tutorials, but permission prompts need different treatment. Besides pointing to a button, the assistant should explain what clicking it will do, so the user can make their own decision.
tilemarch 9 hours ago|
An arrow on the screen solves “where do I click”; it doesn’t solve “should this happen”

To me this feels like it takes away from what the human is supposed to do (read, understand the consequences of the action, then.. consent or abort)

There is a reason your AI Agent won't automate these clicks for you

More comments...