← All posts

Let your AI coding agent see your screen

Frayo has a command-line tool for scripts and agents. It is off until you turn it on.

AI coding agents are good at code and blind to everything else. You change the CSS, and the agent cannot see that the button is now off-screen. A dialog shows an error, and you copy it out by hand. You say "look at the screenshot from earlier" and it cannot.

The usual fix is to take a screenshot, drag it into the chat and wait. It works, and it is slow, and it is yours to do every time.

A tool the agent can call

Frayo ships a command, frayo:

frayo capture window --json
frayo ocr --rect 0,800,1200,300
frayo search "NullPointerException" --json
frayo list --limit 5

The agent runs a command, gets a file path or some text back, and carries on. No dragging.

Text is cheaper than pictures

The command you will use most is ocr. It reads the text on screen, or in an image file, and prints it. An error dialog costs a few dozen tokens as text. As an image it costs a lot more, and the agent can still misread it.

Add --code and the text keeps its indentation, so a screenshot of a function comes back as a function. That is the same layout work as Copy Text in the app.

When the picture does matter, such as a layout bug, capture window --json returns the path of a PNG that the agent can open.

There is also search. Every capture's text is indexed, so "find the screenshot with the stack trace" is one command.

You stay in control

A tool that reads your screen needs rules. Here are ours.

• Off by default. Nothing can connect until you open Settings → Integrations and switch on Allow scripts and agents to capture. Switch it off and it stops.
• Same user only. The app listens on a private socket. Your macOS user can reach it. Other accounts cannot.
• Never silent. Every capture made through the tool shows a "Captured by command line" notice.
• The app holds the permission. The tool does not get Screen Recording rights of its own. It asks the app, and the app asks macOS as it always does.
• Pickers wait for you. capture area and capture window without a rectangle open the normal picker and wait for you to choose.

One thing to be plain about: once access is on, anything running as you can use it. Agents can see what is on your screen when they ask. Turn it on when you want that, and off when you do not.

Setup in two steps

1. Settings → Integrations → Command-line tool → Install. It adds a link at ~/.local/bin/frayo. If your shell cannot find it, add that folder to your PATH.
2. Give your agent the skill file that comes with Frayo. It is a short note that says when to use each command and to prefer ocr when only the text matters. It also tells the agent to ask you to pick a window before it runs a picker, and not to retry in a loop if you cancel.

Exit codes are simple, so scripts can react: 0 is fine, 2 means the app is not reachable, and 3 means permission was denied or you pressed Esc.

What it is good for

• "I changed the layout. Check that it looks right."
• "What does the error in that dialog say?"
• "Find the screenshot I took of the stack trace on Tuesday."
• Scripts that capture a region every hour.

What it is not

It is not a remote desktop and it does not watch you. It does one thing per command, when asked.

Coming soon