Phone Harness Review: Let AI Agents Control Your Phone

Originally published on andrew.ooo — visit the original for any updates, code snippets that aged out, or follow-up posts.

TL;DR

Phone Harness is a thin Python layer that lets an AI agent — Claude Code, Codex, or anything that can run a shell command — drive your actual phone. Not a simulator, not a cloud device farm, not an API pretending to be a phone. The handset on your desk.

The trick is that it doesn't try to be clever. On iPhone it hijacks a window macOS already gives you (iPhone Mirroring), reads it with Apple's own OCR, and posts mouse and keyboard events into it. On Android it just shells out to adb. There is no daemon, no agent app installed on the phone, no WebDriverAgent, no jailbreak.

Key facts, verified August 30, 2026:

• 2,099 GitHub stars, 197 forks, 17 open issues — repo created August 7, 2026, last commit August 29, 2026
• MIT licensed, Python 3.10+, latest tag 0.2.0
• iPhone: macOS Sequoia or later + iPhone Mirroring; needs Accessibility and Screen Recording permissions for your terminal
• Android: adb over USB or Wi-Fi — no Mac window required, nothing needs to be in the foreground
• Ships as an agent skill (SKILL.md) that Claude Code and Codex load automatically
• Stateless per call — every invocation re-queries the screen, so there's no background process to babysit
• Known breakage: macOS 26 drops synthetic vertical drags in the mirroring window (open issue), and there's no multi-touch, no pinch, no Face ID flows

The honest summary: the iPhone path is a genuinely clever hack that inherits every fragility of hacks. The Android path is boring, solid, and the reason to actually install this.

The problem it solves

Agents got good at browsers. computer-use models, Playwright wrappers, browser harnesses — that space is crowded and mature. Phones stayed locked.

The reason is structural. A browser hands you a DOM: every element has a selector, a bounding box, and a semantic role. A phone screen hands you pixels. iOS in particular has no supported way to script a physical device from outside — Apple's automation story is XCUITest, which requires Xcode, a developer signing identity, and a test target compiled into the app you want to drive. That's fine if you own the app. It's useless if you want an agent to check your bank balance, reply to a text, or navigate an iOS-only app that has no web equivalent.

The existing workarounds were all heavy: jailbreak the device, run WebDriverAgent, buy cloud device time, or install a companion app. Each one is a project before you get to the actual task.

Phone Harness's insight is that macOS Sequoia already solved the hard part and nobody noticed. iPhone Mirroring renders your phone as an ordinary Mac window and forwards real mouse and keyboard input to it as touches. Apple built the transport. Phone Harness just points a script at it.

How it actually works

The architecture splits cleanly by platform, but exposes identical helpers.

iPhone. The Mac does all the work:

• Eyes — screencapture grabs just the mirroring window by window ID, then Apple's Vision framework OCRs it. You get back every visible string with a tap-ready center point. The README calls this "the poor man's DOM," which is exactly right.
• Hands — CGEvents posted at the HID tap level: taps, long presses, drags, scroll gestures, unicode typing, plus the mirroring app's own shortcuts (Cmd+1 Home, Cmd+2 App Switcher, Cmd+3 Spotlight).
• Verify — screenshot again. There is no DOM to assert against, so the capture is the ground truth.

Android. adb reaches the phone directly and none of the above is needed:

• screencap for pixels, but also the phone's real accessibility tree via uiautomator — meaning ocr() returns exact strings and exact boxes rather than inferred ones, and tap_ui("url_bar") can find elements OCR could never see.
• input tap/swipe/text for the hands. Coordinates are device pixels, so a screenshot maps 1:1 to tap(x, y) with no conversion.
• Nothing on the Mac needs focus. The phone can be in a drawer.

The harness auto-resolves which phone to use (USB first, then the paired Wi-Fi device), refuses to drive a locked handset, and can keep the screen awake for a session without permanently changing a setting.

Everything is stateless per invocation — window bounds and captures are re-queried each time, and adb runs its own server. Persistent state (default platform, remembered devices) lives in ~/.config/phone-harness.

Writing agent code against it

The interface is a heredoc. That's the whole API surface:

./phone-harness <<'PY'
open_app("Notes")
tap_text("New Note")
type_text("hello from the harness")
print([o["text"] for o in ocr()][:10])
PY

Helpers are pre-imported into the script namespace. The useful ones:

screen_info()                  # {window, frontmost, img_px}
ocr(min_confidence=0.3)        # [{text, confidence, x, y, w, h}]
find_text("Weather")           # visible text matching a query
tap_text("Weather", index=0)   # find + tap, raises with what IS visible
screenshot(path=None)          # PNG path — for icons OCR can't read
tap_image_point(x, y, image_size=...)   # tap using screenshot pixel coords
ui()                           # accessibility tree (Android)
tap_ui("url_bar", clickable_only=True)  # exact-match tap (Android)
scroll("down"); swipe("up"); long_press(x, y); press("return")
scroll_until(done)             # scroll until your predicate on OCR is met
scroll_collect(extract, key=...)  # walk a list, de-duping as it goes

Two design decisions in here are worth calling out, because they're the difference between a demo and something you'd trust.

First: the harness refuses to tell you whether an action worked. Helpers return observations — text, coordinates, before/after state — and never a verdict. The SKILL.md is blunt about why: the maintainer tried to build a generic "did the screen change" check, and "every rule that fit a list broke on a feed, and every rule that fit a feed broke on a strip that scrolls inside a still screen." So the loop is the method: name what should change, do one action, check that one thing, isolate when it fails.

That's unusually honest API design. Most agent tooling returns a confident boolean and lies.

Second: scroll and swipe deliberately disagree on direction.

scroll("down")   # show me what is further down
swipe("up")      # thumb goes up — "next video"

scroll takes content direction; swipe takes finger motion. It looks like a bug until you realise English does the same thing — "scroll down the page" and "swipe up for the next one" describe the same outcome. Note that this was a breaking change: the old scroll("up") is today's scroll("down").

Installation

The README's install path is itself a prompt you paste into your agent, which is a nice touch — the tool onboards itself. Manually:

git clone https://github.com/ShawnPana/phone-harness ~/.phone-harness
cd ~/.phone-harness
pip install -e .                    # global `phone-harness` command

# register as an agent skill
mkdir -p ~/.claude/skills/phone-harness
phone-harness skill > ~/.claude/skills/phone-harness/SKILL.md
mkdir -p "${CODEX_HOME:-$HOME/.codex}/skills/phone-harness"
phone-harness skill > "${CODEX_HOME:-$HOME/.codex}/skills/phone-harness/SKILL.md"

phone-harness config set platform ios      # or android
phone-harness --doctor

For iPhone you also need iPhone Mirroring paired once (requires the physical phone) and two permissions granted to your terminal, not to Python:

open "x-apple.systempreferences:com.apple.preference.security?Privacy_Accessibility"
open "x-apple.systempreferences:com.apple.preference.security?Privacy_ScreenCapture"

Accessibility takes effect immediately; Screen Recording only after you restart the terminal. The docs warn that --doctor can pass while taps still silently do nothing, because a fresh machine may prompt for additional permissions on first real action. Budget time for that.

For Android: brew install android-platform-tools, enable Developer options (tap Build number 7×), then USB debugging or Wireless debugging with a 6-digit pairing code.

The failure modes, documented

The most valuable section of the README is the list of things that don't work, learned the hard way on iPhone:

• AppleScript click at is silently ignored — the mirroring window is a video stream with no accessibility tree
• Unicode key payloads don't work — mirroring forwards raw HID keycodes, so typing must go through keycodes
• Slow touch-drags barely move an iOS list — use wheel scroll for lists, a fast flick for pages
• Input while the window isn't frontmost gets swallowed

That last one has a partial fix. The default build works the phone without stealing your focus: capture is by window ID, and taps and keystrokes are delivered as event records straight to the app. Scrolling is the exception, because macOS routes scrolls to whatever window is under the pointer — so a scroll briefly raises the mirroring window and hands focus back. You'll see a flicker on scrolls and nothing otherwise.

The two open issues on the repo right now are both real-world bites:

1. macOS 26 drops all synthetic scroll and vertical drag in iPhone Mirroring. Taps still work; a flick is the workaround. This is a live regression against a shipping OS.
2. type_text into a pre-filled field appends instead of replacing. Small, but exactly the sort of thing that silently corrupts an agent run.

The commit log from the last few days shows the maintainer actively fighting the scroll problem — scroll: refuse to fire a gesture into another window, fix: is_frontmost() false positive was routing scroll to the wrong app. That's healthy churn, but it's churn.

Community reception

The Show HN post landed with 2 points and no comments — a total miss. The traction came from elsewhere: 2,099 stars in three weeks, nearly 200 forks, a Trendshift listing, and writeups on Medium framing it as "Claude Code can control your iPhone now — no jailbreak, no API, no Xcode."

The adjacent Reddit conversation is where the real signal is. Threads in r/ClaudeAI about giving Claude iPhone control through iPhone Mirroring keep converging on the same observation: "Claude handles these well because it's just another look-at-screen, decide-what-to-tap cycle." That's the honest reason this category is suddenly viable — agents already learned this loop from browser and desktop use, so phone control needed a transport, not new model capability.

It's also not alone. droidrun/mobile-harness ships Markdown-only instructions for Android, iOS, and cloud phones; various mobile-mcp servers wrap the same idea as MCP tools. The r/ClaudeAI consensus on those is that iPhone support is where they fall down — and Phone Harness's mirroring route is currently the least-effort path to a real iOS device.

Who should use this

Good fit:

• You're on a Mac with an iPhone and want an agent to handle iOS-only apps, 2FA-bound flows, or anything tied to your phone number
• You're doing exploratory QA on a real Android device and want an agent that can read the accessibility tree rather than guess at pixels
• You want an editable harness — agent-workspace/agent_helpers.py is explicitly yours to extend, and the docs push you to accumulate working checks there

Bad fit:

• You want reliability. This is a three-week-old 0.2.0 with an active OS-level regression. It's a tool for supervised sessions, not unattended cron jobs.
• You're not on macOS. The iPhone path is macOS-only by construction. Android over adb would work anywhere in principle, but the project is Mac-centric.
• You need pinch, camera, Face ID, or DRM video. All explicitly out of scope.
• The task is doable on the web. The SKILL.md itself has a "When Not to Use" section telling agents to leave the phone alone if a website or API can do the job. Respect that — it's the correct call and it saves you a fragile dependency.

Comparison with alternatives

The differentiator is narrow but real: Phone Harness is the only one of these that drives a physical iPhone without Xcode, a signing identity, or a jailbreak. Everything else either skips iOS or demands the full Apple developer toolchain.

FAQ

Does Phone Harness need a jailbroken iPhone?
No. It uses macOS Sequoia's iPhone Mirroring, which is a supported Apple feature. No jailbreak, no Xcode, no WebDriverAgent, and nothing installed on the phone itself.

Does it work on Windows or Linux?
The Android path is adb-based and portable in principle, but the project targets macOS. The iPhone path is macOS-only by definition, since iPhone Mirroring is a macOS Sequoia feature.

Which AI agents can use it?
Any agent that can run a shell command. It ships a SKILL.md that registers as a native agent skill for Claude Code and Codex, so those two pick it up automatically. Anything else can invoke phone-harness with a heredoc.

Is it safe to give an agent control of my phone?
Treat it as high-risk and supervise it. The agent gets the same reach you have — banking apps, messages, purchases — with no permission scoping and no undo. The harness refuses to drive a locked phone and connecting the device is always a human action, but there is no sandbox once you're in. Run it on a session you're watching.

Why does swipe("up") scroll down?
By design. scroll takes the direction of the content you want to see; swipe takes the direction the finger moves. Only swipe uses finger motion — scroll, scroll_screen, scroll_until, and scroll_collect all take content direction.

Does scrolling work on the latest macOS?
Not reliably on iPhone. There is an open issue confirming macOS 26 drops synthetic scroll and vertical drag events in iPhone Mirroring; taps still work and a fast flick is the workaround. Android scrolling via adb is unaffected.

What's the license?
MIT. Free to use commercially. The maintainer funds it through GitHub Sponsors.

Verdict

Phone Harness is the clearest example this year of a tool that wins by not building anything. Apple shipped the transport; adb has existed for fifteen years; Vision OCR is a system framework. The contribution is 2,000 lines of Python that notice this and get out of the way.

The iPhone path is the headline and the weaker half — a clever exploitation of a window Apple never intended to be scripted, currently losing a fight with macOS 26 over scroll events. The Android path is unglamorous and genuinely good: real accessibility tree, exact element matching, no focus requirements, works while the phone sits in a drawer.

Install it for Android automation today. Use the iPhone side for supervised, exploratory work and expect to hit at least one of the documented failure modes in your first session. At 0.2.0 and three weeks old, that's a fair trade.

Sources

• ShawnPana/phone-harness on GitHub — README, SKILL.md, install.md, issues and commit history (accessed August 30, 2026)
• phone-harness.com — project site
• Show HN: Phone-harness — launch thread
• r/ClaudeAI: iPhone control through macOS iPhone Mirroring — community discussion of the approach
• droidrun/mobile-harness — alternative referenced in the comparison