My AI Agent Drove My iPhone for a Day. Then I Parked It.

Saturday, coffee in hand, the dangerous kind of simple idea in my head.

On my phone sits an app I reach for constantly — no API, no web version worth the name, everything locked behind its own screens. Every engineer I know is busy wiring AI agents into terminals, browsers and incident channels. But agents that can only talk are half an agent, and the phone is where the world actually transacts. If an agent is ever going to be useful beyond a chat window, it has to work where the work is: inside apps like that one.

So: give my AI agent (Muse — the same one that runs my blog pipeline) hands on my real iPhone. Open the app, read its screens, tap through it with me in the loop. Eyes and fingers, under orders — nothing gets tapped behind my back.

Down this road, before you start:

• The exact build and link fixes for Xcode 27.1 / iOS 27 — including two gates nobody warns you about.

• Two small Objective-C patches for private Apple methods iOS 27 silently deleted.

• A 14-step copy-paste recipe ending in an automation server on your phone.

• The proof: no cable, control over Tailscale, a live screen read, one remote tap.

• And the honest bill — the friction that made me park it the same day.

What I wanted my agent to do

The standard tool for handing an agent an iPhone is WebDriverAgent (WDA). Plain words: a test-runner app that moonlights as a remote control. Born at Facebook, now maintained by the Appium project. You build it in Xcode (Apple’s development environment), sign it with your own Apple ID, and install it on the phone like any app you wrote yourself.

Once it runs, WDA starts an HTTP server on port 8100. That is the whole trick — the phone becomes a tiny web API. Send a GET for the source and it walks the current app’s view hierarchy — every button, label and field — and hands it back as XML. Send a POST with coordinates and it taps the glass. No computer vision, no screenshot-guessing. Just HTTP.

Or at least, that is how it is supposed to go.

The morning: a “successful” test that never ran

The rig: a current iPhone on iOS 27.0.1, an Apple-silicon Mac. Even step zero had a cover charge — the Mac had to hop from macOS 26.5.1 to 26.7.1 first, because current Xcode refuses to install below macOS 26.6.

Then the first self-inflicted wound. The WDA source cloned that morning was branch 3.17 of the old Facebook repo — a vintage tree, discovered to be vintage only in the evening. Maintenance had long moved to Appium. Half the day’s fires trace back to that: old code, brand-new operating system.

Signing was easy: personal Apple ID, free “Personal Team”, automatic signing, unique bundle IDs — the stock com.facebook.* IDs can’t be provisioned by a personal team, so they became com.yourname.WebDriverAgentRunner and com.yourname.WebDriverAgentLib.

Hit build. Three brick walls in under five minutes.

Wall 1 — the test runs died in 0.6 seconds: IPHONEOS_DEPLOYMENT_TARGET is set to 10.0, supported range is 15.0 to 27.1.x. Xcode 27 looked at a deployment target from the iOS 10 era and laughed. Both targets went to 15.0.

Wall 2 — the project’s base config (IOSSettings.xcconfig) shipped GCC_TREAT_WARNINGS_AS_ERRORS = YES paired with -Weverything. Against the new iOS 27 SDK headers, warnings breed like rabbits, and every one was a build error. That went to NO.

Wall 3 — the linker, on Apple’s private automation framework: Cannot link directly with 'XCTAutomationSupport' because product being built is not an allowed client of it. XCTAutomationSupport is the engine room under Apple’s UI-testing stack, protected by an allowlist — only binaries presenting an approved client name may link it. The surgery: build WebDriverAgentLib as a static library (MACH_O_TYPE = staticlib), link XCTAutomationSupport into the Runner with the PrivateFrameworks search path, and pass -Wl,-client_name,XCTest so the bundle presents itself as client “XCTest” — an approved name.

By midday the build went green: “Testing…”, app installed, Stop button live. I walked away pleased.

All afternoon Xcode showed the test running, and every fresh start was rejected — “There is already a test running.” When the session record finally closed it showed 20,301.4 seconds: 5.6 hours of “testing.” Nothing was listening on port 8100. No runner process existed on the phone. The evening’s evidence proved the test bundle could not load at all — the morning “success” was a session record that never died, not a server that ever ran.

The build was green, the app was installed, and the test had been “running” for 20,301 seconds. It had never loaded.

The evening: two gates and a missing framework

That evening I stopped trusting the GUI and asked the phone directly, via devicectl (Apple’s command-line device tool). Root Cause A, in its own words: Installing cryptex com.apple.MobileAsset.DDI is disallowed because developer mode is not enabled. (CoreDeviceError 12040/12052)

Developer Mode was OFF. Without it, the Developer Disk Image — the “DDI”, iOS’s bundle of developer services — cannot mount, the phone lists as connected (no DDI), and every launch dies at step 0. Fix: Settings → Privacy & Security → Developer Mode → ON, survive the reboot, confirm again after it.

Next, Root Cause B. The build now installed the runner; iOS refused to launch it. Xcode’s own alert: The application could not be launched because the Developer App Certificate is not trusted. Fix: Settings → General → VPN & Device Management → trust the Apple Development certificate under my own Apple ID. Two gates, both on the phone, neither mentioned by any happy-path tutorial I have ever read.

Then Root Cause C, the real one. A detached launch died instantly (exit code 83), and xcodebuild showed why: Library not loaded: @rpath/XCTAutomationSupport.framework/XCTAutomationSupport.

We had LINKED the framework but never EMBEDDED it — and it lives in none of the searched places: not the app bundle, not /System/Developer/Library/Frameworks, not the cryptex copies. It lives in PrivateFrameworks, and nothing had told the binary’s runpath (the @rpath list the loader searches at launch) to look there. Self-inflicted, too: the static merge made the test binary reference the framework directly, discarding the search paths the stock dynamic library carries.

The fix: two Runner build settings — add /System/Developer/Library/PrivateFrameworks and /private/preboot/Cryptexes/OS/System/Developer/Library/PrivateFrameworks to LD_RUNPATH_SEARCH_PATHS. On the rebuild: bundle loaded, runner started, automation screen up, and the console printed the line you wait all day for: ServerURLHere->http://<phone-wifi-ip>:8100<-ServerURLHere.

iOS 27 removed two methods. We patched both.

The server lived about three and a half minutes before it crashed with an NSInvalidArgumentException: Enqueue Failure: -[__NSXPCInterfaceProxy_XCTMessagingChannel_RunnerToDaemon _XCT_exchangeProtocolVersion:reply:]: unrecognized selector.

Plain words: WDA asks the phone’s test daemon “which protocol version do you speak?” In iOS 27, the daemon no longer answers that call. The old code — FBTestmanagerdVersion in FBXCodeCompatibility.m — asked with no guard and no error handling, so the exception executed the runner. The rude part: the proxy still claims to support the call. The interface declares it; the daemon just dropped it.

That number is informational only — it lands in session info, nothing steers by it — so a fallback of 0 is harmless. My agent read the source, found the send site, and patched it: respondsToSelector check with an early return, the exchange wrapped in @try/@catch, 0 kept on failure.

Crash 2 surfaced on the first real screen read (GET /source): -[XCUIApplicationProcess waitForQuiescenceIncludingAnimationsIdle:]: unrecognized selector. WDA waits for the app to go idle — “quiescence” — before snapshotting, through another private method iOS 27 removed. There is a settings knob, waitForIdleTimeout, that sounds exactly like the fix. We set it to 0. Nothing changed: the screen-read path funnels through a different timeout into the same raw send, in XCUIElement+FBUtilities.m (fb_waitUntilStableWithTimeout:). The knob exists. The knob is decorative. Same patch: guard it, catch it, and if the method is gone, skip the wait — best-effort.

// The pattern that fixes both iOS 27 crashes: never call a private
// method you have not checked for, and never let the check kill you.
if ([object respondsToSelector:@selector(thePrivateMethod:)]) {
  @try {
    [object thePrivateMethod:];
  } @catch (NSException *exception) {
    // Selector vanished mid-flight — fall back. Do not crash the runner.
  }
}

The recipe

Everything above, compressed into the checklist I wish had existed on Saturday morning. Every name and value is load-bearing.

# Project / target settings that made it build on Xcode 27.1 / iOS 27
IPHONEOS_DEPLOYMENT_TARGET = 15.0 # both targets (was 10.0)
GCC_TREAT_WARNINGS_AS_ERRORS = NO # IOSSettings.xcconfig
MACH_O_TYPE = staticlib # WebDriverAgentLib only

# Runner target
OTHER_LDFLAGS = "-Wl,-client_name,XCTest"
FRAMEWORK_SEARCH_PATHS = "$(PLATFORM_DIR)/Developer/Library/PrivateFrameworks"
LD_RUNPATH_SEARCH_PATHS = "/System/Developer/Library/PrivateFrameworks" "/private/preboot/Cryptexes/OS/System/Developer/Library/PrivateFrameworks"

1. Install current Xcode plus the iOS platform component; pair the phone and trust the Mac.

2. Clone the maintained Appium WebDriverAgent repo — not an old branch of the original.

3. Personal Team signing on both targets; unique bundle IDs (com.yourname.*).

4. Deployment target 15.0 on both targets.

5. IOSSettings.xcconfig: GCC_TREAT_WARNINGS_AS_ERRORS = NO.

6. WebDriverAgentLib: MACH_O_TYPE = staticlib.

7. Runner: link XCTAutomationSupport, add PrivateFrameworks to FRAMEWORK_SEARCH_PATHS, and OTHER_LDFLAGS += "-Wl,-client_name,XCTest".

8. Runner: LD_RUNPATH_SEARCH_PATHS += both PrivateFrameworks paths from the settings block.

9. On the phone: Developer Mode ON (Settings → Privacy & Security; expect a reboot and a second confirmation).

10. On the phone: trust the developer certificate (Settings → General → VPN & Device Management).

11. Patch FBXCodeCompatibility.m (FBTestmanagerdVersion): respondsToSelector guard + @try/@catch, fallback 0.

12. Patch XCUIElement+FBUtilities.m (fb_waitUntilStableWithTimeout:): guard + @try/@catch around the quiescence send; skip the wait if the method is gone.

13. Start with the xcodebuild test command below (~90 seconds to server-up). Optional, once, while cabled: tick “Connect via network” in Xcode’s Devices window so later starts run over Wi-Fi.

14. Drive it: GET /status → POST /session with empty capabilities (target app in the foreground) → GET /source → POST /wda/tap/0 with coordinates read from the XML.

The proof: no cable, one tap, real data

Control never needed the Mac relay we had built — a USB port-forwarder that never answered cleanly and turned out to be furniture. Plain curl from the agent’s machine, straight to the phone over Tailscale, the private mesh VPN where every device keeps a stable private address:

# Start (Mac, ~90 seconds to server-up). Full path — bare xcrun may
# point at the Command Line Tools and find nothing.
/Applications/Xcode.app/Contents/Developer/usr/bin/xcodebuild test \
  -project ~/Documents/WebDriverAgent/WebDriverAgent.xcodeproj \
  -scheme WebDriverAgentRunner \
  -destination id=<your-device-udid>

# Drive it from anywhere (here: over Tailscale, no cable)
curl http://<phone-tailscale-ip>:8100/status
curl -X POST http://<phone-tailscale-ip>:8100/session \
  -d '{"capabilities":{"alwaysMatch":{},"firstMatch":[{}]}}'
curl http://<phone-tailscale-ip>:8100/session/<session-id>/source
curl -X POST http://<phone-tailscale-ip>:8100/session/<session-id>/wda/tap/0 \
  -d '{"x":67,"y":904}'

GET /status answered: "WebDriverAgent is ready to accept commands", ready: true, iOS 27.0.1 — and the patch’s fingerprint, "testmanagerdVersion": 0. It answered again and again. A session was created, and /status stayed healthy after session creation — the runner had survived the exact call that killed it before the patch.

One driving landmine, learned live: creating a session bound to SpringBoard (the home-screen process) timed out and wedged the single-threaded server solid. Attach to a normal app in the foreground. Never target system apps.

Then the full loop. “Connect via network” was on. The cable run stopped, the cable came OUT, the start command ran again — session hosted over Wi-Fi. The agent attached to a target app (com.example.targetapp) and read its current screen back live — every label and figure exactly as displayed.

Element search by accessibility ID is broken on this tree (Invalid key path wdName specified in predicate format string), so the coordinates came straight from the source XML: a tab’s rectangle, center (67, 904). One POST to /wda/tap/0 — the tab switched on the phone in my hand — and the next screen read back cleanly: a list, every row and count intact.

Read + tap, end to end, zero cables anywhere in the workflow.

Why I parked it the same day

The control never missed. Every status, session, read and tap landed, all evening, over a VPN, on a phone with no cable in it. As a technology demo, flawless.

And that same evening I told the agent to switch it off and park the whole approach. The bill, unsoftened:

• While a session runs, the phone is not your phone. The automation screen owns the display. You are carrying a test rig, not a handset. There is no background mode — iOS does not offer one for this.

• Every start needs the Mac awake and a human pasting a Terminal command. Ninety seconds, yes — but a Mac, a human, and a paste, every single time. That same evening, starting it by driving Xcode’s GUI remotely failed twice. The paste is the only start that ever worked.

• Free signing expires every 7 days. Miss the weekly re-run and the runner is dead until somebody re-signs it. A paid developer account stretches that to a year — for $99 a year, rent on automating your own pocket hardware.

• The stack is an old branch held together by two source patches and build-setting surgery. iOS 27 deleted two private methods it stood on and told nobody. The next update can delete two more, and each one costs another evening exactly like this one.

The technology verdict and the ownership verdict are different verdicts. If this ever comes back, it comes back on a dedicated spare phone on a dock, next to an always-on Mac — never on the daily driver. Until then: parked, with the recipe left running in the one place it costs nothing — this page.

Save the recipe if you are heading down this road anyway. When your agent finally reads your phone screen, enjoy the ten minutes. Then count what it costs to keep.

© 2026 Honey Kiran. All rights reserved.

Honey Kiran · DevOps & Platform Engineering

I write about DevOps, platform engineering, Kubernetes, and putting AI agents to work — with guardrails, receipts, and the occasional parked project.

Running an agent against your own phone, or thinking about it? Tell me where yours broke — the replies are where the real field notes collect.

_Disclaimer

• This post describes my personal experience with my own systems and accounts, in my own words. It is a first-person account of what I observed and what I do in my own work — not a statement of any company's intentions, policies, or internal decisions. Where I do not know something, the post says so.
• I am not affiliated with, endorsed by, or speaking for any company, product, or vendor named in this post. All product and company names are trademarks of their respective owners and are mentioned only to describe the experience accurately.
• Any error messages, documentation excerpts, or figures quoted here are reproduced from my own records or from the public sources linked in the post, believed accurate at the time of writing. Product behavior changes over time; verify against current official documentation before relying on any detail.
• Nothing in this post is professional, financial, legal, or security advice for your specific situation. Examples are illustrative — validate in your own environment before running anything in production.
• Corrections are welcome. If any company named here publishes an official statement, or if any detail is wrong, I will update this post and note the correction. The goal is an accurate public record, not a verdict.

Examples are illustrative — validate in your own environment before running in production._