How AI agents use mobile apps: reading the screen, choosing an action, and checking what happened.
Facts from mobile-mcp, Droidrun, AppAgent, AndroidWorld, Anthropic's computer use docs, Google's screen automation help page and AppFunctions, checked November 16, 2026. Phonebox output is copied from its docs.
The loop
- Observe. Capture the screen in a form the model can read.
- Decide. The model compares the screen with the goal and chooses one next step, or a few it's sure of.
- Act. A control layer turns the choice into a tap, typed text, a swipe or a key press on the phone.
- Check. Read the screen again. Did the expected screen appear? If not, decide again from what's actually there.
The check is what separates an agent from a script. A script assumes the tap worked; an agent looks. Most failures in practice are unexpected screens: a permission dialog, a cookie banner, an update prompt, a slow network spinner.
How the agent reads the screen
| Screenshot only | Accessibility tree | Both | |
|---|---|---|---|
| What the model gets | An image; it acts by pixel coordinates | Elements with text, resource ID, description, bounds and flags such as clickable | The element list, plus an image for what the list can't describe |
| Good at | Any app, including games and canvas UIs | Precise taps by label; cheap to send; quotable in reports | Most apps, with a fallback for the rest |
| Weak at | Small targets; coordinates break when the image is resized | Custom-drawn UIs and images with no labels | More to send per step |
| Used by | Anthropic computer use and OpenAI's computer tool (desktop) | UI Automator dumps, AppAgent's labeled elements | mobile-mcp (accessibility-first), Droidrun, Phonebox |
On Android the tree comes from the accessibility framework, the same one screen readers use; uiautomator dump writes it as XML. Phonebox condenses it into one line per element, numbered so the model can refer to it, and sends a screenshot alongside:
snapshot snp_4f2k7m3q6z3a
app com.android.launcher3
keyboard hidden
screen 1080x2400
[1] EditText "Search apps" #search clickable editable (540,250)
[2] TextView "Chrome" #icon clickable (200,1900)
[3] TextView "Settings" #icon clickable (500,1900)
[4] TextView "Play Store" #icon clickable (800,1900)Each line is a ref, the element's type, its text, its resource ID after #, its flags and its center in device pixels. The screenshot is 720 pixels wide, so a model that picks a point from the image divides by scale to get device pixels. Agents that skip this step tap the wrong place, which is the most common bug in screenshot-driven Android agents.
How it targets a tap
Coordinates are the least reliable target: a moved button or a different screen size breaks them. Phonebox takes five target forms: coordinates, a ref from the last observation, text, a resource ID or a content description. A batch of up to 20 actions runs in order and stops at the first failure, which saves a round trip per step:
{
"actions": [
{ "type": "open_app", "package": "com.android.settings" },
{ "type": "wait_for", "target": { "text": "Network & internet" }, "timeout_ms": 10000 },
{ "type": "tap", "target": { "text": "Network & internet" } },
{ "type": "scroll", "direction": "down", "amount": 0.5 },
{ "type": "back" }
],
"observe": "ui"
}A text target tries an exact match, then a case-insensitive one, then text that contains it, and prefers clickable elements. wait_for checks the screen every half second until the text appears, so the agent doesn't guess how long a screen takes to load.
How it recovers from a failed step
A failure is only useful if it tells the model what to do next. When a target matches several elements, Phonebox returns the candidates instead of guessing:
{
"index": 1, "type": "tap", "ok": false,
"error": {
"code": "ambiguous_target",
"message": "Several elements on the screen match the target.",
"retryable": false,
"next": "Add nth to the target, or tap by ref from the returned observation.",
"details": {
"candidates": [
{ "ref": 3, "type": "Button", "text": "Add to cart", "center": [860, 620] },
{ "ref": 5, "type": "Button", "text": "Add to cart", "center": [860, 900] },
{ "ref": 7, "type": "Button", "text": "Add to cart", "center": [860, 1180] }
]
}
}
}The model picks the right "Add to cart" by the product next to it and taps by ref. Two other failures matter: stale_snapshot means the screen changed since the ref was read, so observe again; action_outcome_unknown means a tap may or may not have happened. Phonebox never retries an action itself, because the tap might have been a payment. The agent reads the screen and decides.
Where the phone runs
| The user's own phone | Emulator or USB phone | Cloud phone | |
|---|---|---|---|
| Example | Gemini screen automation (beta) | mobile-mcp, Droidrun, Appium MCP over ADB | Phonebox, over REST, CLI, SDK or MCP |
| Who can use it | Gemini users on Pixel 10 and 11 and recent Galaxy devices, in five countries | Anyone with the Android SDK, or a phone and a cable | Any agent with an API key |
| Which apps | Rides, food and grocery (more on some devices) | Any app you install | Any APK you install, or the Play Store after a human signs in |
| Runs where the agent runs | On the device | Next to the agent's machine | Anywhere: CI, cloud sandboxes, servers |
| Keeps sign-ins | Yes, it's the user's phone | While the emulator image lasts | Yes; a parked phone keeps apps and sign-ins and costs nothing |
A fourth path skips the screen entirely: Android 16's AppFunctions let an app expose functions an agent can call directly. It's an experimental preview, and it only covers apps that adopt it. For the apps that don't, an agent still has to use the screen.
What still needs a human
- Sign-ins with a second factor. The code goes to the account owner. Phonebox creates a live view link the owner opens in a browser to enter it; the agent waits for the next screen, and the phone stays signed in after that.
- CAPTCHAs and bot checks. They exist to stop automation; hand them to a person, or use an account and app that allow automation.
- Irreversible steps. Payments, sends and deletions: have the agent stop and ask before it taps.
- Terms of service. Some apps forbid automated use. Check before you automate someone else's app.
How good are agents at it?
Good enough that the standard benchmark ran out of headroom. AndroidWorld, from Google researchers, has 116 tasks across 20 real apps on an emulator; by late 2026 several teams self-reported 100%, and the leaderboard is archived as saturated. Real use is harder than a benchmark: logins, pop-ups, slow networks and apps that change every week. That's why the check step, readable failures and a human handoff matter more than raw model skill.
Questions
How do AI agents use mobile apps?
In a loop: the agent reads the screen (as a screenshot, the accessibility tree's list of elements, or both), the model chooses the next action toward the goal, a control layer taps, types or swipes on the phone, and the agent reads the screen again to check the result. The phone can be the user's own, an emulator or USB device driven over ADB, or a cloud phone reached through an API or MCP server such as Phonebox.
Can an AI assistant use apps on my phone for me?
On some phones, yes: Google's Gemini screen automation, in beta in late 2026, runs tasks in ride, food and grocery apps on Pixel 10 and 11 and recent Galaxy devices in five countries. For any app and any agent, a cloud phone works instead: Phonebox gives an assistant its own Android phone, which you sign into once through a live view link.
Do AI agents read the screen with screenshots or the accessibility tree?
Both are used. Screenshot-only agents act by pixel coordinates and work on any UI, but miss small targets. Accessibility-tree agents get each element's text, ID and position, which makes taps precise and reports quotable, but miss custom-drawn UIs. Tools such as mobile-mcp, Droidrun and Phonebox send the element list and fall back to the screenshot.
Get started with Phonebox. New accounts start with a free 7-day trial, card required.
The Phonebox team