Actions
Run a batch of actions, with every action type, target form and limit, the result format, and the errors of a batch and of each action.
Actions and targets explains how batches, targets and failures work, with worked examples. This page is the exact reference.
Run actions
POST /v1/phones/{id}/actions needs the phones:control scope.
Runs up to 20 actions on a ready phone, in order, and stops at the first one that fails. The body is required and may be up to 200 KB.
| Field | Type | Default | Rules |
|---|---|---|---|
actions | array | None | From 1 to 20 actions. |
observe | string | ui | none, ui, screenshot or both: what the response shows of the screen afterwards. |
The ms of every wait and the timeout_ms of every wait_for in a batch may add up to 60,000 at most. A wait_for without timeout_ms counts as 10,000.
{
"actions": [
{ "type": "tap", "target": { "id": "email" } },
{ "type": "type", "text": "alex@example.com", "clear": true },
{ "type": "tap", "target": { "text": "Continue" } },
{ "type": "wait_for", "target": { "text": "Enter your password" }, "timeout_ms": 15000 }
],
"observe": "none"
}curl https://phonebox.dev/v1/phones/ph_7kx2m6q4v3ta/actions \
-H "Authorization: Bearer $PHONEBOX_API_KEY" \
-H "Content-Type: application/json" \
-d '{"actions": [{"type": "tap", "target": {"id": "email"}}, {"type": "type", "text": "alex@example.com", "clear": true}, {"type": "tap", "target": {"text": "Continue"}}, {"type": "wait_for", "target": {"text": "Enter your password"}, "timeout_ms": 15000}], "observe": "none"}'A batch that ran answers 200 OK, whether or not every action succeeded:
{
"results": [
{ "index": 0, "type": "tap", "ok": true, "ms": 231, "resolved": { "ref": 2, "center": [540, 700] } },
{ "index": 1, "type": "type", "ok": true, "ms": 1406 },
{ "index": 2, "type": "tap", "ok": true, "ms": 198, "resolved": { "ref": 3, "center": [540, 950] } },
{ "index": 3, "type": "wait_for", "ok": true, "ms": 2517 }
],
"completed": 4
}| Field | Meaning |
|---|---|
results | One entry for each action that ran, in order, ending with the one that failed if one did. Actions after a failure don't run and have no entry. |
results[].index, results[].type | The action's position in the batch, from 0, and its type. |
results[].ok | Whether the action succeeded. |
results[].ms | How long it took, in milliseconds. |
results[].resolved | For tap, double_tap and long_press: the center acted on and, when the target was an element, its ref on the screen as it was read for the action. |
results[].error | For a failed action: code, message, retryable, next and details, as in the action errors. |
completed | How many actions succeeded. |
observation | The screen afterwards, as observe asked. |
observe decides what observation holds:
observe | After a batch that completed | After a batch that stopped early |
|---|---|---|
none | No observation. | The element list. |
ui | The element list. | The element list. |
screenshot | Only taken_at and a JPEG screenshot 720 pixels wide. | The element list and the screenshot. |
both | The element list and the screenshot. | The element list and the screenshot. |
The element list is a full observation, with a new snapshot_id whose refs you can target. A batch that stops early includes it, so you can see where it stopped, as long as the screen can be read. Reading the screen afterwards gets only the time the request has left. If it fails or doesn't finish in time, the response leaves observation out, and the results still come back, so observe the screen before you continue.
When a batch stops
This batch opened Settings, then found nothing labeled Bluetooth:
{
"actions": [
{ "type": "open_app", "package": "com.android.settings" },
{ "type": "tap", "target": { "text": "Bluetooth" } }
]
}{
"results": [
{ "index": 0, "type": "open_app", "ok": true, "ms": 1284 },
{
"index": 1, "type": "tap", "ok": false, "ms": 367,
"error": {
"code": "target_not_found",
"message": "No element on the screen matches the target.",
"retryable": false,
"next": null,
"details": {
"target": { "text": "Bluetooth" },
"candidates": [
{ "ref": 1, "type": "TextView", "text": "Search settings", "desc": null, "id": "com.android.settings:id/search", "center": [540, 210] },
{ "ref": 2, "type": "TextView", "text": "Network & internet", "desc": null, "id": null, "center": [540, 520] },
{ "ref": 3, "type": "TextView", "text": "Connected devices", "desc": null, "id": null, "center": [540, 760] },
{ "ref": 4, "type": "TextView", "text": "Apps", "desc": null, "id": null, "center": [540, 1000] }
],
"snapshot_id": "snp_4f2k7m3q6z3a"
}
}
}
],
"completed": 1,
"observation": {
"snapshot_id": "snp_4f2k7m3q6z3a",
"taken_at": "2026-09-29T14:36:02.337Z",
"app": { "package": "com.android.settings", "activity": null },
"keyboard": { "visible": false, "focused_editable": false },
"screen": { "width": 1080, "height": 2400 },
"elements": [
{ "ref": 1, "type": "TextView", "text": "Search settings", "desc": null, "id": "com.android.settings:id/search", "bounds": [60, 160, 1020, 260], "center": [540, 210], "state": ["clickable"] },
{ "ref": 2, "type": "TextView", "text": "Network & internet", "desc": null, "id": null, "bounds": [0, 460, 1080, 580], "center": [540, 520], "state": ["clickable"] },
{ "ref": 3, "type": "TextView", "text": "Connected devices", "desc": null, "id": null, "bounds": [0, 700, 1080, 820], "center": [540, 760], "state": ["clickable"] },
{ "ref": 4, "type": "TextView", "text": "Apps", "desc": null, "id": null, "bounds": [0, 940, 1080, 1060], "center": [540, 1000], "state": ["clickable"] }
],
"truncated": false,
"text": "app com.android.settings\nkeyboard hidden\nscreen 1080x2400\n[1] TextView \"Search settings\" #search clickable (540,210)\n[2] TextView \"Network & internet\" clickable (540,520)\n[3] TextView \"Connected devices\" clickable (540,760)\n[4] TextView \"Apps\" clickable (540,1000)"
}
}The action read the screen before the batch's observation did, and the screen can move in between. So Phonebox looks up each candidate of a target_not_found or ambiguous_target in the returned observation, gives it that element's ref and center, and names the observation's snapshot_id in details.snapshot_id. When any candidate isn't there anymore, details keeps the refs as the action saw them and names no snapshot, so observe again before you act by ref. Here, the next step is to tap one of the candidates, such as {"ref": 3, "snapshot_id": "snp_4f2k7m3q6z3a"}, and look again.
The actions before the one that failed have already run, so the next batch starts at the failed action, changed as needed, followed by the actions that came after it. Never send the whole batch again: here, a new batch would not open Settings a second time.
When nothing ran
Every error that Phonebox sends from this endpoint, in the error envelope with its error.code, means that no action ran, except 500 internal_error. That covers the request's own problems, such as a bad body or a phone that isn't ready, and a phone service that failed before the first action reached the phone:
{
"error": {
"type": "provider_error",
"code": "phone_unavailable",
"message": "The phone is temporarily unavailable.",
"retryable": true,
"next": "Nothing was done on the phone; retry this action in a few seconds.",
"request_id": "req_t3v5x2z6b4d7",
"details": { "retry_after_seconds": 2 }
}
}A batch refused this way is safe to send again after the wait. Any other failure leaves the outcome unknown: a 500 internal_error, a 502 or 504 without the envelope, which a proxy or the hosting platform sends, a timeout on your side, a dropped connection, or no answer at all. The batch may have run, in part or in full, so observe before you send anything again. Failures without the envelope explains why.
| Error | When |
|---|---|
400 validation_failed | The body breaks a rule: an unknown action type or field, a value out of range, more than 20 actions, or more than 60 seconds of waits. |
413 payload_too_large | The body is larger than 200 KB. |
503 phone_unavailable | The first action couldn't reach the phone, so nothing ran. This error is retryable. When the phone's control service was out of reach, Retry-After and details.retry_after_seconds are 2 seconds. Otherwise the response has no Retry-After, so wait a few seconds. |
503 capacity_unavailable | The phone service had no capacity for the first action, so nothing ran. Try again after Retry-After. |
429 rate_limited | The phone service asked Phonebox to slow down before the first action, so nothing ran. Retry-After and details.retry_after_seconds say how long to wait. |
502 provider_error | The phone service failed before the first action reached the phone, for example while reading the screen to find the target. Nothing ran, and this error is retryable. |
The readiness errors in Common errors apply too.
Action types
type | Fields | Rules and defaults |
|---|---|---|
tap | target | Any target. |
double_tap | target | Two taps 80 ms apart. |
long_press | target, duration_ms | duration_ms from 300 to 5000, 800 by default. |
swipe | from, to, duration_ms | from and to are points, {"x": …, "y": …}. duration_ms from 50 to 5000, 300 by default. |
scroll | direction, target, amount | direction is up, down, left or right. target is optional: an element to scroll inside, or the whole screen when you leave it out or give a point. amount from 0.1 to 1, 0.6 by default. |
type | text, clear, submit | text has 1 to 5000 characters and goes into the focused field. clear empties the field first and submit presses Enter afterwards, both false by default. |
key | key | A key name, or an Android key code from 0 to 400. |
back, home, recents, notifications | None | Press Back, go home, open the recent apps, or open the notifications. |
open_app | package, activity | package is an Android package name. activity is optional, up to 300 characters. |
close_app | package, clear_data | clear_data, false by default, also erases the app's data and its sign-ins. |
open_url | url, package | url is a web address or a deep link, 3 to 2000 characters. package is optional: the app to open it in. |
wait | ms | From 1 to 10000 milliseconds. |
wait_for | target, gone, timeout_ms | target is a text, ID or description target. gone, false by default, waits for the target to disappear instead. timeout_ms from 100 to 30000, 10000 by default. It checks the screen every 500 ms. |
The key names are enter, delete, tab, escape, space, up, down, left, right, page_up, page_down, move_home, move_end, menu, search, volume_up and volume_down. A package name has at least two parts separated by dots, such as org.wikipedia, and at most 255 characters.
Targets
A target takes exactly one of these forms. Targets explains how each one finds its element.
| Form | Rules |
|---|---|
{"x": 540, "y": 950} | A point in device pixels. Each coordinate is an integer from 0 to 10000. |
{"ref": 3, "snapshot_id": "snp_4f2k7m3q6z3a"} | ref from 1 to 1000, and the snapshot_id of an observation from the last 15 minutes. The element must still be on the screen, unchanged. |
{"text": "Sign in", "nth": 2} | text from 1 to 500 characters, matched against text and descriptions. nth is optional, from 1 to 50. |
{"id": "sign_in"} | 1 to 300 characters: a full resource ID, or the part after :id/. |
{"desc": "Back"} | 1 to 500 characters, matched against content descriptions. |
Text, ID and description targets see the same elements that an observation lists, which are at most 250.
Action errors
A failed action's error has the same fields as an error envelope, without type and request_id. These are the codes it can carry:
| Code | Retryable | Details | What to do |
|---|---|---|---|
target_not_found | false | target, candidates (up to 10), snapshot_id, and matches when nth was larger than the number of matches | Use one of the candidates, scroll to bring the element into view, or observe again. |
ambiguous_target | false | candidates (up to 10), snapshot_id | Target a candidate by ref with details.snapshot_id, or add nth to a text target. Only text targets take nth, so narrow an ID or description target instead, for example with the full resource ID. |
stale_snapshot | false | snapshot_id | The element behind the ref changed or went away, or the snapshot is older than 15 minutes. Observe again. |
no_focused_field | false | None | A type action found no focused text field. Tap the field, then type. |
wait_timeout | true | timeout_ms, target | A wait_for ran out of time. Observe, and wait again with a longer timeout_ms if the screen is still loading. |
app_not_found | false | package | The app isn't installed, or its install hasn't finished. Install it, or check the package name. |
action_outcome_unknown | false | None | The phone didn't confirm the action, which may or may not have happened. Observe before anything else. |
phone_unavailable | true | retry_after_seconds, when the phone's control service was out of reach | This action didn't run. After a few seconds, send it and the actions after it again. |
rate_limited | true | retry_after_seconds | This action didn't run. Wait, then send it and the actions after it again. |
capacity_unavailable | true | None | This action didn't run. Try it and the actions after it again later. |
provider_error | true | None | The phone service refused or failed the action. Observe, then decide whether to try again. |
provider_timeout | true | not_run | The batch had already run for 90 seconds, so this action and the ones after it didn't run. not_run counts them. Observe, then send them in a new batch. |
internal_error | true | None | Something failed on our side. Observe before you decide. |
Partial actions
A type with submit types and then presses Enter, and a double_tap taps twice. When the second step fails after the first reached the phone, the error has details.partial set to true and retryable set to false, whatever its code, and next says which step was done. Observe before you do anything else. The second step may itself have happened, so send it only when the screen shows that it didn't, for example a form still waiting with your text in it. Never repeat the whole action.
Unknown outcomes shows how to recover from these cases.