Actions and targets

Every action type, how targets find elements, the rules of a batch, and what to do when an action fails or its outcome is unknown.

You act on a phone by sending a batch of actions to POST /v1/phones/{id}/actions. A batch holds up to 20 actions, which run in order, and its response tells you how each one went. The phone must be ready, and the key needs the phones:control scope.

POST /v1/phones/{id}/actions

{
  "actions": [
    { "type": "open_app", "package": "com.android.settings" },
    { "type": "wait_for", "target": { "text": "Network & internet" }, "timeout_ms": 10000 },
    { "type": "tap", "target": { "text": "Network & internet" } },
    { "type": "scroll", "direction": "down", "amount": 0.5 },
    { "type": "back" }
  ],
  "observe": "none"
}

curl https://phonebox.dev/v1/phones/ph_7kx2m6q4v3ta/actions \
  -H "Authorization: Bearer $PHONEBOX_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"actions": [{"type": "open_app", "package": "com.android.settings"}, {"type": "wait_for", "target": {"text": "Network & internet"}, "timeout_ms": 10000}, {"type": "tap", "target": {"text": "Network & internet"}}, {"type": "scroll", "direction": "down", "amount": 0.5}, {"type": "back"}], "observe": "none"}'

Response: batch

{
  "results": [
    { "index": 0, "type": "open_app", "ok": true, "ms": 1311 },
    { "index": 1, "type": "wait_for", "ok": true, "ms": 1024 },
    { "index": 2, "type": "tap", "ok": true, "ms": 206, "resolved": { "ref": 4, "center": [540, 620] } },
    { "index": 3, "type": "scroll", "ok": true, "ms": 688 },
    { "index": 4, "type": "back", "ok": true, "ms": 143 }
  ],
  "completed": 5
}

Each result has the action's index and type, whether it succeeded (ok), and how long it took in milliseconds (ms). A tap, double tap or long press also has resolved: the center it acted on and, when the target was an element, that element's ref. A failed action has an error instead. completed counts the actions that succeeded.

Action types

`type`	Fields	What it does
`tap`	`target`	Taps the target's center.
`double_tap`	`target`	Taps it twice, 80 ms apart.
`long_press`	`target`, and `duration_ms` from 300 to 5000, 800 by default	Presses and holds the target.
`swipe`	`from` and `to` as `{"x": …, "y": …}`, and `duration_ms` from 50 to 5000, 300 by default	Swipes from one point to the other.
`scroll`	`direction` (`up`, `down`, `left` or `right`), an optional element `target`, and `amount` from 0.1 to 1, 0.6 by default	Scrolls the screen, or inside the target element. The swipe stays within the middle 60% of the element or screen, and `amount` is how much of that span it covers.
`type`	`text` of up to 5000 characters, `clear` (default `false`) and `submit` (default `false`)	Types into the focused text field. `clear` empties the field first, and `submit` presses Enter afterwards.
`key`	`key`: a key name, or an Android key code from 0 to 400	Presses one key.
`back`, `home`, `recents`, `notifications`	None	Presses Back, goes to the home screen, opens the recent apps, or opens the notifications.
`open_app`	`package`, and an optional `activity`	Opens an installed app, at that activity if you name one.
`close_app`	`package`, and `clear_data` (default `false`)	Stops an app. `clear_data` also erases the app's data, its signed-in accounts included.
`open_url`	`url`, and an optional `package`	Opens a web address or a deep link, in that app if you name one.
`wait`	`ms`, from 1 to 10000	Pauses for that many milliseconds.
`wait_for`	A text, ID or description `target`, `gone` (default `false`), and `timeout_ms` from 100 to 30000, 10000 by default	Checks the screen every half second until the target appears, or with `gone` until it disappears.

The key names are enter, delete, tab, escape, space, up, down, left, right, page_up, page_down, move_home, move_end, menu, search, volume_up and volume_down.

Targets

Each target takes exactly one of these forms:

Form	Example	How it finds the element
Coordinates	`{"x": 540, "y": 950}`	A point in device pixels.
Ref	`{"ref": 3, "snapshot_id": "snp_4f2k7m3q6z3a"}`	The element with that ref in an earlier observation, found again on the screen as it is now.
Text	`{"text": "Sign in"}` or `{"text": "Sign in", "nth": 2}`	An element whose text or description matches, as described below.
ID	`{"id": "sign_in"}`	An element with that resource ID, either in full or the part after `:id/`.
Description	`{"desc": "Back"}`	An element whose content description matches, in the same way as text.

A text target tries three levels, in order:

text or a description that is exactly the target;
the same match, ignoring case;
text or a description that contains the target, ignoring case.

The first level that finds anything decides, and within it clickable elements win over the rest. If one element remains, that is the target. nth picks among several, counting from 1 in reading order. If several remain and there is no nth, the action fails with ambiguous_target. Only text targets take nth: an ID or description target that matches several elements fails with ambiguous_target too, and you narrow it, for example with the full resource ID, or target one of the candidates by ref.

A ref target is checked before it is used. The element must still be on the screen with the same type, resource ID, text and description, and overlap its old position by at least half. Otherwise the action fails with stale_snapshot, and you observe again.

When a target finds nothing, the action fails with target_not_found. Its details.target repeats your target, and details.candidates lists up to 10 elements that share a word with it, or else clickable elements. When nth is larger than the number of matches, details.matches says how many there were.

Batch rules

A batch has from 1 to 20 actions, and the ms of its wait actions and the timeout_ms of its wait_for actions may add up to 60 seconds at most. A wait_for without a timeout_ms counts as 10 seconds. A batch that breaks either rule fails with 400 validation_failed.
Actions run in order, and the batch stops at the first action that fails. results has an entry for each action that ran, ending with the one that failed, and the actions after it never run. The actions before the failed one did run, so when you try again, send only the failed action, changed as needed, and the ones after it.
A batch that has run for 90 seconds starts no more actions. The next one fails with the retryable provider_timeout, and its details.not_run counts the actions that didn't run.
observe chooses what the response shows of the screen afterwards. ui, the default, adds the element list as an observation. screenshot adds only a 720-pixel-wide JPEG screenshot, both adds both, and none adds nothing.
A batch that stops early shows the element list, whatever observe asked for, so you can see where it stopped, unless the screen can't be read, as the next rule describes. none and ui add it without a screenshot, and screenshot and both add it with one.
Reading the screen after a batch gets only the time the request has left. When the read doesn't finish in time, the response has results and completed but no observation, so observe the screen before you continue.
A request that runs a batch may take up to 300 seconds, so give your HTTP client a timeout of about 300 seconds for batches.
Phonebox never retries an action for you, and it never replays a batch. A retry is always your decision, made after you look at the screen.

Problems with the request itself fail the whole request with an error envelope and an HTTP error status. That covers the key, the body, a phone that isn't ready, rate limits, and a phone that couldn't be reached before the first action. Once the batch runs, a failed action is part of a 200 response, with ok: false and an error in its result.

When an action fails

This batch stops at its second action, because three buttons on the screen read "Add to cart":

POST /v1/phones/{id}/actions

{
  "actions": [
    { "type": "scroll", "direction": "down" },
    { "type": "tap", "target": { "text": "Add to cart" } },
    { "type": "wait_for", "target": { "text": "Added to cart" }, "timeout_ms": 5000 }
  ]
}

Response: batch

{
  "results": [
    { "index": 0, "type": "scroll", "ok": true, "ms": 702 },
    {
      "index": 1, "type": "tap", "ok": false, "ms": 391,
      "error": {
        "code": "ambiguous_target",
        "message": "Several elements on the screen match the target.",
        "retryable": false,
        "next": "Add nth to the target, or tap by ref from the returned observation.",
        "details": {
          "candidates": [
            { "ref": 3, "type": "Button", "text": "Add to cart", "desc": null, "id": "com.example.shop:id/add_to_cart", "center": [860, 620] },
            { "ref": 5, "type": "Button", "text": "Add to cart", "desc": null, "id": "com.example.shop:id/add_to_cart", "center": [860, 900] },
            { "ref": 7, "type": "Button", "text": "Add to cart", "desc": null, "id": "com.example.shop:id/add_to_cart", "center": [860, 1180] }
          ],
          "snapshot_id": "snp_4f2k7m3q6z3a"
        }
      }
    }
  ],
  "completed": 1,
  "observation": {
    "snapshot_id": "snp_4f2k7m3q6z3a",
    "taken_at": "2026-09-29T11:15:09.240Z",
    "app": { "package": "com.example.shop", "activity": null },
    "keyboard": { "visible": false, "focused_editable": false },
    "screen": { "width": 1080, "height": 2400 },
    "elements": [
      { "ref": 1, "type": "EditText", "text": "Search products", "desc": null, "id": "com.example.shop:id/search", "bounds": [60, 140, 1020, 240], "center": [540, 190], "state": ["clickable", "editable"] },
      { "ref": 2, "type": "TextView", "text": "Wireless earbuds", "desc": null, "id": null, "bounds": [60, 560, 640, 680], "center": [350, 620], "state": [] },
      { "ref": 3, "type": "Button", "text": "Add to cart", "desc": null, "id": "com.example.shop:id/add_to_cart", "bounds": [700, 560, 1020, 680], "center": [860, 620], "state": ["clickable"] },
      { "ref": 4, "type": "TextView", "text": "Phone stand", "desc": null, "id": null, "bounds": [60, 840, 640, 960], "center": [350, 900], "state": [] },
      { "ref": 5, "type": "Button", "text": "Add to cart", "desc": null, "id": "com.example.shop:id/add_to_cart", "bounds": [700, 840, 1020, 960], "center": [860, 900], "state": ["clickable"] },
      { "ref": 6, "type": "TextView", "text": "USB-C cable", "desc": null, "id": null, "bounds": [60, 1120, 640, 1240], "center": [350, 1180], "state": [] },
      { "ref": 7, "type": "Button", "text": "Add to cart", "desc": null, "id": "com.example.shop:id/add_to_cart", "bounds": [700, 1120, 1020, 1240], "center": [860, 1180], "state": ["clickable"] }
    ],
    "truncated": false,
    "text": "app com.example.shop\nkeyboard hidden\nscreen 1080x2400\n[1] EditText \"Search products\" #search clickable editable (540,190)\n[2] TextView \"Wireless earbuds\" (350,620)\n[3] Button \"Add to cart\" #add_to_cart clickable (860,620)\n[4] TextView \"Phone stand\" (350,900)\n[5] Button \"Add to cart\" #add_to_cart clickable (860,900)\n[6] TextView \"USB-C cable\" (350,1180)\n[7] Button \"Add to cart\" #add_to_cart clickable (860,1180)"
  }
}

The screen may move between the moment an action fails and the moment the batch reads the screen. Phonebox therefore matches each candidate to the element in the returned observation, gives it that element's ref, and names the observation's snapshot_id in details.snapshot_id. If any candidate is no longer on the screen, details keeps the refs as the action saw them and names no snapshot, and you observe again before acting by ref.

To tap the phone stand's button, target it by ref with that snapshot, or by its text with nth:

POST /v1/phones/{id}/actions

{
  "actions": [
    { "type": "tap", "target": { "ref": 5, "snapshot_id": "snp_4f2k7m3q6z3a" } },
    { "type": "wait_for", "target": { "text": "Added to cart" }, "timeout_ms": 5000 }
  ]
}

POST /v1/phones/{id}/actions

{
  "actions": [
    { "type": "tap", "target": { "text": "Add to cart", "nth": 2 } }
  ]
}

These are the errors an action's result can carry, and what to do about each:

Code	Meaning	What to do
`target_not_found`	No element matches the target.	Observe, then use one of `details.candidates`, or scroll to bring the element into view.
`ambiguous_target`	Several elements match.	Target a ref from `details.candidates` with `details.snapshot_id`, or add `nth` to a text target. Only text targets take `nth`, so narrow an ID or description target instead, for example with the full resource ID.
`stale_snapshot`	The element behind a ref changed or went away.	Observe again and use the new refs.
`no_focused_field`	A `type` action found no focused text field.	Tap the field first, then type.
`wait_timeout`	A `wait_for` ran out of time. It is retryable, and `details` has the `timeout_ms` and `target`.	Observe. If the screen is still loading, wait again with a longer `timeout_ms`.
`app_not_found`	`open_app` or `close_app` named an app that isn't installed.	Install it, or check the package name.
`action_outcome_unknown`	The phone didn't confirm the action. It may or may not have happened.	Observe before anything else. See below.
`phone_unavailable`	The phone was briefly out of reach.	When `retryable` is true, the action didn't run, and you can send it and the rest of the batch again.
`rate_limited`	The phone service was busy.	Wait `details.retry_after_seconds`, then send the rest of the batch again.
`provider_error`	The phone service refused the action.	Observe, then decide whether to try again.
`provider_timeout`	The batch ran out of time before this action. `details.not_run` counts the actions that didn't run.	Observe, then send them again in a new batch.
`capacity_unavailable`	The phone service had no capacity for this action, so it didn't run.	Send it and the actions after it again later.
`internal_error`	Something failed on our side, and the action may have run.	Observe before you decide what to send again.

Nothing ran

When the first action of a batch couldn't reach the phone at all, the whole request fails with 503 phone_unavailable, and both Retry-After and details.retry_after_seconds say to wait 2 seconds. Nothing was done on the phone, so sending the same batch again is safe:

Response: error

{
  "error": {
    "type": "provider_error",
    "code": "phone_unavailable",
    "message": "The phone is temporarily unavailable.",
    "retryable": true,
    "next": "Nothing was done on the phone; retry this action in a few seconds.",
    "request_id": "req_w5x3y6z2a4b7",
    "details": { "retry_after_seconds": 2 }
  }
}

This happens most often in the first seconds after a phone becomes ready. Other failures before the first action mean that nothing ran, too: 503 capacity_unavailable and 429 rate_limited come with a Retry-After, while 502 provider_error, and a 503 phone_unavailable during maintenance, come without one. When a later action couldn't reach the phone, the earlier ones already ran, so the batch returns 200 as usual. The failed action's result then has phone_unavailable with retryable set to true, which means that action didn't run, and you can send it and the ones after it again.

These rules hold only for Phonebox's own errors, which carry the error envelope with an error.code. A 502 or 504 without it, which a proxy or the hosting platform sends, a timeout on your side, or a dropped connection says nothing about the batch, which may have run: observe before you send it again.

Unknown outcomes

action_outcome_unknown means the phone didn't confirm the action: it may have happened or not. Its retryable is always false. Never repeat such an action blindly, since a second tap on "Pay" or a second copy of a message can do real harm. Observe the screen, work out whether the action took effect, and continue from there.

Partial actions

Some actions take two steps. type with submit types the text and then presses Enter, and double_tap taps twice. When the second step fails after the first one reached the phone, the result has details.partial set to true and retryable set to false, whatever the error, and next says what was done. In this batch, the text was typed but Enter never reached the phone:

Response: batch

{
  "results": [
    { "index": 0, "type": "tap", "ok": true, "ms": 197, "resolved": { "ref": 2, "center": [540, 700] } },
    {
      "index": 1, "type": "type", "ok": false, "ms": 1214,
      "error": {
        "code": "phone_unavailable",
        "message": "The phone is temporarily unavailable.",
        "retryable": false,
        "next": "The text was typed but Enter wasn't pressed. Observe the screen, then send a key action for enter.",
        "details": { "retry_after_seconds": 2, "partial": true }
      }
    }
  ],
  "completed": 1,
  "observation": {
    "snapshot_id": "snp_7d3h6j2n5p4r",
    "taken_at": "2026-09-29T11:32:47.615Z",
    "app": { "package": "com.example.app", "activity": null },
    "keyboard": { "visible": true, "focused_editable": true },
    "screen": { "width": 1080, "height": 2400 },
    "elements": [
      { "ref": 1, "type": "TextView", "text": "Welcome back", "desc": null, "id": null, "bounds": [60, 400, 1020, 520], "center": [540, 460], "state": [] },
      { "ref": 2, "type": "EditText", "text": "alex@example.com", "desc": null, "id": "com.example.app:id/email", "bounds": [60, 640, 1020, 760], "center": [540, 700], "state": ["clickable", "editable", "focused"] },
      { "ref": 3, "type": "Button", "text": "Sign in", "desc": null, "id": "com.example.app:id/sign_in", "bounds": [60, 890, 1020, 1010], "center": [540, 950], "state": ["clickable"] }
    ],
    "truncated": false,
    "text": "app com.example.app\nkeyboard visible, text field focused\nscreen 1080x2400\n[1] TextView \"Welcome back\" (540,460)\n[2] EditText \"alex@example.com\" #email clickable editable focused (540,700)\n[3] Button \"Sign in\" #sign_in clickable (540,950)"
  }
}

Observe before you continue, and never press Enter blindly. Here phone_unavailable means Enter never reached the phone, and the observation still shows the filled field, so a key action for enter finishes the step. Don't send a second type, which would type the text again. When the error is action_outcome_unknown instead, Enter may have been pressed after all: check that the screen hasn't moved on before you press it.

Wait for the screen

Screens take time to change after a tap. Instead of fixed pauses, use wait_for to wait for what you expect to see, or for a spinner to go away:

POST /v1/phones/{id}/actions

{
  "actions": [
    { "type": "tap", "target": { "text": "Sign in" } },
    { "type": "wait_for", "target": { "text": "Loading" }, "gone": true, "timeout_ms": 20000 },
    { "type": "wait_for", "target": { "text": "Welcome" }, "timeout_ms": 5000 }
  ],
  "observe": "ui"
}

wait_for finds its target the same way as a text, ID or description target, and it checks the screen every 500 ms. When the time runs out, it fails with wait_timeout, which is retryable. The CLI's phonebox wait exits with code 3 in that case.

In the CLI and the SDK

The CLI's tap, type, key, swipe, scroll, open, wait, back, home and recents commands, and the matching SDK methods such as tap(), type() and waitFor(), each send a batch of one action with "observe": "none". When that action fails, the CLI prints its error and exits with code 1, or 3 when wait runs out of time, and the SDK throws a PhoneboxError. phonebox act and the SDK's act() send a whole batch and return every result, so check completed and each ok yourself.

Action typesCopy Anchor Link

TargetsCopy Anchor Link

Batch rulesCopy Anchor Link

When an action failsCopy Anchor Link

Nothing ranCopy Anchor Link

Unknown outcomesCopy Anchor Link

Partial actionsCopy Anchor Link

Wait for the screenCopy Anchor Link

In the CLI and the SDKCopy Anchor Link

On this page

Action types

Targets

Batch rules

When an action fails

Nothing ran

Unknown outcomes

Partial actions

Wait for the screen

In the CLI and the SDK