# Actions and targets Source: https://phonebox.dev/docs/using-phones/actions > Every action type, how targets find elements, the rules of a batch, and what to do when an action fails or its outcome is unknown. You act on a phone by sending a batch of actions to `POST /v1/phones/{id}/actions`. A batch holds up to 20 actions, which run in order, and its response tells you how each one went. The phone must be `ready`, and the key needs the `phones:control` scope. ```json title="POST /v1/phones/{id}/actions" { "actions": [ { "type": "open_app", "package": "com.android.settings" }, { "type": "wait_for", "target": { "text": "Network & internet" }, "timeout_ms": 10000 }, { "type": "tap", "target": { "text": "Network & internet" } }, { "type": "scroll", "direction": "down", "amount": 0.5 }, { "type": "back" } ], "observe": "none" } ``` ```bash curl https://phonebox.dev/v1/phones/ph_7kx2m6q4v3ta/actions \ -H "Authorization: Bearer $PHONEBOX_API_KEY" \ -H "Content-Type: application/json" \ -d '{"actions": [{"type": "open_app", "package": "com.android.settings"}, {"type": "wait_for", "target": {"text": "Network & internet"}, "timeout_ms": 10000}, {"type": "tap", "target": {"text": "Network & internet"}}, {"type": "scroll", "direction": "down", "amount": 0.5}, {"type": "back"}], "observe": "none"}' ``` ```json title="Response: batch" { "results": [ { "index": 0, "type": "open_app", "ok": true, "ms": 1311 }, { "index": 1, "type": "wait_for", "ok": true, "ms": 1024 }, { "index": 2, "type": "tap", "ok": true, "ms": 206, "resolved": { "ref": 4, "center": [540, 620] } }, { "index": 3, "type": "scroll", "ok": true, "ms": 688 }, { "index": 4, "type": "back", "ok": true, "ms": 143 } ], "completed": 5 } ``` Each result has the action's `index` and `type`, whether it succeeded (`ok`), and how long it took in milliseconds (`ms`). A tap, double tap or long press also has `resolved`: the `center` it acted on and, when the target was an element, that element's `ref`. A failed action has an `error` instead. `completed` counts the actions that succeeded. ## Action types [#action-types] | `type` | Fields | What it does | | ------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `tap` | `target` | Taps the target's center. | | `double_tap` | `target` | Taps it twice, 80 ms apart. | | `long_press` | `target`, and `duration_ms` from 300 to 5000, 800 by default | Presses and holds the target. | | `swipe` | `from` and `to` as `{"x": …, "y": …}`, and `duration_ms` from 50 to 5000, 300 by default | Swipes from one point to the other. | | `scroll` | `direction` (`up`, `down`, `left` or `right`), an optional element `target`, and `amount` from 0.1 to 1, 0.6 by default | Scrolls the screen, or inside the target element. The swipe stays within the middle 60% of the element or screen, and `amount` is how much of that span it covers. | | `type` | `text` of up to 5000 characters, `clear` (default `false`) and `submit` (default `false`) | Types into the focused text field. `clear` empties the field first, and `submit` presses Enter afterwards. | | `key` | `key`: a key name, or an Android key code from 0 to 400 | Presses one key. | | `back`, `home`, `recents`, `notifications` | None | Presses Back, goes to the home screen, opens the recent apps, or opens the notifications. | | `open_app` | `package`, and an optional `activity` | Opens an installed app, at that activity if you name one. | | `close_app` | `package`, and `clear_data` (default `false`) | Stops an app. `clear_data` also erases the app's data, its signed-in accounts included. | | `open_url` | `url`, and an optional `package` | Opens a web address or a deep link, in that app if you name one. | | `wait` | `ms`, from 1 to 10000 | Pauses for that many milliseconds. | | `wait_for` | A text, ID or description `target`, `gone` (default `false`), and `timeout_ms` from 100 to 30000, 10000 by default | Checks the screen every half second until the target appears, or with `gone` until it disappears. | The key names are `enter`, `delete`, `tab`, `escape`, `space`, `up`, `down`, `left`, `right`, `page_up`, `page_down`, `move_home`, `move_end`, `menu`, `search`, `volume_up` and `volume_down`. ## Targets [#targets] Each target takes exactly one of these forms: | Form | Example | How it finds the element | | ----------- | -------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------- | | Coordinates | `{"x": 540, "y": 950}` | A point in device pixels. | | Ref | `{"ref": 3, "snapshot_id": "snp_4f2k7m3q6z3a"}` | The element with that ref in an earlier [observation](/docs/using-phones/observe), found again on the screen as it is now. | | Text | `{"text": "Sign in"}` or `{"text": "Sign in", "nth": 2}` | An element whose text or description matches, as described below. | | ID | `{"id": "sign_in"}` | An element with that resource ID, either in full or the part after `:id/`. | | Description | `{"desc": "Back"}` | An element whose content description matches, in the same way as text. | A text target tries three levels, in order: 1. text or a description that is exactly the target; 2. the same match, ignoring case; 3. text or a description that contains the target, ignoring case. The first level that finds anything decides, and within it clickable elements win over the rest. If one element remains, that is the target. `nth` picks among several, counting from 1 in reading order. If several remain and there is no `nth`, the action fails with `ambiguous_target`. Only text targets take `nth`: an ID or description target that matches several elements fails with `ambiguous_target` too, and you narrow it, for example with the full resource ID, or target one of the candidates by ref. A ref target is checked before it is used. The element must still be on the screen with the same type, resource ID, text and description, and overlap its old position by at least half. Otherwise the action fails with `stale_snapshot`, and you observe again. When a target finds nothing, the action fails with `target_not_found`. Its `details.target` repeats your target, and `details.candidates` lists up to 10 elements that share a word with it, or else clickable elements. When `nth` is larger than the number of matches, `details.matches` says how many there were. ## Batch rules [#batch-rules] * A batch has from 1 to 20 actions, and the `ms` of its `wait` actions and the `timeout_ms` of its `wait_for` actions may add up to 60 seconds at most. A `wait_for` without a `timeout_ms` counts as 10 seconds. A batch that breaks either rule fails with `400 validation_failed`. * Actions run in order, and the batch stops at the first action that fails. `results` has an entry for each action that ran, ending with the one that failed, and the actions after it never run. The actions before the failed one did run, so when you try again, send only the failed action, changed as needed, and the ones after it. * A batch that has run for 90 seconds starts no more actions. The next one fails with the retryable `provider_timeout`, and its `details.not_run` counts the actions that didn't run. * `observe` chooses what the response shows of the screen afterwards. `ui`, the default, adds the element list as an `observation`. `screenshot` adds only a 720-pixel-wide JPEG screenshot, `both` adds both, and `none` adds nothing. * A batch that stops early shows the element list, whatever `observe` asked for, so you can see where it stopped, unless the screen can't be read, as the next rule describes. `none` and `ui` add it without a screenshot, and `screenshot` and `both` add it with one. * Reading the screen after a batch gets only the time the request has left. When the read doesn't finish in time, the response has `results` and `completed` but no `observation`, so observe the screen before you continue. * A request that runs a batch may take up to 300 seconds, so give your HTTP client a timeout of about 300 seconds for batches. * Phonebox never retries an action for you, and it never replays a batch. A retry is always your decision, made after you look at the screen. Problems with the request itself fail the whole request with an [error envelope](/docs/interfaces/rest#errors) and an HTTP error status. That covers the key, the body, a phone that isn't `ready`, rate limits, and a phone that couldn't be reached before the first action. Once the batch runs, a failed action is part of a `200` response, with `ok: false` and an `error` in its result. ## When an action fails [#when-an-action-fails] This batch stops at its second action, because three buttons on the screen read "Add to cart": ```json title="POST /v1/phones/{id}/actions" { "actions": [ { "type": "scroll", "direction": "down" }, { "type": "tap", "target": { "text": "Add to cart" } }, { "type": "wait_for", "target": { "text": "Added to cart" }, "timeout_ms": 5000 } ] } ``` ```json title="Response: batch" { "results": [ { "index": 0, "type": "scroll", "ok": true, "ms": 702 }, { "index": 1, "type": "tap", "ok": false, "ms": 391, "error": { "code": "ambiguous_target", "message": "Several elements on the screen match the target.", "retryable": false, "next": "Add nth to the target, or tap by ref from the returned observation.", "details": { "candidates": [ { "ref": 3, "type": "Button", "text": "Add to cart", "desc": null, "id": "com.example.shop:id/add_to_cart", "center": [860, 620] }, { "ref": 5, "type": "Button", "text": "Add to cart", "desc": null, "id": "com.example.shop:id/add_to_cart", "center": [860, 900] }, { "ref": 7, "type": "Button", "text": "Add to cart", "desc": null, "id": "com.example.shop:id/add_to_cart", "center": [860, 1180] } ], "snapshot_id": "snp_4f2k7m3q6z3a" } } } ], "completed": 1, "observation": { "snapshot_id": "snp_4f2k7m3q6z3a", "taken_at": "2026-09-29T11:15:09.240Z", "app": { "package": "com.example.shop", "activity": null }, "keyboard": { "visible": false, "focused_editable": false }, "screen": { "width": 1080, "height": 2400 }, "elements": [ { "ref": 1, "type": "EditText", "text": "Search products", "desc": null, "id": "com.example.shop:id/search", "bounds": [60, 140, 1020, 240], "center": [540, 190], "state": ["clickable", "editable"] }, { "ref": 2, "type": "TextView", "text": "Wireless earbuds", "desc": null, "id": null, "bounds": [60, 560, 640, 680], "center": [350, 620], "state": [] }, { "ref": 3, "type": "Button", "text": "Add to cart", "desc": null, "id": "com.example.shop:id/add_to_cart", "bounds": [700, 560, 1020, 680], "center": [860, 620], "state": ["clickable"] }, { "ref": 4, "type": "TextView", "text": "Phone stand", "desc": null, "id": null, "bounds": [60, 840, 640, 960], "center": [350, 900], "state": [] }, { "ref": 5, "type": "Button", "text": "Add to cart", "desc": null, "id": "com.example.shop:id/add_to_cart", "bounds": [700, 840, 1020, 960], "center": [860, 900], "state": ["clickable"] }, { "ref": 6, "type": "TextView", "text": "USB-C cable", "desc": null, "id": null, "bounds": [60, 1120, 640, 1240], "center": [350, 1180], "state": [] }, { "ref": 7, "type": "Button", "text": "Add to cart", "desc": null, "id": "com.example.shop:id/add_to_cart", "bounds": [700, 1120, 1020, 1240], "center": [860, 1180], "state": ["clickable"] } ], "truncated": false, "text": "app com.example.shop\nkeyboard hidden\nscreen 1080x2400\n[1] EditText \"Search products\" #search clickable editable (540,190)\n[2] TextView \"Wireless earbuds\" (350,620)\n[3] Button \"Add to cart\" #add_to_cart clickable (860,620)\n[4] TextView \"Phone stand\" (350,900)\n[5] Button \"Add to cart\" #add_to_cart clickable (860,900)\n[6] TextView \"USB-C cable\" (350,1180)\n[7] Button \"Add to cart\" #add_to_cart clickable (860,1180)" } } ``` The screen may move between the moment an action fails and the moment the batch reads the screen. Phonebox therefore matches each candidate to the element in the returned observation, gives it that element's ref, and names the observation's `snapshot_id` in `details.snapshot_id`. If any candidate is no longer on the screen, `details` keeps the refs as the action saw them and names no snapshot, and you observe again before acting by ref. To tap the phone stand's button, target it by ref with that snapshot, or by its text with `nth`: ```json title="POST /v1/phones/{id}/actions" { "actions": [ { "type": "tap", "target": { "ref": 5, "snapshot_id": "snp_4f2k7m3q6z3a" } }, { "type": "wait_for", "target": { "text": "Added to cart" }, "timeout_ms": 5000 } ] } ``` ```json title="POST /v1/phones/{id}/actions" { "actions": [ { "type": "tap", "target": { "text": "Add to cart", "nth": 2 } } ] } ``` These are the errors an action's result can carry, and what to do about each: | Code | Meaning | What to do | | ------------------------ | --------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `target_not_found` | No element matches the target. | Observe, then use one of `details.candidates`, or scroll to bring the element into view. | | `ambiguous_target` | Several elements match. | Target a ref from `details.candidates` with `details.snapshot_id`, or add `nth` to a text target. Only text targets take `nth`, so narrow an ID or description target instead, for example with the full resource ID. | | `stale_snapshot` | The element behind a ref changed or went away. | Observe again and use the new refs. | | `no_focused_field` | A `type` action found no focused text field. | Tap the field first, then type. | | `wait_timeout` | A `wait_for` ran out of time. It is retryable, and `details` has the `timeout_ms` and `target`. | Observe. If the screen is still loading, wait again with a longer `timeout_ms`. | | `app_not_found` | `open_app` or `close_app` named an app that isn't installed. | [Install it](/docs/using-phones/apps), or check the package name. | | `action_outcome_unknown` | The phone didn't confirm the action. It may or may not have happened. | Observe before anything else. See below. | | `phone_unavailable` | The phone was briefly out of reach. | When `retryable` is true, the action didn't run, and you can send it and the rest of the batch again. | | `rate_limited` | The phone service was busy. | Wait `details.retry_after_seconds`, then send the rest of the batch again. | | `provider_error` | The phone service refused the action. | Observe, then decide whether to try again. | | `provider_timeout` | The batch ran out of time before this action. `details.not_run` counts the actions that didn't run. | Observe, then send them again in a new batch. | | `capacity_unavailable` | The phone service had no capacity for this action, so it didn't run. | Send it and the actions after it again later. | | `internal_error` | Something failed on our side, and the action may have run. | Observe before you decide what to send again. | ## Nothing ran [#nothing-ran] When the first action of a batch couldn't reach the phone at all, the whole request fails with `503 phone_unavailable`, and both `Retry-After` and `details.retry_after_seconds` say to wait 2 seconds. Nothing was done on the phone, so sending the same batch again is safe: ```json title="Response: error" { "error": { "type": "provider_error", "code": "phone_unavailable", "message": "The phone is temporarily unavailable.", "retryable": true, "next": "Nothing was done on the phone; retry this action in a few seconds.", "request_id": "req_w5x3y6z2a4b7", "details": { "retry_after_seconds": 2 } } } ``` This happens most often in the first seconds after a phone becomes ready. Other failures before the first action mean that nothing ran, too: `503 capacity_unavailable` and `429 rate_limited` come with a `Retry-After`, while `502 provider_error`, and a `503 phone_unavailable` during maintenance, come without one. When a later action couldn't reach the phone, the earlier ones already ran, so the batch returns `200` as usual. The failed action's result then has `phone_unavailable` with `retryable` set to `true`, which means that action didn't run, and you can send it and the ones after it again. These rules hold only for Phonebox's own errors, which carry the [error envelope](/docs/api-reference/errors) with an `error.code`. A `502` or `504` without it, which a proxy or the hosting platform sends, a timeout on your side, or a dropped connection says nothing about the batch, which may have run: observe before you send it again. ## Unknown outcomes [#unknown-outcomes] `action_outcome_unknown` means the phone didn't confirm the action: it may have happened or not. Its `retryable` is always `false`. Never repeat such an action blindly, since a second tap on "Pay" or a second copy of a message can do real harm. Observe the screen, work out whether the action took effect, and continue from there. ## Partial actions [#partial-actions] Some actions take two steps. `type` with `submit` types the text and then presses Enter, and `double_tap` taps twice. When the second step fails after the first one reached the phone, the result has `details.partial` set to `true` and `retryable` set to `false`, whatever the error, and `next` says what was done. In this batch, the text was typed but Enter never reached the phone: ```json title="Response: batch" { "results": [ { "index": 0, "type": "tap", "ok": true, "ms": 197, "resolved": { "ref": 2, "center": [540, 700] } }, { "index": 1, "type": "type", "ok": false, "ms": 1214, "error": { "code": "phone_unavailable", "message": "The phone is temporarily unavailable.", "retryable": false, "next": "The text was typed but Enter wasn't pressed. Observe the screen, then send a key action for enter.", "details": { "retry_after_seconds": 2, "partial": true } } } ], "completed": 1, "observation": { "snapshot_id": "snp_7d3h6j2n5p4r", "taken_at": "2026-09-29T11:32:47.615Z", "app": { "package": "com.example.app", "activity": null }, "keyboard": { "visible": true, "focused_editable": true }, "screen": { "width": 1080, "height": 2400 }, "elements": [ { "ref": 1, "type": "TextView", "text": "Welcome back", "desc": null, "id": null, "bounds": [60, 400, 1020, 520], "center": [540, 460], "state": [] }, { "ref": 2, "type": "EditText", "text": "alex@example.com", "desc": null, "id": "com.example.app:id/email", "bounds": [60, 640, 1020, 760], "center": [540, 700], "state": ["clickable", "editable", "focused"] }, { "ref": 3, "type": "Button", "text": "Sign in", "desc": null, "id": "com.example.app:id/sign_in", "bounds": [60, 890, 1020, 1010], "center": [540, 950], "state": ["clickable"] } ], "truncated": false, "text": "app com.example.app\nkeyboard visible, text field focused\nscreen 1080x2400\n[1] TextView \"Welcome back\" (540,460)\n[2] EditText \"alex@example.com\" #email clickable editable focused (540,700)\n[3] Button \"Sign in\" #sign_in clickable (540,950)" } } ``` Observe before you continue, and never press Enter blindly. Here `phone_unavailable` means Enter never reached the phone, and the observation still shows the filled field, so a `key` action for `enter` finishes the step. Don't send a second `type`, which would type the text again. When the error is `action_outcome_unknown` instead, Enter may have been pressed after all: check that the screen hasn't moved on before you press it. ## Wait for the screen [#wait-for-the-screen] Screens take time to change after a tap. Instead of fixed pauses, use `wait_for` to wait for what you expect to see, or for a spinner to go away: ```json title="POST /v1/phones/{id}/actions" { "actions": [ { "type": "tap", "target": { "text": "Sign in" } }, { "type": "wait_for", "target": { "text": "Loading" }, "gone": true, "timeout_ms": 20000 }, { "type": "wait_for", "target": { "text": "Welcome" }, "timeout_ms": 5000 } ], "observe": "ui" } ``` `wait_for` finds its target the same way as a text, ID or description target, and it checks the screen every 500 ms. When the time runs out, it fails with `wait_timeout`, which is retryable. The CLI's `phonebox wait` exits with code 3 in that case. ## In the CLI and the SDK [#in-the-cli-and-the-sdk] The CLI's `tap`, `type`, `key`, `swipe`, `scroll`, `open`, `wait`, `back`, `home` and `recents` commands, and the matching SDK methods such as `tap()`, `type()` and `waitFor()`, each send a batch of one action with `"observe": "none"`. When that action fails, the CLI prints its error and exits with code 1, or 3 when `wait` runs out of time, and the SDK throws a `PhoneboxError`. `phonebox act` and the SDK's `act()` send a whole batch and return every result, so check `completed` and each `ok` yourself.