# Observe the screen

Source: https://phonebox.dev/docs/using-phones/observe

> Read the screen as JSON or as compact text, with optional screenshots, and use snapshots to act on elements by ref.



Observing reads the phone's screen: which app is in front, whether the keyboard is showing, and every element you can act on or read. Each element gets a ref number that actions can target. The phone must be `ready`.

```bash
curl https://phonebox.dev/v1/phones/ph_7kx2m6q4v3ta/observe \
  -H "Authorization: Bearer $PHONEBOX_API_KEY"
```

```json title="Response: observation"
{
  "snapshot_id": "snp_4f2k7m3q6z3a",
  "taken_at": "2026-09-29T10:01:05.662Z",
  "app": { "package": "com.android.launcher3", "activity": null },
  "keyboard": { "visible": false, "focused_editable": false },
  "screen": { "width": 1080, "height": 2400 },
  "elements": [
    {
      "ref": 1, "type": "EditText", "text": "Search apps", "desc": null, "id": "com.android.launcher3:id/search",
      "bounds": [60, 200, 1020, 300], "center": [540, 250], "state": ["clickable", "editable"]
    },
    {
      "ref": 2, "type": "TextView", "text": "Chrome", "desc": null, "id": "com.android.launcher3:id/icon",
      "bounds": [100, 1800, 300, 2000], "center": [200, 1900], "state": ["clickable"]
    },
    {
      "ref": 3, "type": "TextView", "text": "Settings", "desc": null, "id": "com.android.launcher3:id/icon",
      "bounds": [400, 1800, 600, 2000], "center": [500, 1900], "state": ["clickable"]
    },
    {
      "ref": 4, "type": "TextView", "text": "Play Store", "desc": null, "id": "com.android.launcher3:id/icon",
      "bounds": [700, 1800, 900, 2000], "center": [800, 1900], "state": ["clickable"]
    }
  ],
  "truncated": false,
  "text": "app com.android.launcher3\nkeyboard hidden\nscreen 1080x2400\n[1] EditText \"Search apps\" #search clickable editable (540,250)\n[2] TextView \"Chrome\" #icon clickable (200,1900)\n[3] TextView \"Settings\" #icon clickable (500,1900)\n[4] TextView \"Play Store\" #icon clickable (800,1900)"
}
```

| Field         | Meaning                                                                       |
| ------------- | ----------------------------------------------------------------------------- |
| `snapshot_id` | The ID of this reading of the screen. Actions use it with a ref.              |
| `taken_at`    | When the screen was read.                                                     |
| `app`         | The package in front. `activity` is often `null`.                             |
| `keyboard`    | Whether the keyboard is showing, and whether a text field has focus.          |
| `screen`      | The screen's size in pixels.                                                  |
| `elements`    | The elements, in reading order.                                               |
| `truncated`   | `true` when the screen had more than 250 elements and the rest were left out. |
| `text`        | The same screen in the compact text form.                                     |
| `screenshot`  | The screenshot, when you ask for one.                                         |

Each element has these fields:

| Field    | Meaning                                                                                                                                       |
| -------- | --------------------------------------------------------------------------------------------------------------------------------------------- |
| `ref`    | Its number in this snapshot, from 1.                                                                                                          |
| `type`   | Its Android class name without the package, such as `Button`, `EditText` or `TextView`.                                                       |
| `text`   | Its text, or `null`.                                                                                                                          |
| `desc`   | Its content description, or `null`. Icons often have one instead of text.                                                                     |
| `id`     | Its full resource ID, such as `com.android.launcher3:id/search`, or `null`.                                                                   |
| `bounds` | Its rectangle as `[left, top, right, bottom]`.                                                                                                |
| `center` | The point a tap on it uses, as `[x, y]`.                                                                                                      |
| `state`  | Its flags: `clickable`, `long_clickable`, `editable`, `focused`, `checkable`, `checked`, `selected`, `scrollable`, `disabled` and `password`. |

All coordinates are device pixels, the same pixels that `screen` measures and that `{x, y}` targets use.

## Which elements are listed [#which-elements-are-listed]

The list keeps what an agent can use and drops layout containers. An element is listed when it is visible, has a size, and either can be acted on (it is clickable, long-clickable, editable, checkable or scrollable) or has text or a description. Elements of the system status bar are left out unless you can act on them.

Refs run from 1 in reading order: top to bottom, then left to right. At most 250 elements are returned. When a screen has more, `truncated` is `true` and the text form ends with a line that says so. Scroll to reach the others.

## The text form [#the-text-form]

Add `format=text` for the screen as plain text, made for models to read. It is the `text` field on its own, and the response's `X-Snapshot-Id` header carries the snapshot ID:

```bash
curl -i "https://phonebox.dev/v1/phones/ph_7kx2m6q4v3ta/observe?format=text" \
  -H "Authorization: Bearer $PHONEBOX_API_KEY"
```

```text
app com.android.launcher3
keyboard hidden
screen 1080x2400
[1] EditText "Search apps" #search clickable editable (540,250)
[2] TextView "Chrome" #icon clickable (200,1900)
[3] TextView "Settings" #icon clickable (500,1900)
[4] TextView "Play Store" #icon clickable (800,1900)
```

The CLI's `phonebox look` and the SDK's `look()` print the snapshot ID on a first line of its own, `snapshot snp_…`, as MCP's `observe` does, and then this text.

The first three lines of the text are always the same header:

| Line                                  | Meaning                                                                                                                                |
| ------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------- |
| `app com.example.shop (MainActivity)` | The package in front, followed by its activity in parentheses when it is known. It reads `app unknown` when the package can't be read. |
| `keyboard hidden`                     | Or `keyboard visible`, with `, text field focused` added when a text field has focus. The same suffix can follow `keyboard hidden`.    |
| `screen 1080x2400`                    | The screen's width and height in pixels.                                                                                               |

Each element line after the header reads `[ref] Type "text" desc="description" #id flags (x,y)`. A part the element doesn't have is left out, and `desc` is shown only when it differs from the text. `#id` is the part of the resource ID after `:id/`, and `(x,y)` is the element's center. When the list was truncated, the last line is `… more elements are on screen but not listed`.

The text form never includes a screenshot, even when you ask for one.

## Screenshots [#screenshots]

Add `screenshot=jpeg` or `screenshot=png` to include a screenshot in the JSON, and `max_width` to scale it down to at most that many pixels wide, from 160 to 2000:

```bash
curl "https://phonebox.dev/v1/phones/ph_7kx2m6q4v3ta/observe?screenshot=jpeg&max_width=720" \
  -H "Authorization: Bearer $PHONEBOX_API_KEY"
```

The response then carries a `screenshot` object with `format`, `width`, `height`, `scale` and `data`, the image in base64. Element coordinates stay in device pixels. `scale` is the image's width divided by the screen's, so an element's center falls on the image at its coordinates times `scale`, and a point on the image maps back to the phone at its coordinates divided by `scale`. A 1080-pixel-wide screen at `max_width=720` has a `scale` of 0.666667. Phonebox keeps the whole response under 4 MB, and it shrinks a screenshot further when it has to.

To get only the image, use the screenshot endpoint. It returns the image itself, not JSON:

```bash
curl -o screen.jpg "https://phonebox.dev/v1/phones/ph_7kx2m6q4v3ta/screenshot?format=jpeg&max_width=720&quality=80" \
  -H "Authorization: Bearer $PHONEBOX_API_KEY"
```

| Parameter   | Meaning                                                                                                    |
| ----------- | ---------------------------------------------------------------------------------------------------------- |
| `format`    | `png`, the default, or `jpeg`.                                                                             |
| `max_width` | The most pixels wide the image may be, from 160 to 2000. By default, the image has the screen's full size. |
| `quality`   | JPEG quality, from 30 to 95. The default is 70.                                                            |

The response carries the screen's size in device pixels in the `X-Screen-Width` and `X-Screen-Height` headers, and the image's `scale` in `X-Screen-Scale`, so you can map a point on a scaled image back onto the phone.

Use screenshots when the element list doesn't show what you need, as in games, maps, drawings, images and web pages that expose no elements. Then act by coordinates.

## Snapshots and refs [#snapshots-and-refs]

Each observation stores its element list as a snapshot for 15 minutes, under its `snapshot_id`. An action can then target an element as `{"ref": 3, "snapshot_id": "snp_4f2k7m3q6z3a"}`. Before acting, Phonebox finds that element on the screen as it is now: it must have the same type, resource ID, text and description, and overlap its old position by at least half. If it doesn't, the action fails with `stale_snapshot`, and you observe again.

A ref means something only together with its snapshot. Refs from different snapshots can point at different elements, so always pass the `snapshot_id` from the same observation.

Observing a phone counts as activity, so it keeps an idle phone from parking. A Read-only key's requests are the exception: they never keep a phone awake. The CLI's `phonebox look` prints the text form, and `phonebox look --json` prints the JSON.