Observe the screen
Read the screen as JSON or as compact text, with optional screenshots, and use snapshots to act on elements by ref.
Observing reads the phone's screen: which app is in front, whether the keyboard is showing, and every element you can act on or read. Each element gets a ref number that actions can target. The phone must be ready.
curl https://phonebox.dev/v1/phones/ph_7kx2m6q4v3ta/observe \
-H "Authorization: Bearer $PHONEBOX_API_KEY"{
"snapshot_id": "snp_4f2k7m3q6z3a",
"taken_at": "2026-09-29T10:01:05.662Z",
"app": { "package": "com.android.launcher3", "activity": null },
"keyboard": { "visible": false, "focused_editable": false },
"screen": { "width": 1080, "height": 2400 },
"elements": [
{
"ref": 1, "type": "EditText", "text": "Search apps", "desc": null, "id": "com.android.launcher3:id/search",
"bounds": [60, 200, 1020, 300], "center": [540, 250], "state": ["clickable", "editable"]
},
{
"ref": 2, "type": "TextView", "text": "Chrome", "desc": null, "id": "com.android.launcher3:id/icon",
"bounds": [100, 1800, 300, 2000], "center": [200, 1900], "state": ["clickable"]
},
{
"ref": 3, "type": "TextView", "text": "Settings", "desc": null, "id": "com.android.launcher3:id/icon",
"bounds": [400, 1800, 600, 2000], "center": [500, 1900], "state": ["clickable"]
},
{
"ref": 4, "type": "TextView", "text": "Play Store", "desc": null, "id": "com.android.launcher3:id/icon",
"bounds": [700, 1800, 900, 2000], "center": [800, 1900], "state": ["clickable"]
}
],
"truncated": false,
"text": "app com.android.launcher3\nkeyboard hidden\nscreen 1080x2400\n[1] EditText \"Search apps\" #search clickable editable (540,250)\n[2] TextView \"Chrome\" #icon clickable (200,1900)\n[3] TextView \"Settings\" #icon clickable (500,1900)\n[4] TextView \"Play Store\" #icon clickable (800,1900)"
}| Field | Meaning |
|---|---|
snapshot_id | The ID of this reading of the screen. Actions use it with a ref. |
taken_at | When the screen was read. |
app | The package in front. activity is often null. |
keyboard | Whether the keyboard is showing, and whether a text field has focus. |
screen | The screen's size in pixels. |
elements | The elements, in reading order. |
truncated | true when the screen had more than 250 elements and the rest were left out. |
text | The same screen in the compact text form. |
screenshot | The screenshot, when you ask for one. |
Each element has these fields:
| Field | Meaning |
|---|---|
ref | Its number in this snapshot, from 1. |
type | Its Android class name without the package, such as Button, EditText or TextView. |
text | Its text, or null. |
desc | Its content description, or null. Icons often have one instead of text. |
id | Its full resource ID, such as com.android.launcher3:id/search, or null. |
bounds | Its rectangle as [left, top, right, bottom]. |
center | The point a tap on it uses, as [x, y]. |
state | Its flags: clickable, long_clickable, editable, focused, checkable, checked, selected, scrollable, disabled and password. |
All coordinates are device pixels, the same pixels that screen measures and that {x, y} targets use.
Which elements are listed
The list keeps what an agent can use and drops layout containers. An element is listed when it is visible, has a size, and either can be acted on (it is clickable, long-clickable, editable, checkable or scrollable) or has text or a description. Elements of the system status bar are left out unless you can act on them.
Refs run from 1 in reading order: top to bottom, then left to right. At most 250 elements are returned. When a screen has more, truncated is true and the text form ends with a line that says so. Scroll to reach the others.
The text form
Add format=text for the screen as plain text, made for models to read. It is the text field on its own, and the response's X-Snapshot-Id header carries the snapshot ID:
curl -i "https://phonebox.dev/v1/phones/ph_7kx2m6q4v3ta/observe?format=text" \
-H "Authorization: Bearer $PHONEBOX_API_KEY"app com.android.launcher3
keyboard hidden
screen 1080x2400
[1] EditText "Search apps" #search clickable editable (540,250)
[2] TextView "Chrome" #icon clickable (200,1900)
[3] TextView "Settings" #icon clickable (500,1900)
[4] TextView "Play Store" #icon clickable (800,1900)The CLI's phonebox look and the SDK's look() print the snapshot ID on a first line of its own, snapshot snp_…, as MCP's observe does, and then this text.
The first three lines of the text are always the same header:
| Line | Meaning |
|---|---|
app com.example.shop (MainActivity) | The package in front, followed by its activity in parentheses when it is known. It reads app unknown when the package can't be read. |
keyboard hidden | Or keyboard visible, with , text field focused added when a text field has focus. The same suffix can follow keyboard hidden. |
screen 1080x2400 | The screen's width and height in pixels. |
Each element line after the header reads [ref] Type "text" desc="description" #id flags (x,y). A part the element doesn't have is left out, and desc is shown only when it differs from the text. #id is the part of the resource ID after :id/, and (x,y) is the element's center. When the list was truncated, the last line is … more elements are on screen but not listed.
The text form never includes a screenshot, even when you ask for one.
Screenshots
Add screenshot=jpeg or screenshot=png to include a screenshot in the JSON, and max_width to scale it down to at most that many pixels wide, from 160 to 2000:
curl "https://phonebox.dev/v1/phones/ph_7kx2m6q4v3ta/observe?screenshot=jpeg&max_width=720" \
-H "Authorization: Bearer $PHONEBOX_API_KEY"The response then carries a screenshot object with format, width, height, scale and data, the image in base64. Element coordinates stay in device pixels. scale is the image's width divided by the screen's, so an element's center falls on the image at its coordinates times scale, and a point on the image maps back to the phone at its coordinates divided by scale. A 1080-pixel-wide screen at max_width=720 has a scale of 0.666667. Phonebox keeps the whole response under 4 MB, and it shrinks a screenshot further when it has to.
To get only the image, use the screenshot endpoint. It returns the image itself, not JSON:
curl -o screen.jpg "https://phonebox.dev/v1/phones/ph_7kx2m6q4v3ta/screenshot?format=jpeg&max_width=720&quality=80" \
-H "Authorization: Bearer $PHONEBOX_API_KEY"| Parameter | Meaning |
|---|---|
format | png, the default, or jpeg. |
max_width | The most pixels wide the image may be, from 160 to 2000. By default, the image has the screen's full size. |
quality | JPEG quality, from 30 to 95. The default is 70. |
The response carries the screen's size in device pixels in the X-Screen-Width and X-Screen-Height headers, and the image's scale in X-Screen-Scale, so you can map a point on a scaled image back onto the phone.
Use screenshots when the element list doesn't show what you need, as in games, maps, drawings, images and web pages that expose no elements. Then act by coordinates.
Snapshots and refs
Each observation stores its element list as a snapshot for 15 minutes, under its snapshot_id. An action can then target an element as {"ref": 3, "snapshot_id": "snp_4f2k7m3q6z3a"}. Before acting, Phonebox finds that element on the screen as it is now: it must have the same type, resource ID, text and description, and overlap its old position by at least half. If it doesn't, the action fails with stale_snapshot, and you observe again.
A ref means something only together with its snapshot. Refs from different snapshots can point at different elements, so always pass the snapshot_id from the same observation.
Observing a phone counts as activity, so it keeps an idle phone from parking. A Read-only key's requests are the exception: they never keep a phone awake. The CLI's phonebox look prints the text form, and phonebox look --json prints the JSON.