The screen reads empty

An observation lists the elements that Android's accessibility layer reports: views that you can act on or that carry text or a description. Most apps report their buttons, fields and labels this way. When an observation lists few elements, or none, while the screen clearly shows something, one of these is usually the reason.

The app draws its own screenCopy Anchor Link

Games, canvas drawings, maps, many video players and some apps built with cross-platform frameworks draw their content themselves. Android then sees one large view with nothing inside it, so the observation lists little or nothing, and text, ID and description targets can't find anything.

Work from a screenshot and coordinates instead:

1. Observe with a screenshot, such as GET /v1/phones/{id}/observe?screenshot=jpeg&max_width=720, or phonebox look --screenshot screen.jpg.
2. Find what you want on the image, and convert the point to device pixels by dividing its coordinates by screenshot.scale. On a 1080-pixel-wide screen at max_width=720, the scale is 0.666667, so a point at (360, 800) on the image is (540, 1200) on the phone.
3. Tap, press or swipe at that point:

{
  "actions": [
    { "type": "tap", "target": { "x": 540, "y": 1200 } },
    { "type": "wait", "ms": 1500 }
  ],
  "observe": "screenshot"
}

"observe": "screenshot" returns a new screenshot with the results, so you can check the effect of the tap without another request. When the screen couldn't be read, or not in time, the response has no observation, so observe before you go on. Since such an app offers nothing to wait for, a short wait gives it time to draw.

The screenshot endpoint returns the image alone, with the screen's size and the scale in its headers, when you don't need the JSON.

The app is still loadingCopy Anchor Link

Right after an app opens, it may show a splash screen or a spinner with nothing to act on. Wait for what you expect, instead of observing again and again:

{
  "actions": [
    { "type": "open_app", "package": "com.example.shop" },
    { "type": "wait_for", "target": { "text": "Search products" }, "timeout_ms": 20000 }
  ],
  "observe": "ui"
}

The same applies to a slow network inside an app. If the wait_for runs out with wait_timeout, the returned observation shows what is there instead. When the screen couldn't be read, or not in time, observation is missing, and you observe the screen yourself.

The observation is truncatedCopy Anchor Link

An observation lists at most 250 elements, in reading order from top to bottom. On a longer screen, truncated is true, and the text form ends with a line saying that more elements are on screen but not listed.

Text, ID and description targets search the same 250 elements, so an element beyond them can't be found by its label. Scroll until it moves up into the listed part, or scroll inside the list that holds it with a scroll action whose target is that list.

Something covers the screenCopy Anchor Link

• The keyboard. When keyboard.visible is true, the keyboard may hide elements at the bottom of the screen. Press back once to close it, or scroll.
• A dialog or a permission prompt. The observation shows the dialog's elements rather than the screen behind it. Answer the dialog first.
• The wrong app. The first line of the text form, app …, names the app in front. It reads app unknown when Phonebox couldn't tell which app it is. Open the app you meant with open_app.

A web pageCopy Anchor Link

Browsers usually report a page's links, buttons and fields as elements. A page drawn on a canvas, or a web game, reports nothing, so use screenshots and coordinates as above.