Skip to content

ui ​

The ui module reads the accessibility element tree of the frontmost app. It is useful for locating controls by text or role instead of relying only on screenshots and color matching.

TIP

All coordinates returned by this module are screen pixels, so they can be passed directly to touch.click().

WARNING

The element tree is sampled from the frontmost app. It is not available when SpringBoard is in the foreground, and it may take a few seconds on complex screens.

ui.tree() ​

Get the UI element tree of the frontmost app.

python
ui.tree() -> dict[str, object]

Arguments ​

None.

Return ​

python
{
    "app": str,          # Bundle identifier of the frontmost app.
    "pid": int,          # Process id of the frontmost app.
    "screen": {
        "w": int,        # Screen width in pixels.
        "h": int,        # Screen height in pixels.
    },
    "elements": [
        {
            "role": str,                 # Display text of the element.
            "label": str,                # Optional.
            "value": str,                # Optional.
            "frame": [int, int, int, int],  # [x, y, width, height] in pixels.
            "center": [int, int],           # Center point in pixels.
            "center_norm": [float, float],  # Center point normalized to 0.0–1.0.
        }
    ],
}

label and value are omitted when they are empty or identical to role.

Raises RuntimeError if the accessibility tree is unavailable, the frontmost app is SpringBoard, or the IPC request fails.

Example ​

python
from catscript import sys, ui

if __name__ == "__main__":
    tree = ui.tree()
    sys.log(f"app={tree['app']} elements={len(tree['elements'])}")
    for element in tree["elements"]:
        sys.log(element.get("role"))

ui.elements() ​

Get only the element list from ui.tree().

python
ui.elements() -> list[dict[str, object]]

Arguments ​

None.

Return ​

The elements list described in ui.tree().

Example ​

python
from catscript import sys, ui

if __name__ == "__main__":
    for element in ui.elements():
        sys.log(element.get("role"))

ui.find() ​

Find the first matching element.

python
ui.find(
    role: str | None = None,
    label: str | None = None,
    value: str | None = None,
    contains: bool = False,
    index: int = 0,
) -> dict[str, object] | None

Arguments ​

NameTypeDefaultDescription
rolestr | NoneNoneMatch the role field.
labelstr | NoneNoneMatch the label field.
valuestr | NoneNoneMatch the value field.
containsboolFalseUse substring matching instead of exact matching.
indexint0Index of the match to return.

Return ​

The matching element dictionary, or None if no element matches.

  • Multiple criteria are combined with AND.
  • If an element has no label field, label matching falls back to role.
  • Raises ValueError if index < 0.

Example ​

python
from catscript import touch, ui

if __name__ == "__main__":
    element = ui.find(label="Wi-Fi")
    if element:
        touch.click(*element["center"])

ui.wait() ​

Wait until a matching element appears.

python
ui.wait(
    role: str | None = None,
    label: str | None = None,
    value: str | None = None,
    contains: bool = False,
    timeout: float = 10.0,
    interval: float = 0.5,
) -> dict[str, object] | None

Arguments ​

NameTypeDefaultDescription
rolestr | NoneNoneMatch the role field.
labelstr | NoneNoneMatch the label field.
valuestr | NoneNoneMatch the value field.
containsboolFalseUse substring matching instead of exact matching.
timeoutfloat10.0Timeout in seconds.
intervalfloat0.5Polling interval in seconds.

Return ​

The matching element dictionary, or None if the timeout expires.

Raises ValueError if interval <= 0.

Example ​

python
from catscript import touch, ui

if __name__ == "__main__":
    element = ui.wait(label="Allow", timeout=5)
    if element:
        touch.click(*element["center"])

ui.tap() ​

Tap an element.

python
ui.tap(
    element: dict[str, object] | None = None,
    role: str | None = None,
    label: str | None = None,
    value: str | None = None,
    contains: bool = False,
) -> bool

Arguments ​

NameTypeDefaultDescription
elementdict[str, object] | NoneNoneElement to tap. Takes priority over the criteria.
rolestr | NoneNoneMatch the role field.
labelstr | NoneNoneMatch the label field.
valuestr | NoneNoneMatch the value field.
containsboolFalseUse substring matching instead of exact matching.

Return ​

TypeDescription
boolTrue if an element was found and tapped, otherwise False.

Raises ValueError if the element has no center field.

Example ​

python
from catscript import ui

if __name__ == "__main__":
    ui.tap(label="Wi-Fi")