Playback events

Search…
Playback events
To playback a recorded event, send a cross-document message with type field set to playEvent, and value set to a playback object. We will attempt to locate the element of the recorded event and repeat the action.
A playback object should have the fields:
event - the entire event object that was previously recorded
timeout - (default 10) the number of seconds to wait for a match of the recorded element
strictMatch - (default false) if true, element must match all recorded attributes exactly. Otherwise, we will attempt to find the best matching element
id - (optional) a string to identify this playback attempt, to help identify success or failure.
If playback is successful, the iFrame will post a cross-document message with type set to playbackFoundAndSent. Its value field will be an object with the fields:
id - the id string that was optionally provided to playEvent
playback - if id was not provided, the entire playback object
matchedElement - the element that matched the original recorded event
matchedOn - an array of attributes where the new element matched the recorded element
If the playback failed the iFrame will post a cross-document message with type set to playbackError. Its value field will be an object with the fields:
id - the id string that was optionally provided to playEvent
playback - if id was not provided, the entire playback object
errorId - a string that may be one of:
notFound - we timed out waiting for a matching element to appear
uiError - we were unable to retrieve the UI heirarchy for the app
unknown - unknown error
message - a string describing the error
Playback element matching
A central problem when playing back recorded events is to find the recorded element in the new UI during playback. Apps may differ between recording and playback because of changes to the content, design, device dimensions, or language.
When searching for correct element during playback, we attempt to find a unique match based on the rules listed below. Each rule defines the set of fields we will consider. If there are multiple elements that satisfy a rule, we will attempt to "break the tie" by considering defining characteristics of sibling and parent elements. If remain multiple matches or there are no matches, we move on to the next rule.
In scrollable views, the path can be misleading as the exact scroll position may differ between playback and recording. Therefore, path is removed from the rules if the element is inside of a scrollable view. (Not yet implemented on iOS, see Known Issues above.)
On Android, we match using these rules:
1.content-desc, text, path, class, resource-id
2.content-desc, path, class, resource-id
3.content-desc
4.text, class, resource-id
5.text, class
6.text
7.path, class, resource-id
8.path, class
9.path, resource-id
10.class, resource-id
11.class
On iOS, we match using these rules:
1.accessibilityIdentifier, text, path, class, baseClass
2.accessibilityIdentifier, text, class, baseClass
3.accessibilityIdentifier, path, class, baseClass
4.accessibilityIdentifier
5.text, path, class, baseClass
6.text, class, baseClass
7.text, class
8.path, class, baseClass
9.path, class
10.path, baseClass
11.class, baseClass
12.class
13.baseClass
App recorder (beta) - Previous
Record events
Next - App recorder (beta)
Retrieving the user interface
Last modified 1yr ago
Copy link