To playback a recorded event, send a cross-document message with type
field set to playEvent
, and value
set to a playback
object. We will attempt to locate the element of the recorded event and repeat the action.
A playback
object should have the fields:
event
- the entire event object that was previously recorded
timeout
- (default 10) the number of seconds to wait for a match of the recorded element
strictMatch
- (default false) if true, element must match all recorded attributes exactly. Otherwise, we will attempt to find the best matching element
id
- (optional) a string to identify this playback attempt, to help identify success or failure.
If playback is successful, the iFrame will post a cross-document message with type
set to playbackFoundAndSent
. Its value
field will be an object with the fields:
id
- the id string that was optionally provided to playEvent
playback
- if id
was not provided, the entire playback object
matchedElement
- the element that matched the original recorded event
matchedOn
- an array of attributes where the new element matched the recorded element
If the playback failed the iFrame will post a cross-document message with type
set to playbackError
. Its value
field will be an object with the fields:
id
- the id string that was optionally provided to playEvent
playback
- if id
was not provided, the entire playback object
errorId
- a string that may be one of:
notFound
- we timed out waiting for a matching element to appear
uiError
- we were unable to retrieve the UI heirarchy for the app
unknown
- unknown error
message
- a string describing the error
A central problem when playing back recorded events is to find the recorded element in the new UI during playback. Apps may differ between recording and playback because of changes to the content, design, device dimensions, or language.
When searching for correct element during playback, we attempt to find a unique match based on the rules listed below. Each rule defines the set of fields we will consider. If there are multiple elements that satisfy a rule, we will attempt to "break the tie" by considering defining characteristics of sibling and parent elements. If remain multiple matches or there are no matches, we move on to the next rule.
In scrollable views, the path can be misleading as the exact scroll position may differ between playback and recording. Therefore, path is removed from the rules if the element is inside of a scrollable view. (Not yet implemented on iOS, see Known Issues above.)
On Android, we match using these rules:
content-desc, text, path, class, resource-id
content-desc, path, class, resource-id
content-desc
text, class, resource-id
text, class
text
path, class, resource-id
path, class
path, resource-id
class, resource-id
class
On iOS, we match using these rules:
accessibilityIdentifier, text, path, class, baseClass
accessibilityIdentifier, text, class, baseClass
accessibilityIdentifier, path, class, baseClass
accessibilityIdentifier
text, path, class, baseClass
text, class, baseClass
text, class
path, class, baseClass
path, class
path, baseClass
class, baseClass
class
baseClass