Touch interactions

Targeting Elements

Touch interactions may accept a target element to play the interaction on. An element is described with an "Element Selector"; a convenient to way to target a UI element on the device.

An Element Selector describes the element in your application by its attributes and other properties. Below is a list of accepted attributes for each platform.

Element Selectors work regardless of the device or screen size, meaning you can run the same set of actions on both a phone and tablet

Element Attributes

iOS

Attribute	Description
accessibilityIdentifier	The element's accessibilityIdentifier
accessibilityLabel	The element's accessibilityLabel
accessibilityHint	The element's accessibilityHint
accessibilityValue	The element's accessibilityValue
baseClass	Name of the element's base (or inherited) class
class	Name of the element's class
text	Text content of the element and its children (case sensitive)

Android

Attribute	Description
resource-id	Value of the element's Resource ID ex: com.example:id/icon
content-desc	Value of the element's Content Description (case sensitive)
class	Name of the element's class ex: android.widget.TextView
text	Text content of the element and its children (case sensitive)

Any mixture of these attributes can be used to describe your element:

// tap on an element by text
await session.tap({
  element: {
    attributes: { 
      text: "OK"
    }
  }  
})

// tap on an element by accessibilityIdentifier
await session.tap({
  element: {
    attributes: {
      accessibilityIdentifier: "dialog-confirm-button"
    }
  }  
})

// tap on an element by *both* text and accessibilityIdentifier
await session.tap({
  element: {
    attributes: {
      text: "OK",
      accessibilityIdentifier: "dialog-confirm-button"
    }
  }  
})

Best Practices

When developing your app, we recommend adding accessibility identifiers wherever possible to aid you when automating interactions with Appetize. This will allow for simpler queries and also help your app be more accessible.

iOS

On iOS, you should be using accessibilityIdentifier.

Android

On Android, you should be using resource-id.

React Native

On React Native, a testID property on your component will map to accessibilityIdentifier on iOS and resource-id on Android.

Some components may not (yet) support mapping testID back to a resource-id. In these cases the fallback is to look at tag or content-desc for the value.

Sample

// ios
await session.tap({
  element: {
      attributes: {
          accessibilityIdentifier: "dialog-confirm-button"
      }
  }
})

// android
await session.tap({
  element: {
      attributes: {
          'resource-id': "dialog-confirm-button"
      }
  }
})

If you do not have an accessibilityId to reference it is best to query elements by text or accessibility attributes such as accessibilityLabel (iOS) or content-desc (Android).

Methods

findElement

Returns an element that matches the selector. This is useful for waiting until an element appears.

const element = await session.findElement({
    attributes: {
        text: "Home"
    }
})

console.log(element)

// output
{
  attributes: { text: "Home", class: "UILabel" },
  path: "0/0/0/0/1/2/3",
  bounds: { x: 10, y: 100, width: 100, height: 20 }  
}

If multiple elements are found it will return the first element.

findElements

Returns an array of all elements matching that selector.

const elements = await session.findElements({
    attributes: { 
       accessibilityLabel: 'Like post'
    }
})

swipe

Swipes from the screen position or element

By position or coordinates

// swipe up at middle of the screen
session.swipe({
    gesture: 'up', // can be 'left', 'down', 'right', or a function
    position: {
        x: '50%', // 50% of screen width
        y: '50%', // 50% of screen height
    },
    duration: 1000 // optional, in ms
})

// swipe up at x/y coordinate
session.swipe({
    gesture: 'up',
    coordinates: {
        x: 100,
        y: 100
    }
})

By element

// swipe left from the middle of an image element
session.swipe({
    gesture: 'left'
    element: { 
        attributes: {
            accessibilityIdentifier: 'image-carousel' 
        }        
    },
    // optional, defaults to 50%,50%
    localPosition: { 
        x: '50%', // middle of element on x-axis
        y: '50%'  // middle of element on y-axis
    }
})

Complex Gestures

You can describe a more complex gesture by providing a function to gesture.

// swipe from middle of screen to the left and then down
session.swipe({
    position: {
        x: '50%',
        y: '50%',
    },
    // use up(), left(), right(), or down() for simple movements
    gesture: g => g.left('25%').down('25%')
})

// swipe diagonally up and right
session.swipe({
    position: {
        x: '50%',
        y: '50%',
    },
    // .to(x, y) will move to that position on screen
    gesture: g => g.to('25%', '-25%')
})

// swipe up, wait 500ms, and then swipe right
session.swipe({
    position: {
        x: '0%',
        y: '100%',
    },
    // wait(ms) will hold the swipe gesture in place
    gesture: g => g.up('25%').wait(500).right('50%')
})

tap

Taps on the specified element or coordinates

You can set a duration for a tap to simulate holding it down. For example, if you set it to 2 seconds, the tap will stay pressed for 2 seconds before releasing.

await session.tap({ element, duration: 2000 })

By position or coordinate

// tap at middle of the screen
await session.tap({
    position: {
        x: '50%',
        y: '50%'
    },
    // optional, defaults to 50%,50%
    localPosition: { 
        x: '50%', // middle of element on x-axis
        y: '50%'  // middle of element on y-axis
    }
})

// tap at x/y coordinate
await session.tap({
    position: {
        x: 100,
        y: 250
    },
    // optional, defaults to 50%,50%
    localPosition: { 
        x: '50%', // middle of element on x-axis
        y: '50%'  // middle of element on y-axis
    }
})

By element

await session.tap({ 
    element: {
        attributes: {
            text: 'submit'
        } 
    },
    // optional, defaults to 50%,50%
    localPosition: { 
        x: '50%', // middle of element on x-axis
        y: '50%'  // middle of element on y-axis
    }
})

Timeouts

Interactions that target an element will wait up to 10 seconds to find the element. You may lower these timeouts by passing a second parameter:

await session.tap({ ... }, { timeout: 5000 })
await session.swipe({ ... }, { timeout: 5000 })
await session.findElement({ ... }, { timeout: 5000 })