Pointing and selection with finger

User goal / Problem this is trying to solve

• Point at an object or option in the menu that the user would like to select or interact with (e.g. a menu button)
• Pressing a UI button
• Combining pointing and selection into one natural and fluid movement

Interaction

1. The users hand is displayed in the virtual environment (usually by tracking the hand held controller)
2. The index finger is extended and becomes the active selection device
3. The end of the finger may have a glowing ‘ET effect’ to indicate it’s function
4. The user moves the finger to make contacts with buttons and other interactive UI elements through direct contact in the virtual environment
5. Pushing down on a button will fully trigger the input (note that unlike Ray Cast and Gaze the user must be within touching distance, so all menus and interactive UI elements must be within arms length of the users position or a movement pattern will also be required)
6. There is often audio or haptic feedback to indicate a press has been executed (note this will be covered in more depth in a forthcoming pattern)

Good

• It’s intuitive and natural as we are used to pressing buttons with our index finger
• Can help with data entry tasks with a keyboard as we can invoke are encoded spatial and muscle memory for speed typing to some extent
• Works better with Oculus Touch controllers which track when the index finger is actually extended, improving the virtual and physical mapping

Bad

• Cannot easily activate items or options in the distance without switching input mode or moving closer to them
• Lack of realistic haptic feedback can make the interaction feel artificial (although the audio and controller haptics can be engineered to minimise this)
• There is typically no hover state (so soft activation or on-hover disclosure is not supported)
• If the virtual finger is not spatially mapped to the actual finger it can make the interactions feel less real
• Only works with 6DOF controller tracking (or tracking hands directly)

Related to: Pointing and selection with finger (AR)

Examples

Facebook Spaces (Oculus Rift)

Threesixty Reality concept examples