Displaying controllers

User goal / Problem this is trying to solve

• Allow the user to see the physical handheld controllers in the VR environment.
• This allows the user to put down and pick up the controllers without removing the headset
• It allows users to see the physical buttons and inputs on the controller, making it easier to input commands whilst in the VR space

Interaction

An exact virtual representation of the controllers is shown in the VR space. They need to be 1:1 scale and tracked accurately, so their position in the VR space matches exactly their position in the real world.

Since they are hand held, this means the user will always need to see them moving realistically as they move their hands.

This has the added benefit of increasing presence, especially if the tracking accuracy and latency are reduce to imperceivable levels.

1. Show the controllers in the virtual space
2. Track their movement and orientation
3. Show all input controls in precise locations, or at least the ones that are relevant to the experience
4. Visually change the appearance on the input controls when the user interacts with them to provide feedback about where the user is pressing (e.g. Oculus shows the joysticks moving and the button being pressed in)

This is particularly effective on the Vive and Oculus Rift platforms.

Note that designers need to decide when to show the controller and when to show the hand and this really depends on the context and application.

The controllers are best shown when the user does not need to manipulate virtual objects with hand like gestures, or when they need to interact by pressing the correct button, especially if pressing a different button could have an undesirable effect.

Linked to: Controller Hints (overlay of action labels over the top of the controller)

Good

• Allows the user to minimise the need to remove the headset and break presence.
• Controllers can be easily found and picked up whilst in VR
• The user can see where the buttons and inputs are and easily execute commands in this way

Bad

One drawback is that this pattern is limited to scenarios where the user only needs to activate certain inputs on the controller.

It’s not easy to show the controller in this way when for example the user is holding virtual objects in their hands.

PlayStation VR applications do not typically visualise the controllers within the VR environment, and the difference on the experience is immense as every time the headset needs to be removed to view the controller, the user experiences a frustrating disruption to presence.

Examples

Onwards, Oculus

Oculus controllers shown in Onward game with real time input feedback

Google Block, HTC Vive

Vive controllers visualised is a stylised way in Google Blocks

Oculus touch

Due to the touch sensitivity and high fidelity spatial tracking both the hands and the controllers can be visualised simultaneously in a very believable way.
Users have commented that it really feels like their actual hands