Design patterns for Immersive Tech

Virtual Reality patterns

Augmenting controllers

User goal

  • Allow users to easily perform a wide range of tasks using virtual handheld tools
  • Help the user to understand how to use the virtual tool by using it’s appearance to indicate function
  • Allow the user to complete tasks one handed
  • Maintain high levels of usability by incorporating patters such as controller hints maintaining a visual mapping to the physical controller inputs


This article covers a wide range of specific examples and implementations. The core premise is that of transforming the controller’s appearance to more closely resemble the tool needed by the user. The appearance of the controller in the VR environment can be changed to almost anything. It’s common to show a representation of the hand (particularly for Oculus), a gun or a sword (or other handheld object), however here we will focus on interactive tools that combine this change of appearance with intuitive mapping to physical controls.

Some important key principles include:

  • Maintaining the spatial relationship between buttons and inputs on the augmented tool and where the actual buttons are located on the real controller
  • Maintaining an intuitive relationship between the input mechanics on the real controller and the visualised affordance in the VR environment. For example a directional touch-up (e.g. Vive controller) could become a 4 button control pad, a D-pad or a radial menu of options in VR.
  • Using controller hints to indicate the functionality of each input and button on the augmented visualisation
  • Making the top of the controller visually resemble the tool being used: it could become a paintbrush showing the selected colour of paint, a pen, a hammer head, a screwdriver, a spray paint can, a hand to grab things etc.

Linked to: Displaying controllers


  • Makes it clear which tool has been selected
  • Makes the function of the tool easy to understand
  • Takes advantage of the unlimited augmentation potential of VR
  • Adheres to consistent principles that apply even when the control is displayed without augmentation


Steam VR Home: Augmented Oculus controller becomes a multi function Ray Cast

Steam VR Home: Spray can tool using the joystick as a 360 degree colour palette that can be operated one handed

Google Blocks: Hand tool used for grabbing objects

Google Blocks: User can switch the function and appearance of the tool using a menu attached to the other controller