Towards Immersive Virtual Reality
Why eye tracking is the natural next step for VR
A paradigm shift in virtual reality: Your eyes touch the screen
In 2007 Apple launched the iPhone, and the whole concept of using mobile devices changed. How? By replacing the joystick/keypad with the touch screen, the user now interacts directly with the screen using his/her fingers.
An analogous step in virtual reality (VR) is eye tracking. Rather than 1) looking, 2) turning your head or pointing with the hand controller and 3) selecting, you simply 1) look and 2) select. An entire step is removed in every interaction.
Over the past 15 years, gradually fine-tuned eye tracking technology has been successfully applied in several demanding applications, from medicine to marketing. Within the next couple of years, it is bound to change the entire concept of VR as well.
This document briefly summarizes the challenges for the VR market, the possibilities that eye tracking opens up for VR and how it will shape the future of VR.
Oscar Werner, President Tobii Tech:
“Eye tracking eliminates an entire step of interaction in VR, just like touch interfaces do compared to the mouse or the touchpad. In this way, VR interaction becomes more human. Let’s face it, we don’t establish eye contact by turning our forehead to the person we talk to. We don’t turn our forehead to point where we want to throw an object. We look. That is how we as humans aim. To bring true reality to VR, VR headsets need to adopt to the way humans naturally interact.”
The VR-race is on, and some of the world’s largest consumer electronics brands are investing in it to win. The business opportunities and challenges are massive, and whoever succeeds in creating truly natural, immersive experiences will play a leading role in this fast-growing market.
Reasonably priced head-mounted displays (HMDs) from major tech companies are expected to hit the shelves in time for the 2017 holiday season. In other words, VR-hardware will take the first step towards becoming mass-market products.
According to a recent market survey, 15 % of respondents in developed regions intend to buy an HMD within the next 12 months. In China, fully 45 % expressed this intention. Source: UBS Global Research, Is VR becoming an Investment Reality? 30 January 2017
Mass-market adoption of VR faces a number of demanding challenges:
- Pricing. The tipping point for mass-market adoption of HMDs is expected to be $500. Source: UBS Global Research, Is VR becoming an Investment Reality? 30 January 2017
- “VR Nausea” following extended use.
- Poor rendering and image quality.
- Lack of attractive content.
The first wave of mass-market HMDs expected to come out in 2017 will be near or below the price range indicated above, and volume growth will continue to reduce the price. The bigger question is whether the challenges related to user experience, image output and application development will be sufficiently resolved in order to entice consumer demand.
One thing is certain: The winners in the VR-race will be players who successfully innovate to create naturally immersive VR experiences.
VR-Immersion through eye tracking
Eye tracking enables a unique tool set for creating a natural, immersive VR experience. In the following section, the basic functionality enabled by eye tracking is described and exemplified.
Use your gaze – a new paradigm in VR
Eye tracking technology enables the VR headset to register eye movement and positioning. Hence, the VR environment can be built around the fact that the software is responsive to the user’s gaze. This naturally translates into a whole new range of possibilities in terms of interacting with and experiencing the VR environment.
It is important to emphasize eye tracking as a new dimension in VR. New opportunities open up for software developers to artfully employ and play with the basic human ability of focusing the gaze, for instance when we perform actions and express intentions, reactions and emotions. The objective is to create intuitive, fluid and immersive experiences. Some of the benefits are:
- Human eye movement representation. In multiplayer mode, your avatar’s eye movement mirrors your actual eye movement. This is to avoid avatars staring straight ahead with fixed eyes, and adds a strong emotional element to social interaction.
- Eye tracking shortens the distance between intent and action, enabling the user to perform the same action through fewer steps. You look, you click. Only two steps, and the experience feels natural and direct.
- Eye tracking removes unnecessary and unnatural head movements: In real life, you don’t point with your forehead. If you have a menu in front of you, you don’t navigate it by moving your forehead. You look at the intended part of the menu, and then take action using, for example, your hands.
- Eye tracking enables the headset to know what you are interested in: Non-eye tracking VR headsets (incorrectly) assume that your intent is equal to the direction of your forehead. In real life, we often look in one direction (our real area of interest) with our head pointing in another direction.
- Eye tracking enables the VR environment to add implicit interaction based on gaze. The VR environment can respond to the gaze regardless of any user intention. For instance, NPCs (none playable character) can react to your gaze. Just like humans.
Pushing VR towards widespread market adoption
The emergence of eye tracking in VR is analogous to the implementation of touch screens in mobile phones. In both cases, a new paradigm is set through the integration of sensors adapted to basic human sensual abilities that allow the user to interact with the device in a more intuitive, precise and direct way.
Eye tracking in VR represents a paradigm shift and may well be a key factor pushing VR towards widespread market adoption.
Examples of user scenarios
Feature: Avatar Eye Movements
Scenario: An online multiplayer VR-community where you as an avatar interact with other avatars.
VR without ET: Eye movement of the avatar does not represent eye movement of the user. The avatar’s eyes stare straight ahead without making eye contact with other players and face is as stiff as a doll’s face.
VR with ET: Eye movement of the avatar represents eye movement of the user. The avatar’s eyes convey the user’s emotions and thoughts.
Natural targeting: Throw/aim at gaze and pick up at gaze
Scenario: You are a hunter-gatherer in the Neolithic era. You pick up rocks and throw them at bears attacking your cave.
VR without ET: The precision of your throw is determined by the direction and speed when moving the controller. The rocks are on a pile right next to you and you pick them up by touching them with your controller.
But it is pretty hard to hit the bears. This is due to three reasons (i) your are holding a lightweight controller and trying to replicate the hand movement of throwing a heavy stone without feeling the weight of the stone, (ii) the accuracy of hand controllers is not perfect and (iii) you are (probably) not as skilled as our prehistoric ancestors at throwing stones or you would have to train a life time acquire the skillset.
The developers regret that the bears cannot move very fast, or be too far away. Otherwise the game would become too difficult or too easy, as they would have to reduce the number of bears to enlarge the area that represents a hit. Also, it is a challenge to integrate additional elements, such as picking up rocks from different locations in and around the cave without breaking up the natural flow of the game.
VR with ET: The precision of your throw is determined by the direction and speed in which you move the controller – and also aided by what you look at in the VR environment. The bears can move fast and the difficulty is tweaked by the extent to which your gaze assists your throw.
The developers have fine-tuned how much aid, depending on the intended gameplay. The fact that the developers know your intent, i.e. the target you want to hit, is extremely powerful. Which makes it easier to create more immersive gameplay. Now the bears can move fast, and the difficulty is tweaked by the extent to which your gaze assists your throw.
Moreover, the stones are scattered all around the cave entrance. You find them by looking at them (the stone starts to glow) and pick them up with just one click before you throw. Later in the game you acquire a bow and arrows, and here too your gaze assists your aim.
Eye tracking use cases
The above examples do not exhaust all the possibilities that eye tracking brings to VR. But it indicates how eye tracking can be implemented in game mechanics in order to improve user interaction and immersion.
Eye tracking in VR opens up new possibilities for software developers to create experiences through a VR environment responsive to the user’s eyes.
Below is a list of use cases that eye tracking enables in VR. In going forward towards deeper immersion through eye tracking in VR, developers will further refine and expand these concepts. To this end, strong partnerships between hardware vendors and content developers will be key.
- Feature: Avatar eye movements: Your avatar’s eye movement mirrors your actual eye movement, adding a strong emotional element to social interaction.
- Feature: Pick up at gaze: Gaze assisted object interaction - look at an object and press a button, and it is picked up.
- Feature: Throw at gaze: Gaze assisted throwing - hit the object you are looking at much easier.
- Feature: Fire at Gaze: Projectiles converge and curve into the target you are looking at.
- Feature: Select at gaze: Look at an object and manipulate it with the hand controllers.
- Feature: Teleport/move to gaze: Press the teleportation button and look at a teleportation station and release the button, and thus teleport/move there
- Feature: Foveated Rendering: Do less rendering on parts of the screen you are not looking at. This is unnoticeable and frees up processing to improve the graphical experience.
- Feature: Dynamic Depth of Field: Put the objects you are looking at in focus.
The natural next step forward
Towards eye tracking in VR
A successful implementation and roll out of eye tracking in an HMD requires overcoming, and piecing together, three major technical challenges.
- Optomechanics. Eye tracking begins with high frequency series of pictures of your eyes. Integrated cameras deliver high quality images that enable the system to register your gaze and your eye movement.
- Advanced computer vision algorithms transform the pictures into gaze x and y coordinates, eye open/close signals, interpupillary distance measurements and pupil dilation measurements.
- User experience design and easy to use Software Development Toolkits enable developers to rely on eye tracking features as they build their applications.
For each of the challenges listed above there are proven technical solutions. However companies that master all these technology areas are few. *)
Move fast. Go big.
The touch screen analogy aims to stress the fact that eye tracking in VR represents a real paradigm shift and a huge opportunity. Providing a new level of immersion and ease of use, it may well be a key factor propelling VR towards widespread market adoption.
Also, fully functional eye tracking demos are available today **), and the forerunners in the VR industry are planning future product releases. Eye tracking is definitely part of these plans. The fact that both Google and Facebook acquired eye tracking companies in 2016 is a testimony to the technology’s importance.
The opportunity to make your VR HMD responsive to the user’s gaze is simply too great to be ignored. That is why eye tracking is the next natural step for VR.
- Tobii is the world’s largest provider of eye tracking (ET) technology, knowledge and services. The company offers ET solutions for gaming, augmentative and alternative communication, behavioral research and technology integration.
- **) Working closely with leading game studios and hardware suppliers around the world, the company offers unique ET knowledge and resources, fully functional demos, a qualified development platform and support, as well as business insight and opportunities for joint marketing activities.
- Tobii is actively involved in the development of the Khronos’ VR standard to be released in 2018. The standard will include cross-platform APIs for tracking headsets, controllers and other objects, and rendering to a diverse set of display hardware.
- *) Tobii has extensive experience in the three critical knowledge areas highlighted above – optomechanics, advanced computer vision algorithms and User experience design and Software Development Toolkits