Navigating the Digital Frontier: The Evolution of Three-Dimensional User Interfaces through VR and AR Technologies

Expert analysis by Yashwanth Kotha, Senior UI/UX Designer

In the 1960s, researchers kickstarted the exploration of 3D interaction and display. A significant breakthrough came in 1962 when Morton Heilig invented the Sensorama simulator. This device offered 3D video feedback, motion, audio, and other sensations to create a virtual environment.

However, progress was slow due to limited technology and high costs until the 1980s. Initially, virtual environments were mainly used for military purposes in the United States. Over time, technological advancements led to new opportunities in education, entertainment, manufacturing, and other fields. Today 3D user interfaces are becoming an essential part of product design.

Unlike conventional 2D UIs that confine interaction to flat screens and tools like a mouse or a keyboard, 3D UIs provide a more immersive method for navigating and manipulating digital content. With the help of special devices such as data gloves, motion trackers, VR headsets, or 3D mice, 3d UIs enable users to interact more naturally and directly with the digital environment. In this article we’ll explore the differences in 2d and 3D UIs and will have a detailed look at challenges both approaches can pose. Besides, we’ll see how 3-dimensional interfaces evolved and developed. Hop in to learn more.

The essence of 3D interfaces deviates from traditional 2D interfaces in significant ways. They offer a heightened level of immersion and interactivity. While 2D interfaces are flat and rely on x and y coordinates for navigation and interaction, 3D interfaces introduce a z-axis, adding depth to the user experience. This depth fosters more organic interactions, mirroring real-world behaviours.

2D UIs compared to 3D UIs: challenges and limitations

The CEO of Experience Dynamics, Frank Spillers defines the main differences in the following way. In 3D, your view changes based on head movement, adding a layer of comfort consideration for users wearing head-mounted devices. Unlike 2D, there’s no fixed canvas or frame in VR, no familiar keyboard or mouse. Instead, users navigate through a spatial environment where sound, movement, and space interaction play crucial roles.

Despite these differences, some elements remain constant. Both 2D and 3D UIs have user goals and affordances, indicating interactable elements like buttons. However, in 3D, these elements take on more realistic forms, like tangible buttons or levers, requiring clear affordances for users to identify.

In 2D, users follow linear flows, moving from screen to screen. In contrast, 3D experiences are more narrative-driven, guiding users through immersive storylines or interactions. While 2D deals with states of interaction, 3D interfaces present a series of connected episodes.

Emotions play a role in both types of interfaces, but in 3D, they induce moods that persist throughout the experience, shaping the user’s journey. Navigation in 2D is straightforward, while in 3D, it becomes specialised, allowing users to move freely within a virtual environment.

Tools in 2D are static icons or menus, whereas in 3D, they become body- or world-locked tools. Body-locked tools stay close to the user, whole world-locked tools interact with the environment. Additionally, UI elements in 2D are non-diegetic, existing separately from the user’s surroundings. In 3D, UIs become diegetic, seamlessly integrating into the virtual world.

While transitioning from 2D to 3D offers exciting possibilities, it comes with its set of challenges. Designers must ensure user comfort, address motion sickness in VR environments, and navigate the complexities of rendering 3D graphics effectively. Despite these obstacles, 3D interfaces open doors to immersive experiences and innovative design solutions beyond traditional 2D interfaces.

Now let us conclude the list of limitations 2D UIs pose:

– Lack of depth and spatial awareness limits immersion.

– Unable to create a sense of presence like 3D interfaces.

– Struggles to convey spatial relationships effectively.

– Relies on traditional input methods, restricting interaction possibilities.

– Numerous static elements that do not respond to user actions or environmental changes.

– Overall, inherent limitations hinder truly immersive and engaging experiences.

Emergence of Three-Dimensional User Interfaces

As we have already stated, a three-dimensional interface enables users to interact with virtual objects, environments, or information using direct 3D input in physical and/or virtual space.

The history of 3D interactions dates back to the 1960s, pioneered by researchers such as Ivan Sutherland, Fred Brooks, and Morton Heilig, who invented the Sensorama simulator in 1962. Dr. Ivan Sutherland further advanced 3D interfaces with his work on the Sword of Damocles in 1968, creating a head-mounted display for immersive experiences.

Types of 3D interfaces include:

1. Virtual reality interfaces

Fully immersive environments accessible through VR headsets, allowing users to interact with virtual worlds using hand controllers, gestures, or voice commands like Oculus Rift S.

2. Mixed reality interfaces

Combining elements of VR and AR, MR interfaces enable virtual and real elements to coexist, offering complex and interactive experiences. For example, HoloLens.

3. 3D desktop interfaces

Extending traditional 2D desktop environments into three dimensions, these interfaces improve navigation and organisation through spatial arrangement.

Virtual Reality: Immersive Experiences

By virtual reality we understand a simulated 3D environment that lets users explore and interact with a digital world that mimics reality. Users may need special devices like helmets or goggles to fully immerse themselves in the virtual environment.

There are three main types of VR:

1. Non-immersive VR

Non-immersive VR space is accessed through a computer screen, where users have some control over the virtual environment but it doesn’t directly interact with them. Examples include video games and online room design tools.

2. Semi-immersive VR

It provides a partial virtual experience through a screen or special glasses/headset, focusing on visual 3D aspects without physical movement. Flight simulators are a common example.

3. Fully immersive VR

Fully immersive VR offers the highest level of immersion by completely placing users in a 3D world with sight, sound, and sometimes touch or smell. Users wear special equipment and can interact fully with the environment. This technology is still developing but has applications in gaming and healthcare. Additionally, collaborative VR allows people from different locations to interact in a shared virtual space, communicating through microphones and headsets.

Case Study

One of the most interesting examples is the VR UI of The Economist VR app. In line with The Economist’s dedication to offering content across all digital platforms and their venture into immersive storytelling, they have teamed up with Visualise to create a specialised VR app for the publication.

The Economist VR app currently features three immersive experiences, including two collaborative projects: the acclaimed RecoVR Mosul: a collective reconstruction and a new VR travel film, Passport: Osaka. Additionally, they have partnered with US studio Object Normal on Oceans VR: Net Positive.

In response to the growing trend of traditional media outlets launching apps, The Economist VR app aims to showcase top-notch VR design and innovation. To achieve this, Visualise collaborated with product design studio Kickpush, known for their advanced UX design approach. Their in-house development team then translated these design concepts into a captivating virtual experience.

Augmented Reality: Bridging Digital and Physical Worlds

Augmented reality uses the existing real-world environment and adds virtual information on top of it. Unlike virtual reality, where users are immersed in entirely digital space, AR integrates digital elements into the physical world. AR applications range from visualising furniture in a room to interactive educational tools and enhanced retail experiences.

While AR is often considered a subset of VR, it is increasingly recognised as a distinct field. Mixed reality (MR) is another related concept that combines physical and virtual worlds seamlessly. Together, VR, AR, and MR are often referred to as “extended reality” (XR).

AR has diverse applications across industries. The evolution of Augmented Reality is closely connected with the advancements in hardware, software, and connectivity, transitioning from simple overlays on computer screens to sophisticated experiences using smartphones, smart glasses, and other wearable devices.

Early applications of AR centred around entertainment and gaming, with apps like Pokemon Go introducing the concept to a broad audience. Smartphone AR technology further democratised AR, allowing users to interact with virtual elements through their mobile devices.

The development of AR wearables, such as smart glasses, has enabled hands-free, immersive experiences across different fields. AR is applied in education, healthcare, retail, and enterprise. It has become a useful tool for interactive learning experiences, aiding medical procedures, enhancing shopping experiences, and improving workplace productivity.

Looking ahead, the future of AR holds promising advancements such as the AR Cloud, which facilitates persistent AR experiences, and the integration of 5G connectivity for faster and more reliable AR content delivery. Spatial computing, focusing on the interaction between physical and digital spaces, will continue to drive innovation in AR, paving the way for seamless integration of digital content into our spatial environment.

Case Study

As an example, I would like to mention the IKEA Kreativ app. It allows users to scan their room and design their own space by incorporating IKEA’s 3D products into their home environment. Users can experience this in mixed reality or choose from over 50 virtual showrooms to visualise their designs.

Here is another example by IKEA: IKEA Place app. It represents a significant step forward in changing the way people shop for furniture.

With this app, users can virtually place true-to-scale 3D models of IKEA furniture in their own homes. By simply using their smartphone camera, users can visualise how different pieces of furniture will look and fit in their space before making a purchase. This immersive experience allows for confident decision-making and ensures that the chosen furniture complements the user’s existing decor and layout seamlessly.

Final Thoughts

In the future, we can expect even greater progress in 3D user interfaces, especially in integrating them into traditional 2D environments like websites. This will add a new layer of complexity to interface design, prompting considerations for how interfaces behave in immersive environments such as Virtual Reality. Interactivity will remain a key trend, with animation continuing to play a crucial role in interface design.

Looking forward, there’s potential for further exploration of multi-dimensional design, while recognizing the ongoing relevance of 2D interfaces when adapted to 3D environments. Embracing design fictions can stimulate imagination and innovation. As we embrace the trend of Immersive UI, we should explore ways to incorporate more sensorial experiences and foster creativity. This prompts us to consider the challenges and opportunities associated with designing immersive and engaging interfaces.