The most profound technologies are those that disappear. They weave themselves into the fabric of everyday life until they are indistinguishable from it — Mark Weiser

The tree of human computer interaction continues to grow. There are historic and established branches with examples like the mouse and keyboard, touch, and gaming controllers. Recent trends include ambient computing and human understanding, such as speech, hand, and body tracking. Spatial computing encompasses the sum of all academic and industry work in these areas, alongside research from HCI-adjacent fields like psychology, sociology, cognitive neuroscience, and artificial intelligence.

Through the lens of HCI, spatial computing is distinct from the tactical execution of AR/VR/MR. While a subset of technologies within computer vision, machine intelligence, and display/optical systems are shared, design and user experience diverges quickly depending on product capabilities and intent: for example, is the product optical see-through or not? Or, should it be worn in public? What gels spatial computing across new product categories is an intent to change how we compute, increasing the utility of computation with specific technologies and design systems that mitigate the challenges of the when and where.

All-day wearable augmented reality requires intentional design to support diverse environments and situations. To cherry-pick one example, imagine future social scenarios concerning our attention. Do we want to compute while engaging in conversation with others? (Ever talked to someone wearing AirPods? The social friction is immense). On the other hand, gaming or enterprise virtual reality platforms belong in room-scale spaces, divorced from the world and commanding the wearer's full attention. Both product categories are deeply entangled in body-scale input and the related network of questions around speed, precision, and fatigue.

With respect to input, an AR/VR product might look at one of these scales with a sampling of questions to surface micro-level challenges such as:

• Social acceptability & input modality;
• ergonomics including length & effort of interaction;
• and latency & fidelity of the feedback loop.

If these bullet points are familiar, it's because they’re fundamental concerns of human computer interaction. The point is that many of elemental components of spatial computing may be decomposed through well-known HCI research and design methodologies. But not everything.

Many in the augmented reality community are in love with the notion of superpowers. The hypothesis is that pervasive environment understanding coupled with a tight and effortless feedback loop enables new modes of computation. Another form of superpower re-examines the impact of embodied cognition on learning. Research in embodied cognition conjectures that VR might accelerate thinking and creativity by increasing the surface area of possible brain/body experiences. Alas, situated activities with head-mounted devices were explicitly out of scope for Weiser when he penned his original thoughts on Ubiquitous Computing decades ago at PARC. Otherwise, AR might resemble an extension of UbiComp theory.

Hypothesis such as these, backed by classically studied interaction problems, cements spatial computing as the latest branch of HCI. Paradigm shifts and their impact may only be measured in retrospect – but the move from pictures under glass to body-centric computation situated in diverse environments – will hopefully be clear looking back ten years from now. Thus, spatial computing is not a revolutionary movement to re-think interaction; it is a steady march towards frictionless and pervasive computation backed by the rich history of HCI.

Ingredients for Evolution

Inspired by Schniderman's (et al.) Grand Challenges for HCI, the following list distills some of the remaining challenges in bringing AR/VR/MR products to scale over the next ten years. Fair warning: this list only scratches the surface of the remaining work to be considered, but offers a few sounding points to continue the exploration.

(1) Friction to engage

A well-designed spatial computing environment will enable humans to express thoughts easily, but platforms need to reduce the friction of transitioning in & out of AR/VR both cognitively, ergonomically, and socially (i.e. the AirPods problem). Furthermore, spatial computing provides an opportunity to fundamentally reconsider the accessibility of computational systems for those with sensory or physical disabilities (for more, please see the Inclusive Design Kit from Kat Holmes).

(2) Input at the speed of thought

Backed by contextual awareness of the environment and personalized data graphs, how can we implement efficient multi-modal input systems that encourage us to treat computers as collaborators rather than machines? Will these devices ever be less than machines? Traditional screen-based metaphors and interactions may be helpful to establish familiarity, but should ultimately take less precedence as we discover new computational behaviors that transcend the (primarily) screen-based input paradigms in use today.

(3) Seamless across devices, environments, and contexts

This notion is what Timoni West calls, "A really good internet of things [6]." Spatial computing devices contend with myriad cloud services and devices speaking different protocols, both open and closed (what some may call the AR Cloud). These layers should exist invisibly between platform and user. Furthermore, a seamless experience needs to be contextually adaptive across social situations where attention might be fractured between the device and the world.

(4) Frameworks for trust and privacy

AR/VR/MR platforms operate on personal data captured or derived from users and their environment. Platform owners must be pressed to provide clear and accountable systems for auditing collected information, and provide strong safeguards to protect the security of data, both locally and remotely. For more, read The Eyes Are the Prize by Avi Bar-Zeev, The Mirrorworld Bill of Rights by Paul Hoover, and Jessica Outlaw's work on XR Privacy.

(5) Real world valence

Spatial computing systems persist in reality, where objects and tools have semantics and purpose. Spatial computing should recognize that environmental elements have intrinsic meaning and often utility in our daily lives (this concept based on ideas outlined by Simon Greenwold in his 2003 MIT Thesis).

(6) Tools made for the medium

Developer experience should be a chief consideration to reduce research and development time. Spatial computing commands novel tools designed from the ground up that capture essential patterns. The first generation of tools offers a solid foundation, with software like Unity's EditorXR and Project MARS, Torch3D, Apple's RealityKit, SketchBox, Microsoft's Maquette, Tivori, and others.

Pervasive adoption of AR/VR/MR platforms is contingent on solving these problems deeply. Spatial computing assists in giving us a practical and theoretical foundation to solve critical interaction issues by chiefly considering the human elements of the problem space.

An ideal path forward involves identifying more problems, at both micro and macro levels, rather than immediately looking to technology for solutions. With forward momentum in spatial computing, AR/VR/MR technologies might conceivably dissapear into the background, bringing us closer to Weiser's vision of invisible computation and what Ken Perlin sees as the magical future of being human.

For more in-depth study of spatial computing principles, check out this index of talks, research papers, and articles.