Designing for misalignment in Augmented Reality displays: Towards a Theory of Contextual Aids

Augmented Reality (AR) displays show virtual information spatially in the physical world. For example, in the surgical domain, AR displays can overlay medical imaging on top of the patient to help surgeons distinguish healthy tissue from cancerous tissue. This affordance of adding visual information near the area of action has led to a gamut of AR applications with one caveat: their design assumes a flawless alignment between the virtual information and its physical context. If this alignment fails, the system might mislead users by presenting the virtual information out of context.Fortunately, technical advancements in tracking, calibration, and optics have helped improve and create a more seamless augmented reality experience; however, even in these improved systems, alignment failure will lead to user-task performance errors. In my dissertation, I argue that AR interface design needs to account for such misalignments and limitations of AR technology. First, I investigated the case of a situated checklist for cardiopulmonary resuscitation (HoloCPR), which helped deliver accurate guidance even in the presence of tracking errors. Then, I investigated a telementoring system for trauma procedures (ARTEMIS) which helped remote surgeons communicate with novices despite a constant registration error. Finally, I showed that slight optical deviation can drastically impact overall user-task performance and cognitive workload, even in an idealized environment with perfect tracking and calibration.I sum the findings of this final piece, along with the systems designed and developed in this dissertation, to point in the direction of a new type of AR interface building block: contextual aids and how they may have significant implications for AR interface design for surgical procedures, paving the way for the development and integration of more advanced AR systems in surgical practice

The VideoMob Interactive Art Installation Connecting Strangers through Inclusive Digital Crowds

VideoMob is an interactive video platform and an artwork that enables strangers visiting different installation locations to interact across time and space through a computer interface that detects their presence, video-records their actions while automatically removing the video background through computer vision, and co-situates visitors as part of the same digital environment. Through the combination of individual user videos to form a digital crowd, strangers are connected through the graphic display. Our work is inspired by the way distant people can interact with each other through technology and influenced by artists working in the realm of interactive art. We deployed VideoMob in a variety of settings, locations, and contexts to observe hundreds of visitors’ reactions. By analyzing behavioral data collected through depth cameras from our 1,068 recordings across eight venues, we studied how participants behave when given the opportunity to record their own video portrait into the artwork. We report the specific activity performed in front of the camera and the influences that existing crowds impose on new participants. Our analysis informs the integration of a series of possible novel interaction paradigms based on real-time analysis of the visitors’ behavior through specific computer vision and machine learning techniques that have the potential to increase the engagement of the artwork's visitors and to impact user experience. </jats:p