Colloquy Cycle on Wednesday, 14. July 2021, 10:00

Augmented Reality (AR) and Virtual Reality (VR), collectively known as eXtended Reality (XR) experiences are built on rich, complex real-time interactive systems (RISs) that require the integration of numerous components supporting everything from rendering of virtual content to tracking of objects and people in the real world. Game engines such as Unity and Unreal currently provide a significantly easier pipeline than in the past to integrate different subsystems of XR applications. But there are a number of development questions that arise when considering how interaction, visualization, rendering, and application logic should interact, as developers are often left to create the “logical glue” on their own, leading to software components with low reusability. In this talk, I present a new software design pattern, the Relay & Responder (R&R) pattern, that attempts to address the concerns found with many traditional object-oriented approaches in XR systems. The R&R pattern simplifies the design of these systems by separating logical components from the communication infrastructure that connects them, while minimizing coupling and facilitating the creation of logical hierarchies that can improve XR application design and module reuse. 

Additionally, I explore how this pattern can, across a number of different research development efforts, aid in the creation of powerful and rich XR RISs. I first present related work in XR system design and introduce the R&R pattern. Then I discuss how XR development can be eased by utilizing modular building blocks and present the Mercury Messaging framework (https://github.com/ColumbiaCGUI/MercuryMessaging), which implements the R&R pattern. Next I delve into three new XR systems that explore complex XR RIS designs (including user-study-management modules) using the pattern and framework. I then address the creation of multi-user, networked XR RISs using R&R and Mercury. Finally I end with a discussion on additional considerations, advantages, and limitations of the pattern and framework, in addition to prospective future work that will help improve both.

Dr. Carmine Elvezio recently received his PhD in CS from Columbia University, studying AR/VR/MR and 3D graphics, and interaction and visualization techniques in the Computer Graphics and User Interfaces Lab, advised by Prof. Steven Feiner. He develops 3D systems across several domains including medicine, remote maintenance, space, music, and rehabilitation, working with many technologies including Microsoft HoloLens, Oculus Rift, Unity, Unreal, and OpenGL. He has participated in projects sponsored by NSF,  Google, Verizon, Canon, and NASA, amongst others. He has also contributed to a number of open-source frameworks, including MercuryMessaging (https://github.com/ColumbiaCGUI/MercuryMessaging) and GoblinXNA.

Carmine managed the CGUI Lab at Columbia with Prof. Feiner from 2013 until 2021, where he advised over 150 independent research projects, assisted in teaching courses on 3D user interfaces, AR, and VR, and participated in many multi-disciplinary university initiatives.