Platform Experience / Augmented Reality
Prototyping GeoSpatial Augmented Reality Experience for Explorative Research
Executive Summary
We prototyped a spatial storytelling experience using a geo-locative AR technology. We digitally reconstructed Rochester's Third Ward from the mid-twentieth century in the exact location before urban renewal altered the neighborhood. Through this immersive journey, viewers are able to view 3D model of the old neighborhood and gain a deep understanding of the history and significance of the African American experience in the urban North.
My Role
I performed background research on Location based AR technology, created a roadmap for an AR prototype, and developed the experience. I was part of first set of explorative research with AR prototype to understand the requirement for building a full fledged mobile app.
Why this Project?
The Annual Clarissa Street Reunion, established in 1996, honors the values, relationships, music, and culture of the neighborhood. Its rich history is portrayed through museum exhibits with photos and videos. The Jazz Club of Rochester plans to feature Clarissa Street's iconic 1960s Jazz Club, emphasizing the area's cultural vibrancy through a engaging in-location experience.
What's the Opportunity?
Given the diverse age groups in attendance of the annual reunion, there exists a unique opportunity for our team at MAGIC Spells Studios to engage with the community interactively, telling the compelling story of their neighborhood's cultural heritage and embraces the history through augmented reality experience.
Vision
Leverage the use of AR experience to educate the current generation about historically significant places in their neighborhoods that no longer exist.
Approach
It's crucial to understand that we won't be creating the entire app in one go. The initial AR experience serves as a foundation for exploratory research, involving a significant number of participants using the AR prototype and sharing their insights. Their invaluable insights, gathered are the building blocks for a follow-up user-centered design approach to design and deploy the final product.
Understanding the Context
In the exploration of Clarissa Street's jazz heritage, the "Pythodd Club" emerges as a paramount venue where internationally acclaimed musicians graced the stage during the vibrant 1960s jazz era. This club stood as a beacon of musical brilliance, becoming an integral part of Clarissa Street's cultural identity. Our team meticulously assessed the ambiance of the Pythodd Club by engaging with government officials from the city of Rochester. Through these interactions, we gathered images and video recording of the Pythodd Club.
Additionally, we reviewed the playlists of the renowned jazz tunes that once reverberated within the club's walls during the 1960s. We ideated to use these as interactive elements in the AR experience.
Observational Study
We visited the Pythodd Club location to comprehend the spatial layout, determining the placement of our 3D model and assessing the surrounding environment. During our visit, we identified real-world obstacles that might obstruct the 3D model in the AR space. Additionally, we conducted a safety audit for users, particularly considering the location's positioning at a busy intersection where awareness of moving traffic is crucial. Finally, we measured the available dimensions of the space to ensure our real-life-sized virtual building fits seamlessly within the environment.
What to build?
Following initial field research and stakeholder discussions, we defined the scope of the project. We decided to rebuild the historic "Pythodd Club" immersive experience via AR, recreating Clarissa Street's jazz ambiance for an immersive cultural experience in an outdoor location.
What content should we use?
To bring the space to life, we needed to gather:
- Historical archives (photos, blueprints, audio recordings)
- Musical assets that captured the essence of mid-20th-century jazz
- Spatial data of the original building footprint for accurate AR reconstruction
- Firsthand oral histories from community members and historians
How should we build it?
We decided to use Unity's AR Foundation framework, enabling cross-platform development for both iOS and Android, integrating features from ARCore for Android and ARKit for iOS. This approach would streamline development, eliminating the need for separate coding for each platform and reducing overall development time.
Phase 1: Location Based Tracking System using GPS (2022)
Initially, we had built an in-house location-based tracking algorithm for Unity AR to place 3D models in world coordinates. Simultaneously, we explored other outdoor location tracking Unity integrations. Each option had its pros and cons, but none met our team's requirements in terms of pricing and functionality flexibility.
Updating the Model
While experimenting with a generating label in the geo-location, our 3D modeling artist provided a draft version of the Pythodd Club. We subsequently replaced the labels with the draft version.
The solution faced challenges due to inconsistent GPS calibration across different devices, leading to varying precision. The virtual object frequently had an offset height in real-world space until the GPS stabilized. Moreover, under low-light conditions, especially at night, the real environment lacked detail, resulting in a less realistic visual experience in AR space.
Building the AR Interactions
At this point, we updated our model further and Rather than offering users a static model for research purposes, we opted to include interactive elements. This approach allows users to explore the prototype's flexibility freely and offer suggestions without constraints. We implemented ray-based interactions around specific objects in the model, like windows, doors, and entrances. Interacting with these objects provides users with media and music experiences
"The known issue at this point was fluctuating gps values for same location. My co-op at MAGIC Spell Studios concluded at this point (Jul 2022), leading to reduced involvement in the project. However, after the summer 2023, I rejoined the team and resumed my contributions"
New Updates in the AR Dev Tools and APIs
At this time, both the Google Geospatial API, which supports VPS with GPS, and Niantic Flagship VPS were released. These updates proved to be more accurate in determining real-world locations, significantly enhancing the precision of location-based tracking.
After evaluating these toolkits, we decided to use Google Geospatial API to the team. This choice was influenced by our team's familiarity with ARCore SDK and the limited location options provided by Niantic at that time (July 2023) which didn't support for Rochester. This further convinced our decision to opt for the Google Geospatial API for its flexibility in terms of location availability.
Migrating Prototype to Google's Geospatial API
Transitioning to the Geospatial API was seamless because most of our AR interactions were closely integrated with the model. Clear documentation and helpful examples from Google simplified our task, making the migration process smooth and efficient.
When we tested the prototype first in our nearby location to check the preciseness with our model, it was way better than our previous algorithm.
Later we tested at out Clarissa street before the Reunion where we planned to do an explorative research. We noticed some misalignments in terms of orientation of the building which we fixed before the Reunion.
Milestone
Stable location-based AR prototype with spatial interactions used for the user study during Clarissa Street Reunion on Aug 19, 2023.
During the reunion, my co-developer and I presented the prototype to individuals of all age groups. Our team administered a survey containing Likert scale questions and a few open-ended questions for participants to answer. Since this write-up focuses on the prototyping workflow rather than explaining the contextual inquiry we conducted, I will swiftly move on to the findings of this study.
Few Other Interesting Observations
Sunlight Glare on Display
The sunlight glare made it difficult for the participants to view the prototype, so we had to use umbrellas to shield the display from the sunlight. Since it is an outdoor only experience, it is crucial to tackle this situation
Handling Occlusion
During the prototyping phase, we accounted for car occlusion in the parking lots. However, at the reunion, the parking lot was crowded, and our prototype did not handle people occlusion. Consequently, the experience was not fully seamless.
Scalability
We are currently exploring on scalability of the app for media management by retrieving it from database like Firestore.
AR Experience Enhancements
Working on enhancing the AR experience with new guidelines based on feedback and observations while the UX team works on the app design and flow.
Interior Design
Detailing out the interior AR experience and exploring ways to integrate with the outdoor experience.
Reflections
Balancing the roles of User Researcher and AR Experience Prototyper posed a significant challenge. During the development process, numerous questions arose regarding user needs, leading to the incorporation of additional features for testing. While this enriched the experience, it also extended the development timeline. Therefore, prioritizing essential elements for the demo became paramount.
I gained extensive experience in utilizing various Google AR Core APIs, including Occlusion Manager, Light Estimation API, and Geospatial API. These experiences provided valuable insights into the significance of scene understanding for achieving more realistic rendering of virtual objects.
As the AR development tools and frameworks continue to evolve, what we strived to achieve over several months last year in terms of location tracking has now become a simple API call, offering enhanced precision. This illustrates the beauty of AR development and motivates me to explore new possibilities for improving immersion.
Next Project