Prototyping AR in a University Museum: How User Tests Informed an Accessibility Plan Including and Beyond the Museum

Max Evjen, Michigan State University, USA

Abstract

Keywords: Augmentation, Immersion, Evaluation, Accessibility

Background

In summer and fall of 2017, the Michigan State University Museum, a university museum of science and culture, worked with a Michigan State University student studying in the art, art history, and design department on the development of an augmented reality (AR) pilot project, focused on a mural depicting a Michigan wetland area in the Museums’ Hall of Animal Diversity.

The Hall of Animal Diversity is a gallery with taxidermied birds, mammals, and insects arranged by theme, for instance, animals of mid-Michigan (Figure 1), camouflage, etc. There is a large brown bear (who has become the museum’s mascot and can be followed on Twitter @MSUMuseumBear), a couple of simple activity stations, an interactive video of different bird sounds, and the Michigan wetland mural (Figure 2), created by a local artist, that includes a key at the bottom for identifying animals featured in the wetland scene.

Figure 1: Animals of mid-Michigan display in the Hall of Animal Diversity

Figure 2: The wetland mural in the Hall of Animal Diversity

Goals of the Project

Staff at the museum had been discussing the possibility of creating AR for a variety of experiences throughout the museum’s ten long-term galleries as a way to update them in an engaging way without completely changing the exhibitions themselves. Some of the exhibitions in the galleries at the museum hail from the 1960s and have few, if any, interactive elements. The mural was an attractive option for a pilot study since repeated anecdotal reports indicated that visitors rarely stopped to even look at the animals in the picture. We, therefore, wanted to use AR to encourage close looking and, in addition, engage visitors with the central themes of the gallery (animal adaptation and behavior).

A clear benefit of working in a museum in an academic environment is the potential collaboration with faculty, staff, and students from a variety of disciplines. We were able to work with a student who had experience working in AR environments in a museum setting using Unity 3D and Vuforia, and an arrangement was made for the student to be hired under a student-work contract by the College of Arts and Letters to work with the museum to develop the AR pilot experience.

The museum purchased four Samsung Nexus 10 tablets to begin testing of the pilot program. The museum staff did not want to create an app to download since data repeatedly shows that most people do not download museum apps or if they do, they only open them once (Dilenschneider (2017), Green (2016), and we provided the devices to visitors during user tests with the program pre-loaded on each device.

Since we only had one student to work on the project and did not have a designer assigned to the project, we were constrained in our design to what we could find in the Unity Asset store that could possibly appear in a Michigan wetland. The outcome was that the experience merely included a few additional animals (a bird, a turtle, a duck, and a dragonfly) that appeared in different areas, and additional text interpretations appeared with AR animals, as well as with animals already painted in the mural (Figures 3-5).

Figure 3: Augmented duck and vegetation

Figure 4: Augmented bird with interpretation

Figure 5: Interpretation with animals in mural

Evaluation Plan

User tests were conducted in fall of 2017 and spring of 2018. (Data was collected by Max Evjen, who was the project manager, and three student volunteers). Analysis of the user tests commenced in fall of 2018 (conducted by the project manager, an intern assigned to the project, and five student volunteers), and results are set to be reported to the museum leadership by February of 2019.

The goals, evaluation questions, and evaluation protocols of the project follow:

Project Goals

Institutional Goals

• Update long-term exhibitions by augmentation rather than replacement
• Engage MSU students particularly, in addition to other visitors

Visitor-Experience Goals

• Encourage close looking at the wetlands mural in the Hall of Animal Diversity
• Encourage visitor learning about animal adaptation and behavior in the context of diverse ecosystems
• Encourage surprise and wonder about the experience

Evaluation Questions

The following evaluation questions were posed to address these goals:

• How much time do visitors spend at the mural during the AR experience?
• Do visitors engage in close looking with the mural during the AR experience?
• Do visitors learn anything about animal behavior and/or adaptation?
• Are visitors delighted while using the AR program?
• Who uses the program?
• What other outcomes do visitors report?

Evaluation Protocols

The following evaluation protocols will be employed to assess the experience, learning, and affect of the visitors as they relate to the institutional and visitor experience goals:

1. Questionnaires: Diamond, Luke, and Uttal (2009) state, “Questionnaires sometimes have advantages over interviews because they can be given out to participants without the evaluator being present, and the evaluator may be less likely to have an influence on participants’ responses.” A survey was offered in the gallery to address evaluation questions 3, 4, 5, and 6. Our final n = 66.
2. Behavior Sampling: Visitors were be observed in the gallery through video. Diamond, Luke, & Uttal (2009) state that continuous recording of an event can be beneficial for “detailed, or sometimes even frame by frame analysis.” Indeed, these continuous recordings allow for a more detailed picture of how visitors are interacting with the exhibits and each other, and what aspects of an exhibition visitors will be most drawn toward. Behavior sampling was conducted to address evaluation questions 1, 2, 4, and 6. Our final n= 81.

Analysis

Analysis was performed in fall 2018 and spring 2019. Evjen and a group of six MSU students, including one intern working on the project, performed the analysis of the data. This included an inductive analysis of the behavior observations, wherein the team watched the videos and wrote on post-it notes what they saw visitors do or hear them say. The team then grouped all of the post-its into themes that would dictate how we designed the behavior-observation-coding scheme. The team used a code book to record instances of those themes and entered them into a Google Form. Quantitative data was tabulated, and qualitative data was coded and analyzed. The team also split up questionnaires and entered the data into a Google Sheet, after which quantitative data was tabulated and qualitative data was coded and analyzed. The report and a presentation are set to be presented to the museum senior staff on February 21st, 2019.

Notable Results

• Results of the analysis of the questionnaires show that 56% of those who engaged in the experience were MSU students, 12.1% were MSU Alums, 12.1% were MSU Employees, and 22.7% were family of MSU students, alums, staff, or faculty.
• When asked if visitors learned anything from the experience, most (60.6%) indicated that they learned about animal behaviors and/or adaptations, the central themes of the gallery. 12.1% responded that they did not learn anything.
• When asked how the experience made visitors feel 50% indicated fun and/or excitement, 39.3% indicated surprise and/or wonder, and 4.5% indicated it encouraged slow looking of the mural. All of these align to the central goals of the project.
• When asked if anything surprised them about the experience, 51% indicated the augmented animals were surprising, 4.5% indicated augmented text was surprising, and 16.6% indicated aspects of the experience as a whole were surprising.
• When asked if there was anything else they wish they could do with the program, 21.2% indicated they wished for more interactivity with the program, 19.6% indicated they wanted the experience throughout the museum—an institutional goal of the project, 15.1% made recommendations about the animals in the program, 12% made recommendations about the overall experience, and 6% made recommendations about the hardware used.

Results of the Behavior Observation

• The gender of the participants appeared to be 55.6% female and 44.4% male. The approximate age of the participants was 63% young adult, 27.2% under 18, and 9.9% older adult (50+). The average duration during which the program was used at the mural was one minute 40 seconds; the median use was one minute 10 seconds, and the mode was three minutes.
• 60.5% of participants showed delight through facial or verbal expressions. 12.4% of participants asked additional questions of the facilitator. 3.8% of participants asked about using AR throughout the museum. 5.1% of participants asked about more functionality in the program (sound or movement).
• 50.7% of participants discussed the program with other people in their group. 43.2% of participants called others over to view something in the program. 4.9% of participants related aspects of the program to past experience or understanding.

Outcomes

The fact that 60.1% of respondents in the questionnaire indicated they learned something about animal adaptation and/or behavior is significant in that in previous evaluations for exhibitions at various museums, very few typically respond with answers relating to the central themes of exhibitions, despite, in some cases, themes written in multiple places in exhibitions.

Since 50% of respondents in the questionnaire indicated fun and/or excitement, and 39.3% indicated surprise and/or wonder, and 60.5% of those observed in behavior observation indicate delight through verbal or facial expression, this speaks to augmented reality as a program that has potential to increase visitor satisfaction with their visit.

Indeed, 19.6% of respondents to the questionnaire, and 3.8% of those observed indicated the desire to have the experience throughout the museum—an institutional goal—and that desire for more experiences like this speaks well for visitor reception to the future of the project.

Since 50.7% of participants in the behavior observation discussed the program with other people in their group, and 43.2% of participants called others over to view something in the program, this data shows that while the AR is on single devices for personal use, the experience is still a social experience. Dilenschneider (2012) says, “People don’t go to a museum to see the newest exhibit . . . people go to a museum to see the newest exhibit with people they care about.” It follows that social experiences will satisfy visitor expectations.

Future of the Project

At the outset of this project, the development team was trying to explore ways to make the program accessible. We had the idea of creating sound for the program late in the time that we could work with the student who was hired for working on the program, so we could not include it before testing. We also desired interactivity in the program but were constrained by the knowledge of the programmer, who did not have experience in creating the desired interactivity. Even before deploying the experience for testing, we knew that future iterations of the project should likely include sound or other immersive experiences, and possibly more interaction. We also know that should we take AR throughout the long-term galleries; goals for the projects will have to be redefined to address learning goals specific to each gallery.

While the user tests shows promise for developing the AR experience throughout remaining long-term galleries, and appears to demonstrate learning about exhibition goals, all indicating the museum should pursue AR further, the museum was at a crossroads: either find additional individuals with experience in developing AR around MSU who could develop this project further, or contract with third-party developers. Working with developers at MSU might take more time, and the expertise is uncertain, whereas third-party vendors might have the expertise and capacity for creating AR experiences quickly, but that could be cost prohibitive for a very resource-strapped organization.

More recent conversations between the MSU Resource Center for Persons with Disabilities (RCPD) and the museum during the user tests were eventually centered on the possibility of using AR as a constituent part of an overall accessibility plan that the museum had yet to create and implement. Michigan State University, broadly, is in the middle of a five-year accessibility plan, but that plan only applied to academic units, not areas like the museum. The museum is now working toward developing an accessibility plan and intends to create additional AR experiences throughout the museum as a means of making the museum more accessible, since AR has been shown to successfully address accessibility, including offering experiences for individuals with low- or no vision (Coughlin & Miele, 2017), or individuals with mobility challenges (Tecla Blog, 2018).

During these conversations, we found that RCPD has been working with a PhD student, who has experience developing AR with iBeacons; they have indicated that there may be interest in coordinating an AR program with other areas of the campus, including possibly the Eli and Edythe Broad Art Museum (the campus contemporary art museum), and other individuals within the College of Arts and Letters, who have been brainstorming about the use of AR and iBeacons throughout the campus. This partnership, and thoughts about exactly how the accessibility plan will dictate how we will create AR experiences throughout the museum to address accessibility issues, are emergent. But the opportunity to continue working with individuals across MSU’s campus shows great benefits: communication across a campus as large as MSU has been key to avoiding duplicative work, and the ability to pool our resources means working toward successful projects that we could not accomplish with our own limited resources.

References

J. M. Coughlan and J. Miele (2017). “AR4VI: AR as an Accessibility Tool for People with Visual Impairments.” 2017 IEEE International Symposium on Mixed and Augmented Reality (ISMAR-Adjunct), Nantes, 288-292.
Available at: http://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=8088507&isnumber=8088417

Diamond, J., Luke, J., & Uttal, D. (2009). Practical Evaluation Guide: Tools for Museums and Other Informal Educational Settings. AltaMira Press.

Dilenschneider, Colleen (2012). “According to Visitors, THIS is the Best Part About Going to a Museum (Hint: It’s Not The Exhibits).” Know Your Own Bone, Feb. 12, 2012. Consulted February 12th, 2019.
Available at: https://www.colleendilen.com/2012/02/12/according-to-visitors-this-is-the-best-part-about-going-to-a-museum-hint-its-not-the-exhibits/

Dilenschneider, Colleen (2017). “Are Mobile Apps Worth It For Cultural Organizations? (DATA).” Know Your Own Bone, Apr. 9, 2017. Consulted September 25th, 2018.
Available at: https://www.colleendilen.com/2017/04/05/are-mobile-apps-worth-it-for-cultural-organizations-data/

Evjen, M, Montiel deShuman, A., White-Owen, R., Winesmith, K., Gangsei, E., Dawson, B., Fernandez-Duque, S. (2017). “Best Practices for Creating Meaningful AR/VR Experiences in Museums.” Museum Computer Network, 2017, Pittsburgh, PA. Available at: https://www.youtube.com/watch?v=auj3a-V_O6Y

Green, Lindsey. (2016). “What we know about mobile experiences in Museums after 6 years of research.” Medium. Post Aug. 19, 2016. Consulted September 25, 2018. Available at: https://medium.com/frankly-green-webb/what-we-know-about-mobile-experiences-in-museums-after-6-years-of-research-42117def2c49

S. Manion. (2012). “Beyond Cool: Making Mobile Augmented Reality Work for Museum Education.” Museums and the Web 2012, N. Proctor & R. Cherry (eds). Silver Spring, MD. Consulted September 25, 2018. Available at: https://www.museumsandtheweb.com/mw2012/papers/beyond_cool_making_mobile_augmented_reality_wo.html

R. McKinley and A. Damala (2013). “ARtSENSE and Manifest.AR: Revisiting Museums in the Public Realm through Emerging Art Practices.” Museums and the Web 2013, N. Proctor & R. Cherry (eds). Silver Spring, MD. Consulted September 25, 2018. Available at: https://mw2013.museumsandtheweb.com/paper/artsense-and-manifest-ar-revisiting-museums-in-the-public-realm-through-emerging-art-practices/

A. Pardes. (2018). “For Museums, Augmented Reality Is The Next Frontier.” Wired. Available at: https://www.wired.com/story/museums-augmented-reality-next-frontier/amp?mbid=social_twitter_onsiteshare&__twitter_impression=true

Rothfarb., R. (2011). “Mixing Realities to Connect People, Places, and Exhibits Using Mobile Augmented-Reality Applications.” Museums and the Web 2011: Proceedings. J. Trant and D. Bearman (eds). Toronto: Archives & Museum Informatics. Consulted September 25, 2018. Available at: http://conference.archimuse.com/mw2011/papers/mixing_realities_connect_people_places_exhibits_using_mobile_augmented_reality

Schavemaker, M., et al. (2011). “Augmented Reality and the Museum Experience.” Museums and the Web 2011: Proceedings. J. Trant and D. Bearman (eds). Toronto: Archives & Museum Informatics. Consulted September 25, 2018. Available at: http://conference.archimuse.com/mw2011/papers/augmented_reality_museum_experience

Tecla Blog. (2018). “Augmented Reality and Accessibility.” Tecla. Available at: https://gettecla.com/blogs/news/augmented-reality-and-accessibility