Labs as an Agile Instrument to Supercharge the Development of a Tech Heavy Museum: Making informed choices by failing early and learning by doing

Maarten Brinkerink, Netherlands Institute for Sound and Vision, The Netherlands, Karen Drost, Netherlands Institute for Sound and Vision, The Netherlands

Abstract

Keywords: innovation, experiment, agile, lab, research, technology

Abstract

This paper describes how the Netherlands Institute for Sound and Vision in Hilversum uses labs as a flexible instrument to experiment with different solutions at an early stage in the realization of a new visitor experience. This fits within a broader development of the institute – and its museum – towards rapid prototyping, multidisciplinary teamwork, and involving our visitors in research and product development (Oomen, 2016).

Labs allow us to assess different technological solutions for the many challenges involved with our ambitious vision for the new museum. They provide us insight into the functional performance, user experience, and practicalities involved with implementing these technological solutions. These assessments are partly conducted ‘in the wild’ on the museum floor, with test subjects representing the new prospective museum visitors. In the coming years these labs will allow us to fail early, make informed choices, and build internal knowledge on key aspects of our new museum experience and the solutions required to realize our vision.

This paper describes the topics we identified for potential labs related to our new museum concept and the rationale behind them. It will also describe how we’ve structured the labs as an agile instrument to inform the development process and how they are implemented in the larger structure of the program that delivers the new museum. All of this will be illustrated with examples and lessons learned.

Introduction

The Experience at the Netherlands Institute for Sound and Vision in Hilversum celebrated its 12th birthday in December 2018. This personalized visitor experience – which is based on an RFID chipped ring handed out to all visitors – is filled with interactive exhibitions and still proves to be an example for a lot of similar institutions. However, we are now on the cusp of a large scale transformation into an institute for media. The institute wants to be more than a national audiovisual archive with playful access to it via our museum. In response to a ongoing change in the media landscape, the storyline for the new museum is based on the notion that everyone plays a significant role in the world of media: as a consumer/recipient, as a subject, and (increasingly) as a participant/maker.

The visit to the new museum – estimated to open in 2021 – starts by logging the visitor in with a face scan and creating a basic personal profile. During the visit, exhibits and interactive installations identify the visitor and adapt to their personal profile. This profile becomes more detailed through interaction with the exhibits and tweaking the results, resulting in a profile that uncovers the media personality of the visitor. At the end of the visit, you retrieve your media profile as a resumé of your activities in the museum and their significance to your role within in the world of media.

The realization of this personalized visitor journey is a collaboration between our museum specialists, our R&D department, our IT department, and external agencies. Given the need for implementing cutting edge technology – that often has never been applied to a museum context – we use labs as a flexible instrument to experiment with different solutions and implementations at an early stage.

Lab Topics for the Museum Renewal

The masterplan and initial design of our new museum encompasses many exciting and innovative concepts, which make use of cutting edge technology. This raises the question if these ideas deliver their promise in reality, and if the practicalities involved influence the feasibility of this vision.

Labs allow us to put these ideas into practice in (as) real (as possible) circumstances and with actual museum visitors. This allows us to test – and potentially fail – early, resulting in changing how we inform our museum development. This is what Hegley (2016) calls the “agile museum”.

In order to focus the efforts of our labs, we made the strategic decision to only run labs on topics that benefit the overall knowledge position of the institute (and leave other, more general, topics up to our suppliers). This resulted in an initial list of six lab topics related to multimedia exhibits and user interaction.

Conversational Agents

This topic deals with the possible deployment of embodied (robots, androids) and virtual agents (assistants, conversational bots) for communicating to and with the museum visitors. This can happen on different levels and touch points in the visitor journey:

• Welcome and goodbye
• Questionnaires and feedback
• Museum guides
• Narration
• Contextualization
• Q&A related to the media landscape
• Archive access

Interaction Models

Like many other experience and interaction-driven museums, our current exhibit is saturated with touch screen interfaces. In our new museum we aim to use more natural and seamless interaction models, so that the emphasis is more on the experience and content and less on the mediating screen, interface, and mechanics. To enable this we will investigate various modalities as input, like:

• Speech (see topic “Conversational Agents” above)
• Gestures
• Physical input
• Eye-tracking
• Brain waves
• Facial recognition
• Biometrics

Personal Devices

Given the subject matter of our new museum – media and its omnipresence in society – we’ve decided to embrace the devices visitors always carry with. These devices – starting with the smartphone – will be interwoven in the visitor journey and interaction with the museum exhibits. Apart from the undeniable role of the smartphone we are also looking ahead to the role other personal device will play in the media landscape in the near future. Think of devices like:

• Activity trackers
• Smart glasses
• Smart watches
• Head-up displays
• Clothing

User Tracking

When visitors enter the museum, we want them to feel like they’ve entered an environment that responds to their presence. This not only translates to the personalized experience and seamless interaction models, but also to how the museum responds to the entirety of visitors present at a given time. This requires different ways to track users throughout their visitor journey in the museum. Like user tracking through:

• Beacons
• Motion sensors
• Face recognition
• Signal tracing
• Emotion/sentiment analysis

Immersive Experiences

Creating a fully immersive experience is still considered to be the ultimate form of a mediated experience. Although VR, AR, MR and XR have seen a resurgence in the hype curve over the last five years, we still haven’t realized a Holodeck. As a museum about media, these experiments with virtual space can’t be ignored. But especially in the museum context this creates two main challenges we then need to face:

1. Combining the physical and virtual dimensions in one meaningful experience
2. Turning what is often an individual experience into a social activity

Responsive Design

In the new museum we want our exhibits to be suitable for a diverse range of visitors, including people of various ages, cultural backgrounds, and with different possible physical handicaps or mental challenges. We believe responsive design can help with making our exhibits more universally accessible, by striving to provide the best possible user experience for each possible user. We want to experiment with design features like:

• Multimodal interaction
• Multimodal information presentation
• Different levels of sensory stimulus
• Personalization of content

Museum Development Labs

In this section of this paper we want to illustrate our use of labs as an agile instrument to supercharge the development of a tech-heavy museum, by providing four concrete examples that illustrate different stages of museum development.

R&D Labs: Facial recognition for visitor identification

An obvious stage of museum development to introduce a lab is at the research and development stage of new museum concepts. Here the labs allows you to experiment with untested ideas or assumptions in a controlled environment. A concrete example of this type of lab is our exploration of facial recognition as a means to identify our visitors during their visitor journey.

Providing visitors with a personalized museum experience is a fundamental principle within the vision for our new museum. This ambition comes with the challenge of identifying the museum visitor throughout their journey. Given the advent of facial recognition in modern consumer electronics – but also given how it symbolizes the surveillance threat of AI – our art director imagined facial recognition as an elegant and metaphorically powerful solution for visitor identification.

Although conceptually sound, the deployment of facial recognition for visitor identification in our new museum left us with many questions:

1. Is the state-of-the art reliable enough?
2. How does the technology perform under the constraints of the museum environment?
3. What ethical and privacy considerations do we need to take into account? Do visitors accept facial recognition to be applied on them in a museum context?
4. What other possibilities – beyond identification – does facial recognition offer us?

Given the lack of reference implementations of this technology in the museum context – in the Netherlands the closest example we found was innovation center ICER – we decided to deploy a lab to explore these questions. To focus the lab, we formulated the following use case:

In the new museum we want to identify visitors through facial recognition, in order for us to know what specific visitors are interacting with the different exhibits throughout the museum, so we can provide them with personalized experiences.

To research and develop the practical possibilities to uniquely identify the museum visitors, we combined a quantitative evaluation of off-the-shelve facial recognition systems with a qualitative evaluation of a self-produced proof-of-concept. We didn’t only want to evaluate the performance of current technology, but we also wanted gain some hands-on experience with the implementation of facial recognition in an interactive application.

For our quantitative evaluation we selected four typical museum exhibit scenarios with varying conditions (light intensity and source, number of subject in view, and viewing angle of the camera). We then recorded scripted test videos with a group of twelve test subjects (colleagues and volunteers), who represented the biggest possible diversity in age, gender and ethnicity. We also recorded a scripted enrollment video for each test subject, that could be used to train the facial recognition systems. All this video data was then ran through the systems we evaluated (FaceReader and Microsoft Azure Face API).

Because the group of test subjects can’t be considered a representative sample of our museum population, our result were merely indicative, and not representative. But in our tests facial recognition (both systems) performed well in optimal circumstances. This means that in the first scenario with good (artificial) lighting conditions and the camera facing the front of the subject, all test subjects were recognized.

In less ideal circumstance the amount of detected faces and identified subjects dropped significantly. The third scenario, with no artificial light, in particular performed very poorly (less than 10%, compared to the optimal scenario). It is however important to note that in all scenarios (even the least ideal) the success rate of identification – detected faces that could also be identified – was very high (the lowest amount of successful identifications was 95% for one particular subject). This means that once a face is actually detected by the system, it could also successfully identify that face in most cases.

For our qualitative evaluation we produced a proof-of-concept of an interactive application. This application consisted of a mobile setup that could be placed in the offices of our institute and would greet colleagues with a random selected pre-scripted sentences, which – as a dynamic variable – included the name of the identified person. In order to realize this concept, we engineered an application that combined an off-the-shelve facial recognition SDK and accompanying hardware from Ensura and speech synthesis software from ReadSpeaker.

At an all staff meeting we asked our colleagues to voluntarily go through the enrollment procedure of the facial recognition software, which a third of them actually did (over 50 people). The following three days the proof-of-concept was placed at the main entrance of our office. During the first two days, we tweaked the configuration. We documented our experience with producing and configuring the setup, and also recorded the interactions with the system and our colleagues that resulted from their encounters with the proof-of-concept. After a week we circulated a survey among our colleagues to gather feedback on their experience with the proof-of-concept.

The qualitative approach provided us with a plethora of learnings. We learned that in most cases enrollment could be completed well within a minute, with just one or two attempts at following the standard procedure needed per individual. The system status messages that the facial recognition system provided were very helpful in ensuring a successful enrollment in most cases. We did however encounter two subjects that couldn’t be enrolled at all. This probably also had to do with our lack of experience with the system, but this would of course be unacceptable in a real museum context.

The proof-of-concept reinforced our findings from the quantitative analysis: It took us two days to get the calibration of the system and its surrounding good enough for it to recognize all subjects that interacted with the installation. Apart from the amount of artificial light (and diffusion of the light), we learned that guiding the gaze of the subject directly towards the lens was of utmost importance. In our proof-of-concept we solved this by placing a sign that said “smile at the little birdy” just above the camera. We also experienced that fast moving subjects were hard to track, which is something to consider when designing the interaction model for museums exhibits that rely on facial recognition for visitor identification.

After the trial with the proof-of-concept, over 40 colleagues that had used the office space during that week responded to our survey. More than half of them reported that the system had either identified them “sometimes” (27%), or “always” (32%). Of the remaining 41%, 12% reported they weren’t identified, because they didn’t go through the enrollment procedure (which makes sense of course). The remaining 29% still leaves quite some room for improvement. However, we also have to take into account that we were constantly optimizing the system while running the trial.

Most respondents (58%) reported that they enjoyed the experience of being recognized. Another 30% reported a “neutral” response. A minority of 12% found the experience to be (somewhat) “creepy”.

While running these quantitative and qualitative explorations of facial recognition as a means to identify museum visitors, we captured valuable lessons for the further development of our new museum concept:

• Take the substantial effort it takes to configure, calibrate and optimize the conditions for facial recognition technology to perform well in the museum environment into account.
• Consider the amount of available light and direct camera angles needed for facial recognition to be able to detect faces (and then identify them) in the overall spatial design of the museum environment. Especially enrollment of visitors during the start of their museum journey requires optimal (artificial) conditions.
• Although the reliability of the identification – especially in the quantitative evaluation – was high, we did encounter situations where subjects couldn’t be enrolled, were not detected or were even identified as another person (false positive). While facial recognition can provide a more seamless experience, at the moment it is not as reliable as a more traditional token (like an RFID chip or QR code). In most cases this makes a fallback to other identification methods a necessity.

Our lab report also made note of the ongoing social and political debate which surrounds facial recognition. This debate touches upon various (critical) reflections on the current limitations and threats of the technology. This ranges from privacy and civil liberty concerns (Lynch, 2018), lack of public regulation and industry best-practices (Smith, 2018), to issues with biases towards minorities (Moore, 2018).

Apart from the technical performance of facial recognition, we also need to relate to the social and political debates surrounding facial recognition, especially since developments in media technology and their impact on society are an integral part of our new museum storyline. Having said that, our own experience with these concerns within the museum context showed that these concerns weren’t a big issue for the parties involved. Most subject were willing to freely provide informed consent to their participation in the facial recognition experiments we did, the suppliers we interacted with have all have already implemented GDPR style privacy-by-default mechanisms, and performance across different types of subjects was very similar.

A final point for discussion the lab report touched upon was the fact that most of the current facial recognition solutions go (well) beyond our use case of identification. In their output they also provide additional analysis of the faces it processes. This includes determination of gender, age and the emotional state of a subject. This provides interesting possibilities to utilize this analysis as part of the interaction models for the museum exhibits. But it also asks for further investigation (in a lab?) into the reliability, utility, and possible privacy concerns these forms of analysis raise.

Prototype Labs: Cutting edge screen and projection technology

A second type of lab that we can distinguish as an instrument to aid our museum development are labs aimed at (rapid) prototyping the actual museum exhibits. This allows us to explore the technical and economical feasibility of our concepts and the underlying design principles in how we want to realize our new museum vision. This type of labs is becoming more relevant since we have recently started the preliminary design phase for our new museum.

The rough design of our new museum includes concepts for exhibits that often require solutions that go beyond the cutting edge of screen and projection technology. To determine the feasibility of these exhibits – before we include them in our preliminary and final designs – we need to prototype these solutions ourselves. This will help us to determine suitable hardware and technology for the solutions, while it allows us to gather experience with the realization of these types of exhibits and an opportunity to test with real museum visitors at an early stage.

The first prototype lab we are currently running focuses on one of our main design principles for the new museum: In the new museum we want to move away from the traditional rectangular two dimensional flat screen, and rather present media as layered projections on various surfaces that combine media content with contextual layers and user interfaces elements superimposed on each other in three dimensional space. To explore the possibilities we decided to prototype with various projection technologies and surfaces and see how far we could push them to produce transparent layers and depth in the resulting images.

During one week of prototyping we ran tests with various setups that combined various types of projectors and surfaces (screen, foil, gauze) and a transparent OLED screen. In doing so we found potential problems with all solutions. Traditional screens can only be used for the final layer of an image, since they don’t provide transparency. The foils we tested produced very little contrast in the projected images. The projection gauze offered relatively low resolution, making it only suitable for distant viewing. Finally the transparent OLED is susceptible to producing a moiré effect, depending on the viewing angle and its relative placement from the other projection surfaces. After having gained these initial insights and experience, we are now further investigating the affordability, availability and sustainability of these different solutions.

Project Labs: Experimental installations in the exhibition space

A more challenging form of labs in the museum context are experiments within the (permanent) exhibition space. In the new museum we have the ambition to showcase the cutting edge of media innovation to our visitors. With project labs we are already familiarizing yourselves with the implications of putting experimental installations on the museum floor and accepting the risk that installations will not function as expected, or don’t live up to their hype.

Given the challenges involved with curating cutting edge media technology in a fast moving environment, like the need for new partnerships (enterprises, start-ups, artists), and loss of control over the content of the exhibition, we decided to add dedicated a space within the current museum: The Project Space (an experimental space where your can touch the future of media). Exploiting this space lets us meet all of these challenges head on. At the same time it introduces our visitors to new developments in media technology, content, and channels and lets them reflect on the possible implications of these developments for themselves. Visitors – especially the so called millennials (Young, 2016) – are increasingly accustomed with being confronted with minimum viable products, especially in the digital domain.

In the Project Space we aim to present at least three but a maximum of five installations at a time. The format of these installations is free form, but they have to be working prototypes – so no mockups, pitches, or concepts – that represent the cutting edge of media innovation. Apart from that, these installations need to represent the (possible) future of media and offer the visitor some way to experience this for themselves through interaction with the installation.

An example of an experimental installation that featured in the Project Space was our collaboration with independent designer and researcher Anja Groten. She developed an art installation (Face the InterFace) about surveillance through (oh, irony) facial recognition. We scouted this installation as a possible addition for our project lab.

The installation explores what visitors actually know about the algorithms looking at them. What decisions do they make? And, more importantly, how do they influence our lives? It lets the visitor experience how our faces are being detected, recognized, flagged, filtered, swiped, targeted, and possibly even criminalized on a daily basis. The project does this by placing the visitor in front of a smart mirror that starts a scripted interactive dialogue with the visitor, based on their facial expressions, as captured by the smart mirror.

Since we started with the Project Space (in October 2018) we’ve learned a lot about the challenges involved with bringing cutting edge media innovation in the exhibition space:

• Transforming part of the exhibition space into a ‘permanent beta’ that explores the future of media is not something that can be designed or planned in advance: Start and learn as we go.
• Managing expectations about the focus of this exhibition space is challenging, given its experimental nature and the cutting edge subject matter: Keep redefining your goals and criteria as you learn more about what is a good fit for staff, partners and visitors.
• Our initial assumptions about the contribution that partners could deliver to the Project Space were too high. We underestimated the time and resources involved with bringing existing prototypes and projects into and preparing them for our museum context, even when partners wouldn’t charge any additional costs: Build long-term strategic partnerships with established innovators in the media space and scout for more mature media innovations that require less adaption to be suitable for a museum audience.
• Initially we wanted to change the content of the Project Space every quarter. However we’ve learned that the burden on our producer and multimedia department is too high to meet this goal.
• Given the experimental nature of the installations, it is essential to have museum guides present in the Project Space to assist visitors in the use of these (prototype) installations and guide them through the interactions they offer.
• Although we have experienced that the thematic focus of the Project Space – the future of media – is clear to the visitors, we expect to further improve the cohesion of the exhibition space by bringing curating installations around a more specific topic within media innovation.

Visitor Research Labs: Eye tracking and news reception

A final type of lab that is relevant to the development of our new museum is what we call an audience research lab. As mentioned in the introduction of this paper the role of the visitor within the media landscape is a central part of our new storyline. As a result we need ways to explore and investigate how our visitors navigate, experience, and cultivate this landscape. Using labs to research the media behavior of our visitors provides us with a tool to gather these insights, while at the same time giving visitors an active role in expanding our shared understanding of media. A telling examples of this type of lab is our collaboration with the JournalismLab at the HU University of Applied Sciences and the University of Amsterdam.

One of the latest media phenomena that has gotten attention in the popular media debate is the concept of ‘filter bubbles’. This phenomenon featured in our previous temporary exhibition “News or Nonsense” (Drost, 2017). Online news consumption has grown to 79% (Newman, 2018) and has lead to a diversification of the news landscape. News consumers are not tied to one media outlet (like their subscription news paper), but consume news via various channels and sources. Despite this diversification, the concern about filter bubbles is based on the assumption that the influence of news aggregators, search engines and algorithms counters this diversification of news by only presenting similar news to an individual, based on their personal profile. While the filter bubble theory remains contested in academic circles, the related idea of selective exposure – the theory that people prefer information that supports their existing beliefs and try to avoid information that contradicts these believes – has longer standing in the field of media studies.

To look at the impact of selective exposure on the diverse online news supply, JournalismLab approached our institute with the proposal to set up an eye-tracking experiment among our museum visitors to further investigate this. The experiment looked at the role of news sources in the selection of news by consumers: Do people prefer sources they know and trust? Do they select news that fits with their existing beliefs, despite the source?

During this two-day eye-tracking experiment, the subjects (N=42) were presented with a prototype of an online newsstand with twelve news items, varying in source, subject, image and number of likes. Subjects were asked to review the items available in the newsstand and select one item. Afterwards subjects were asked to indicated if this item included a source, and if so, which one?

Although analysis of the results is still ongoing (and will be published in an official academic paper later this year) we’ve learned that the visitors paid most attention to the news headings and images, and less attention to the source and number of likes attached to a specific news item. This seems to confirm the selective exposure theory for online news selection, since the content of an article appears more important than its source.

This is just a first example how collaborating in labs with research partners helps us enrich the storyline of our new museum with scientific insights about media reception (in this case) and how our visitors can become active subjects in uncovering these insights. The visitors generally enjoyed the experience of the novelty of an eye-tracking experiment and were happy to contribute to more insight in how we consume media.

Conclusions

This paper described how the Netherlands Institute for Sound and Vision uses labs as a flexible instrument to experiment with different solutions at an early stage in the realization of a new visitor experience.

For certain research topics in particular actively experimenting with these solutions contributes to the overall knowledge position of the institute:

1. Conversational Agents
2. Interaction Models
3. Personal Devices
4. User Tracking
5. Immersive Experiences
6. Responsive Design

The paper shows how labs can be valuable at different stages of museum development;

1. experimenting with untested ideas or assumptions in a controlled environment (R&D Labs),
2. exploring the technical and economical feasibility of our concepts and the underlying design principles through rapid prototyping (Prototype Labs),
3. experimenting within the (permanent) exhibition space to showcase the cutting edge of media innovation and letting our visitors experience these experiments for themselves (Project Labs),
4. and researching the media behavior of our visitors to learn more about their interactions and experiences with the media landscape (Research Labs).

Given the flexible nature of the lab instruments, it has proven challenging to get all colleagues on the same page about these instruments and their goals. At the same time running the actual labs has provided the institution with great opportunities to work on specific challenges in multidisciplinary team (a lager ambition of the institute). Our first experiences with involving our museum visitors in these various experiments is really promising. It adds to their experience at the museum, either through the surprise of new, uncommon interactions, and/or feeling they contribute something meaningful. Finally it helps position our institute and future museum as an active player in the ever-evolving media landscape.

Acknowledgements

We would like to thank Roeland Ordelman, Jesse de Vos, Joyce van Telgen, Kelly Mostert, Johannes Wassenaar, Brigitte Jansen, Frederike Huijgen, Mariska Brasser, Jeroen Korsse and his technical team for their awesome work within the lab teams.

We would also like to thank Johan Oomen, Josefien Schuurman, Ineke Middag, Martijn Laar and Corine Lindenbergh for their managerial support for the labs as an organizational instrument.

For this paper we received input from our colleague Koen van de Brug (on the Prototype Lab) and Gregory Markus on our brutalist use of the English language.

Lastly we would like to to thank our many international colleagues for inspiring us to constantly explore the limits of museum exhibitions.

References

• Oomen, J. (2016). “Changing gears: Fast-lane design for accelerated innovation in memory organisations.”. MW2016: Museums and the Web 2016. Published January 19, 2016. Consulted March 22, 2019. https://mw2016.museumsandtheweb.com/paper/changing-gears-fast-lane-design-for-accelerated-innovation-in-memory-organisations/
• Hegley, D. (2016). “The Agile museum.” MW2016: Museums and the Web 2016. Published January 15, 2016. Consulted March 22, 2019. https://mw2016.museumsandtheweb.com/paper/the-agile-museum/
• Lynch, J. (2018). “Face Off: Law Enforcement Use of Face Recognition Technology”. Published February 12, 2018. Consulted March 22, 2019. https://www.eff.org/nl/wp/law-enforcement-use-face-recognition
• Smith, B. (2018). “Facial recognition technology: The need for public regulation and corporate responsibility”. Published July 13, 2018. Consulted March 22, 2019. https://blogs.microsoft.com/on-the-issues/2018/07/13/facial-recognition-technology-the-need-for-public-regulation-and-corporate-responsibility/
• Moore, S. (2018). “AI Frontiers: Kairos Untangles Face Recognition Bias”. Published October 26, 2018. Consulted March 22, 2019. https://www.kairos.com/blog/ai-frontiers-kairos-untangles-face-recognition-bias
• Young, K. (2016). “What Are Museums Doing to Engage Millennials?” The Iris. Published November 16, 2016. Consulted March 22, 2019. http://blogs.getty.edu/iris/what-are-museums-doing-to-engage-millennials/
• Drost, K. (2017). “News or Nonsense: an exhibition about the working of news”. Published November 3, 2017. Consulted March 22, 2019. https://www.museumscomputergroup.org.uk/events/museumstech-2017/
• Newman, N. (2018). “Reuters Institute Digital News Report 2018”. Published 2018. Consulted March 22, 2019. http://media.digitalnewsreport.org/wp-content/uploads/2018/06/digital-news-report-2018.pdf?x89475