Everything Old is New Again
Wes Lindamood, United States Holocaust Memorial Museum, USA, Michael Haley Goldman, United States Holocaust Memorial Museum, USA
AbstractUsing the recently redesigned Holocaust Encyclopedia (HE) as a primary case study, with supporting examples from the United States Holocaust Memorial Museum’s work on USHMM.org and Experiencing History, this paper will explore how we are taking an audience-centered approach to content strategy for our digital products. As part of our exploration, we will examine the challenges that digital teams face when working with limited resources to support ever-evolving user needs for new and existing products.
Keywords: Content Strategy, Continuous Improvement, User-Centered Design, Web Redesign
The Holocaust Encyclopedia is an authoritative reference tool created by the United States Holocaust Memorial Museum to share the history and contributing factors of the Holocaust with students, educators, and adults. Dating back to 1993 as an interactive computer terminal that visitors could only access at the museum, the encyclopedia now accounts for up to 80% of the Museum’s web traffic.
Figures 1 and 2: Visitors using computer terminals in the Wexner Learning Center at the U.S. Holocaust Memorial Museum circa 1993 (left). Wexner Learning Center computer monitor displaying an oral history interview with survivor Michael (Miso) Vogel circa 1993.
From the first migration of the encyclopedia from a computer terminal to a public website in 2000 to the most recent redesign in 2018, the Holocaust Encyclopedia has gone through multiple iterations over the years. As the museum’s oldest digital educational resource, the encyclopedia provides a compelling look at the museum’s evolving approach to content strategy on the Web.
To begin our paper, we will look at lessons we have learned from the Holocaust Encyclopedia’s content strategy initiatives and how those lessons are informing our thinking about future work. Following our exploration of the Holocaust Encyclopedia, we will examine related content strategy efforts for museum’s main website, (USHMM.org), and Experiencing History, a digital teaching and learning platform that instructors and students use to engage directly with primary sources related to the Holocaust.
In all of these efforts, we strive to follow content strategy best practices to ensure that we have a solid foundation to build upon. We’ll dive in to some of the audience-centered best practices, such as comprehension testing of our content using remote usability testing tools.
Core to taking an audience-centered approach to content strategy is the notion that digital product owners are in constant conversation with their users, and through a process of continuous improvement, are adjusting to ever-evolving user needs and changes in technology. Practicing continuous improvement while juggling limited resources and competing priorities is easier said than done; however, so alongside our look at best practices, we’ll examine some of the challenges we face supporting this approach. We are not alone in our goal to continuously improve our digital products, so we will conclude by looking at examples from other museums and beyond that can inform our thinking about content strategy and continuous improvement.
Lessons from the Holocaust Encyclopedia’s Audience-centered Content Strategy Initiatives
Figures 3 and 4: Holocaust Encyclopedia circa 2015 (left) and 2018 (right)
Audience needs have been a consideration in each version of the Holocaust Encyclopedia. For example, translations of the content began in 2002, aiming to make reliable information about the history of the Holocaust available to broader audiences. The introduction of “Further Reading” suggestions from USHMM library staff in 2007 was an attempt to provide information for educators. Articles on specific topics were added on an ad hoc basis in response to requests from library staff receiving questions from the public. However, these were essentially “add-ons” to the original design, and it was not until the most recent redesign that audience needs served as the primary driver.
The recently redesigned Holocaust Encyclopedia represents a complete overhaul of the site’s content. While the accuracy of the historical facts presented in encyclopedia remains paramount, meeting audience needs became the primary goal. Recognizing students, educators, and the general public as the primary audiences for the Encyclopedia, the redesign addressed the visual design and the content together. For the content, this meant:
- Setting the context so readers can immediately grasp the significance of a topic
- Adjusting the reading level to become more engaging and understandable to student audiences, without alienating other learners. (We learned about specific words that students struggled with.)
- Providing other tools to facilitate student exploration, such as tagging
- Ensuring that there are no dead-ends in the content
- Using media to tell a narrative, moving away from the text-based highly linear approach of the original encyclopedia
- Seeking to identify and dispel common misconceptions
- Providing tools for critical thinking
The audience-centered approach continues now that the site is live. For instance, the most requested feature since the beta period has been to include a published date on each page, which we are in the process of implementing.
What has remained constant is the mission to provide accurate information. The ways to make it usable and compelling have evolved. This overhaul of content was a major undertaking that appears to have paid off, with indicators such as time on page for encyclopedia articles on the rise.
A Look at Content Strategy Best Practices Across the Holocaust Museum
Figures 5 and 6: USHMM.org circa 2000 (left) and 2018 (right)
The Holocaust Encyclopedia is not the only product at the Holocaust Museum that has benefited from audience-centered content strategy. Like the encyclopedia, the museum’s main website USHMM.org, is a good example of a product that evolved and changed several times over the years in response to user needs and organizational goals.
A newer digital product, Experiencing History, which is used by teachers and students to explore primary sources related to the Holocaust is another good example. This relatively recent digital product has incorporated audience-centered content strategy best practices into its work from the very beginning. Starting with a pilot version featuring two collections, the Experiencing History team worked closely with a target audience of teachers to shape the product’s content and features for an initial launch. Through workshops with product users, and usability testing during each iteration of the product’s development, the team then refined their understanding of the content needs of the audience. The work of conducting user research continues today with regular comprehension testing of the content of new collections that get added to the product.
While the product goals and user needs of Experiencing History, USHMM.org, and the Holocaust Encyclopedia are varied, they share a set of audience-centered best practices that inform our approach to content strategy at the museum. Starting with the identification of a product owner who is accountable for aligning the product’s goals with audience needs on an ongoing basis, the following best practices highlight major activities that help us position our content to support and respond to ever evolving user needs.
Figure 7: Screenshots of Experiencing History, (https://perspectives.ushmm.org/) a teaching and learning tool that instructors and students use to engage directly with primary sources related to the Holocaust.
Identify a Product Owner Who is Accountable for Aligning the Product’s Goals with Audience Needs on an Ongoing Basis
The first step in establishing an audience-centered content strategy is identifying a product owner who will be accountable for aligning the product’s goals with audience needs on an ongoing basis. While this may seem obvious, product owners in our museum have multiple responsibilities, and dedicating sufficient time to research, planning, and production on a product outside the context of a launch or redesign can be a challenge. In addition, because ownership of content is often shared by multiple groups, designating a single owner who is empowered to represent the views of multiple stakeholders is an emergent practice at the museum.
For the Holocaust Encyclopedia and Experiencing History, we have dedicated product owners who manage the product strategy and content road map for their products. On the main website, ownership of the content is shared, with editing and publication of the content being guided and implemented by a small digital content team.
Understand Audience Needs Through Both Qualitative and Quantitative Analysis
Product owners at the museum are often content experts who are new to product strategy and user-needs analysis. While support for defining product strategy comes from museum leadership, product owners are responsible for planning user research and understanding the product’s content from the perspective of the end user. To do that, product owners at the museum work with the user-experience team at the museum to conduct qualitative and quantitative research.
Qualitative research like usability testing helps us to evaluate proposed designs and conduct targeted research based on patterns we’re seeing in our site analytics. In the early stages of the Holocaust Encyclopedia redesign, we suspected that users would respond favorably to overviews and key fact sections on article pages. When a series of in-person usability tests during the prototype stage of the project showed promise, the team proceeded with reformatting key articles for the beta launch of the product and turned to quantitative analysis to track the performance of the rewrites. So far, the quantitative analysis has validated the approach.
For quantitative research, product owners at the Holocaust Museum rely on Google Analytics and site surveys. For example, at the start of the Holocaust Encyclopedia redesign, the team looked at top performing pages (based on page views and time on page) in Google Analytics to help set priorities for rewrites and reformatting of articles. To learn more about user needs for the content, we will often turn to site surveys. On USHMM.org, we have used site surveys to learn more about why users visit different sections of the site and if they were able to find what they were looking for. We can then follow up with more targeted tests like card sorting, in which users are asked to group, rank, and label content in a way that makes sense to them. Click tests are another way we learn about users’ expectations for navigating USHMM.org. In a click test, users are asked how to find something specific on our site, and then prompted to click on the right place on a screenshot or series of screenshots to find the information. Applying what we learned from these tests has led to improvements in content labeling and categorization.
Figure 8: These click test maps show the navigation labels that usability test participants selected when asked, “Where would you go to find out how much it costs to visit the museum?” We initially referred to timed entry passes required to visit the museum as “passes,” but we learned through testing that the word “ticket” was more meaningful to our audience.
Another survey method we regularly use to inform our content strategy is comprehension testing. When Experiencing History introduces new content, it is vetted by historians who are experts in the subject, teachers who will be using the content in classrooms as part of their curriculum, and students recruited through remote usability testing services. The recruited students are presented with surveys that help the product team better understand if the writing style is engaging and understandable.
Audit Content Through the Lens of Audience Needs
The insights gleaned from qualitative and quantitative research are invaluable as we begin the process of auditing our existing content to make decisions about updates and additions.
The first step in auditing our content is to create an inventory of all the existing and proposed content we plan to include in a product. Our inventories typically take the form of spreadsheets that document key metadata about each content item, such as the type of media, location of the content, the owner of the content, and engagement analytics that provide us with insight into the level of interest users have in the content.
After an inventory has been created, the next step is to analyze the content to assess how well it supports the product’s goals and audience needs. By looking at our content holistically, key patterns emerge that help us:
- Assess the completeness of the content
- Identify and remove redundant and outdated content
- Establish guidelines for plain language that encourage clear and effective communication
- Structure new and existing content in a consistent way to improve both the findability and usability of the content
As a snapshot in time, an audit helps establish a baseline understanding of a product’s content in the current moment. During a new product launch or a redesign, an audit helps pave the way for the establishment of content guidelines that inform the creation of new content.
Define Audience-centered Guiding Principles
With an audit documenting where we are, we have a have a much better sense of where we need to go. The next step in the content strategy process is to define guiding principles that will inform the creation of content going forward. In our products, guiding principles have taken the form of editorial style guides and vision documents.
In the vision document for the redesigned Holocaust Encyclopedia, we identified “connective and engaging” as pillars of our approach to guide the creation of new content. In the old version of the Holocaust Encyclopedia, we discovered dead ends in our articles and found that users were not exploring related and relevant content. In the new version of the encyclopedia, we prioritized creating more connections between content and surfacing related content in contextually relevant ways. To make the content more engaging, we focused on the writing level and word choice in our articles, so that they were written in an age-appropriate way that would not be intimidating to students. We also focused on incorporating more personal stories and introducing more hierarchy into the structure of our articles (e.g., by including subheads and lists) to make the articles easier to scan.
In the content guidelines for USHMM.org, we established a similar set of recommendations. Two key guiding principles for the site include engaging young learners and communicating visually. Young learners are a key audience for us, and different audiences have different needs. By focusing our writing for young learners, we are encouraged to use non-academic, everyday language that will be understandable and useful to a wide range of site visitors. By prioritizing the need to communicate visually, we are supporting multi-modal learning, improving the scanability of the site and taking advantage of evocative content from our collections that creates emotional resonance.
Set Up a Process for Continuous Improvement
The creation of editorial calendars, content life cycles, and content governance marks the transition point between content strategy work for a launch and the ongoing content strategy work required to help a product succeed.
On USHMM.org, we use an editorial calendar to provide direction on the prioritization of new content and to select relevant content to be featured throughout the year. Tied to key dates, anniversaries, and news events, the calendar provides a structure for organizing work on the site and related products.
The Holocaust Encyclopedia refers to the same editorial calendar as USHMM.org, so that updates to content and approach are coordinated across products. In addition to staying in sync with USHMM.org, the other major drivers of new content are new artifacts that are added to the museum’s collections and new scholarship about the history of the Holocaust. As relevant items are added to the collection, Holocaust Encyclopedia articles are updated to make them more comprehensive.
While key facts in our articles do not change, scholarship and our approach to the study of the Holocaust have evolved since USHMM.org and the Holocaust Encyclopedia were created—and continue to evolve. For example, many people are aware that Nazi Germany established a camp system, and within that system were camps dedicated to mass killings and that one of those camps was Auschwitz. Less well-known is the part of the Holocaust now known as “Holocaust by bullets”—the mass shootings of Jews and Roma in eastern Europe. Many of these graves were undiscovered until recent years. As scholars and investigators uncovered more about this largely unknown chapter of the Holocaust, our texts were updated accordingly. Each update is carefully vetted by historians and stakeholders at the museum, which is a time-consuming process, so planning update work well in advance with the help of an editorial calendar is very important.
A notable example of a workflow designed to support continuous improvement emerged during the last major redesign of USHMM.org. As part of the redesign process, the team established a governance model with the full life cycle of the product’s content in mind. Devised as a distributed publishing model, the workflow called for a clear hierarchy of content revisions on the site that distinguished minor changes that could be made at the departmental level from major changes that would need to be reviewed and edited by museum leadership. The workflow was designed to increase our publishing efficiency by making it easier for individual content creators to produce publication-ready content without sacrificing consistency or the high-quality standards we maintain for all content.
A core component of this workflow was the creation of a “Digital Content Panel” that could facilitate cross-departmental content development by identifying content priorities across the site.
Responsibilities of the Content Panel Included:
- Creating an umbrella editorial calendar for the museum as a whole that was informed by departmental editorial calendars
- Developing a distributed review process to ease publishing bottlenecks and improve quality by ensuring more than one person is looking at every aspect of content quality and appropriateness
- Maintaining and updating a content style guide to stay in sync with changes to the site
- Establishing a content life cycle by regularly auditing the site’s content to be kept up-to-date and aligned with the museum’s larger strategies
These combined efforts were designed to help fight the forces of entropy that make large sites increasingly unmanageable over time. In practice, the Digital Content Panel did not take shape as a centralized governance group. Instead, the responsibilities of the group were absorbed by a variety of sub-groups tasked with taking on different parts of the Digital Content Panel mission. Cross-departmental content development is coordinated at the producer- and content-contributor level, but governance of the overarching content strategy for the site has proven difficult to establish. Through staffing changes and the growth of the museum’s digital product portfolio, the focus of key staff responsible for maintaining an overarching content strategy was broadened and redirected. An important takeaway from this work is an improved understanding of the complexity and time involved in supporting and sustaining an evolving content strategy. While content (the text, images, video, etc., . . . ) is the tangible expression of a content strategy effort, the strategy (the people and the process) is where the complexity lives.
The End of a Redesign is Just the Beginning
We know from experience that creating a content strategy through the launch or redesign of a product is just the beginning of work for content teams and that the challenges we face are not unique. In a cautionary tale from Museums and the Web 2010, “Tales of the Unexpected: A Pragmatic and Candid View of Life Post-Launch” (Burnette, 2010), the authors acknowledged the lack of regard for the day-to-day realities of the post-launch period and note:
It’s easy for your colleagues to see your updated visual design, but much harder for them to ‘see’ and appreciate the layers of expanded content, new information architecture, and behind-the-scenes code, metadata, and tagging that is consuming your Web team. This lack of understanding of the complexities of on-line publications will inevitably lead to moments of frustration when trying to balance increased expectations against what’s actually possible.
These words still hold true in 2019.
As we have learned from our work on several products, taking an audience-centered approach to content strategy sets us up for success, but the lack of a governance structure to maintain it will result in a decreased product benefit to users over time. Change is inevitable, and there are no guarantees that we will ever have a stable governance structure to support the content strategies of our products, but there are steps we can take to make our content strategy work more resilient and responsive to audience needs over the long term.
In “A Four-step Road Map for Good Content Governance,” Joseph Phillips of GatherContent identifies clearly defined documentation, outlining strategic authority for the content and implementation accountability, as key to the process (Phillips, 2015). Of course, a content strategy workflow is only going to be as effective as the people supporting it, so hiring and training must be included in the content strategy plan. There is no short cut here. As we noted in this paper, content may be the output of content strategy, but it’s the people who make the difference. Governance can feel overwhelming, so Tenessa Gemelke, Director of Marketing & Events at the content strategy firm Brain Traffic, reminds us that “small steps can make a big difference” when putting a content strategy plan in place (Gemelke, 2018).
Some of the small steps we are taking at the Holocaust Museum to improve our content strategy on an ongoing basis include embracing small audience-centered experiments, sharing more information about user research activities across product teams, and improving documentation of what teams are learning from their experiments.
The introduction of small audience-centered experiments and iterative learning has influenced our work and how we think about content strategy in a big way. From regular comprehension testing of new primary source material that gets added to Experiencing History to the introduction of user stories into the rewrite process of Holocaust Encyclopedia articles, we are transforming how we work as we go.
User stories, which are typically thought of as a way of framing a software development task in a user-centered way, are helping our content producers frame writing assignments to focus on outcomes for our audience that are targeted to product goals. For product owners, the concreteness of this approach helps to document what we have learned about our approach in a measurable way.
As we mature in the process of regularly gathering user feedback, we are also working to improve the process of documenting and sharing what we have learned across the organization. We have multiple initiatives with overlapping audiences, so when thinking about information sharing across products we want to do a better job of combining related audience research activities into one cohesive effort. Improved documentation of what we have learned from our user research will make it easier to share across groups, discover patterns that apply broadly to all or products, and learn more about how we can better serve audiences in contextually relevant ways.
While clear governance rules for the content strategy of our products will remain a work in progress, we believe these small steps are helping to fulfill the goal of continuous improvement.
Burnette, A. et al., “Tales of the Unexpected: A Pragmatic and Candid View of Life Post-Launch.” In J. Trant and D. Bearman (eds). Museums and the Web 2010: Proceedings. Toronto: Archives & Museum Informatics. Published March 31, 2010. Consulted February 4, 2019. Available at: http://www.archimuse.com/mw2010/papers/burnette/burnette.html
Gemelke, T. (2018). “How Nonprofits Can Profit From Content Strategy.” Brain Traffic. Consulted February 4, 2019. Available at: https://www.braintraffic.com/blog/how-nonprofits-can-profit-from-content-strategy
Phillips, J. (2015). “A four step road map for good content governance.” Gather Content. Consulted February 4, 2019. Available at: https://gathercontent.com/blog/a-four-step-road-map-for-good-content-governance
Lindamood, Wes and Haley Goldman, Michael. "Everything Old is New Again." MW19: MW 2019. Published January 16, 2019. Consulted .