Catching Carbon in Augmented Reality

Toward the end of the day, a 9-year-old girl sitting near the back of the room raised her hand. 

“This might be a stupid question,” she said, “but what’s carbon?”

The adults in the room — a team of climatologists, data scientists, educators, museum professionals, and digital media producers — looked at each other. After spending months developing an augmented reality (AR) experience for museum visitors, they had gathered the group of young beta testers at the American Museum of Natural History (AMNH) to try out the app in spring 2024.

“We all took a step back,” said Christina Torres, a PhD student in science education at Teachers College. “We’d been focusing on specific interactions happening within the museum’s dioramas, but we hadn’t thought to define carbon and explain why it’s important to climate change.”

While the designers hadn’t expected that particular question, it was exactly the kind of feedback they were looking for. After decades of struggling to communicate about climate science effectively, experts know that listening — and responding to what people say — is a crucial part of creating effective communication. 

“Ultimately, there's only so much that we can do with our adult hats on. We had to give the mic over to the kids,” Torres said.

After many rounds of development and feedback, the team produced Carbon Catchers, an augmented reality (AR) experience that guides museum guests on a scavenger hunt through the Milstein Hall of Ocean Life at AMNH. Once a guest navigates to the experience, they can use it to encounter games and experiences that build on the museum’s famous exhibits.

In one round of the game, users dive into the carbon cycle by helping a sea otter eat its fill of sea urchins. In another, they help a manatee graze on sea grasses. The app was piloted at the museum’s EarthFest on Earth Day 2024.

Bringing cutting-edge climate science to AMNH visitors

Carbon Catchers is the first product in a standing collaboration between AMNH and LEAP, a climate data science research center funded by the National Science Foundation and hosted by Columbia Engineering. An NSF Science and Technology Center, LEAP has the dual mandate of advancing research into climate data science and connecting that research with the public. 

Tian Zheng, the center’s deputy director, chief convergence officer, and education director, said the museum is an ideal partner for advancing LEAP’s goal of creating convergence between research and education.

“As a museum with a wide range of exhibits about the natural world, AMNH is well positioned to integrate conversations about climate change directly into their exhibits,” she said. As part of its efforts to bridge climate science and machine learning, LEAP works closely with educational institutions and community groups to open dialogues about the climate. The center also collaborates with Teachers College to run a summer institute that convenes faculty, doctoral students, expert teachers, and leaders from the NYC Public Schools Office of Sustainability to continue integrating climate change into the school system’s curriculum. 

“For us, communication is not a one-directional process where we say, ‘we’re doing exciting research, so let’s tell the whole world about it,’” she said. “It’s really bidirectional, so that we’re finding opportunities for education and outreach where we’re able to listen and hear how people think about this research and its impact.”

Charles Lang, senior executive director at the Digital Futures Initiative at Teachers College, said the collaboration offers students a chance to work on technologies that promote the public good. 

“Teachers College trains many students to work in the technology sector and while they are here we want to give them experiences working on projects that clearly benefit the community — AMNH has been a great partner in this endeavor,” he said.

A new tool for sharing climate data science

The project started in late 2023 with a group of researchers and designers from across Columbia and from teams at the AMNH who were enthusiastic about finding new ways to start conversations about Earth’s climate and how it’s changing. 

For more than six months, representatives from LEAP, AMNH, the Brown Institute for Media Innovation at the Columbia Journalism School, and the Digital Futures Institute (DFI) Scholars Program and Communication, Media, and Learning Technologies Design (CMLTD) Program at Teachers College had been developing an innovative concept for interpreting data about the climate for museum visitors. 

They began by “thinking deeply about what topics would be appropriate to communicate to the public, including families with children,” said Torres, who is also a PhD fellow at LEAP. After considering ideas such as iterating on the museum’s existing climate visuals or finding ways to illustrate predictions about the future climate, the team came across a new paper that reviewed the role of organisms and food chains in the carbon cycle.

“We had come to the collective sense that we wanted to do something that felt positive,” said Eric Chen, director of creative research at the Brown Institute. “We were motivated to give people a sense of agency.” 

“At AMNH, we've spent years developing interactive prototypes and digital products to find creative ways of enhancing our static and permanent exhibits,” said Eozin Che, the museum’s lead creative technologist. “By using emerging technologies like AR, we bring in new layers of information and multimedia, making scientific concepts and discoveries more engaging for our visitors.”

After several rounds of development and testing, the researchers invited a dozen kids and their parents to the museum one afternoon to test out a beta version of the experience. 

“We collected specific feedback from the kids and adults, and we observed how the app seemed to affect the broader interactions people were having with each other and with the exhibits,” Chen said. “It pushed us to think more deeply about how effectively the experience itself was conveying the messages we were trying to get across.” 

That feedback — including the girl’s question — led the developers to restructure the experience before its debut at AMNH’s EarthFest on Earth Day 2024.

By focusing on the basic principle that animals capture and store carbon by simply eating food, the team was able to tell a different kind of story about the climate. By using AR and games, they put the audience in a position to take part in a planet-scale process that so often seems overwhelmingly abstract. 

“Emerging technologies, such as AR, enable visitors to explore the physical museum while interacting with digital content, making even the oldest permanent halls feel modern,” Che said. “AR has great potential to enhance the educational experience by creating compelling narratives that make scientific concepts more accessible and engaging.”