How Engineering Researchers are Reimagining Roadways and Sidewalks

At the Center for Smart Streetscapes (CS3), dozens of researchers collaborate with more than 80 non-academic community members, such as industry partners, community organizations, municipalities, and K-12 schools. These collaborations are designed to produce new technologies and to build trust in those technologies through accountability and transparency.

“One of the strengths of a center like CS3 is that it keeps researchers thinking about the real-world problems that industry and communities are trying to solve,” said Brian Smith, an assistant professor of computer science at Columbia Engineering.

For Carl Vondrick, YM Associate Professor of Computer Science at Columbia Engineering, a good starting point for collaboration happens when a potential partner identifies a practical challenge they can’t solve alone.

“A lot of interesting questions in computer science and machine learning have come from the issues and challenges that emerge in practice when these systems are deployed,” he said.

Sharon Di, associate professor of civil engineering and engineering mechanics, added that partnering with the public sector can result in tangible benefits for the public.

“Government agencies have a lot of data that we don’t have,” she said. “If we can help with the analysis and do modeling and simulations, hopefully we can find solutions to problems that departments of transportation and local communities are facing.”

These three CS3 members described their ongoing research projects at the center’s 2024 Innovation Summit.

Safer, smoother traffic

Smart streetscape technology enables traffic managers to better manage traffic — and to give pedestrians an extra measure of safety.

“As a transportation engineer, I am interested in how we can predict estimated traffic conditions by creating a digital twin of the streetscape,” Di said, referring to a digital representation that’s constantly updated with data from the real world. “To do this, we leverage the COSMOS testbed to collect and process real-time data from the physical world,” Di said. 

Located beside Columbia’s campus, the COSMOS testbed offers researchers a chance to experiment with next-generation sensors, wireless communication infrastructure, and edge-cloud computing in a functioning intersection. 

In this work, Di uses some of those systems to rapidly collect, process, and transmit data about the position and movement of vehicles, cyclists, pedestrians, and other objects in the roadway. Certain cameras and edge-cloud computing capabilities enable “object detection and tracking, as well as trajectory prediction” within the digital twin of that intersection. 

Crucially, the system can quickly process and send information via industry-standard protocols. 

“What’s more important than the capabilities of the digital twin itself is what we can do with these messages,” Di said. 

In one project, Di mentored a high school student who developed a phone app that used these messages to warn a pedestrian of an impending crash. 

“With a birds eye view camera, we were able to identify objects, make trajectory predictions, and issue warnings about potential risk,” she said. 

In another project, Di designed a method for smoothing traffic flow by intelligently coordinating traffic lights along the Amsterdam Avenue corridor in upper Manhattan.

“We developed a federated learning technique to protect privacy and reduce communication costs,” she said. In simulations, the method reduced traffic delay for vehicles moving through the testbed by 13 percent.

Using AI to see the unseen

AI systems have a serious shortcoming when it comes to analyzing images and video: if an object isn’t visible, even a state-of-the-art system will struggle to use context clues to compute that it is there. For example, in the many systems that can identify and label a pedestrian, a pedestrian will inconspicuously “disappear” any time a vehicle obscures the camera’s view. Researchers call this problem “occlusion.”

“It’s a very challenging problem in computer vision,” said Carl Vondrick, associate professor of computer science at Columbia. “If you’re on the streetscape in New York City, there’s a lot of occlusion. All you see are partial bits of each object.”

Vondrick is developing approaches to give AI systems the kind of intuitive sense of physics that tells a person — even a two-year-old child — that an object hasn’t disappeared just because it isn’t currently visible. 

“You need to build visual systems that can reconstruct our environment in 3D,” he said. 

While vast quantities of 2D images have enabled progress in machine learning and computer vision systems, there is very little data that contains the information necessary to reconstruct 3D reality. 

“We've been developing approaches to build 3D representations from 2D data,” he said. “We’re building systems that can make sense of many types of data, ranging from images taken on the ground to surveillance cameras mounted around the city. It’s an opportunity to combine data from drones and satellites to build a global representation as to what's happening on a street or inside of a city.”

Improving the sidewalk experience

There is more to streetscape technology than tracking pedestrians and vehicles. In his work, Brian Smith advances accessibility and community through digital media technologies that often incorporate augmented reality (AR) experiences in unexpected ways.

“I'm really fascinated by the fact that with smart city technology, we have all of these intelligent computing possibilities, sensors and cameras and things like that,” Smith said. “But for them to really have an impact on daily life, people need to have access to their capabilities.”

Smith shared two projects that aim to bring smart city technologies to pedestrians. In one, the technologies make the streetscape safer and more accessible to visually impaired people. In the second project, Smith draws on his experience working as a research scientist in industry to bring local history to life.

“I think these technologies are almost like actors in a stage play, where all these different sensors and chips are like actors putting together a great performance for the audience,” Smith said. “They’re giving the audience a great user experience.”

Accessibility has long been one of Smith’s primary research themes. His work making sports broadcasts and video games equally accessible to low-vision users has opened a new world of experiences to these communities.

“With CS3, we’ve been thinking a lot about how we can take this research to the next level by moving towards making our physical cities accessible to people who are blind, low-vision, or mobility impaired,” Smith said. Smith and members of his lab have used surveillance camera footage to power an app that provides real-time assistance to help a blind user avoid obstacles and navigate to buildings, bus stops, benches, and other points of interest.

Smith’s research in building technologies that connect members of the community is based on what he has heard from CS3’s community partners.

“One of the big takeaways from those conversations is the need for community members to preserve and share their history so it isn’t wiped away as the city expands,” he said. That input — along with Smith’s experience designing AR and social projects for Snapchat — inspired several efforts to record and share memories. 

In one project, community members have the chance to record a video that can later be replayed as an immersive memory. In another project, technologies allow community members to take photographs and annotate them with their memories and knowledge about the neighborhood. 

“The idea is that each point of interest in the neighborhood has an associated discussion where people can keep the conversation going through comments and stories,” Smith said.