Three Columbia Engineering Researchers Win Amazon Research Awards

Three Columbia Engineering researchers– Matei Ciocarlie, Tal Malkin, and Carl Vondrick–recently received Amazon Research Awards (ARA) for their trailblazing work in a range of fields, from robotics to cryptography to computer vision. 

Ciocarlie, associate professor of mechanical engineering, is well-known for his pioneering work in robotics and creating artificial mechanisms that can interact with the world as skillfully as biological organisms. Computer Science Professor Malkin, a leader in cybersecurity research, explores the intersections of privacy, security, and theoretical computer science, aiming to create systems that are provably secure and to understand the limits of what is possible to achieve. Vondrick, associate professor of computer science, is focused on computer vision, machine learning, and predictive modeling to significantly improve how machines understand and interact with the visual world.

They are among 98 Fall 2023 recipients representing 51 universities in 15 countries. The awards provide unrestricted funds and AWS (Amazon Web Services) Promotional Credits to academic researchers investigating various research topics in multiple disciplines. Recipients can access more than 300 Amazon public datasets and utilize AWS AI/ML services and tools through their AWS Promotional Credits. Recipients are also assigned an Amazon research contact who offers consultation and advice, as well as opportunities to participate in Amazon events and training sessions. Amazon encourages the publication of research results, presentations of research at Amazon offices worldwide, and the release of related code under open-source licenses.

In the AWS AI category, Ciocarlie proposed a project called “Do you Speak EMG? Generative Pre-training on Electromyographic Signals for Controlling a Rehabilitation Robot after Stroke.” He is collaborating with Computer Science Professor Carl Vondrick to develop machine learning models related to those used for large language models but aimed at understanding a different type of language -- electromyography, a technique that measures the muscle response, or electrical activity, to a nerve’s stimulation of the muscle.

A wearable robotic orthosis (an external device like a brace or splint) that assists the movements of a stroke survivor can use this language to know what activity the user is trying to perform at any given moment, and provide the correct type of physical assistance at the right time. Such a device could improve the lives of millions of people in the U.S. alone who live with motor impairments. The team proposes to advance the state of generative pre-training machine learning methods and to leverage AWS-scale computing to bring these methods to bear on a new world of languages -- those of the human body. The overall project that this initiative is a part of is a collaboration with Dr. Joel Stein at Columbia University Irving Medical Center’s Rehabilitation Medicine.

In the new AWS Cryptography and Privacy category, Malkin won an ARA for her project, “Cryptographic Techniques for Machine Learning.” She seeks to apply a cryptographic perspective to Machine Learning (ML). Specifically, she is focused on the problem of quality-preserving watermarking for large language models, proposing ways to use cryptographic techniques to reconcile two seemingly contradictory goals. On one hand, it is desirable to watermark AI-generated content to distinguish it from human-generated content. On the other hand, the watermark should only minimally alter the quality of the AI-generated content. As part of the project, Malkin also proposes several concrete problems in different ML settings where privacy is required, along with approaches to address these issues using secure computation and differential privacy techniques.

Vondrick won an ARA for his proposal, “Viper: Visual Inference via Python Execution for Reasoning,” also in the AWS AI category. His project aims to improve visual query answering by addressing limitations in current end-to-end models, such as lack of mathematical and logical reasoning, difficulty separating visual processing from reasoning, inconsistency, and lack of explainability. His team plans to integrate code generation into computer vision and natural language processing methods, building on his preliminary ViperGPT framework. The proposal focuses on two main tasks: replanning, which develops an approach for adjusting programs when they fail, and learning to use tools, which incorporates learning into the process for more efficient program generation. By combining these tasks, Vondrick aims to create a comprehensive solution for visual queries that overcomes the limitations of current models and advances the fields of computer vision and natural language processing. He also won a 2018 ARA for his project, “Anticipating Human Behaviors from Unlabeled Video.”