Project CETI (Cetacean Translation Initiative) aims to collect millions to billions of high-quality, highly contextualized vocalizations in order to understand how sperm whales communicate. But finding the whales and knowing where they will surface to capture the data is challenging — making it difficult to attach listening devices and collect visual information.
Today, a Project CETI research team led by Stephanie Gil, Assistant Professor of Computer Science at the Harvard John A. Paulson School of Engineering and Applied Sciences (SEAS), have proposed a new reinforcement learning framework with autonomous drones to find sperm whales and predict where they will surface.
The research is published in Science Robotics.
This new study uses various sensing devices, such as Project CETI aerial drones with very high frequency (VHF) signal sensing capability that leverage signal phase along with the drone’s motion to emulate an ‘antenna array in air’ for estimating directionality of received pings from CETI’s on-whale tags. It demonstrates that it’s possible to predict when and where a whale may surface by using these various sensor data as well as predictive models of sperm whales dive behavior. With that information, Project CETI can now design algorithms for the most efficient route for a drone to rendezvous — or encounter — a whale at the surface. This also opens up possible conservation applications to help ships avoid striking whales while at the surface.
Presenting the Autonomous Vehicles for whAle Tracking And Rendezvous by remote Sensing, or AVATARS framework, this study jointly develops two interrelated components of autonomy and sensing: autonomy, which determines the positioning commands of the autonomous robots to maximize visual whale encounters; and sensing, which measures the Angle-of-Arrival (AOA) from whale tags to inform the decision-making process. Measurements from our autonomous drone to surfaced tags, acoustic AOA from existing underwater sensors, and whale motion models from previous biological studies of sperm whales are provided as inputs to the AVATARS autonomous decision-making algorithm, which in turn aims to minimize missed rendezvous opportunities with whales.
AVATARS is the first co-development of VHF sensing and reinforcement learning decision-making for maximizing rendezvous of robots and whales at sea. A well-known application of time-critical rendezvous is used with rideshare apps, which uses real-time sensing to note the dynamic paths and positions of drivers and potential riders. When a rider requests a ride, it can assign a driver to rendezvous with the rider as efficiently and as timely as possible. Project CETI’s case is similar in that they are real-time tracking the whale, with the goal of coordinating the drone’s rendezvous to meet the whale at the surface.
This research advances Project CETI’s goal of obtaining millions to billions of high-quality, highly contextualized whale vocalizations. The addition of diverse types of data will improve location estimates and routing algorithms — helping Project CETI meet that goal more efficiently.
“I’m excited to contribute to this breakthrough for Project CETI. By leveraging autonomous systems and advanced sensor integration, we’re able to solve key challenges in tracking and studying whales in their natural habitats. This is not only a technological advancement, but also a critical step in helping us understand the complex communications and behaviors of these creatures,” said Gil.
“This research is a major milestone for Project CETI’s mission. We can now significantly enhance our ability to gather high-quality and large-scale dataset on whale vocalizations and the associated behavioral context, putting us one step closer to better listening to and translating what sperm whales are saying,” said David Gruber, Founder and Lead of Project CETI.
“‘This research was an amazing opportunity to test our systems and algorithms in a challenging marine environment. This interdisciplinary work, that combines wireless sensing, artificial intelligence and marine biology, is a prime example of how robotics can be part of the solution for further deciphering the social behavior of sperm whales,” said Ninad Jadhav, Harvard University PhD candidate and first author on the paper.
“This project provides an excellent opportunity to test our algorithms in the field, where robotics and artificial intelligence can enrich data collection and expedite research for broader science in language processing and marine biology, ultimately protecting the health and habitat of sperm whales,” said Sushmita Bhattacharya, a postdoctoral researcher in Gil’s REACT Lab at SEAS.
More information: