MIT CSAIL Director Daniela Rus Presents New Self-Driving Models

When you read this, you'll never think about self-driving vehicles the same way again!

I want to talk about some conversations I’ve been having with Daniela Rus, the director of our own CSAIL lab at MIT.

Highlighting some groundbreaking work on autonomous driving as we’re brainstorming, she talks about what people are already doing (with Dall-E, etc.) generating images from text.

“What else can we use text to generate?” she asks, specifically mentioning text-to-drive, which is one of her projects at CSAIL right now. And I’m proud of what they’re doing!

Recently, Rus has talked about how to come at these kinds of goals.

She suggests that first, you need an autonomous vehicle and you can get one by using a “recipe” for self-driving that's been used in the field by various companies and research teams. It combines a drive-by-wire system with sensors, software for perception, an estimation model to localize the vehicle and a planning and control module to tell the vehicle what to do. There are solutions for each of these modules — these solutions are brittle in that they need carefully engineered parameters for each road situation,

With machine learning, she adds, we get more robust solutions that can learn from humans what to do in a wide variety of situations without the need to manually engineer the parameters.

Here are some quotes from her on the subject:

“We have AI solutions. These solutions need data.”

“We need to do so much more … We need to think about physics, we need to think about geometry - we need to think about the rules of the road - all of those are coming together in this project.”

“They require that we specifically program each type of road situation that we encounter,” (of systems representing self-driving 1.0) (my term). “So you have to carefully engineer your parameters for nighttime driving, daytime driving, driving in rain, driving in sunshine, driving with or without lane markers… this gets to be tricky.”

“In simulation, we can turn that data into anything that we want,” she said. “With this approach, we can train self-driving vehicles to cope with challenging driving situations.”

A major part of improving self-driving, she explains, involves figuring out how to help the robotic systems respond to black swan crisis conditions, and that can be costly, and complex.

What can we do?

Rus describes systems that rely on the versatility of data inputs and applied reasoning. She references the concept of end-to-end learning and real-world transferable reinforcement learning with a triangle approach combining these three elements: data-model-deploy. You can see a lot more in her slide deck or the presentation itself. It is compelling! Here’s some more:

“Can we get our robots to understand their environment?” she asked, then answered. “We can begin to make headway in this direction.”

A goal, she noted, is to “connect the world's knowledge to visual learning systems” for spatial reasoning.

“The end result is systems that allow us to reason about input streams using language,” she said, “generalizing to new situations.”

In explaining how this works, Rus mentioned a set of descriptions that you curate in order to drive this engine - in other words, you can move from a simple perception-to-action model based on visual pixels, to something where the vehicle starts to understand how to talk using words about objects like deer, road and sky.

Think about that – instead of just “seeing” the road and responding in statistical ways, the programs may read text descriptions of how to handle things, learn, and respond accordingly. Thinking of it this way, it’s easy to imagine how an AI will become a much better driver than the average human!

“This is truly accelerating the way our autonomous systems can understand their world,” she said. “We get them to a point where they are much easier to interact with … all of this will take us into a world where we can begin to think of our cars as our friends.”

I would say “that’s it,” but really, what a concept! A lot of us have had important reservations about the limits of self-driving, and news around Tesla autopilot didn’t help. But this talk really helped me to understand how to get over some of these problems in a fundamental way. When AI learns to reason from written input, it will be very capable, and we have to think about the ways in which it will outperform a human. This presentation, to me, was one of the high points of our conference.