To maintain the character.
Professionally, you basically need to live and think in the future, anticipating trends and the use of technology. At the same time, both of you share a passion for modifying and renovating historic houses. What makes the past so appealing when it comes to your living situation?
Shapiro: I live in a historic home, which is over 110 years old. When I bought it over 20 years ago, it needed a great deal of renovation. However, it was important to maintain the character of this beautiful craftsman home, all while infusing it with modern conveniences. So, we worked with an architect to preserve everything and make modifications that would make it more energy-efficient, including installing solar panels. It was early on in the "solarisation" of houses in the US, and in California in particular, when I started this endeavour. Just the concept of doing something that was good for the environment was motivation enough. And, it ultimately had a positive return on investment. We would produce more energy than we consumed during the day, and we could push energy back out to the grid. Then, in the evening when the sun went down, we would consume energy back from the grid, but at a lower rate. It was an interesting early adoption of the technology. It would be ideal to have panels that rotate and follow the sun. But unfortunately, my system doesn't have that, yet.
Mankowsky: We have an old house in Swabia; it's from 1835. You can't insulate it. It's a similar issue that you have, Danny. But we don't want to destroy the original character. So, I built a greenhouse on one side to the street where the sun is shining. It was made from high-tech polycarbonate with seven walls, which has good insulation that also blocks out street noise. We made many other improvements, while at the same time you don't want to ruin these old buildings. We will install solar panels in the next phase.
Danny Shapiro is NVIDIA's Vice President of Automotive, where the company is spearheading innovation in car design, autonomous driving and AI-enabled in-vehicle experiences. With over 25 years of computer graphics and semiconductor industry experience, he's one of the leading experts in AI solutions for self-driving cars, trucks and shuttles.
With a background in social sciences, philosophy and psychology, and early encounters with the first wave of AI, Alexander Mankowsky is the futurologist at Mercedes-Benz. His coveted research and studies hint at what the future of transportation and urbanism holds for humanity.
Many iconic models.
This process of finding a solution for a day-to-day problem is similar to the professional work both of you do. To succeed in your jobs, is it helpful to always have one foot in the past, or in the present at least?
Shapiro: When you look at things around us, there's an appreciation for art, or heritage – something like an old home, or a beautiful classic car. Mercedes-Benz has a rich history with many iconic models and we love their appearance, and the stories connected to them. Being able to take vintage items forward, to merge them with new technology, is a big challenge, but also exciting. I constantly look for examples or possibilities to fuse quality, aesthetics, and technology in the best possible way. I was so excited when Mercedes-Benz was bringing electric cars to the market. I was one of the first people in the United States to get the all-electric B-Class. So being able to drive around in Silicon Valley in an eco-friendly Mercedes-Benz was a great feeling. But it was also a symbol for what I described above. The B-Class that brought together quality, craftsmanship and the aesthetics Mercedes-Benz is known for with cutting-edge electric vehicle technology. That's also what's fascinating about my job, where we get to look at different dynamics and technologies, then fuse them together in order to build a better product.
Mankowsky: I completely understand this, because in a company like Mercedes-Benz, you must be a pragmatic visionary. It's meaningless to come up with so-called futuristic fantasies, which are then turned around and disrupted because they were nothing more than a trend of the time. For example – like installing a fusion energy reactor in your car. Pragmatism is way more productive: it's like in these houses with all these old things, and old technologies. I love old technologies. And I believe many still have a value for us today. I prefer to own music physically for example, because I don't want to be interrupted by a phone call or an email on the computer. When I put on a record, it's just music and nothing else. These technologies may have disadvantages, but we shouldn't throw old solutions away just because something new is coming up.
Safety is the number-one priority
Mr Shapiro, how do you see the balance between analogue and digital?
Shapiro: So much of design and development today is hybrid, where the physical and digital merge. Digital's role of course has grown, enabling products and user experiences to get better and better over time. When the iPhone first came out, it had just a few functions and was therefore somewhat limited in its capabilities. Now, just look at the millions of apps out there and how it's evolved: it's phenomenal. This concept of software-defined products and over-the-air updates has rapidly become the expectation. And companies like NVIDIA are dedicated to continually making that software better, throughout the life of the product. As a result, the product is never done; it's always evolving. You don't have to throw things away; rather, you can make them better. Take for example the future fleets of Mercedes-Benz vehicles that we're working together on. Starting in 2024, these will be software-defined cars. Soft­ware and hardware from NVIDIA and Mercedes-Benz will make the cars safer, they'll make them more enjoyable, and these cars will do so many more things than what's on the road today. Plus, they'll be able to do things that we haven't even thought of yet. At NVIDIA, we've built a platform that is perpetually upgradeable and leaves the door wide open to future innovation.
Mr Mankowsky, you've worked on a project about autonomous driving and empathy. How are those two topics connected?
Mankowsky: As a social scientist, I looked at mobility and saw that it's a cooperative task. That means that we are predicting what the others are doing and then we act accordingly. Then the next question was, how does it work? And what will happen if you introduce a robot to such a communicational context? Think of skaters: they manage to move around without accidents because they can predict what the others are doing. And now introduce a robot: the skaters would struggle to avoid this thing, because they could not predict its movements. Together with neuroscientists, we found out that our perception works with empathy in a way that we continually perceive information from others unconsciously. A robot car, though, is not "alive" – it has no intentions. It is driven algorithmically. Our learned prediction mechanism will very likely run into trouble. We might get anxious or do the wrong things when reacting to the car. What we asked ourselves next was how to communicate without language. We did an experiment and placed cameras on the heads of horses. This way we learned that horses were in a constant stream of communication with us humans and other horses. We then applied this learning to self-driving cars. One main aspect is that cars need to communicate their intentions – they need to show that they are in automatic mode. That's important. Imagine crossing a road without knowing if the car has sensed you.
What role does safety play in future autonomous driving cars?
Shapiro: Safety is our number-one priority. Our aim is to train a car to drive much better than a human. Perception is a key aspect of achieving that goal, as we develop a system of achieving super-human abilities of detection and understanding of the environment. We have many algorithms and deep neural networks that run inside the car, detecting everything from the lanes, signs and lights to other road users. This is all performed over 360 degrees and more precisely than we as humans could. Through this process, we're building a digital twin of the real world as we drive. That way, the car knows everything that's there, whether moving or static. And then there are behaviours that are tied to each of the objects in the environment. A car, for example, moves differently than a truck, which moves differently than a motorcycle, which moves differently than a pedestrian. With safety at the forefront, our self-driving cars are not going to be aggressive. They're not going to put themselves in hazardous situations. In that regard, they're empathetic to the world and the situation. They will proceed with caution and in a predictable manner. We developed a safety system that's called Safety Force Field: it is designed to not create or contribute to an unsafe situation. But there's no way to predict everything. In today's world, crazy events can happen all the time, whether initiated by humans or not. We'll never be able to account for everything. But what we can do is create systems that are far safer than current human-driven systems on our roads today in order to dramatically reduce accidents, injuries and fatalities.
Mankowsky: We're still missing a model and theory for cooperation between human perception and machines. You can compare this to a dance floor. Dancing is pure empathy, because otherwise you would stomp on the feet of the others all the time [laughs]. You have to predict the movement, and the rhythm helps you to do so, just like traffic rules, per se. This kind of model is missing in self-driving technology.
Shapiro: I think the ability to communicate is not an absolute requirement for autonomous vehicles, but it is a huge bonus when we can have vehicle-to-vehicle communication or vehicle-to-infra- structure communication. Part of the problem we have today is that there are a billion cars that do not currently have this capability. So if we were to introduce vehicle-to-vehicle communications, there would still be legacy vehicles on the road that won't be using it. Of course, the way to solve this is get rid of all cars and just put new cars out that are autonomous and talk to each other. But it's not realistic. Over time, if we have cars that are all talking to each other and the infrastructure, we can eliminate the street lights. You could have that dance floor experience where cars will just safely pass through intersections.
Mankowsky: As great as that sounds, I also see a problem here: I fear a future in which cars are communicating invisibly. And the people are standing there, not being able to decide what to do, because they can't predict the algorithm.
Who is actually shaping the future: is it us with our needs, or is it technology that provides certain possibilities we adapt?
Mankowsky: If we're using a very simple model, there are two relevant spheres: culture and technology. Until around 2010, the driving force of our field of work was technology. Now, culture has the lead. And social innovation, or social change, can be abrupt. Think of the fall of the Berlin Wall. For technology on the other side, the time for development is slow. We are still working with programming language C, we still use HTML and nobody can change that because there are so many who still use it and need it. The inertia is so heavy on the technological side. On the social side, that's not the case. We have the climate crisis. We will have many people fleeing their countries roaming around the planet. That is totally predictable, since it's already there. So, the cultural wheel will be the dominant one. Whether that's a good or a bad thing, that's another question.
Shapiro: I agree. Both worlds are merging more and more now. In some cases, it's the technology that has enabled society to move at a different pace, for things to shift radically. Think about social media: there are tools that have enabled massive shifts in ways people communicate and organise, for good or bad.
Mankowsky: Now, the cultural and the political are starting to hit back. You can see that by the internet falling apart – everybody is creating their own rules and regulations. This openness, which was strong at the beginning of the digital revolution, is not there anymore. It's not an open field anymore. Therefore, the social part is the bigger driving force. And we have to make the technology flexible enough that it fits the gear on the social side.
We already talked about self-driving cars making roads safer. What are other aspects that could change because of this technology?
Shapiro: It's about giving people back their time. The number of hours people spend in their cars, stuck in traffic or just driving on long trips is staggering. As a passenger in a self-driving car, there is no need to pay attention to the traffic; instead they can work, sleep, read books, watch movies, play games or other things – even video conferences are possible. In addition, most cars stay parked the majority of the time, and there are large parking structures and lots throughout cities. But, if we had much more efficient autonomous transportation, we could replace many of those parking lots with parks to infuse more nature into our cities.
Mankowsky: We had beautiful visions about shared spaces, where the shared spaces were open for people, for cars, and moving infrastructure. I know it sounds futuristic, but it's not so futuristic after all. Think of a farmer's market: a booth can come to where the people are instead of the people having to come to the booths. Architects love that idea, and if you provide the technology, it will be feasible. I wouldn't focus so much on the passenger car, which is a nice thing, but it's not the whole picture.
Shapiro: Absolutely. We're not just focused on moving people. A big part of what we're doing is moving goods. Trucking is a huge aspect of our roadways, with a great deal of safety issues that need to be addressed. Also, there's a global shortage of truck drivers which is further exacerbated by our society increasing online shopping and the need for shipping. In addition to long-haul trucking, we see last-mile delivery to homes increasing, whether it's vans delivering groceries, or sidewalk robots delivering meals to homes. The benefit, of course, is that the core technology being developed for cars is exactly the same for trucks. We are able to modify the use cases, but leverage all the same AI.
Mankowsky: That's fashionable in architecture, in city planning. To make many things movable or mobile. So you don't need so many things anymore, it's way more flexible. That's a good thing.
Blending the digital side with the physical side.
Smart cities, so to speak. Speaking of smart homes: are there some home improvement measures that you have planned for the future?
Shapiro: I have a new home renovation project. This home was built in the 1960s and needs to be modernised. I want to make it a smart home, like a software-defined home with new technologies, new electronics. There's also the physical side – for example, we'll replace the original windows with new, energy-efficient windows.
I want to focus on making the house green by equipping it with smart tech­nol­ogies for heating, cooling and even lighting that's more environmentally friendly. It's again about blending the digital side with the physical side, and creating a great environment and living experience.
Mankowsky: Sounds nice! Our plan is to have a larger garden. For that, we will try to grow more vegetables, for example. And this will also include some automation – because we cannot do that on our own with all the maintenance.
So there will be robots who pick the apples in the garden and bring them to you?
Mankowsky: I would love to build machinery that is able to do something like that. There are already machines for that. The funniest machines you can imagine you can find in agriculture. I would love to build one on my own – it would be possible because recognising an apple that's mostly red and round should be easy.
Shapiro: There's some really exciting work we're doing in the agricultural space with autonomous vehicles and tractors. They can use AI to determine how to precisely target fertiliser or weedkiller in the most efficient manner. We see AI and robots helping in so many different industries. Robots are of course used in manufacturing cars. But with more AI and technology used in automating factories, we can have robots safely working alongside humans. These cobots improve quality and efficiency, and help prevent human workforce injuries due to repetitive motion. There are so many exciting ways to use this technology to improve the quality of life for all.