Karpathy claimed they worked for years and could not reap the benefits from multiple sensors however hard they tried. He seemed really convinced and does not get to me as one who tells stuff just to justify cost reductions, like Elon sometimes is carried away.
It’s also possible they weren’t able to see benefits given the processors that were available to them at the time.
It’s also possible that they were so far behind Waymo in the journey to FSD that they weren’t yet at a point where multiple sensors would make a significant difference.
This is my read on it as well. I listened to the same interview (it was definitely on the front page, if not at #1 for a while). I stepped away feeling like Karpathy had described a lot of good business reasons for not using other sensors, but not a lot of good technical reasons. Sensor fusion is hard, yes, but maybe not harder than perfectly re-projecting 2D pixel images into 3D vector space.
Just my interpretation, but it felt to me like a hail mary because they were fully committed to being the first mover. Waiting around for LiDAR prices to come down would have meant that Waymo would have beat them.
The problem is, in reality, we have Tesla vehicles that sold with other sensors that are now disabled, and those cars are now very difficult to drive with any automation enabled, even simple cruise control, due to the limitations of vision-only driving. And this is leading to them slamming on brakes at inopportune times, blinding oncoming drivers with the bright headlights, etc. - issues that did not happen when other sensors were used.
In reality, we have Tesla vehicles that sold with other sensors that are now disabled, and those cars are now better to drive with automation enabled than with the extraneous sensors, even simple cruise control is better and phantom braking is actually decreased vs spurious radar returns that existed previously, due to the limitations of low resolution radar. Auto high beams is now better at not blinding oncoming drivers with bright headlights, etc. - issues that happened all the time when other sensors were used previously.
As someone who owned a 2019 Tesla and who owns a 2023 Tesla, the older car had better autopilot. It was starting to degrade in 2020. The new one is worse. Phantom braking was quite rare in 2019. The very first night I drove home the new car, it phantom braked on a lonely, empty stretch of I-5.
FSD stack is disabled on highways. You are using the years old code. Beta v11 when it comes out will enable OP's FSD referenced improvements for highways.
I used to mock my relatives for not wanting to put their kids in a Tesla. But yeah, I'm not subjecting my family to beta software for a critical safety feature, nor should anyone else.
Wasn't the Summon feature downgraded when they moved to a vision-only approach? One YT video I saw compared a version 1 and a more recent version and the vision-only wasn't able to do as much. Contradicts the idea additional sensors do not add value.
I think the idea is less "additional sensors don't add value in various scenarios" and more "we are 100% certain a vision-only system can perform well on existing road infrastructure; we are not sure about sensor fusion systems".
You are misunderstanding the situation. Karpathy claimed that sensor fusion of radar, sonar and vision isn’t working well. He made no such claim about Lidar. Lidar is the sensor that is the crucial difference between Waymo and Tesla's approach to self driving.
The reason he claimed sensor fusion was not working well was due to vendor versioning. He claimed the same sensor from different manufacturing batches performed differently and thus needed to be re-characterized, which then has follow on effects in various math models. Multiply this by many sensors, and the need for replacement parts inventory for a decade or two and the problem becomes intractable was his claim on his most recent appearance on the LF podcast.
Then something else was the bottleneck at the time. It is very easy to prove that some sensors in some situations will be able to perceive things that other sensors cannot. In those situations the additional sensors are crucial first steps. I would guess the bottleneck is shitty reliance on statistical machine learning with a long tail of unhandled edge cases. Each case very uncommon, but in aggregate a very important sum.
If you're referring to the Lex Friedman interview, at the start of his answer, he mentions it was a cost-based decision. And that radar/ultrasonic wasn't worth it for them, due to the additional time it took. Not that it wasn't helpful, just more effort that could be better spent elsewhere.
While technically true that extra sensors contribute noise, surely with the proper programming they should be helpful.
You just need to weight the information from you extra sensors appropriately based on your confidence the signal is accurate.
For example, if your lidar sensor is 99.999% percent sure there is an obstacle in front of you, surely it's helpful to take that information into account, even if it is a tiny bit uncertain/noisy.
Wouldn't that be more related to parallel processing power and throughout capabilities than feasibility?
More data in any situation where bandwidth is already maximized will lead to entropy and noise. However, if the capabilities were there to process all of that data in low latency scenarios with headroom to spare, surely adding additional sensors and data points would lead to a more complete model of spatial awareness for the car.
That's all hypothetical and reliant on ignoring the realities of running a business and tech development lol
A bit like our discussion here. More standpoints can ideally be merged into a coherent, more complete picture. The key is to sort out disagreements as coming from different backgrounds and biases and everybody admitting she is not 100% right. Otherwise it is a quarrel of stubborn knowitalls that can't agree.
Fusion is hard. As hard as getting humans to agree. Been there. In both situations.
And of course you can concentrate on improving one echo chamber, err single sensor. But you can never come past its fundamental limitations.
I always thought it came down to two reason, costs and looks. Telsa has to sell a car people want to drive daily. They can't have a bunch of lidar sensors on their cars no one would buy them even if they could drive themselves. Also they would most likely cost a lot more due to lidar sensors not being cheap compared to normal cameras.
Unlike Waymo who doesn't care about selling cars to people to drive daily. No one going to care that the taxi they are taking looks ugly as long as it get them to the place they are going for cheaper. The cost is also a less of a factor due to them being able to produce an income by charging people to ride in them.
