Continuing with your hypothetical, even though we’d be 90% safer as a collective, the safety of the individual feels compromised: the risk of an accident is non-uniform when involving humans (depending on e.g. age, experience, safety, alertness, etc.), but becomes uniform (or at least more uniform) with an algorithm in charge.
A very good observation. Based on your comment, I think we can relax the requirement stated by OP by saying:
"Until Waymo's cars reduce any individual's chance of an accident."
So for example, suppose a Waymo car is better than humans overall, but tends to do worse than humans when there's a small bump on the road. And suppose that all humans (in a given regulator's area, e.g., California) tend to encounter such bumps at roughly the same rate (per mile driven) over their lifetime. In that case, it's probably going to be acceptable, since every individual is better off.
I don't know, maybe this is not impactful / obvious enough for people to care about?
What certainly is obvious is that the safest drivers are much safer than an average driver (does anyone know of a study that estimates this ratio?). Therefore, at the very least, the threshold for Waymo should be not the average accident rate, but the accident rate for the safest drivers.
Continuing with your hypothetical, even though we’d be 90% safer as a collective, the safety of the individual feels compromised: the risk of an accident is non-uniform when involving humans (depending on e.g. age, experience, safety, alertness, etc.), but becomes uniform (or at least more uniform) with an algorithm in charge.
That’s a tough thing for people to buy into.