Yeah, that's one of the caveats. The auth email or SMS to the previous owner would allow them to indicate they moved. That would probably handle most cases quickly. But, if no response in x time, it could default to allow and/or if the new owner wants to be proactive, they can send proof of residence. A little bit of a PITA, but doesn't add much to the PITA moving already is. And, moving is pretty infrequent.
There's also billing address validation, as that frequently matches shipping address anyway.
This kind of validation would also help with other scams.
When ordering online, I, not from Amazon as I don't use it, quite regularly get a 60 second timeout requesting a payment be confirmed in my bank's app.
VisionPlus or one of the many other systems that may be employed in the chain of payment are configurable for many many such triggers or rules, change of address IIRC is a pretty basic one.