That's true only if you use Vec<T> instead of a specialized arena, either append only, maybe growable, or generational, where access invalidation is tracked for you on access.
Yeah if you go with Vec, you have to accept that you can't delete anything until you're done with the whole collection. A lot of programs (including basically anything that isn't long running) can accept that. The rest need to use SlotMap or similar, which is an easy transition that you can make as needed.
