You also lose time by slowing down and speeding up but that notwithstanding, you may have fewer intermediate stops with HSR (as is the case with Acela vs. Regional service) but you can't cut out intermediate stops. All you have to do is look at Shinkansen routes.