The height is not only to avoid terrain, it is also to buy you time to move horizontally. It is more efficient to go higher at the beginning of the flight and taking a parabolic path than to try to maintain altitude any lower. Even with the braking burn, which for the same reason should occur as low as possible. Your engine is most efficient when your craft is moving quickly - the Oberth effect.
