One of the reasons fairly wealthy people can't retire early is that a major medical event in their family could wipe them out financially. 65 is the standard retirement age in the U.S. for a very good reason - after 65, you are covered under Medicare. Even then, Medicare only covers 80%.

http://en.wikipedia.org/wiki/Medicare_(United_States)