But this means you need diff tools, version control systems and editors (edit: and merge tools) for every single format, which while nice in theory (hey plugins!), in practice doesn't work well. Text is nice because it sort of a minimum common denominator.

Of course semantic diffs would be nice; even, or especially, for code for example but while they do exist they haven't seen much uptake because they do not integrate well with existing tooling.