I am an Indian and it shocks me that Indians are still blaming the British after 70 yrs of independence.
Is 70 years of Independence not enough to make your language "first class citizen" ? Ofcourse Bengali is second class language because Bengalis didn't invent the standard.
Can we stop blaming white people for everything. Seriously WTF.
Hebrew (and I'd guess Arabic and other right-to-left languages) work rather badly in Unicode when it comes to bidirectional rendering; however, to the extent that it's a result of Israeli/Egyptian/Saudi/etc. companies and/or governments failing to pay $18K (the figure from TFA) to pay for the consortium membership, I kinda blame them and not the consortium. I mean, it's not a whole lot of money, even for a smallish company, not to mention a biggish state.
Also this bit was just amazing: "It took half a century to replace the English-only ASCII with Unicode, and even that was only made possible with an encoding that explicitly maintains compatibility with ASCII, allowing English speakers to continue ignoring other languages."
Seriously? Nobody forces anyone to use UTF-8, and even if it were the only encoding available - by how much the cost of these extra bytes reduces the standard of living in non-English-speaking countries, exactly?
It is in fact unfortunate that the pile of poo is standardized while characters from languages spoken by hundreds of millions are not, however. A pity that a good and surprising (to some) point was mixed with all that "white men" business.
The pile of poo is a great example. It is supported because the unicode consortium pays attention to the need of non western languages, not because it ignores them and would rather do silly things.
Japan has had a tendency to includes all sorts of crap, including this example of actual crap, into its character sets/encodings. Japan also has a history of working with the various standard bodies like the Unicode Consortium. Because Japanese systems had the pile of poo, unicode has it.
> Hebrew (and I'd guess Arabic and other right-to-left languages) work rather badly in Unicode when it comes to bidirectional rendering; however, to the extent that it's a result of Israeli/Egyptian/Saudi/etc. companies and/or governments failing to pay $18K (the figure from TFA) to pay for the consortium membership
I'm sorry? How does this have anything to do with encoding, versus software toolkits? Unicode and encoding generally has little to nothing to do with input methods or rendering, and if you can explain why Unicode in particular works better in one direction I am genuinely curious.
> It is in fact unfortunate that the pile of poo is standardized while characters from languages spoken by hundreds of millions are not, however
I don't think in 2015 Unicode lacks codepoints for any glyph in regular usage by hundreds of millions. OP has not cited any; the one he explicitly complains of was in fact added in 2005.
Before someone says Chinese, that's a bit more like citing some archaic English word missing from a spellchecker. I am skeptical that there is a Han character missing from Unicode that >> 1 million people recognize much less use on a regular basis.
Unicode defines a "logical order" and a "rendering order". The logical order is supposed to be the order in which you read the text - letters read earlier are closer to the beginning of the buffer. The rendering order is how the stuff appears on screen - where an English word inside Hebrew text, if you count the letters from right to left, will obviously have its last letter assigned a smaller number than its first letter.
This means that any program rendering Unicode text - which is stored in logical order - needs to implement the crazy "bidi rendering" algorithm which includes a guess of the "document language" (is it a mostly-English or a mostly-Hebrew text? that fact changes everything in how the character order affects on-screen positioning) and the arbitrary constant 64 for nesting levels.
In this sense Unicode has a lot to do with rendering. As to input methods...
Could your program ignore it all and just let users edit text however you choose? Well, you'd need to translate the on-screen result to a logical order that can then be rendered by someone else the way Unicode prescribes, and you'd be breaking a near-universal (though terrible) UI standard, confusing users mightily. For instance users are accustomed to Delete deleting a different character when the cursor is between two words, one English one and one Hebrew one, depending on where you came from (a single on-screen position corresponds to two logical order positions, see?) You also can't store any hints wrt how things should be interpreted in the Unicode text that are not a part of the Unicode standard, you can only store such hints in a side-band channel which is not available if you store plain text someone else might read. And if you don't do that, well, you can ignore Unicode altogether... except when handling paste commands which will force you to implement bidi in all its glory, including a guess of what the document language was.
Now the part that I don't remember (it's been a decade since I last touched this shit) is whether Unicode mandates anything wrt editing, versus just implicitly compelling you to standardize on the same methods that make sense in light of their logical & rendering orders...
BTW - would I, or some bunch of Jews, Arabs etc. with a background in programming and linguistics beat the white folks in the quality of their solution? I dunno, it's a really really tough problem because you kinda don't want to reverse words in the file, on the other hand you kinda ought to "reverse" them on the screen - that is, spell some in the opposite direction of some others. You could keep them reversed in the file - a different tradeoff - or you could have some other bidirectional rendering algorithm, perhaps with less defaults/guesses. How it "really" ought to work I don't know, obviously neither left-to-right nor right-to-left writing systems were designed with interoperability in mind. Much as I hated bidi when I dealt with it and much as I despise it every day when it routinely garbles my email etc., it's hard to expect much better...
As to your comparison of Chinese characters in Unicode and old English words in a spellchecker... The latter is what, 10000x easier to fix/work around? (Not that I know anything about Chinese; I do see people in the comments counter his point about his own language, I just can't know who is right.)
Okay, how would you do it?
If you want bidirectional mixed language support, you're not going to get around implementing bidi rendering at some layer.
Character encoding is not that layer. You are making legitimate complaints, but they are largely in the realm of poor internationalization support. There may be finger-pointing between applications, toolkits, and OS middleware, but the character encoding should not be taking the bum rap for that.
Internationalization is a fundamentally difficult engineering problem and no character encoding is going to solve it.
> Could your program ignore it all and just let users edit text however you choose?
Sure, that's what happens in my text editor in my terminal, the unicode rendering rules are ignored, each code point gets one character on the screen, from left to right. Once things look how I expect them to in there, I test on real devices, in the real output context to make sure things are looking how they should; often I also have someone who can read and understand the text check too.
This works for me, but I'm a software developer, and I'm also not fluent in many languages, so seeing code points in logical order doesn't cause me problems with readability.
Was British rule actually a net negative, in retrospect? Have there been studies done using objective criteria (not emotional) over counties that were colonies versus ones that weren't? I suppose you can't really quantify the value of people that were destroyed by colonization, but you can look at the current population.
Also I just gotta wonder: suppose European or other relatively simple-to-encode languages didn't exist, and everyone used the OPs language. How would they have handled advancing computers? I've seen photos of Japanese typewriters and they look... unwieldy to say the least. And graphics tech took a while to get advanced enough to handle such languages, let alone input. (MS Japanese IME uses some light AI to pick the desired kanji, right?)
Disclaimer: I don't mean this in an offensive way, just a dispassionate curiosity.
>"Was British rule actually a net negative, in retrospect? Have there been studies done using objective criteria (not emotional) over counties that were colonies versus ones that weren't?"
It's both a positive, and a negative. But if you want to calculate whether it was a "net" positive or "net" negative, then I'm afraid you're going to have to quantify very difficult-to-quantify concepts. And you'll have to do it on behalf of a lot of people, some of which don't exist anymore to answer your question. In essence, next to impossible.
However, particularly in the case of colonialism. I will add that because it's such a difficult concept, it is being exploited by manipulative scholars and politicians for personal/political agendas. So you have politicians in these countries claiming that it was a net negative, without any way to quantify it, and without acknowledging any good that came along with it. Now, of course, colonialism was bad, period. It can never be justified to enslave people, and it should never have happened.
Well that's why I said without taking into account people that were destroyed by it. That is, if we look purely at modern-day circumstances, are they better?
A comparison might be a country with overpopulation. If you kill x% of the population and enact some birth control plan, then 100 years later you might end up with a "net positive" discounting the people that were killed and the emotional effects of their loved ones.
Are there countries in Africa that weren't colonized, or some islands that weren't, that we could compare to ones that were and make some inferences?
Alternatively, you could compare countries that had "more" or "less" colonialist control/influence, and see if there is any correlation to their current performance. Though, as with all social-sciency it's very difficult to isolate the variables to get deep insight into possible causal links.
It is really hard to quantify something like that. While British siphoned off large amounts of natural resources and often caused widespread famines in India. They also gave us western democracy, education in western science,access to all the western knowledge.
You speak as though, these things wouldn't have come without the British; I'm sure Japan today is a poor backwards country, for they weren't blessed with the act of "civilizing" by the cross.
All this is out of context and has nothing to do with what OP talks about.
About what you are talking about, it is a fact that India was the richest countries in the world before British colonised it, and after British left it was one of poorest.
> suppose European or other relatively simple-to-encode languages didn't exist, and everyone used the OPs language. How would they have handled advancing computers?
Bengali isn't all that complicated; more so than English, yes, but definitely surmountable. Printing presses, typewriters, etc. have been used without incident for Bengali (and for other Indian languages) for a long time.
P.S.: It's not really a great idea to open a post with "Was British rule actually a net negative, in retrospect?" I understand you did not intend to offend, but at the very least it's a touchy subject, and a contentious and complicated history.
Although I agree with a lot of what you said, I perceive the last part, your post-scriptum, to be a little bit distorted. The Renaissance had many sources. One of them was the massive intellectual immigration from the crumbling Byzantine Empire, thus serving for much of intellectual works as a bridge over the time (from antiquity to enlightened medieval period) and space (near east to all over the Europe). And it was mostly Greek in its cultural background.
Is 70 years of Independence not enough to make your language "first class citizen" ? Ofcourse Bengali is second class language because Bengalis didn't invent the standard.
Can we stop blaming white people for everything. Seriously WTF.