As a simplification, I did not draw any of the blocks contributing to code signing. RSA is used everywhere to ensure that the code is signed. If you consider the signature as the MAC, it does "Authenticate Then Encrypt" (the signature is over the NCCH or FIRM header. The header contains hash + size of each section.)
Help me understand a bit more about how a bogus key was able to generate a gibberish text segment that code could successfully return into if all program text was protected by RSA signatures. I may have misread the article.
So we have FIRM, the main firmware image. It has three segments: 1) main ARM11 kernel modules, 2) ARM11 kernel, 3) ARM9 kernel/ARM9 process. The FIRM header has a SHA256 hash over all the segments and the size of all segments. The FIRM header also has a RSA2048 signature over the header. Then everything (sig + header + FIRM) is encrypted with AES-CTR and placed on the NAND.
On system start, the whole chunk is decrypted, the signature is verified, and everything works as expected. Until in the New 3DS, they decide to also additionally encrypt segment 3 (the ARM9 stuff) with a separate key on the NAND. That's what led to the whole mess. So I guess their mistaken assumption was that since FIRM was signed, the encrypted ARM9 section was protected. However, they didn't take account of the fact that the key to decrypt it can be corrupted. It's a bit subtle.
I follow. But, if they'd encrypted every segment with AES-OCB or AES-GCM, rather than raw AES-CTR and then relying on the RSA signature of a header with hashes, this bug wouldn't have been possible, right?
(I get that there are multiple ways to break a bug. :)
Of course, in a perfect world we would all be doing that. However, there is a tradeoff of boot-time/power consumption/hardware complexity (the hw AES engine only does CCM/CTR/CBC). Given their track record though, I wouldn't be surprised if things still broke if they used GCM. I think the main takeaway isn't that there's a couple of things they could have done differently to prevent this (which is always true in hindsight), but that certain major design decisions made such bugs inevitable.
