How many people would've owned CD-ROM drives between 1984 (when CD-ROMs first hit the market) and 1988 (when ISO 9660 was finalized), though?

The bigger pain point for me was around booting from CDs; even though El Torito was around by the time I was potty trained, there were still plenty of PCs out there from before - and it's those PCs on which I cut my teeth as a kid :)

I don’t recall that being the case, but my memory could be faulty.

As I recall, the High Sierra Format - which served as the basis for ISO 9660 — was widely used prior to ISO 9660’s standardization and adoption.

Widely, but not exclusively used. Some big vendors at the time that made things like library indexes and legal references used their own completely wacky in-house formats that required lots of fiddling with device drivers. They'd often only work with a handful of CD-ROM devices as well, meaning you're stuck with your $2000 single-speed top-loading NEC drive for eternity.

High Sierra won out, it was the logocal winner, and there was, thankfully, no standards war like with HD-DVD vs. Blu-Ray.