"Multiply colors" in pure physics is ill-defined, but we're talking bout a rendering engine. Engineers are allowed to make approximations whenever they want, and approximations aren't restricted from introducing new concepts/terminology.
The closest thing to "multiply two colors" in physics would be taking a product state:
|ψ⟩ = |ψ1⟩ ⊗ |ψ2⟩
Here, we are expanding a Hilbert space to include both unentangled particles (color == photon).
The RGB vectors are more or less based on biology, they're just a toy model[0]. Plus, physics is nothing more than a series of better and better approximations to things we observe -- it's all made up at some point. I'm guessing you're upset that the "made up" stuff in this case leads to no new physical insight and deviates from what we know to be more consistent with observation. That doesn't mean that "made up" stuff is not useful. Toy models are everywhere, even in fundamental research.
There's a very good reason why people don't solve the kinematics of a tennis ball as if it were a 10^23 quantum-many-body problem. If you're constrained on time and money, the smartest thing to do is to get the closest result you can in the smallest time with the least amount of money.
