I believe it was either the last MICRO* or the one before that when Dally addressed this point. The specialized hardware for graphics ends up comprising such a small portion of the overall chip that it wasn't worth it to remove it. The "GPUs were made for graphics thus aren't good for DL" argument really doesn't hold a lot of water IMO.
* It might've been a difference conference now that I think of it.
The microarchctiecture has many unnecessary things and it's not optimized as a whole for deep learning.