Sparse stuff is not efficient to compute unless it's really, really sparse, or at least partially dense in large enough chunks that your compute can efficiently do its job. If L1 hit is counting to three or four depending on the arch, a full cache miss is counting to 200+. If you miss your cache all the time (which with sparse stuff you will) things get really slow. And that's _before_ you consider that GPU programs can't really do different branches across threads, and non-coalesced memory access absolutely crushes their memory throughput, and CPUs have to blow their pipeline out on branch misprediction, so you want very predictable branches. It all looks good on paper, but most researchers do not have the engineering chops to validate these ideas in practice properly.
I think these pruning methods do work if you put some effort into engineering them properly.
But I came here to agree about most researchers having no interest in actual inference time performance. I just tried a library that was meant to be a "drop in replacement" for an embedding table that as meant to use a lot less memory, and after a little bit of fiddling, yes, it was a good drop in replacement for an embedding table if all you wanted to do was write papers and compute compression rations using theoretical bits needed. In practice, only the training/eval version of the code was written and nobody had actually written the the theoretically possible efficient inference path in the 3 implementations I looked at, and all the numbers in the paper were on theoretical memory savings, so my memory usage actually went up...
I don't see how they would work, TBH. I wrote a good amount of low level, accelerated code for deep learning kernels, and I've yet to see a case where sparse stuff is faster than dense. Moreover, based on my knowledge of low level details, I don't see how typical "academic" pruning can be made fast if you aren't using model-specialized hardware. The way you make things fast on CPU/GPU/TPU is by loading as wide a vector as possible, having as few branches as possible, and helping your prefetcher as much as possible. Sparsity gets in the way of all three, _especially_ on the GPU.
