I guess what I'm doing is not considered that useful then? I usually only have zero, one, or occasionally two things actively doing inference at a time, be it claude code sessions or one of the chatgpt/claude web interfaces, and i bet that's true for like 95% of people using llms. And anyway i bet even the hardcore people using a bunch of parallel agents would appreciate having access to local, private inference for some things.
You're obviously right though that cloud inference isn't going away anytime soon
What about when claude code starts spawning sub-agents? it's pretty good about doing that now even if a user doesn't request it. Does this could as parallel inference?
I haven't seen it do it by itself that much so far. Even then, a lot of the time is spent waiting for some shell command to complete, which isn't inference. I guess being able to do parallel inference is like having multiple cores in a cpu, you can get by without it by just sharing that single core, and having it just gives you a performance boost under certain circumstances
Interesting. For me it seems to be much more common that even basic requests start spinning up sub-agents. But I also routinely ask it to do this so maybe it's stored this as a preference somewhere deep in its bowels.
And yes, I agree. I find the experience better less for speed and more for context management. But it's far from necessary.
I guess what I'm doing is not considered that useful then? I usually only have zero, one, or occasionally two things actively doing inference at a time, be it claude code sessions or one of the chatgpt/claude web interfaces, and i bet that's true for like 95% of people using llms. And anyway i bet even the hardcore people using a bunch of parallel agents would appreciate having access to local, private inference for some things.
You're obviously right though that cloud inference isn't going away anytime soon