Which is fine? It's getting capped to 3 at your modem anyways... And the difference here is reaching the "doesn't matter" phase.
Like - 50GB at 3gbps is ~130 seconds, ~160 seconds at 2.5gbps, and ~400 seconds at 1gbps
You're always waiting at least a couple minutes, and never more than 10 for that download.
Personally - I have a couple 10gbps switches that run my backbone in my house (NAS, k8s cluster, HomeAssistant, etc) and I have a 2gbps symmetric wan, but all of the offices (my wife and I both work from home, plus the media room, plus the kid's playroom) all just run 1gbps switches.
They're cheap, normal office work basically never saturates those speeds (even multiple conference calls with video is a-ok), and there's just not enough value in bumping up to 10gbps across the whole house compared to the equipment cost.
The 10gbps switches live in the basement and just connect those zones.
You are completely right that I can always just wait. I prefer not to though... there are a lot of times when it is amazing to have 10G between my main machines.
The devices attached to the 2.5G switch, will be limited to 2.5G per connected port. There won't be any slow down on any other connections. If you still want 10G in the basement, put in a 10G SFP+ switch (they can be had for $15-20 per port and use DACs to control the heat some). You'll only have one shared 10Gbps to the rest of the house, but that is really unlikely to be saturated.