Most have internal batteries and are still "on" to a certain extent unless the battery is completely discharged.

It's "on" enough to detect the activity needed to wake it back up. But will a powered-off phone still be pinging cell towers or making WiFi requests?

Yes, that's how Find My works.

If you cover your phone with an antielectrostatic bag it can't communicate; that is a Faraday cage.

Since people around you will think you are also wearing a tinfoil hard, you had better stick to the phones with hardware switches as sibling comment mentions

Most of those bags are total BS

Seems easy to verify with an iPhone. Put it in the bag, transport the phone, look it up on Find My.

Even easier, put any phone in the bag, call it. Does it ring? IME, yes.

Use one of the few phones with hardware kill switches or removable batteries.

I get that anything emitting can be tracked and stuff. I'm looking to take a baby step where I'm at least not having every possible detail recorded and sold. That Phreeli recommendation from another user seems like exactly what I want (paired with other things like a VPN of course).