You don't need to flush predictor state on every task switch, only when entering a less private context from a more private one. Systems with a feature known as multiple-level security (or "multiple levels of security") can be designed to track these privacy/information requirements automatically.