There are fundamental limits to prediction. Characterizing these limits is an essential part of algorithmic information theory[0] (AIT) and is an operative bound on AI improvement. One of the strongest arguments against "hard take-off" scenarios is that AIT constrained by physics doesn't really allow it.

[0] https://en.wikipedia.org/wiki/Algorithmic_information_theory