I actually just added a full-screen video popup to a site a few days ago, and I made it so that it won't appear to googlebot. This wasn't for deceptive purposes, just because it wasn't relevant. I only show the video once to users, and it doesn't appear when they go back to the site...it's just a walkthrough of our products, which many people seem to need. I suspect many sites will check for googlebot and not show their popups to avoid this penalty.
In my own case, I'm showing the same thing as users who have been there > 1 times. I could remove the check and it wouldn't make much difference, as gbot uses cookies anyway.
For sites that always show the popup to users and never show it to gbot, I agree. However I've seen a lot of scammy sites that do this kind of time so I don't know how likely google is to find out about it.
You're not getting "caught", and Google can fuck right off with any "terms of service" that I never agreed to about how I respond when they scrape my site without permission.
I know that there are practical concerns here, but Google has done a masterful job of shaping the conversation so that many people think anything not sanctioned by Almighty Google with regard to SEO (let alone actual "blackhat" tactics) is illegal or immoral.
It's not illegal, but if you're hiding content from Googlebot because you don't want it to derank that page, once they crawl it with an agent that hides the fact that it's Google, they will then derank your whole website.
So it's not illegal, but if you think you're beating Google by doing it, they're well aware of this workaround and will punish you harder.
If you don't care about ranking, then no biggie. But why would you manipulate your web server like that if you weren't trying to rank.
No one needs permission to request a web page, and that includes bots. But if that still bothers you, why not create arobots.txt with your preferences?
You’re giving permission to scrape by way of having a page accessible to the general public. If googlebot is bothering you so much, install a two line robots.txt and it’ll go away.
I'm not so sure about that. They change up things a lot. Search is only part of the equation however, people can still buy an add to show you anything they like. So these practices aren't going away, they're just pushed into an arbitrage situation.
