That’s funny, cloudflare is pretty easy to get around

Google uses its monopoly position in search to evade crawler blocks that AI companies get caught in. Site owners need Google crawlers, but Google won’t separate the AI and search crawlers.

Not surprising at all most small sites and blogs get 50k bot hits a day

I had to set up fail2ban + bot detection just to block the suckers. Its mostly amazon and openai for me and my small sites. Google/bing look at robots.txt and respect it. Openai in particular slams any site to the ground quick…