A year of fighting scrapers on my 1.5 million-page website. What I tried, how I failed, and what's working now.

A year of fighting scrapers on my 1.5 million-page website. What I tried, how I failed, and what’s working now.

  • fubarx@lemmy.worldEnglish
    5·
    3 days ago

    I thought Cloudflare’s anti-AI defenses were meant to dynamically catch all of this. That’s a lot of work to do to keep bot traffic away.

    FWIW, I tried loading it with a VPN, it showed the content, but not all the images. Turned off VPN, it fully loaded. Maybe the blog has different rules.