A year of fighting scrapers on my 1.5 million-page website. What I tried, how I failed, and what’s working now.

  • CodexArcanum@lemmy.dbzer0.com
    link
    fedilink
    English
    arrow-up
    12
    ·
    3 days ago

    I work on an ecomm site and it’s nightmarish. We have all the cloudflare protections turned on but do try to tune out system so american AI companies will scrape us (we want chatgpt to send us customers and use our data).

    Still, we have a country allow list because it’s easier to admit North American traffic and the specific IPs our contractors use, then block everyone else. The amount of bot traffic flooding in globally is insane. I don’t think the current day internet infrastructure is going to be able to absorb the abuse much longer.