FOSS infrastructure is under attack by AI companies
LLM scrapers are taking down FOSS projects' infrastructure, and it's getting worse.
You're viewing a single thread.
I too read Drew DeVault's article the other day and I'm still wondering how the hell these companies have access to "tens of thousands" of unique IP addresses. Seriously, how the hell do they have access to so many IP addresses that SysAdmins are resorting to banning entire countries to make it stop?
If you get something like 156.67.234.6, then 7, then 56 etc just block 156.67.0.0/24
Sure, network blocking like this has been a thing for decades but it still requires ongoing manual intervention which is what these SysAdmins are complaining about.
There are residential IP providers that provide services to scrapers, etc. that involves them having thousands of IPs available from the same IP ranges as real users. They route traffic through these IPs via malware, hacked routers, "free" VPN clients, etc. If you block the IP range for one of these addresses you'll also block real users.
There are residential IP providers that provide services to scrapers, etc. that involves them having thousands of IPs available from the same IP ranges as real users.
Now that makes sense. I hadn't considered rogue ISPs.
It's not even necessarily the ISPs that are doing it. In many cases they don't like this because their users start getting blocked on websites; it's bad actors piggy-packing on legitimate users connections without those users' knowledge.