I’m trying to understand the bot problem in the internet and finding more ways to defend myself. One thing that I can’t seem to understand is why most bots, scrapers and crawlers seem to have residential IPs.
- Is it that ISPs are being paid by tech-bros to assign them these IPs?
- Is it that residential devices have been hacked /contain malware that does this?
- Is it trivial for companies to assign themselves residential IPs?
- Paid volunteers are doing this for AI companies?
Or is there is some other reason for this?
Obviously this is a problem because one can rotate / cycle through residential IPs and if I aggressively block each offender in my logs permanently, then the next person assigned this IP who may be a legitimate user will be unable to access my site.
I worked as a network analyst for a provider for several years and during that time I’d say ~90% of the issue stemmed from sketchy apps/services that the user loaded from their end.
A lot of “free” VPN services will basically allow bad actors (the paid tier) to use your connection. A lot of IoT devices are also just openly available on the Internet to route through.
From the ISP perspective, we managed the roads, not your car. There is a push to blame the ISP as it’s their network, but realistically how are they meant to provide security (in the context that is being asked) to any device that gets plugged into that network? We even had business customers demand we add clauses to contracts where we would accept responsibility for any malware they sent between sites over an MPLS setup.
In the end, a lot of people seem to want this impossible scenario of the ISP managing security for them but also not inspecting their traffic.
@Mordikan @Maroon A lot of people _don’t know what their ISP does_. Many seem to think that the ISP is selling them the entire internet as a product, and so from that logic why shouldn’t the ISP be liable for whatever mayhem they get into online?
Source: worked for a little while as dial-up ISP support.
Wouldn’t even mind the option to let someone else use my connection a bit for a free VPN tbh, that is no worse than running a TOR node. What I dislike is the dishonesty side of it. Be open and honest then it’s all good.
<.< Let me just throw two scenarios at you real quick to show you why that’s a problem:
That third party can easily just use your connection to view and download CSAM and all external monitoring would suggest that activity was coming from within your network. Because it is.
That third party has full network access to anything your PC does (or at least that nic) which includes anything in your local network. Insecure IoT, other PCs, etc.
First scenario is no different to running a TOR exit node. The second is why it needs to be built securely, which can be done though probably isn’t in these cases.
It would be like running TOR, but not a relay, it would be like an exit node.
That should be enough to warn anyone away from using them.
Running an exit node is perfectly legal though. There would be no evidence you have done anything wrong very quickly.
You can just look at the testimonies from others who have run exit nodes. The cost of your “free” VPN is that law enforcement will constantly be in contact and investigating you because your network/machine is being used to download CSAM.
There is no “oh don’t worry, A.B.C.D is just a tor node, we can give it a pass”. Every time that happens, it has to be treated with a full investigation.
Let them waste their time investigating, actually how do they even know your address?
Wdym wasting their time? They will search your home and confiscate all electronics, including the hardware running your TOR exit node and your phone, and your PC, and your notebook, and your gaming consoles, and of course any of such devices belonging to all other members of your household. And even if they cannot found anything that you have done wrong, you won’t get your stuff back for a long time (think half a year to multiple years).
Seems more like they would be wasting your time. And your money.
Ok, this I can answer personally as we did multiple cases of this happening (CSAM, bomb threats, etc) at work.
So, anonymity on the Internet is not actually a thing. Whether its an IP address or telecom switch or whatever, there is a path back to you even if only for either billing or connectivity purposes. So, for IP, we would receive a subpoena signed by a judge to hand over any and all information regarding the identify of the a given IP address (they include a long list of things whether applicable or not in the order so every potential base is covered). Once legal was able to review and handed it off to us, we take that and look at the DHCP logs to see that on a given date at a given time that the IP address was assigned as part of shelf A / slot B / port C. That shelf/slot/port combination is tied physically to an address/account. We provide the relevant logs and personal information of that user to law enforcement.
For bomb threats over the phone, telecom switches love to tell every other telecom switch who they are (again, connectivity purposes). So, when you make a call to a business/school doing that, their PBX is going to log to the millisecond when that call occurred and who the switch was. Again, subpoena and we pull the SIP logs. We can even provide the RTP/RTCP packets and reconstruct the phone call audio if the subpoena asks for that.
I can pay for internet in cash and the only details I gave them are fake and a random username. 4G internet isn’t even tied to the same location as I can move freely.
As I’m not bothered about doing cybercrimes I don’t bother doing that much about security though.
4G/5G cellular? So, in some ways you’re actually easier to find. Your cell gateway is connecting to a tower which is logged and includes cell strength metrics. That gets compared to other towers and via trilateralization your location is determined.
Again, going back to what I previously said: there is a path back to you even if only for either billing or connectivity purposes.
They’d not only waste their time though, they’d also confiscate all the electronics they can find at your place