EDIT
TO EVERYONE ASKING TO OPEN AN ISSUE ON GITHUB, IT HAS BEEN OPEN SINCE JULY 6: https://github.com/LemmyNet/lemmy/issues/3504
June 24 - https://github.com/LemmyNet/lemmy/issues/3236
TO EVERYONE SAYING THAT THIS IS NOT A CONCERN: Everybody has different laws in their countries (in other words, not everyone is American), and whether or not an admin is liable for such content residing in their servers without their knowledge, don’t you think it’s still an issue anyway? Are you not bothered by the fact that somebody could be sharing illegal images from your server without you ever knowing? Is that okay with you? OR are you only saying this because you’re NOT an admin? Different admins have already responded in the comments and have suggested ways to solve the problem because they are genuinely concerned about this problem as much as I am. Thank you to all the hard working admins. I appreciate and love you all.
ORIGINAL POST
You can upload images to a Lemmy instance without anyone knowing that the image is there if the admins are not regularly checking their pictrs database.
To do this, you create a post on any Lemmy instance, upload an image, and never click the “Create” button. The post is never created but the image is uploaded. Because the post isn’t created, nobody knows that the image is uploaded.
You can also go to any post, upload a picture in the comment, copy the URL and never post the comment. You can also upload an image as your avatar or banner and just close the tab. The image will still reside in the server.
You can (possibly) do the same with community icons and banners.
Why does this matter?
Because anyone can upload illegal images without the admin knowing and the admin will be liable for it. With everything that has been going on lately, I wanted to remind all of you about this. Don’t think that disabling cache is enough. Bad actors can secretly stash illegal images on your Lemmy instance if you aren’t checking!
These bad actors can then share these links around and you would never know! They can report it to the FBI and if you haven’t taken it down (because you did not know) for a certain period, say goodbye to your instance and see you in court.
Only your backend admins who have access to the database (or object storage or whatever) can check this, meaning non-backend admins and moderators WILL NOT BE ABLE TO MONITOR THESE, and regular users WILL NOT BE ABLE TO REPORT THESE.
Aren’t these images deleted if they aren’t used for the post/comment/banner/avatar/icon?
NOPE! The image actually stays uploaded! Lemmy doesn’t check if the images are used! Try it out yourself. Just make sure to copy the link by copying the link text or copying it by clicking the image then “copy image link”.
How come this hasn’t been addressed before?
I don’t know. I am fairly certain that this has been brought up before. Nobody paid attention but I’m bringing it up again after all the shit that happened in the past week. I can’t even find it on the GitHub issue tracker.
I’m an instance administrator, what the fuck do I do?
Check your pictrs images (good luck) or nuke it. Disable pictrs, restrict sign ups, or watch your database like a hawk. You can also delete your instance.
Good luck.
There’s one more option. The awesome @db0@lemmy.dbzer0.com has made this tool to detect and automatically remove CSAM content from a pict-rs object storage.
https://github.com/db0/lemmy-safety
This is a nice tool but orphaned images still need to be purged. Mentioned on the other thread that bad actors can upload spam to fill up object storage space.
That is also very true. I think better tooling for that might come with the next pict-rs version, which will move the storage to a database (right now it’s in an internal ky-value storage). Hopefully that will make it easier to identify orphaned images.
I tried getting this to run in a container, but I was unable to access my GPU in the container. Does anyone have any tips on doing that?
Sorry I haven’t ran this myself yet nor have any experience with that kind of issues. But may I ask why you were concerned with running it inside of a container? Seems rather unnecessary to me.
Running anything in a container isn’t necessary. It just makes it easier to run, as it comes with all the dependencies. And if you decide you don’t want it anymore, you can just remove the container and it and all its dependencies are gone, which is really clean. It also makes the environment extremely repeatable, so people on all distros can run it with the exact same steps. And you don’t need to worry about what version of python you have and if it’s compatible with the dependencies. For example, the dependencies for this script require python 3.10 exactly. You can’t use 3.9 or 3.11.
So really the only reason was I wanted to make it easier for everyone.
I see. I considered the dependency problem but only thought of using a venv to fix that, however you are right, the python version is also often the cause of compatibility issues.
You can go one step further and use a conda env, which would also include the proper python version. All you need then is the micromamba binary. I might develop that as all it would need is to run a shell script to start
Honestly, I’m a little sick of needing to make a venv for each python script, which is why I’m trying to put all my python scripts in containers. I already got db0’s project to the point where anyone can run it with one single command line that you can just copy/paste (assuming you have docker installed already). It is just running on the CPU, which is painfully slow.
Same, it’s the reason why I can’t stand working with python.
Thank you for doing this, btw. Once you have something working on your hands you could consider spreading the word, maybe to db0 himself. I sure would love a convenient way to run that script, and many other admins probably would too.
Not sure how you’re trying to run it in a container, but the answer would depend on a bunch of different factors. Nvidia has a utility you can install that assists in exposing the GPU to the container, documentation found here.
If you’re using docker compose to run it as a service, there’s a doc page for that too. Note that it uses the previous page I mentioned as prerequisite.
There’s another way to get it working from within kubernetes that comes up every now and then on stackoverflow.
If it’s Intel or AMD, no idea if this still applies.
Yes, this is exactly what I had trouble with. The Nvidia container runtime seems to not support my distro. But even when I tried running it on my Ubuntu machine, I was getting tons of dead links using Nvidia’s instructions. And even when I fixed the links. I was getting issues like the apt repository was throwing errors. IIRC, it was some kind of signature issue, and I’m not sure I want to ignore that, especially considering I had to fudge the URL.
I’m thinking the best option is to build from source, but I don’t think that’s easier than just running this in a non-container.
You need a GPU for that. Most $5 VPSs don’t have that.
Yeah I know. It’s supposed to be ran from your computer, not the VPS.
Would I mount the the pictrs folders as a network folder locally?
No. Unfortunately it only works with storages on object storages like S3 buckets, not with filesystem storages. Meaning it access the files remotely one at a time from the bucket, downloading them over the internet (I assume, I didn’t make this).
But the more important thing is that, as it states in the readme, no files get saved to your disk, they only stay in your RAM while they are being processed and everything is deleted right after. This is relevant because even having had CSAM on your disk at some point can put you in trouble in some countries, with this tool it never happens.
Which btw is the same reason why mounting the pict-rs folder to your local computer is probably not a good idea.
theoretically this tool could be adjusted to go via scp and read your filesystem pict-rs storage as well, Just someone has to code it.
Interesting. That would be a nice extension, I think most small admins are using the filesystem (I know I am lol).