Proxmox is sort of the gold standard for homelab server operating systems. Runs containers and VMs.
If you’re not into Proxmox, look into Fedora Server with Cockpit. Web UI for server management. Fedora CoreOS is an immutable variant of Server that would make more sense for a hypervisor, IMO.
First and foremost, backups. Back up everything and back up often. Immutability can’t do anything for critical hardware failure.
Issues happening on something only running container workloads isn’t common but I think it’s worth the extra little effort to reduce the risk even further. Fedora CoreOS or Flatcar is ideal since its declarative nature makes it easily reproducible. Fedora IOT can get you there too, but it doesn’t use ignition so you’ll be setting the server up manually.
Immutability is good. Declarative configuration is good. Manage cattle, not a pet.
Traditional RAID isn’t very flexible and is meant/easiest for fresh disks without data. Since you’ve already got data in place, look into something like SnapRAID.
Intel Arc A310. They’re $100, support AV1 and powered completely by the PCIe bus. Combine it with Tdarr and you can compress your media library down to half the size easily while still being able to easily stream to any device you have.
Nearly any SBC you’d buy would beat the pants off it. If you’re shopping by price then check out a Libre Computer Sweet Potato or Renegade, or a Friendly Elec NanoPi R2S+. They’re <=$40 and should be able to run at least the services you mentioned. If you have more budget, there are $100 mini PCs on Amazon that are great for self hosting tons of stuff, like a Bmax B1 Pro.
Something headless for just running containers? Alpine.
It’s small, boots fast, simple, can run from RAM and Docker is available in its software repository.
An HP Elitedesk mini PC would be small enough to tuck away somewhere and have hardware accelerated transcoding support. Jellyfin recently enabled hardware acceleration for the latest Rockchip boards like the Orange Pi 5 so that’s an option too. If you pre-encode your media into a format compatible with everything you want to stream to then it doesn’t matter, just pick any hardware than can get on your network and run Linux.
Deploy code-server and either connect to it with a VPN or open the port needed to connect over the internet.
I’ve used lots of different boards. The Radxa Rock 3c is cheap and has decent performance, but the official OS support is a bit old. The Libre Computer boards are also good and have Armbian support. Libre Computer is releasing a couple more this year too. BananaPi has good options that aren’t expensive, like the BananaPi M5. Friendly Elec has some boards like the NanoPi R2C and R5C that aren’t pricey and have Armbian support. Any one of these boards are fine for a small home lab. Just boot Armbian, install Docker, and add your containers.
Give webtop a try? Granted I haven’t tried anything heavy on it, but it’s been performant enough for me. Here’s a compose file if it stays formatted correctly:
services: webtop: image: lscr.io/linuxserver/webtop:latest # alpine - xfce # other tags with different bases and desktops: https://github.com/linuxserver/docker-webtop container_name: webtop #security_opt: # - seccomp:unconfined #optional environment: - PUID=1000 - PGID=1000 - TZ=America/Los_Angeles - TITLE=my_desktop #optional volumes: - config:/config #- /var/run/docker.sock:/var/run/docker.sock #optional ports: - 3000:3000 - 3001:3001 restart: unless-stopped volumes: config: {} networks: {}