DISCLAIMER: I never used Tailscale. All I know about Tailscale I learned reading their “How it works” blogpost and documentation, because I wanted to understand the hype.

Since nobody answered your questions, I’ll try my best. Just trust that I spent most of the last 25 years configuring security systems, including but not limited to VPNs.

Hmm, I guess my question would be how does this all work?

See my 2 links above.

I mean, is it not possible to configure STUN/DERP services yourself?

Of course it is, but it will be additional work, that most users are not willing/confident to do and Tailscale provides this service.

Or add control lists yourself? […] For ACLs, I guess Apparmor and/or SELinux profiles would be configured?

Deploying network ACLs on your hosts indeed does not require you to use Tailscale. However they provide an centralised way to manage and deploy them, without worrying about the underlying OS and ACL system. Or even requiring you to have access to the host, it could be an authorised user trying to access your Tailscale network.

Note: AppArmor/SELinux are more “system/process ACLs”, not directly related to network ACLs. I’m oversimplifying a lot, they’re difficult to describe without knowing your sysadmin skills.

The removing a key I can understand why it’s be a nightmare yourself, but how does Tailscale do it where it’s just so simple?

Simple: they ask you to run an agent on all of your Tailscale hosts and connect to their centralised platform. To paraphrase their blogpost: config management is centralized, but that doesn’t matter because it carries virtually no traffic. It just exchanges a few tiny encryption keys and sets policies. The VPNs and their traffic are a distributed mesh.

EDIT: Another question I have is how does Tailscale work when I have a VPN for securing network traffic when browsing the internet etc.? Or is that just seamless?

I’m not sure to understand this question, so I’ll make an asumption: you’re asking what happens if you run Tailscale on a host that already has a VPN configured to access the Internet.

Tailscale (and Wireguard under it) is already a VPN solution, and tunneling a VPN inside another VPN is generally discouraged. But as Tailscale is providing STUN/DERP, if they manage correctly the MTU issues and things like that, I don’t see an immediate reason why it should not work at all.

You can configure Tailscale or Wireguard to create a VPN to access the Internet though.

Once again, if you try to understand how Tailscale works, please read the links at the start of this post. RTFM, kids!

On a more personal opinion, I find their solution clever and elegant. If I have the need for a distributed VPN solution in the near future, I will definitively consider it (or Headscale’s). For the moment, I’m fine with all my hosts connecting to my homelab, configuring a Wireguard tunnel for each roaming host, and opening ports and creating rules on my firewall. Compared to IPSec or OpenVPN tunnels, it seems almost too easy each time.