I run some services on machine A. I want to backup the data to machine B. It’s around 40 gigabytes in thousands of small files. I would prefer to compress the data, but there’s not enough space on machine A to keep the data twice (original + tar archive). I would like to avoid copying that many files via scp/rsync since it’s taking forever and a bit fragile.

Any solutions on how to solve this? Is it possible to compress live on machine A and stream into the archive on machine B without the need to keep the big archive file on A?

Eventually, it’s supposed to be automated and B has like the last 3 dailys, 1 weekly, 1 montly.

B can ssh into A.

  • tofu@lemmy.nocturnal.gardenOP
    1·
    6 days ago

    It’s a VPS and I don’t want to increase the monthly cost - in fact I want to decrease it by disabling the backups the hoster offers. Machine B is my NAS at home.

    Of course, incremental would be preferable, but I’m not sure how that would work in that constellation.

    I wonder if I can compress to B directly. Maybe I could mount a volume from B into A via wireguard and nfs, now that I think about it.

    • redlemace@lemmy.world
      2·
      6 days ago

      Do you have unlimited data transfer with your vps? Transferring 40Gb three times each and every day is quite some traffic.

      Why not host it at home then? I bought a used lenovo thinkcenter (m720q on top of my head) for 75€ It uses only 35 watt, it has 750 Gb storage and I only added some RAM.

      I pay 1€ per month for a fixed public ip to avoid ddns. My NAS is at a family member (over vpn to home) for offsite backups.

      • tofu@lemmy.nocturnal.gardenOP
        2·
        6 days ago

        I do have a someserver, but it’s a fediverse instance with around 60 monthly active users, and the server provider is more reliable than my ISP at home, so any downtime has a bit more impact. There’s also another admin who needs access. Not saying never, but at the moment I’m just looking for a different backup solution, moving the whole thing has some more aspects to consider.

        The data would only be copied once a day. I have to check the limits though, good point.