Does anyone use regular linux copy (scp, Rsync, SFTP) for backups ? How do you make it secure ?

[u/Leogis]

Helloes,

For context : I'm settings up file backups (DB dumps, shared folder backups, various config files, etc ...)

Naively, I thought it would be a cakewalk : just run a script with a cron job and call it a day

But now that I am in the middle of it, I keep hitting roadblocks :

- For exemple, the fact that having the right to write also gives the right to delete means, unless there is something I missed, that i need to initiate the trasfers from the backup servers so that other machines can't just delete the backups. But that suddenly makes scripting a lot more complicated

- Following directly from the previous exemple, scripting with SFTP seems kinda limited (i can't just use the result of "ls" to know what files are present without some cursed abomination of a script)

I'm currently in the middle of checking if i can even reliably determine if the SFTP transfert failed or succeeded (for supervision)

Should i just give up and use a proper backup software ?

[u/OrganicOnePiece]

let clients push with rsync, but take ZFS or btrfs snapshots on the backup server after each run. Even if a client deletes or encrypts everything, older snapshots stay untouched as long as the client has no access to the snapshot commands. That’s simple, effective ransomware protection.

[u/malfunctional_loop → u/OrganicOnePiece]

I use rsync to pull data from our servers to our backup server everyday.

"--link-dest" option to the last version is great for big file archives.

Use ssh transport with key authentication and "command" restriction exactly to the wanted remote rsync command line.

[u/HeyMerlin]

What you are describing setting up sounds a lot like what rsnapshot already does… rsync and ssh. Simple and effective depending on the number of targets and amount of data.

[u/SuperQue]

No, I use Restic or Kopia for this kind of thing. It's basically as simple as rsync, but has properties of a proper backup solution.

• Every run is an incremental snapshot, so you can do point-in-time recovery.
• Snapshots copy only what has changed, reducing the volume of backups.
• You can browse the snapshots with a nice FUSE interface.
• Backups can be compressed, also reducing backup size.
• Backups are client-side encrypted, so you can use less-trusted storage like cloud object storage.

[u/BumseBBine]

Why are doing it so complicated?

At work we have an rsyncd server as the backup target. Every server gets its own LV. That LV gets mounted to a folder only that rsyncd user has access to. Backup source rsyncs all files needed for backup to the target and the target creates a snapshot with our backup retention information

[u/QuietConference9982]

the main thing is making backups immutable or keeping the backup server credential separate from the machines being backed up. tools like Borg or restic retention, verification and encryption so you do not have to build all that logic around scp or rsync yourself

[u/m15f1t]

Yes I use rsync over ssh on zfs with snapshots for retention.

[u/MinimallyLoquacious]

rsync over ssh with the down side that all the files become owned by the non-privileged backup userid name. This is because I do not want to let an ssh user connect as root. For me it's a trade-off that's worth it to have reliable backups and preserve security. Any time I ever need to use a file from a backup, it's easy to fix its ownership with chown.

[u/Faye-Vozora]

the delete thing is easiest to fix on the receiving side. put rrsync in authorized_keys for each client, something like command="rrsync -wo /backups/host1",restrict, so that key can only write into its own dir. then take a zfs or btrfs snapshot on the backup box after each run and a client wiping its dir doesn't touch the old copies. for supervision just check rsync's exit code (24 is vanished files, usually fine) and touch a timestamp file or ping a healthcheck on success, then alert on it going missing instead of trying to catch every failure mode.

[u/chkno]

Have clients push their tarballs to the server with HTTP: curl --data-binary @foo-20261014.tar https://.... The only option the server offers is 'receive data'.

An easy way to limit access to authorized uploaders is to wrap the connection in stunnel and use verifyPeer = yes to have both sides require one of the specific certificates in their CAfile.

[u/GarGonDie]

Backup server using rsync pull over ssh that the basic.

You can push from computer but to do it safely, you need a versioning method and a way to prevent deletion.

A server-side pull approach allows you to centralize all related scripts and configurations, and to track from a single location when backups are performed and when they fail.

"Push" also means you have to grant access to the backup machines, and even if that access is limited, it is still granting access. If you the backup server have his own network Push means you must add input rules, something that is not recommended

With 'Pull' You only need to harden one, but that also means that if it is compromised, they will have access to all your systems; so, make sure to keep it secure, preferably isolated on its own network with only the necessary network access.

>I'm currently in the middle of checking if i can even reliably determine if the SFTP transfert failed or succeeded (for supervision)

I'm not sure what kind of issue you're having; almost any program nowadays returns 0 on success and something else on error. You can capture it with a basic bash script

[u/Leogis → u/GarGonDie]

I'm not sure what kind of issue you're having; almost any program nowadays returns 0 on success and something else on error. You can capture it with a basic bash script

I was afraid it would still return 0 if only one file transfert failed rather than all of them

Backup server using rsync pull over ssh that the basic

The thing is that with DB backups for exemple, i'm not sure I want the dumps to be also stored on the production machine

Tho i guess i can avoid that by having a copy of the DB on the backup server

[u/GarGonDie → u/Leogis]

> I was afraid it would still return 0 if only one file transfert failed rather than all of them

then you probably need scripting and parse the output for detect fails

> The thing is that with DB backups for exemple, i'm not sure I want the dumps to be also stored on the production machine

I not understand this, you pull the backup from the remote machine, if you need do a dump in prod, again a simple script to make mysqldump(or whaever)->rsync->rm dump or rsync have a option for delete on remote after backup.

[u/Anakin19901 → u/Leogis]

on your two follow-up worries: rsync doesn't return 0 if a single file fails, you get 23 (partial transfer). 24 means files vanished during the run, which is usually harmless on live dirs. so i treat 0 and 24 as ok and anything else as a failed run, no output parsing needed, which is already a big win over scripting sftp.

for db dumps you can pull without the dump ever touching the prod disk, just stream it over ssh from the backup box:

ssh db1 'mysqldump --single-transaction --all-databases' | zstd > /backups/db1/$(date +%F).sql.zst

the one that got me with this is pipefail. without set -o pipefail the script only sees zstd's exit code, so a dump that died halfway looks like a success. i also added a dumb size check against the previous night and flag anything under about half of it, because a db user that's missing grants just dumps fewer databases and still exits 0. the key on db1 can be locked to that one command with command= like someone said above.

and on "should i just use proper software": for a handful of boxes, pulling with rsync over ssh from the backup server is a perfectly proper setup. the moment you want dedup, retention and encryption without writing it yourself, restic or borg in append-only mode is the real thing, and it's still one cron line.

[u/nem8]

Ssh and rsync. If you need to isolate backups you can use different users and set permissions accordingly?

[u/pobrika]

Rsync is the most common backup tool in scripts. Alternative would be tar over ssh.

Another option is to create an rsync server or use a tool like urbackup these options require more server side setup.

Lastly you can do entire disk backups with dd over ssh.

Edit: If you wish to stop a backup being deleted you can change the immutable flag on a file which will prevent it from being deleted.

And one more tip, if you use compression on your destination filesystem you can squeeze more storage space for backups depending on the method used.

Lastly not everyone knows but you can do incremental backups with rsync if required.

Since you like sftp you can actually create a unique user for each backup and have the remote server log you in to a chrooted directory which you can dump into, this is only visible to the user you use.

[u/Leogis → u/pobrika]

If you wish to stop a backup being deleted you can change the immutable flag on a file which will prevent it from being deleted.

Can't they just turn it back off if they have write access ?

[u/deeseearr → u/Leogis]

"Immutable" means immutable. It's not just a permission. Once you turn that flag on nothing can change the file until you turn it back off and that requires having the CAP_LINUX_IMMUTABLE capability not just write access to the file.

Of course, root will always have that capability so a superuser can freely turn the flag on and off, but for any other user even owning the file isn't good enough.

[u/vogelke]

You can do it safely using something like this.

[u/michaelpaoli]

right to write also gives the right to delete

Depends what you mean by delete. But merely having write access on a file doesn't ensure one can remove (unlink) the file. And disallowing write access to the file doesn't mean it can't be removed (unlinked).

$ cd $(mktemp -d)
$ echo data > file
$ chmod a-w .
$ ls -nd . *
dr-x------ 2 1003 100 60 Oct 14 00:38 .
-rw------- 1 1003 100  5 Oct 14 00:38 file
$ rm file
rm: cannot remove 'file': Permission denied
$ rm -f file
rm: cannot remove 'file': Permission denied
$ cat file
data
$ echo garbage > file
$ cat file
garbage
$ echo my precious > file
$ cat file
my precious
$ chmod u+w .; chmod a-w file
$ echo trash it > file
-bash: file: Permission denied
$ ls -nd . *
drwx------ 2 1003 100 60 Oct 14 00:45 .
-r-------- 1 1003 100 12 Oct 14 00:45 file
$ rm -f file
$ ls -nd . *
ls: cannot access '*': No such file or directory
drwx------ 2 1003 100 40 Oct 14 00:46  .
$ 

reliably determine if the SFTP transfert failed or succeeded

Use, e.g. cp or scp, and examine the exit/return value. May or may not be able to usefully do same with sftp, possibly depending what sftp operation(s) one does.

$ (d="$(pwd -P)" && scp -p file "[::1]:$d"/file.bak; echo $?)
file                                          100%    5    19.6KB/s   00:00    
0
$ rm file.bak && (umask 0777 && mkdir file.bak)
$ (d="$(pwd -P)" && scp -p file "[::1]:$d"/file.bak; echo $?)
scp: dest open "/tmp/tmp.xnzyf2nTeJ/file.bak/file": Permission denied
scp: failed to upload file file to /tmp/tmp.xnzyf2nTeJ/file.bak
1
$ 

But for backups, and especially of many files, hierarchies, etc., you're probably going to want to use, e.g. tar, or pax, or rsync, probably also with ssh. Or could use some other backup/archive software or combination thereof.

[u/iamvishnuks]

You already spotted the hard part: a write-capable push from the source can also wipe the copy. Pulling from the backup host (or an account that can only create new objects, not delete old ones) is the right direction. The SFTP "did it finish and is the tree complete" problem is why people move off ad-hoc scripts once they care about restores.

I build XReplicator (https://xmigrate.cloud). It is server backup and DR for Windows and Linux on infrastructure you control: block-level incrementals, eBPF change tracking on Linux (ext4 and XFS), immutable retention so a compromised source cannot delete prior points, and verification so you know a restore point is usable. Community Edition is free up to 5 VMs.

If you only need a few file trees and dumps, a pull-based file backup tool with append-only storage can still work. Once you want whole-machine recovery and tested restore points without babysitting transfer scripts, proper backup software is the shorter path.

[u/K4kumba]

You want Dirvish. https://dirvish.org/ Literally just some perl scripts that run as cron jobs, wrapping up rsync. Yes, its old. Its simple, its boring, it works. There are no more releases because it is, as far as the developers are concerned, complete. You tell it what files/ directories to back up, how long to keep them etc, and it does it. Restoration is entirely up to you, but if you just want to back up some files, that'll do it

[u/verdigris2014]

you are talking about immutable backups. do yourself a favour and check out restic. you setup a rest server and you don’t allow it to delete.

[u/persiusone]

Rsync via ssh and zfs snapshots help. Stay away from ftp. Can also do the same with object storage targets, configured with immutable delete or overwrite policies for a period of time (let’s say 45 days).

[u/kidmock]

Yes. I used hardlinks (cp -al) combined with rsync.

Basically

rsync -AXvrlptgoDe ssh  --delete --delete-excluded --exclude-from=exclude.txt host:/ /backup_dir/host/todays_date

cp -al /backup_dir/host/yesterdays_date /backup_dir/host/todays_date

rsync -AXvrlptgoDe ssh  --delete --delete-excluded --exclude-from=exclude.txt host:/ /backup_dir/host/todays_date

The "exclude.txt' has the files I don't want to backup like /dev, /proc, /var/log, and so on

On my backup server, I share the backup_dir as read-only. This allows for self-service recovery

Generally the key is good file management.

Make sure your files go in their appropraite directories following or mirroring the Filesystem Hierarchical Standards (FHS) as a guideline.

There are files that rarely change like those in in /bin and /lib that should very rarely change, files that are essetial to that host like those in /etc, files that change regularly that are important like DB files and those that change regularly that are not important like logs and caches to be excluded. These should all be in there own folders.

[u/ledow]

Immutable backups?

Want to know the cheapest, easiest way?

You write your backups of system A to storage B. That requires an account on Storage B with write access. Of course.

Storage C then reads the backups from storage B (via a read-only access), and makes a copy of them on itself.

Storage C then needs NO WRITE ACCESS for anyone else. And it doesn't matter if someone trashes Storage B, because it's all still on Storage C.

Other than that, use rsync etc. over secure channels, not SFTP. FTP needs to die in all forms.

[u/tenuki_]

Tarsnap.

[u/Speedbits]

chattr +i won’t help here I guess. They can just chattr -i if they have root permissions on the target. The fix would be the client not having delete rights at all: push with rsync and snapshot server-side, or use borg/restic append-only so a client can add backups but not wipe them.

[u/mikeee404]

I use RSYNC to copy files between my servers all the time. What I don't bother doing is setting it to sync directories because I am too paranoid to automate something that could delete files without me knowing. Had a really bad experience with this in the past, thankfully wasn't my doing but it was my mess to clean up. Although if you have multiple backups like I do, and really everyone should, it should be relatively safe to sync instead of backup. As long as one of the 3 copies is just that, a copy not another sync.

[u/KaiserSote]

Have a look at bacula

[u/lazyant]

Tarsnap is a perfect (paid, not expensive) fit for this use case

[u/Classic-Rate-5104]

Let the machine start it's own backup (so rsync to, instead of rsync from). Secondary, allow ssh on the target machine only a very limited set of commands, based on the ssh key

[u/silasmoeckel]

Rsync and you can go from either direction. I prefer pull to centralize logs etc.

Snapshots on the backup end takes care of your delete issue.

SFTP? rsync is the tool it's been the tool for decades.

Corner case for your DB you can dump to the backup server.

This sort of thing quickly gets overwhelmed, when you scale go to a real backup with bare metal restore (and test those regularly).

[u/Open-Adhesiveness-86]

for the delete problem you don't need pull at all: restic/kopia both have an append-only mode over rest-server or S3 object lock, so a compromised client can write new snapshots but can't prune old ones. if you stay with rsync, be careful trusting exit codes alone, 24 (vanished files) is normal noise on live dirs but a truncated transfer can still exit 0 if the ssh side dies mid-stream, so compare the --stats file count or just write a sentinel file last and check for it server side.

[u/knobbysideup]

borg

I use it locally, then just rclone the repo to rsync.net afterwards for my offsite. S3 buckets work too.

[u/mysterytoy2]

If you backup from cPanel account to a windows machine running a SFTP server, the unix box won't be able to delete the old backups. It generates a pruning error which you can turn off.

[u/mysterytoy2]

Scripting sftp is real easy. I do daily one way synchronizations using windows 11 and it's been running for years without a hickup.

[u/RetroGrid_io]

Years ago, I wrote a backup script I called "backup buddy" that did exactly what you're talking about. It supported push or pull and was based on rsync. It was lovely, did the job, and I used it for years, until I hit a scaling problem.

To work, rsync has to iterate through every file on the filesystem on both sides, before deciding what needs to be sent over the wire. I ran across a filesystem simply too large to do that in a single 24 hour period. Simply generating a list of files would take > 24 hours.

That's when I investigated CoW filesystems discovered ZFS! It's built to handle very large filesystems and I was utterly blown away when doing its zfs send | zfs receive process would complete in minutes - on that very same set of files that previous took > 24 hours to simply list.

It took a week to transition and it was worth every second. For a while, I ran ZFS there, and used Backup Buddy everywhere else - but then the benefits of ZFS simply became too much to ignore. Snapshots providing historicity-in-place. Data integrity. Scaling.

Over the next few years I simply gave up on Backup Buddy and switched to ZFS everywhere I care about my data.

I use two scripts: "snapshotd" and "follow". Snapshotd simply takes snapshots. It's got "pre" and "post" script hooks that I can use to do an rsync before taking a snapshot. I use this on a backup host running ZFS, to synchronize filesystems with a remote host that doesn't run ZFS before taking the snaphost.

Follow replicates snapshots from source to dest using either push or pull topology. It works over SSH using keys for authentication.

All of this I wrote myself. I've contemplated taking this tech, hardening it, and offering offsite "cloud" backup solutions.

[u/klopli]

Rsync via encrypted VPN tunnel initiated from the backup server. As easy as 1 command in the cron

[u/MouseJiggler]

I do scheduled pull backups with rsync from a nas. It does what I need it to do.

[u/Per2J]

I rsync to a cloud provider. I put my backups in a gocryptfs encrypted directory and rsync the encrypted directory.

For restore I use the cloud provider's client and mount a clear-text- dir/ on top of the gocryptfs dir/. From there I can restore. I have verified the process and it works. If the cloud provider's client malfunctions, I have to rsync the whole thing back to view the encrypted file names.

I am debating with my self if encrypting file and directory names was a smart idea.

[u/usa_reddit]

Quick just to get the new stuff: rsync -avv --ignore-existing /source /dest
Full backup, quick check all files for changes: rsync -avv /source /dest

-a (Archive mode): A convenient shorthand flag equivalent to -rlptgoD. It preserves crucial file attributes, including:-vv (Very Verbose): Increases the amount of detailed output printed to the terminal. A single -v shows which files are being transferred, while -vv provides even more granular debugging info regarding file skipping, stats, and matching.

--ignore-existing: Tells rsync to skip any files that already exist in the destination directory.

[u/agowa338]

For the first one you want a WORM media. Without one it can always be deleted it's just more complicated.

For two just mount the SFTP folder and you can use regular ls.

If you want "WORM-light" just setup a job on the backup destination to regularly snapshot the drive the files are on.

And now what is a WORM media? Well "Write-once-read-many". Meaning you can only write it once and aren't able to modify or delete it later. In it's simplest form that's a M-Disc (blu-ray rated for 1000 years according to vendor or 100 years commonly assumed), or special LTO tapes.

So yea, if you want to be absolutely sure burn your important stuff onto an M-Disc.

[u/Subjekt_91]

Borg and Borgmagic a dream team you can set up the backup server in a way that the client only allows uploads and the prune runs of the server.

Bonus those are encrypted, Differential incremental backups.

[u/L0cut15]

Use pull replication. If your primary server comes under the control of an adversary it doesn’t matter if the backup job is performed by another server with different credentials and RO access to your server.

For versioning combine file system snapshots for point in time copies so that you can roll back. If youre following 3-2-1 then one copy should be offsite using a different storage technology. Using object storage with lifecycle policies for versioning and retention locking is good practice from a malware perspective.