Top
Best
New

Posted by indigodaddy 3 days ago

4x faster network file sync with rclone (vs rsync) (2025)(www.jeffgeerling.com)
201 points | 99 commentspage 2
xoa 7 hours ago|
Thanks for sharing, hadn't seen it but at almost the same time he made that post I too was struggling to get decent NAS<>NAS transfer speeds with rsync. I should have thought to play more with rclone! I ended up using iSCSI but that is a lot more trouble.

>In fact, some compression modes would actually slow things down as my energy-efficient NAS is running on some slower Arm cores

Depending on the number/type of devices in the setup and usage patterns, it can be effective sometimes to have a single more powerful router and then use it directly as a hop for security or compression (or both) to a set of lower power devices. Like, I know it's not E2EE the same way to send unencrypted data to one OPNsense router, Wireguard (or Nebula or whatever tunnel you prefer) to another over the internet, and then from there to a NAS. But if the NAS is in the same physically secure rack directly attached by hardline to the router (or via isolated switch), I don't think in practice it's significantly enough less secure at the private service level to matter. If the router is a pretty important lynchpin anyone, it can be favorable to lean more heavily on that so one can go cheaper and lower power elsewhere. Not that more efficiency, hardware acceleration etc are at all bad, and conversely sometimes might make sense to have a powerful NAS/other servers and a low power router, but there are good degrees of freedom there. Handier then ever in the current crazy times where sometimes hardware that was formerly easily and cheaply available is now a king's ransom or gone and one has to improvise.

Dunedan 6 hours ago||
I wonder if the at least partially the reason for the speed up isn't the multi-threading, but instead that rclone maybe doesn't compress transferred data by default. That's what rsync does when using SSH, so for already compressed data (like videos for example) disabling SSH compression when invoking rsync speeds it up significantly:

  rsync -e "ssh -o Compression=no" ...
dspillett 52 minutes ago||
IIRC rsync uses your default SSH options, so turning off compression is only needed if your default config explicitly turns it on (generally or just for that host). If sending compressible content using rsync's compression instead of SSH's is more effective when updating files because even if not sending everything it can use it to form the compression dictionary window for what does get sent (though for sending whoe files, SSH's compression may be preferable as rsync is single threaded and using SSH's compression moves that chunk of work to the SSH process).
nh2 5 hours ago||
Compression is off by default in OpenSSH, at least `man 5 ssh_config` says:

> Specifies whether to use compression. The argument must be yes or no (the default).

So I'm surprised you see speedups with your invocation.

Dunedan 3 hours ago||
Good point. Seems like I enabled it in ~/.ssh/config ages ago and did forget about it. Nonetheless, it's good to check whether it's enabled when using rsync to transfer large, already well compressed files.
aidenn0 6 hours ago||
rclone is not as good as rsync for doing ad-hoc transfers; for anything not using the filesystem, you need to set up a configuration, which adds friction. It realy is purpose built for recurring transfers rather than "I need to move X to Y just once"
ruuda 6 hours ago|
We wrote https://github.com/chorusone/fastsync for fast ad-hoc transfers over multiple TCP streams.
KolmogorovComp 7 hours ago||
Why are rclone/rsync never used by default for app updates? Especially games with large assets.
rjmunro 6 hours ago|
zsync is better for that. zsync precalculates all the hashes and puts them in a file alongside the main one. The client downloads the hashes, compares them to what it has then downloads the parts it is missing.

With rsync, you upload hashes of what you have, then the source has to do all the hashing work to figure out what to send you. It's slightly more efficient, but If you are supporting even 10s of downloads it's a lot of work for the source.

The other option is to send just a diff, which I believe e.g. Google Chrome does. Google invented Courgette and Zucchini which partially decompile binaries then recompile them on the other end to reduce the size of diffs. These only work for exact known previous versions, though.

I wonder if the ideas of Courgette and Zucchini can be incorporated into zsync's hashes so that you get the minimal diff, but the flexibility of not having a perfect previous version to work from.

plagiarist 5 hours ago||
Do a CRDT but for binary executables
packetlost 7 hours ago||
I use tab-complete to navigate remote folder structures with rsync all the time, does rclone have that?
nh2 5 hours ago|
This is not a feature of rsync, but of your shell.

So the question "does rclone have that" doesn't make much sense, because it usually wouldn't be rclone implementing it.

For example, zsh does it here for rsync, which actually invokes `ssh` itself:

https://github.com/zsh-users/zsh/blob/3e72a52e27d8ce8d8be0ee...

https://github.com/zsh-users/zsh/blob/3e72a52e27d8ce8d8be0ee...

That said, some CLI tools come with tools for shells to help them implement such things. E.g. `mytool completion-helper ...`

But I don't get rclone SSH completions in zsh, as it doesn't call `_remote_files` for rclone:

https://github.com/zsh-users/zsh/blob/3e72a52e27d8ce8d8be0ee...

rurban 5 hours ago||
Thanks for the lms tips in the comments. Amazing!
tonymet 2 hours ago||
golang concurrent IO is so accessible that even trivial IO transform scripts (e.g. compression, base64, md5sum/cksum) are very easy to multicore.

You'd be astonished at how much faster even seemingly fast local IO can go when you unblock the IO

baal80spam 8 hours ago||
I'll keep saying that rclone is a fantastic and underrated piece of software.
digiown 7 hours ago|
rclone is super cool, but unfortunately many of the providers it supports has such low ratelimits, that it's fairly difficult to use it to transfer much data at all.
plasticsoprano 7 hours ago||
This has been my problem. Not necessarily that the rate limits are low, many can be gotten around by using multiple users to do the work since the limits are per user, but how rclone handles those rate limits when they hit them. The exponential back off will end up making hours and days long delays that will screw a migration.
PunchyHamster 3 hours ago||
I had to tweak some options because it was hitting ratelimit on B2 and the reaction of rclone to that was just... disabling deletes
gjvc 5 hours ago|
May 6, 2025 May 6, 2025 May 6, 2025 May 6, 2025 May 6, 2025 May 6, 2025 May 6, 2025