Is there any rclone option that will accelerate cloud-to-cloud syncing? (Now with log and a new idea or two)

Help and Support
1

What is the problem you are having with rclone?

I'd like to accelerate the syncing of one directory on a remote to another directory on the same remote. Admittedly, the first of those two directories contains gigabytes. But all I'm after is an answer to the question in this post's title. I know that, for local to cloud transfers, and indeed perhaps vice-versa i.e. cloud-to-local, one can adjust checkers, bandwidth and more. I presume though that those options are irrelevant to cloud-to-cloud operations. I suppose I might be able to buy some fancier package from b2 but I was wondering whether any rclone option could help. (I have not determined the speed of transfer presently, and perhaps that speed is not low given the amount of materials, but, still. I add that mostly the files at issue are smaller than a megabyte.) EDIT: on reflection, I think that these options will help: checkers;; fast-list; transfers. (I was thinking that all the work would be done on the remote, but it will not.) Also: would rclone delete combined with rclone copy be faster, at least in the sense of returning control to the user, i.e. getting the local stuff done?

What is your rclone version (output from rclone version)

1.53.3

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Linux Mint 20.1 x64

Which cloud storage system are you using? (eg Google Drive)

B2.

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone sync <directory-on-remote> <another-directory-on-the-same-remote --quiet --stats=0

The rclone config contents with secrets removed.

Irrelevant, I think.

A log from the command with the -vv flag

Here is a sample.

2021/01/17 22:00:23 DEBUG : rclone: Version "v1.53.3" starting with parameters ["rclone" "sync" "enc-b2:x1/current" "enc-b2:x1/fallback_sync" "--checkers=12" "--fast-list" "--log-file" "/tmp/rclone_fallback_C8z" "-vv"]
2021/01/17 22:00:23 DEBUG : Using config file from "/home/<myUserName>/.config/rclone/rclone.conf"
2021/01/17 22:00:23 DEBUG : Creating backend with remote "enc-b2:x1/current"
2021/01/17 22:00:24 DEBUG : Creating backend with remote "b2:x1-everything/3tma9e10hndcg550rmtolqn99s/vk34smf5cjhe9pjaue55h5vpuk"
2021/01/17 22:00:25 DEBUG : Creating backend with remote "enc-b2:x1/fallback_sync"
2021/01/17 22:00:25 DEBUG : Creating backend with remote "b2:x1-everything/3tma9e10hndcg550rmtolqn99s/pe6cqlhu2ahor9kfhu7gep0cv8"
2021/01/17 22:01:14 DEBUG : usr/share/X11/XErrorDB: Size and modification time the same (differ by 0s, within tolerance 1ms)
2021/01/17 22:01:14 DEBUG : usr/share/X11/XErrorDB: Unchanged skipping
2021/01/17 22:01:14 DEBUG : usr/share/X11/xorg.conf.d/40-libinput.conf: Size and modification time the same (differ by 0s, within tolerance 1ms)
2021/01/17 22:01:14 DEBUG : usr/share/X11/xorg.conf.d/40-libinput.conf: Unchanged skipping
2021/01/17 22:01:14 DEBUG : usr/share/X11/xorg.conf.d/51-synaptics-quirks.conf: Size and modification time the same (differ by 0s, within tolerance 1ms)
2021/01/17 22:01:14 DEBUG : usr/share/X11/xorg.conf.d/51-synaptics-quirks.conf: Unchanged skipping
2021/01/17 22:01:14 DEBUG : usr/share/X11/xorg.conf.d/70-synaptics.conf: Size and modification time the same (differ by 0s, within tolerance 1ms)
2021/01/17 22:01:14 DEBUG : usr/share/X11/xorg.conf.d/70-synaptics.conf: Unchanged skipping
2021/01/17 22:01:14 DEBUG : usr/share/X11/xman.help: Size and modification time the same (differ by 0s, within tolerance 1ms)
2021/01/17 22:01:14 DEBUG : usr/share/X11/xman.help: Unchanged skipping
2021/01/17 22:01:14 DEBUG : home/<myUserName>/.bash_aliases: Size and modification time the same (differ by 0s, within tolerance 1ms)
2

You should find that rclone does server side copies using this

rclone sync b2:enc-b2:x1/current enc-b2:x1/fallback_sync --checkers=12 --fast-list --log-file" "/tmp/rclone_fallback_C8z" -vv

You should see INFO : file.txt: Copied (server-side copy) in the logs

Note that server side copies on B2 aren't instant - they proceed at maybe 100 MB/s.

Yes those will all help.

I'm not following you?

3

Thanks, ncw.

Let me explain my question about the delete-with-copy alternative to rclone sync. I was thinking: rclone sync <source> <destination> proceeds by as follows. My PC fetches from the cloud a pair of meta-data - the timestamp of a source item and the timestamp of a destination item, both items being on the cloud - and comparing them, and then if necessary, doing an inter-cloud copy. But if I deleted any extant <destination> and then used rclone copy <source> <new-cloud-destination>, then perhaps rclone would in effect say: 'OK, cloud, now you do a copy' - thus returning control to my PC faster. But perhaps I get this wrong.

The server-side sync that you describe is what I am doing at present. It still takes a while - for some 10GB of stuff and, I don't know, perhaps a thousand files. I feel like it is the timestamp comparisons that take the time. (I can't give you an exact time because I started running the command with one set of options and then I aborted and ran again with another lot and I don't fancy redoing the operation from scratch.)

4

So far so good :slight_smile:

I don't think it will make a noticeable amount of difference.

It is probably the directory listings which are taking the time. You can make these faster with --fast-list or by increasing --checkers or both.

1 Like
5

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.