Why is this copy command taking so long?

Help and Support
1

Hello,

this is my current copy command and I am not sure why it takes 4 hours.

rclone copy "/share/CACHEDEV1_DATA/User Name/" "gcrypt:shared/redacted" --backup-dir "gcrypt:shared/Backups/User Name/"date -I/date '+%HH%MM' --max-age 10m --no-traverse --checkers 3 --log-file /share/CACHEDEV1_DATA/rclone/copysync.log -v --tpslimit 3 --transfers 3 --config /share/CACHEDEV1_DATA/rclone/rclonegdrivebackup.conf

This is the log:

2018/12/17 08:10:00 INFO  : Starting HTTP transaction limiter: max 3 transactions/s with burst 1
2018/12/17 08:10:41 INFO  : folder/structure/file.pdf: Copied (new)
2018/12/17 08:10:50 INFO  : Encrypted drive 'gcrypt:shared/User Name': Waiting for checks to finish
2018/12/17 08:10:50 INFO  : Encrypted drive 'gcrypt:shared/User Name': Waiting for transfers to finish
2018/12/17 08:11:03 INFO  : 
Transferred:   	    2.243M / 2.243 MBytes, 100%, 36.445 kBytes/s, ETA 0s
Errors:                 0
Checks:                 0 / 0, -
Transferred:            5 / 5, 100%
Elapsed time:        1m3s

and it goes on like this for the next 4 hours

2018/12/17 12:17:03 INFO  : 
Transferred:   	    2.243M / 2.243 MBytes, 100%, 158 Bytes/s, ETA 0s
Errors:                 0
Checks:                 0 / 0, -
Transferred:            5 / 5, 100%
Elapsed time:      4h7m3s

2018/12/17 12:18:03 INFO  : 
Transferred:   	    2.243M / 2.243 MBytes, 100%, 158 Bytes/s, ETA 0s
Errors:                 0
Checks:                 0 / 0, -
Transferred:            5 / 5, 100%
Elapsed time:      4h8m3s

I thought the --no-traverse in combination with the max age would be very very fast.

2

Turn up the logging to debug with -vvv and what is the machine doing? High CPU? High Disk IO?

Post the log.

3

Hello the log is ONLY filled with entries like this:

2018/12/19 04:20:45 DEBUG : Folder/Structture/subfolder/file.pdf: Excluded

I guess it takes 4 hours because it goes through every single file, correct?

In that share on my NAS: /share/CACHEDEV1_DATA/User Name/ and all its diretories and sub directories are in total approx. 750000 files.

Is there any way to speed things up?

4

Can you share the log as guessing isn’t fun? :slight_smile:

5

It should be - are you using the latest beta as --no-traverse doesn’t work in in v1.45?

6

rclone v1.45-012-g6e000d26-no-traverse-beta

  • os/arch: linux/amd64
  • go version: go1.11.2

It should work with that version already, right ? Any idea why it still takes that long ?

7

Is rclone iterating all the files on a NAS? That probably explains why it is is slow if rclone is having to run over the network. Even with --no-traverse rclone still has to iterate all the files on the source - what --no-traverse does is avoid iterating the files on the destination.

Can you run rclone on the NAS itself?

8

I am running rclone on the NAS itself already

What can I do now?

Would logs help? (Can I send them via private message to you?)

9

Ah, OK.

How long does rclone size "/share/CACHEDEV1_DATA/User Name/" take? That will tell you how long it takes rclone to read a directory listing of every file - the sync can’t go faster than that.