Understanding problem. Syncing or Copying a huge kind of data


I am trying to copy a huge couple of data (8 TB) to Amazon Cloud.
The size of the files are between some kb and 500gb.
I copied around one month with rclone.
Now my questions:
1.) A lot of big files (>12GB) are in the Amazon trash folder. Why?

2.) I tried to restart the rclone copy process because auf a reboot of the maschine. I allways wait one or two days but rclone does not copy any files…
2016/11/15 22:17:51
Transferred: 1085.957 GBytes (4.012 MBytes/s)
Errors: 0
Checks: 382
Transferred: 0
Elapsed time: 77h0m2.7s

Why are no files copied?
How to speed up this a little bit?

This is my rlcone command:
rclone copy --bwlimit=4300k --transfers 1 --log-file rclone.log --stats 60m /Source/… amazon:/Destination/…

Many thanks for help

A good question! I suspect they are probably files whose uploads have timed out and been retried. Are you using latest rclone? There was lots of work in this area recently.

At a a guess the transfers have all failed.

Try rclone 1.34 and note that you can’t upload files bigger that 50gb to Amazon Drive.

Hi Nick,

thanks for you help. I’ve use the version 1.32. I have updated to 1.34
and will test the next days.
My line is now:
rclone sync --acd-templink-threshold=8G -v --max-size=50G --checksum --bwlimit=4300k --acd-upload-wait-per-gb=10m --transfers 1 --log-file rclone.log --stats 60m /Source… amazon:/Destiantion/…
Do you see any errors?

You shouldn’t really need to set --acd-templink-threshold.

The completely correct max size is --max-size 50000M which I thought was in the docs but appears to have vanished!

Otherwise looks fine!

Hi Nick,

now rclone closed with this error in Log:
2016/11/18 14:40:27 amazon drive root ‘…’: Waiting for transfers to finish
2016/11/18 14:40:27 amazon drive root ‘…’: not deleting files as there were IO errors
2016/11/18 14:40:27 Attempt 3/3 failed with 1 errors and: not deleting files as there were IO errors
2016/11/18 14:40:27 Failed to sync: not deleting files as there were IO errors
This is the command:
rclone sync -v --max-size=50G --bwlimit=4300k --transfers 1 --acd-upload-wait-per-gb=10m --log-file rclone.log --stats 60m /Source/… amazon:/Destination/…

You will probably find that a few files didn’t transfer and you need to run rclone again.

You should see how many errors and transfers there were in the summary.

You can also investigate increasing the --retries

Wait, does ACD support files larger than 50GB now?

No I don’t think so.