Understanding problem. Syncing or Copying a huge kind of data

Help and Support
1

Hi,

I am trying to copy a huge couple of data (8 TB) to Amazon Cloud.
The size of the files are between some kb and 500gb.
I copied around one month with rclone.
Now my questions:
1.) A lot of big files (>12GB) are in the Amazon trash folder. Why?

2.) I tried to restart the rclone copy process because auf a reboot of the maschine. I allways wait one or two days but rclone does not copy any files…
2016/11/15 22:17:51
Transferred: 1085.957 GBytes (4.012 MBytes/s)
Errors: 0
Checks: 382
Transferred: 0
Elapsed time: 77h0m2.7s

Why are no files copied?
How to speed up this a little bit?

This is my rlcone command:
rclone copy --bwlimit=4300k --transfers 1 --log-file rclone.log --stats 60m /Source/… amazon:/Destination/…

Many thanks for help

2

A good question! I suspect they are probably files whose uploads have timed out and been retried. Are you using latest rclone? There was lots of work in this area recently.

At a a guess the transfers have all failed.

Try rclone 1.34 and note that you can’t upload files bigger that 50gb to Amazon Drive.

3

Hi Nick,

thanks for you help. I’ve use the version 1.32. I have updated to 1.34
and will test the next days.
My line is now:
rclone sync --acd-templink-threshold=8G -v --max-size=50G --checksum --bwlimit=4300k --acd-upload-wait-per-gb=10m --transfers 1 --log-file rclone.log --stats 60m /Source… amazon:/Destiantion/…
Do you see any errors?

4

You shouldn’t really need to set --acd-templink-threshold.

The completely correct max size is --max-size 50000M which I thought was in the docs but appears to have vanished!

Otherwise looks fine!

5

Hi Nick,

now rclone closed with this error in Log:
2016/11/18 14:40:27 amazon drive root ‘…’: Waiting for transfers to finish
2016/11/18 14:40:27 amazon drive root ‘…’: not deleting files as there were IO errors
2016/11/18 14:40:27 Attempt 3/3 failed with 1 errors and: not deleting files as there were IO errors
2016/11/18 14:40:27 Failed to sync: not deleting files as there were IO errors
This is the command:
rclone sync -v --max-size=50G --bwlimit=4300k --transfers 1 --acd-upload-wait-per-gb=10m --log-file rclone.log --stats 60m /Source/… amazon:/Destination/…

6

You will probably find that a few files didn’t transfer and you need to run rclone again.

You should see how many errors and transfers there were in the summary.

You can also investigate increasing the --retries

7

Wait, does ACD support files larger than 50GB now?

8

No I don’t think so.