Copying from one amazon drive to another

Help and Support
1

Since amazon opened the UK drive iam thinking about moving the data from one cloud to another.
Wondering whats best way to do it and if its possible to set it up in such way that i wont need much space on VPS machine.

Need to move around 4TB of data and yea i know it will need to be downloaded first then uploaded and that ok as long as the downloaded file is automatically deleted from system iam doing it. ( will be some vps with around 50GB max disk space )

I wonder what would happen if I just copy it from mounted drive ( with acd_cli ) to another amazon drive.

2

If you do it with rclone, eg

rclone copy oldremote:dir newremote:dir

Then rclone will stream the transfers without storing anything to disk.

2 Likes
3

Good to know, thanks for the info.

4

I started to copy from one drive to another using
/usr/sbin/rclone copy acdus:/encrypted acdde:/encrypted -c -v --min-age 180m --acd-upload-wait-time=10m --transfers=45 --checkers=50 --log-file=/var/log/rclonecopy.log

The min-age part is mainly as iam still uploading some stuff to old drive and dont wanna copy partial files.
Copying speed is amazing basically maxing 1GB line (103 MB)

Iam getting quite few:

2016/10/31 15:33:15 uNliDtMkT7HeihH7uM5180xg/UOr-0SYxOegNKs7fcNqbkkS5/SCZW1ayttAepqRHKa6NvCA0Y/spNe1wMLJbqcdgEhFf-q,z2TYrmU4HFww-qLkuFEy9A-6-: Object not found - waiting (142/144)
2016/10/31 15:33:30 uNliDtMkT7HeihH7uM5180xg/UOr-0SYxOegNKs7fcNqbkkS5/SCZW1ayttAepqRHKa6NvCA0Y/spNe1wMLJbqcdgEhFf-q,z2TYrmU4HFww-qLkuFEy9A-6-: Received error: HTTP code 400: "400 Bad Request", reponse body: {"message":"Cannot complete upload"}
 - low level retry 1/10

Was hoping that --acd-upload-wait-time=10m would solve the problem. ( most of files are bellow 10GB )

5

:slight_smile:

You might need to increase that wait time. rclone will retry the files so they will all get there eventually!

6

did you find a solution?

7

It was fixed with updates. If you encounter those issues just set --acd-upload-wait-per-gb 10m ( default is 3m.

8

i have set yesterday to 20m but the problem is not solved…

2017/03/24 06:10:55 DEBUG : path_to_content.mkv: Object not found - waiting (937/938)
2017/03/24 06:11:00 DEBUG : path_to_content.mkv: Object not found - waiting (938/938)
2017/03/24 06:11:05 DEBUG : path_to_content.mkv: Giving up waiting for object - returning original error: HTTP code 400: "400 Bad Request": response body: "{\"message\":\"Cannot complete upload\"}" ("400 Bad Request")
2017/03/24 06:11:05 DEBUG : pacer: Rate limited, sleeping for 838.182873ms (1 consecutive low level retries)
2017/03/24 06:11:05 DEBUG : pacer: low level retry 1/1 (error HTTP code 400: "400 Bad Request": response body: "{\"message\":\"Cannot complete upload\"}")
2017/03/24 06:11:05 DEBUG : path_to_content.mkv: Received error: HTTP code 400: "400 Bad Request": response body: "{\"message\":\"Cannot complete upload\"}" - low level retry 3/10
2017/03/24 06:11:05 DEBUG : pacer: Resetting sleep to minimum 20ms on success

and the upoad start from new…in the most times, by the second upload it works.

in the last 7 hours i uploaded 54 GB…but only 3 files finished with total 14 GB! so i have lost time for 40GB to upload! this is totaly bad and i musst solve the problem

EDIT: i have restart my router and since 3-4 hours and after 26GB upload there is no error coming up. I`ll get back later if my problem is solved or not

9

problem is not solved…after 18 hours of upload the error comes again…and now every upload run in “object not found” and after the second upload the upload finish…

10

And if you reboot the router again does it get better for a while again?

11

@Berzerk90er Did you find a solution for this?

12

I think it was Amazon itself.Because from time to time the error occurred again. But then it was again a few weeks error-free.But Amazon is now history