Iam making a copy from my encfs unencrypted folder to rclone crypt one on same Amazon Drive.
There is quite big difference with Transffered size and actual folder size
/usr/bin/rclone copy /storage/acd/ acdcrypt: -c --no-traverse --no-update-modtime --transfers=30 --checkers=30 --min-age 180m --log-file=/var/log/acd2crypt.log
2017/01/10 11:00:02
Transferred: 25503.823 GBytes (63.747 MBytes/s)
Errors: 327
Checks: 2750
Transferred: 16764
Elapsed time: 113h48m1s
rclone size acd:/crypt
Total objects: 20148
Total size: 14195.530 GBytes (15242334120077 Bytes)
Both of above stats were taken at same time, and I think i had rclone copy running for few hours prior as I was still tweaking transfers to achieve best performance.
I must point out that there are some internet connections dropped on server ( never happened before I started this copy ) so some of the files were restarted hence the errors but the difference is close to double eg I have 14TB in my new crypt folder while 25TB was transferred.
During this 113h run I had this errors:
2017/01/08 02:09:11 amazon drive root ‘crypt’: 401 error received - invalidating token
2017/01/08 02:09:40 amazon drive root ‘crypt’: 401 error received - invalidating token
2017/01/10 05:17:52 amazon drive root ‘crypt’: 401 error received - invalidating token
2017/01/10 05:17:53 amazon drive root ‘crypt’: 401 error received - invalidating token
2017/01/06 07:35:52 series/The Grand Tour (2016)/Season 01/The Grand Tour (2016) - S01E01.mkv: Failed to copy: HTTP code 400: “400 Bad Request”: response body: “{“message”:“Cannot complete upload”}”
2017/01/06 16:42:51 series/iZombie/Season 02/iZombie - S02E16.mkv: Failed to copy: HTTP code 400: “400 Bad Request”: response body: “{“message”:“Cannot complete upload”}”
2017/01/06 16:42:51 series/Rome/Season 01/Rome - S01E10.mkv: Failed to copy: HTTP code 400: “400 Bad Request”: response body: “{“message”:“Cannot complete upload”}”
2017/01/06 16:42:52 series/Stargate Atlantis/Season 02/Stargate Atlantis - S02E17.mkv: Failed to copy: HTTP code 400: “400 Bad Request”: response body: “{“message”:“Cannot complete upload”}”
2017/01/06 17:50:17 series/Rome/Season 01/Rome - S01E07.mkv: Failed to copy: HTTP code 400: “400 Bad Request”: response body: “{“message”:“Cannot complete upload”}”
2017/01/07 01:35:35 series/Star Trek/Season 03/Star Trek - S03E13.mkv: Failed to copy: HTTP code 400: “400 Bad Request”: response body: “{“message”:“Cannot complete upload”}”
2017/01/07 01:59:54 series/Star Trek/Season 03/Star Trek - S03E23.mkv: Failed to copy: HTTP code 400: “400 Bad Request”: response body: “{“message”:“Cannot complete upload”}”
2017/01/07 02:45:01 series/Scandal (2012)/Season 04/Scandal (2012) - S04E11.mkv: Failed to copy: HTTP code 400: “400 Bad Request”: response body: “{“message”:“Cannot complete upload”}”
2017/01/07 10:06:39 series/Star Trek/Season 01/Star Trek - S01E21.mkv: Failed to copy: HTTP code 400: “400 Bad Request”: response body: “{“message”:“Cannot complete upload”}”
2017/01/07 12:31:30 series/The Blacklist/Season 02/The Blacklist - S02E04.mp4: Failed to copy: HTTP code 400: “400 Bad Request”: response body: “{“message”:“Cannot complete upload”}”
2017/01/07 13:15:36 series/Scandal (2012)/Season 04/Scandal (2012) - S04E16.mkv: Failed to copy: HTTP code 400: “400 Bad Request”: response body: “{“message”:“Cannot complete upload”}”
2017/01/07 13:33:19 series/Star Trek/Season 01/Star Trek - S01E04.mkv: Failed to copy: HTTP code 400: “400 Bad Request”: response body: “{“message”:“Cannot complete upload”}”
2017/01/07 13:37:39 series/Scandal (2012)/Season 01/Scandal (2012) - S01E01.mkv: Failed to copy: HTTP code 400: “400 Bad Request”: response body: “{“message”:“Cannot complete upload”}”
2017/01/07 19:16:51 series/Smallville/Season 05/Smallville - S05E16.mkv: Failed to copy: HTTP code 400: “400 Bad Request”: response body: “{“message”:“Cannot complete upload”}”
2017/01/07 19:42:21 series/Arrow/Season 02/Arrow - S02E15.mkv: Failed to copy: HTTP code 400: “400 Bad Request”: response body: “{“message”:“Cannot complete upload”}”
2017/01/07 20:00:43 series/Person of Interest/Season 03/Person of Interest - S03E04.mkv: Failed to copy: HTTP code 400: “400 Bad Request”: response body: “{“message”:“Cannot complete upload”}”
2017/01/07 20:42:06 series/The Blacklist/Season 03/The Blacklist - S03E14.mkv: Failed to copy: HTTP code 400: “400 Bad Request”: response body: “{“message”:“Cannot complete upload”}”
2017/01/07 21:21:11 series/Stargate Atlantis/Season 03/Stargate Atlantis - S03E14.mkv: Failed to copy: HTTP code 400: “400 Bad Request”: response body: “{“message”:“Cannot complete upload”}”
2017/01/07 21:22:07 series/Stargate Atlantis/Season 05/Stargate Atlantis - S05E20.mkv: Failed to copy: HTTP code 400: “400 Bad Request”: response body: “{“message”:“Cannot complete upload”}”
2017/01/07 21:22:10 series/Sons of Anarchy/Season 06/Sons of Anarchy - S06E08.mkv: Failed to copy: HTTP code 400: “400 Bad Request”: response body: “{“message”:“Cannot complete upload”}”
2017/01/07 21:30:34 series/Person of Interest/Season 03/Person of Interest - S03E17.mkv: Failed to copy: HTTP code 400: “400 Bad Request”: response body: “{“message”:“Cannot complete upload”}”
2017/01/07 21:30:34 series/Scandal (2012)/Season 03/Scandal (2012) - S03E07.mkv: Failed to copy: HTTP code 400: “400 Bad Request”: response body: “{“message”:“Cannot complete upload”}”
2017/01/07 21:30:35 series/Smallville/Season 10/Smallville - S10E08.mkv: Failed to copy: HTTP code 400: “400 Bad Request”: response body: “{“message”:“Cannot complete upload”}”
2017/01/07 21:30:40 series/Arrow/Season 03/Arrow - S03E01.mkv: Failed to copy: HTTP code 400: “400 Bad Request”: response body: “{“message”:“Cannot complete upload”}”
2017/01/07 21:51:29 series/Dancing with the Stars/Season 23/Dancing with the Stars - S23E07.mkv: Failed to copy: HTTP code 400: “400 Bad Request”: response body: “{“message”:“Cannot complete upload”}”
2017/01/07 21:52:41 series/Stargate Atlantis/Season 04/Stargate Atlantis - S04E20.mkv: Failed to copy: HTTP code 400: “400 Bad Request”: response body: “{“message”:“Cannot complete upload”}”
2017/01/07 22:37:25 series/Sons of Anarchy/Season 04/Sons of Anarchy - S04E01.mkv: Failed to copy: HTTP code 400: “400 Bad Request”: response body: “{“message”:“Cannot complete upload”}”
2017/01/07 22:53:21 series/Smallville/Season 10/Smallville - S10E07.mkv: Failed to copy: HTTP code 400: “400 Bad Request”: response body: “{“message”:“Cannot complete upload”}”
2017/01/07 23:19:04 series/Sons of Anarchy/Season 04/Sons of Anarchy - S04E08.mkv: Failed to copy: HTTP code 400: “400 Bad Request”: response body: “{“message”:“Cannot complete upload”}”
2017/01/07 23:19:04 series/Dancing with the Stars/Season 23/Dancing with the Stars - S23E12.mkv: Failed to copy: HTTP code 400: “400 Bad Request”: response body: “{“message”:“Cannot complete upload”}”
2017/01/07 23:44:18 series/Smallville/Season 10/Smallville - S10E10.mkv: Failed to copy: HTTP code 400: “400 Bad Request”: response body: “{“message”:“Cannot complete upload”}”
2017/01/07 23:45:17 series/Smallville/Season 05/Smallville - S05E01.mkv: Failed to copy: HTTP code 400: “400 Bad Request”: response body: “{“message”:“Cannot complete upload”}”
2017/01/08 03:52:45 series/Arrow/Season 01/Arrow - S01E18.mkv: Failed to copy: HTTP code 400: “400 Bad Request”: response body: “{“message”:“Cannot complete upload”}”
2017/01/08 04:33:52 series/Smallville/Season 10/Smallville - S10E19.mkv: Failed to copy: HTTP code 400: “400 Bad Request”: response body: “{“message”:“Cannot complete upload”}”
2017/01/08 05:18:45 series/The X-Files/Season 05/The X-Files - S05E14.mkv: Failed to copy: HTTP code 400: “400 Bad Request”: response body: “{“message”:“Cannot complete upload”}”
2017/01/08 06:32:36 series/Arrow/Season 01/Arrow - S01E20.mkv: Failed to copy: HTTP code 400: “400 Bad Request”: response body: “{“message”:“Cannot complete upload”}”
2017/01/08 06:33:36 series/Sons of Anarchy/Season 01/Sons of Anarchy - S01E01.mkv: Failed to copy: HTTP code 400: “400 Bad Request”: response body: “{“message”:“Cannot complete upload”}”
2017/01/08 06:47:24 series/The X-Files/Season 05/The X-Files - S05E02.mkv: Failed to copy: HTTP code 400: “400 Bad Request”: response body: “{“message”:“Cannot complete upload”}”
2017/01/08 08:08:35 series/Sons of Anarchy/Season 03/Sons of Anarchy - S03E05.mkv: Failed to copy: HTTP code 400: “400 Bad Request”: response body: “{“message”:“Cannot complete upload”}”
2017/01/08 08:31:52 series/The X-Files/Season 08/The X-Files - S08E01.mkv: Failed to copy: HTTP code 400: “400 Bad Request”: response body: “{“message”:“Cannot complete upload”}”
2017/01/08 08:51:36 series/Grimm/Season 05/Grimm - S05E14.mkv: Failed to copy: HTTP code 400: “400 Bad Request”: response body: “{“message”:“Cannot complete upload”}”
2017/01/08 09:00:42 series/Sons of Anarchy/Season 01/Sons of Anarchy - S01E05.mkv: Failed to copy: HTTP code 400: “400 Bad Request”: response body: “{“message”:“Cannot complete upload”}”
2017/01/08 09:38:16 series/Sons of Anarchy/Season 03/Sons of Anarchy - S03E12.mkv: Failed to copy: HTTP code 400: “400 Bad Request”: response body: “{“message”:“Cannot complete upload”}”
2017/01/08 10:14:05 series/Ash vs Evil Dead/Season 02/Ash vs Evil Dead - S02E01.mkv: Failed to copy: HTTP code 400: “400 Bad Request”: response body: “{“message”:“Cannot complete upload”}”
2017/01/08 11:07:29 series/Arrow/Season 01/Arrow - S01E11.mkv: Failed to copy: HTTP code 400: “400 Bad Request”: response body: “{“message”:“Cannot complete upload”}”
2017/01/08 11:20:18 series/Grimm/Season 03/Grimm - S03E18.mkv: Failed to copy: HTTP code 400: “400 Bad Request”: response body: “{“message”:“Cannot complete upload”}”
2017/01/08 11:32:56 series/Grimm/Season 04/Grimm - S04E08.mkv: Failed to copy: HTTP code 400: “400 Bad Request”: response body: “{“message”:“Cannot complete upload”}”
2017/01/08 11:39:43 series/Arrow/Season 01/Arrow - S01E14.mkv: Failed to copy: HTTP code 400: “400 Bad Request”: response body: “{“message”:“Cannot complete upload”}”
2017/01/08 12:14:23 series/Arrow/Season 01/Arrow - S01E08.mkv: Failed to copy: HTTP code 400: “400 Bad Request”: response body: “{“message”:“Cannot complete upload”}”
2017/01/08 12:46:53 series/Arrow/Season 04/Arrow - S04E05.mkv: Failed to copy: HTTP code 400: “400 Bad Request”: response body: “{“message”:“Cannot complete upload”}”
2017/01/08 12:47:16 series/The X-Files/Season 08/The X-Files - S08E06.mkv: Failed to copy: HTTP code 400: “400 Bad Request”: response body: “{“message”:“Cannot complete upload”}”
2017/01/09 12:12:51 movies/Predestination (2014)/Predestination (2014).mp4: Failed to copy: HTTP code 400: “400 Bad Request”: response body: “{“message”:“Cannot complete upload”}”
2017/01/09 12:15:27 series/Boardwalk Empire/Season 03/Boardwalk Empire S03E07.mkv: Failed to copy: HTTP code 400: “400 Bad Request”: response body: “{“message”:“Cannot complete upload”}”
2017/01/09 12:15:33 series/Prison Break/Season 01/Prison Break - S01E12.mkv: Failed to copy: HTTP code 400: “400 Bad Request”: response body: “{“message”:“Cannot complete upload”}”
2017/01/09 12:15:34 series/The Vampire Diaries/Season 05/The Vampire Diaries - S05E16.mkv: Failed to copy: HTTP code 400: “400 Bad Request”: response body: “{“message”:“Cannot complete upload”}”
2017/01/09 12:15:34 series/The Vampire Diaries/Season 04/The Vampire Diaries - S04E09.mkv: Failed to copy: HTTP code 400: “400 Bad Request”: response body: “{“message”:“Cannot complete upload”}”
2017/01/09 12:15:34 series/Boardwalk Empire/Season 05/Boardwalk Empire S05E02.mkv: Failed to copy: HTTP code 400: “400 Bad Request”: response body: “{“message”:“Cannot complete upload”}”
2017/01/09 12:15:34 series/The Vampire Diaries/Season 04/The Vampire Diaries - S04E12.mkv: Failed to copy: HTTP code 400: “400 Bad Request”: response body: “{“message”:“Cannot complete upload”}”
2017/01/09 12:15:34 series/The Vampire Diaries/Season 04/The Vampire Diaries - S04E17.mkv: Failed to copy: HTTP code 400: “400 Bad Request”: response body: “{“message”:“Cannot complete upload”}”
2017/01/09 12:21:48 series/Boardwalk Empire/Season 03/Boardwalk Empire S03E09.mkv: Failed to copy: HTTP code 400: “400 Bad Request”: response body: “{“message”:“Cannot complete upload”}”
2017/01/09 22:19:46 series/True Blood/Season 06/True Blood - S06E03.mkv: Failed to copy: HTTP code 400: “400 Bad Request”: response body: “{“message”:“Cannot complete upload”}”
The 61 files with 400 Bad Request were more then likely then ones that could not be copied due temp internet connection loss ( usually for only 10 to 30 seconds ) . Is there a parameter that would make rclone try a bit longer if it happens ?