Hi asdffdsa,
thanks for your suggestion. I had the chance to think about them and really the situation is now way better even if something should be still improved.
I prefer to keep chunk size to 32M because I want to transfer big files and I don' t want to increase the put requests to S3 to don't increase the costs. Anyway I did not realize that eve if I'm tranferring 2 files a time, if it happpens I'm transferring 2 big files, they are split into 32M chunks, so if I don't limit the --s3-upload-concurrency (default 4) in this case I have 32M42 = 256MB memory allocation and it kills my machine. Now reducing --s3-upload-concurrency at least to 2 (but probably I will reduce it to 1) the things are going way better (maximum memory allocation should be now 128MB) and transfer is up since 1h and I got just 2 errors after 35m. Not sure what is this error related to.Following the log in the phase of error.
Regarding the delete-excluded flag it is ok in my case. For the storage class, yes it is already in the config file but this redundancy should not be an issue. Thanks a lot. Need to fix this error and then I can cosider reliable my rclone script to sync my local file to S3 using my ASUS RT-AC56U router, which gave me so many satisfaction in its 6 years working. I used rclone in this years to sync from my NAS to Google Drive, from my pc to local NAS, to Amazon Cloud (until it was possible
) and thi great software was really very reliable and effective! Thanks to ncw!
====== partial LOG ====
2021/02/23 23:22:13 DEBUG : 2020_02/20200220-WA0001.mp4: MD5 = e894c88dd5953ccccd8aa5228a32d069 OK
2021/02/23 23:22:13 INFO : 2020_02/20200220-WA0001.mp4: Copied (new)
2021/02/23 23:22:16 DEBUG : 2020_02/20200220-WA0002.mp4: MD5 = dd2eb494d2833b5b44da47e791cd8d11 OK
2021/02/23 23:22:16 INFO : 2020_02/20200220-WA0002.mp4: Copied (new)
2021/02/23 23:22:19 DEBUG : 2020_02/20200220-WA0003.mp4: MD5 = 610d8a9952ac03ff0d168a89b89c89e6 OK
2021/02/23 23:22:19 INFO : 2020_02/20200220-WA0003.mp4: Copied (new)
2021/02/23 23:22:21 DEBUG : 2020_02/20200220-WA0004.mp4: MD5 = e35ea7210d447eb226ba887da58e2f98 OK
2021/02/23 23:22:21 INFO : 2020_02/20200220-WA0004.mp4: Copied (new)
2021/02/23 23:22:23 DEBUG : 2020_02/20200225-WA0008.mp4: MD5 = 3940951da593e2b9c6819f0632018804 OK
2021/02/23 23:22:23 INFO : 2020_02/20200225-WA0008.mp4: Copied (new)
2021/02/23 23:22:26 DEBUG : 2020_02/20200227-WA0013.mp4: MD5 = b7a2e407779e247bcb54c766e2144db1 OK
2021/02/23 23:22:26 INFO : 2020_02/20200227-WA0013.mp4: Copied (new)
2021/02/23 23:22:28 DEBUG : 2020_02/20200214_082723.mp4: multipart upload starting chunk 1 size 32M offset 0/534.006M
2021/02/23 23:22:30 DEBUG : 2020_02/20200227-WA0014.mp4: MD5 = 990e3b0ec9d36a68d15f3f9c6821643e OK
2021/02/23 23:22:30 INFO : 2020_02/20200227-WA0014.mp4: Copied (new)
2021/02/23 23:22:31 DEBUG : 2020_02/20200214_082723.mp4: multipart upload starting chunk 2 size 32M offset 32M/534.006M
2021/02/23 23:22:32 DEBUG : 2020_02/20200227_175439.mp4: MD5 = 472b54d17a0eeccc872a43d99880b383 OK
2021/02/23 23:22:32 INFO : 2020_02/20200227_175439.mp4: Copied (new)
2021/02/23 23:22:35 DEBUG : 2020_02/20200228-WA0001.mp4: MD5 = 9d3b1520a3d9f04e6e6c3efcfae89cfe OK
2021/02/23 23:22:35 INFO : 2020_02/20200228-WA0001.mp4: Copied (new)
2021/02/23 23:22:40 DEBUG : 2020_02/20200228-WA0003.mp4: MD5 = cc1f22b417e22d05a9e7bs7d213b1e61 OK
2021/02/23 23:22:40 INFO : 2020_02/20200228-WA0003.mp4: Copied (new)
2021/02/23 23:22:45 DEBUG : 2020_02/20200228-WA0016.mp4: MD5 = 8f9ec13ecs6a4a9fa6b54c7w27cd74d4 OK
2021/02/23 23:22:45 INFO : 2020_02/20200228-WA0016.mp4: Copied (new)
2021/02/23 23:23:26 DEBUG : 2020_02/20200214_082723.mp4: multipart upload starting chunk 3 size 32M offset 64M/534.006M
2021/02/23 23:23:42 DEBUG : 2020_02/20200214_082723.mp4: multipart upload starting chunk 4 size 32M offset 96M/534.006M
2021/02/23 23:23:57 DEBUG : 2020_02/20200228_183704.mp4: MD5 = 4e0c0d82a69c202509a09fbf28304b17 OK
2021/02/23 23:23:57 INFO : 2020_02/20200228_183704.mp4: Copied (new)
2021/02/23 23:24:23 DEBUG : 2020_02/20200229-WA0001.mp4: MD5 = 1327559dac3604a28c92d874fda5774a OK
2021/02/23 23:24:23 INFO : 2020_02/20200229-WA0001.mp4: Copied (new)
2021/02/23 23:24:28 DEBUG : 2020_02/20200214_082723.mp4: multipart upload starting chunk 5 size 32M offset 128M/534.006M
2021/02/23 23:24:33 DEBUG : 2020_02/20200214_082723.mp4: multipart upload starting chunk 6 size 32M offset 160M/534.006M
2021/02/23 23:24:38 DEBUG : 2020_02/20200229-WA0002.mp4: MD5 = 64bb24392204dde4a8786e9a7cde4400 OK
2021/02/23 23:24:38 INFO : 2020_02/20200229-WA0002.mp4: Copied (new)
2021/02/23 23:26:00 DEBUG : pacer: low level retry 1/10 (error RequestTimeout: Your socket connection to the server was not read from or written to within the timeout period. Idle connections will be closed.
status code: 400, request id: 1XT0CK08855HWQWQ, host id: SGlEkygVkQ/RyBG6oYZTdBgChoX0wcaYw8G48LkHagWKMe987Fo9g8+booksgdvRlXujG7q+Y=)
2021/02/23 23:26:00 DEBUG : pacer: Rate limited, increasing sleep to 10ms
2021/02/23 23:26:00 DEBUG : pacer: low level retry 1/10 (error RequestTimeout: Your socket connection to the server was not read from or written to within the timeout period. Idle connections will be closed.
status code: 400, request id: 0VMBRWCGX0N83FX0, host id: vvE5OdwQ/z4BmzvUDwuX2l3o28/Mc1zI9EucZ42CoFKM8fe214uJBA89EU+Ul660EBDMj/74iw=)
2021/02/23 23:26:00 DEBUG : pacer: Rate limited, increasing sleep to 20ms
2021/02/23 23:26:00 DEBUG : pacer: Reducing sleep to 15ms
2021/02/23 23:26:00 ERROR : 2020_02/20200229_123910.mp4: Failed to copy: s3 upload: 400 Bad Request: <?xml version="1.0" encoding="UTF-8"?>
RequestTimeout
Your socket connection to the server was not read from or written to within the timeout period. Idle connections will be closed.DHGA6ASTKXEKZMG3Q/WPm36BHE6jOhSF34WOCQe34Cl0cMnDdMOibne71vRhyf67RT3Na8rX3KulcVai7kZ99Ok4=
2021/02/23 23:26:33 DEBUG : pacer: Reducing sleep to 11.25ms
2021/02/23 23:26:35 DEBUG : pacer: Reducing sleep to 0s
2021/02/23 23:26:35 DEBUG : 2020_02/20200229_124412.mp4: MD5 = 56b7d6d4908c558b167b984a94109ae3 OK
2021/02/23 23:26:35 INFO : 2020_02/20200229_124412.mp4: Copied (new)
2021/02/23 23:26:54 DEBUG : 2020_02/20200214_082723.mp4: multipart upload starting chunk 7 size 32M offset 192M/534.006M
2021/02/23 23:27:06 DEBUG : 2020_02/20200214_082723.mp4: multipart upload starting chunk 8 size 32M offset 224M/534.006M
2021/02/23 23:29:00 DEBUG : 2020_02/20200214_082723.mp4: multipart upload starting chunk 9 size 32M offset 256M/534.006M