STOP and READ USE THIS TEMPLATE NO EXCEPTIONS - By not using this, you waste your time, our time and really hate puppies. Please remove these two lines and that will confirm you have read them.
What is the problem you are having with rclone?
First time using rclone. I'm trying to use rclone sync to sync the contents of an external hard drive to an S3 Glacier Deep bucket through my MacBook M2 Pro 32GB RAM. I have a 500Mbps upload speed. Many of the files are 70GB video files; total directory size about 700GB.
I tried dry-run first and all seemed fine, but then running the command with and without --checksum was extremely slow. Does rclone do all the checksums first, then upload? I saw minimal network activity over the course of 14 minutes. Tips on optimizing this sync command since they're mostly video files?
Run the command 'rclone version' and share the full output of the command.
rclone v1.67.0
-
os/version: darwin 14.6.1 (64 bit)
-
os/kernel: 23.6.0 (arm64)
-
os/type: darwin
-
os/arch: arm64 (ARMv8 compatible)
-
go/version: go1.22.4
-
go/linking: dynamic
-
go/tags: none
Which cloud storage system are you using? (eg Google Drive)
S3 through AWS Glacier Deep class
The command you were trying to run (eg rclone copy /tmp remote:tmp)
rclone sync /Volumes/WDDrive/The\ Fabric\ of\ America tfoa-backup:tfoa-backup-bucket --s3-storage-class DEEP_ARCHIVE --log-file rclone_backup.log --stats 1m -vv
rclone sync /Volumes/WDDrive/The\ Fabric\ of\ America tfoa-backup:tfoa-backup-bucket --s3-storage-class DEEP_ARCHIVE --checksum --transfers 4 --checkers 8 --log-file rclone_backup.log --stats 1m --progress
Please run 'rclone config redacted' and share the full output. If you get command not found, please make sure to update rclone.
[tfoa-backup]
type = s3
provider = AWS
access_key_id = XXX
secret_access_key = XXX
region = us-east-2
location_constraint = us-east-2
acl = private
server_side_encryption = AES256
storage_class = DEEP_ARCHIVE
A log from the command that you were trying to run with the -vv flag
Transferred: 215 MiB / 502.426 GiB, 0%, 1.099 MiB/s, ETA 5d9h59m
Checks: 27 / 27, 100%
Transferred: 0 / 223, 0%
Elapsed time: 14m0.0s
Transferring:
* 0008_2024-08-31/Footage/P1077074.MOV: transferring
* 0009_2024-08-31/Footage/P1077075.MOV: transferring
* 0003_2024-08-31/Footage/P1077070.MOV: transferring
* 0001_2024-08-31/Footage/IMG_7509.MOV: 2% /9.477Gi, 1.147Mi/s, 2h17m52s
2024/09/04 22:51:48 DEBUG : 0001_2024-08-31/Footage/IMG_7509.MOV: Seek from 5242880 to 0
2024/09/04 22:51:48 DEBUG : 0001_2024-08-31/Footage/IMG_7509.MOV: Seek from 5242880 to 0
2024/09/04 22:51:49 DEBUG : 0001_2024-08-31/Footage/IMG_7509.MOV: Seek from 5242880 to 0
2024/09/04 22:51:49 DEBUG : 0001_2024-08-31/Footage/IMG_7509.MOV: Seek from 5242880 to 0
2024/09/04 22:51:50 DEBUG : 0001_2024-08-31/Footage/IMG_7509.MOV: multipart upload wrote chunk 45 with 5242880 bytes and etag "3ba6622d6b730d9c2acb1867d52d430e"
2024/09/04 22:51:50 DEBUG : 0001_2024-08-31/Footage/IMG_7509.MOV: multi-thread copy: chunk 45/1941 (230686720-235929600) size 5Mi finished
2024/09/04 22:51:50 DEBUG : 0001_2024-08-31/Footage/IMG_7509.MOV: multi-thread copy: chunk 48/1941 (246415360-251658240) size 5Mi starting
2024/09/04 22:51:50 DEBUG : 0001_2024-08-31/Footage/IMG_7509.MOV: Seek from 5242880 to 0
2024/09/04 22:51:52 DEBUG : 0001_2024-08-31/Footage/IMG_7509.MOV: Seek from 5242880 to 0
2024/09/04 22:51:52 DEBUG : 0001_2024-08-31/Footage/IMG_7509.MOV: multipart upload wrote chunk 47 with 5242880 bytes and etag "ac1d668d993369915332b80cd1232682"
2024/09/04 22:51:52 DEBUG : 0001_2024-08-31/Footage/IMG_7509.MOV: multi-thread copy: chunk 47/1941 (241172480-246415360) size 5Mi finished
2024/09/04 22:51:52 DEBUG : 0001_2024-08-31/Footage/IMG_7509.MOV: multi-thread copy: chunk 49/1941 (251658240-256901120) size 5Mi starting
2024/09/04 22:51:54 DEBUG : 0001_2024-08-31/Footage/IMG_7509.MOV: Seek from 5242880 to 0
2024/09/04 22:51:56 DEBUG : 0001_2024-08-31/Footage/IMG_7509.MOV: multipart upload wrote chunk 44 with 5242880 bytes and etag "6a9fd9517726ac3f71cd551529172741"
2024/09/04 22:51:56 DEBUG : 0001_2024-08-31/Footage/IMG_7509.MOV: multi-thread copy: chunk 44/1941 (225443840-230686720) size 5Mi finished
2024/09/04 22:51:56 DEBUG : 0001_2024-08-31/Footage/IMG_7509.MOV: multi-thread copy: chunk 50/1941 (256901120-262144000) size 5Mi starting
2024/09/04 22:51:59 DEBUG : 0001_2024-08-31/Footage/IMG_7509.MOV: multipart upload wrote chunk 46 with 5242880 bytes and etag "57b73b987715e780ae464549d583a9fb"
2024/09/04 22:51:59 DEBUG : 0001_2024-08-31/Footage/IMG_7509.MOV: multi-thread copy: chunk 46/1941 (235929600-241172480) size 5Mi finished
2024/09/04 22:51:59 DEBUG : 0001_2024-08-31/Footage/IMG_7509.MOV: multi-thread copy: chunk 51/1941 (262144000-267386880) size 5Mi starting
2024/09/04 22:52:00 DEBUG : 0001_2024-08-31/Footage/IMG_7509.MOV: Seek from 5242880 to 0
2024/09/04 22:52:01 DEBUG : 0001_2024-08-31/Footage/IMG_7509.MOV: Seek from 5242880 to 0
2024/09/04 22:52:05 DEBUG : 0001_2024-08-31/Footage/IMG_7509.MOV: Seek from 5242880 to 0
2024/09/04 22:52:06 DEBUG : 0001_2024-08-31/Footage/IMG_7509.MOV: Seek from 5242880 to 0
2024/09/04 22:52:07 DEBUG : 0001_2024-08-31/Footage/IMG_7509.MOV: multipart upload wrote chunk 48 with 5242880 bytes and etag "3ed656c16805198919aca504070a5a13"
2024/09/04 22:52:07 DEBUG : 0001_2024-08-31/Footage/IMG_7509.MOV: multi-thread copy: chunk 48/1941 (246415360-251658240) size 5Mi finished
2024/09/04 22:52:07 DEBUG : 0001_2024-08-31/Footage/IMG_7509.MOV: multi-thread copy: chunk 52/1941 (267386880-272629760) size 5Mi starting
2024/09/04 22:52:07 DEBUG : 0001_2024-08-31/Footage/IMG_7509.MOV: Seek from 5242880 to 0
2024/09/04 22:52:08 DEBUG : 0001_2024-08-31/Footage/IMG_7509.MOV: Seek from 5242880 to 0
2024/09/04 22:52:08 DEBUG : 0001_2024-08-31/Footage/IMG_7509.MOV: multipart upload wrote chunk 49 with 5242880 bytes and etag "6e2e894bd6fda654eb4a105e841e10b7"
2024/09/04 22:52:08 DEBUG : 0001_2024-08-31/Footage/IMG_7509.MOV: multi-thread copy: chunk 49/1941 (251658240-256901120) size 5Mi finished
2024/09/04 22:52:08 DEBUG : 0001_2024-08-31/Footage/IMG_7509.MOV: multi-thread copy: chunk 53/1941 (272629760-277872640) size 5Mi starting
2024/09/04 22:52:08 DEBUG : 0001_2024-08-31/Footage/IMG_7509.MOV: Seek from 5242880 to 0
2024/09/04 22:52:08 DEBUG : 0001_2024-08-31/Footage/IMG_7509.MOV: multipart upload wrote chunk 50 with 5242880 bytes and etag "9813357cccfdf61564fb11c6cc2fb4ef"
2024/09/04 22:52:08 DEBUG : 0001_2024-08-31/Footage/IMG_7509.MOV: multi-thread copy: chunk 50/1941 (256901120-262144000) size 5Mi finished
2024/09/04 22:52:08 DEBUG : 0001_2024-08-31/Footage/IMG_7509.MOV: multi-thread copy: chunk 54/1941 (277872640-283115520) size 5Mi starting
2024/09/04 22:52:09 DEBUG : 0001_2024-08-31/Footage/IMG_7509.MOV: Seek from 5242880 to 0
2024/09/04 22:52:09 DEBUG : 0001_2024-08-31/Footage/IMG_7509.MOV: multipart upload wrote chunk 51 with 5242880 bytes and etag "32113f7ac8574741027d4188f69644e7"
2024/09/04 22:52:09 DEBUG : 0001_2024-08-31/Footage/IMG_7509.MOV: multi-thread copy: chunk 51/1941 (262144000-267386880) size 5Mi finished
2024/09/04 22:52:09 DEBUG : 0001_2024-08-31/Footage/IMG_7509.MOV: multi-thread copy: chunk 55/1941 (283115520-288358400) size 5Mi starting
2024/09/04 22:52:13 DEBUG : 0001_2024-08-31/Footage/IMG_7509.MOV: Seek from 5242880 to 0
2024/09/04 22:52:13 DEBUG : 0001_2024-08-31/Footage/IMG_7509.MOV: Seek from 5242880 to 0
2024/09/04 22:52:13 DEBUG : 0001_2024-08-31/Footage/IMG_7509.MOV: multipart upload wrote chunk 52 with 5242880 bytes and etag "7178f955a54f02a2c1a2f3041ae75c48"
2024/09/04 22:52:13 DEBUG : 0001_2024-08-31/Footage/IMG_7509.MOV: multi-thread copy: chunk 52/1941 (267386880-272629760) size 5Mi finished
2024/09/04 22:52:13 DEBUG : 0001_2024-08-31/Footage/IMG_7509.MOV: multi-thread copy: chunk 56/1941 (288358400-293601280) size 5Mi starting
2024/09/04 22:52:17 DEBUG : 0001_2024-08-31/Footage/IMG_7509.MOV: Seek from 5242880 to 0
2024/09/04 22:52:18 INFO : Signal received: interrupt
2024/09/04 22:52:18 DEBUG : 0001_2024-08-31/Footage/IMG_7509.MOV: multi-thread copy: cancelling transfer on exit
2024/09/04 22:52:18 DEBUG : 0001_2024-08-31/Footage/IMG_7509.MOV: multipart upload "smmhi5Rpp9InKyUjRYSm22Jfvt7lEOA45MWi4EgygmI.MY9xzxHI9.84G7eTSiWg4YcMqFgqLIN3ak1MlYYntleKZHpX0RjGHFudP0mFoaSSRR39o340YhKG3QBuW_zZ" aborted