Rclone Move large files from one folder to another on destination wasabi bucket does not start

Thomas_Defilippis · November 7, 2022, 1:37pm

What is the problem you are having with rclone?

I am trying to move all files older than 12 weeks on the destination s3 wasabi bucket into a folder called /scheduled-for-completion within the same bucket. When I run the command, large files around 100GB, hang forever and no output occurs. The move commands works with smaller files. I want to see if there is a faster way of moving these files over.

Run the command 'rclone version' and share the full output of the command.

rclone v1.53.3-DEV

os/arch: linux/amd64
go version: go1.18

Which cloud storage system are you using? (eg Google Drive)

Wasabi

The command you were trying to run (eg `rclone copy /tmp remote:tmp`)

rclone --min-age 12w -P move wasabi:cmtv-backups/data/  wasabi:cmtv-backups/scheduled-for-deletion/ --checkers 16 --fast-list --timeout 1m --s3-chunk-size 16M --s3-upload-concurrency 16 --log-file=/var/log/rclone.log

The rclone config contents with secrets removed.

[wasabi]
type = s3
env_auth = false
access_key_id = **********
secret_access_key = ******
region = us-west-1
endpoint = ******
location_constraint = 
acl = 
server_side_encryption = 
storage_class =

A log from the command with the `-vv` flag

2022/11/07 08:35:14 DEBUG : --min-age 2.8M to 2022-08-15 09:35:14.386426311 -0400 EDT m=-7257599.953676156
2022/11/07 08:35:14 DEBUG : rclone: Version "v1.53.3-DEV" starting with parameters ["rclone" "--min-age" "12w" "-P" "move" "wasabi:cmtv-backups/data/" "wasabi:cmtv-backups/scheduled-for-deletion/" "--checkers" "16" "--fast-list" "--timeout" "1m" "--s3-chunk-size" "16M" "--s3-upload-concurrency" "16" "--log-file=/var/log/rclone.log" "-vv"]
2022/11/07 08:35:14 DEBUG : Using config file from "/home/greeneggmedia/.config/rclone/rclone.conf"
2022/11/07 08:35:14 DEBUG : Creating backend with remote "wasabi:cmtv-backups/data/"
2022/11/07 08:35:15 DEBUG : fs cache: renaming cache item "wasabi:cmtv-backups/data/" to be canonical "wasabi:cmtv-backups/data"
2022/11/07 08:35:15 DEBUG : Creating backend with remote "wasabi:cmtv-backups/scheduled-for-deletion/"
2022/11/07 08:35:15 DEBUG : fs cache: renaming cache item "wasabi:cmtv-backups/scheduled-for-deletion/" to be canonical "wasabi:cmtv-backups/scheduled-for-deletion"
2022/11/07 08:35:15 DEBUG : 2022-08-20/cm-prod-api-albert_2022-08-20_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 08:35:15 DEBUG : 2022-08-20/cm-prod-app-joan-1_2022-08-20_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 08:35:15 DEBUG : 2022-08-20/cm-prod-cache-corona_2022-08-20_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 08:35:15 DEBUG : 2022-08-20/cm-prod-db-alexandria_2022-08-20_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 08:35:15 DEBUG : 2022-08-20/cm-prod-fs-martin-1_2022-08-20_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 08:35:15 DEBUG : 2022-08-20/cm-prod-transcoder-newman_2022-08-20_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 08:35:15 DEBUG : 2022-08-27/cm-prod-api-albert_2022-08-27_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 08:35:15 DEBUG : 2022-08-27/cm-prod-app-joan-1_2022-08-27_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 08:35:15 DEBUG : 2022-08-27/cm-prod-cache-corona_2022-08-27_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 08:35:16 DEBUG : 2022-08-27/cm-prod-db-alexandria_2022-08-27_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 08:35:16 DEBUG : 2022-08-27/cm-prod-fs-martin-1_2022-08-27_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 08:35:16 DEBUG : 2022-08-27/cm-prod-transcoder-newman_2022-08-27_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 08:35:16 DEBUG : 2022-09-03/cm-prod-api-albert_2022-09-03_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 08:35:16 DEBUG : 2022-09-03/cm-prod-app-joan-1_2022-09-03_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 08:35:16 DEBUG : 2022-09-03/cm-prod-cache-corona_2022-09-03_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 08:35:16 DEBUG : 2022-09-03/cm-prod-db-alexandria_2022-09-03_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 08:35:16 DEBUG : 2022-09-03/cm-prod-fs-martin-1_2022-09-03_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 08:35:16 DEBUG : 2022-09-03/cm-prod-transcoder-newman_2022-09-03_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 08:35:16 DEBUG : 2022-09-10/cm-prod-api-albert_2022-09-10_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 08:35:16 DEBUG : 2022-09-10/cm-prod-app-joan-1_2022-09-10_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 08:35:16 DEBUG : 2022-09-10/cm-prod-cache-corona_2022-09-10_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 08:35:16 DEBUG : 2022-09-10/cm-prod-db-alexandria_2022-09-10_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 08:35:16 DEBUG : 2022-09-10/cm-prod-fs-martin-1_2022-09-10_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 08:35:16 DEBUG : 2022-09-10/cm-prod-transcoder-newman_2022-09-10_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 08:35:16 DEBUG : 2022-09-17/cm-prod-api-albert_2022-09-17_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 08:35:16 DEBUG : 2022-09-17/cm-prod-app-joan-1_2022-09-17_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 08:35:16 DEBUG : 2022-09-17/cm-prod-cache-corona_2022-09-17_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 08:35:16 DEBUG : 2022-09-17/cm-prod-db-alexandria_2022-09-17_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 08:35:16 DEBUG : 2022-09-17/cm-prod-fs-martin-1_2022-09-17_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 08:35:16 DEBUG : 2022-09-17/cm-prod-transcoder-newman_2022-09-17_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 08:35:16 DEBUG : 2022-09-24/cm-prod-api-albert_2022-09-24_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 08:35:16 DEBUG : 2022-09-24/cm-prod-app-joan-1_2022-09-24_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 08:35:17 DEBUG : 2022-09-24/cm-prod-cache-corona_2022-09-24_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 08:35:17 DEBUG : 2022-09-24/cm-prod-db-alexandria_2022-09-24_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 08:35:17 DEBUG : 2022-09-24/cm-prod-fs-martin-1_2022-09-24_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 08:35:17 DEBUG : 2022-09-24/cm-prod-transcoder-newman_2022-09-24_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 08:35:17 DEBUG : 2022-10-01/cm-prod-api-albert_2022-10-01_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 08:35:17 DEBUG : 2022-10-01/cm-prod-app-joan-1_2022-10-01_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 08:35:17 DEBUG : 2022-10-01/cm-prod-cache-corona_2022-10-01_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 08:35:17 DEBUG : 2022-10-01/cm-prod-db-alexandria_2022-10-01_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 08:35:17 DEBUG : 2022-10-01/cm-prod-fs-martin-1_2022-10-01_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 08:35:17 DEBUG : 2022-10-01/cm-prod-transcoder-newman_2022-10-01_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 08:35:17 DEBUG : 2022-10-08/cm-prod-api-albert_2022-10-08_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 08:35:17 DEBUG : 2022-10-08/cm-prod-app-joan-1_2022-10-08_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 08:35:17 DEBUG : 2022-10-08/cm-prod-cache-corona_2022-10-08_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 08:35:17 DEBUG : 2022-10-08/cm-prod-db-alexandria_2022-10-08_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 08:35:17 DEBUG : 2022-10-08/cm-prod-fs-martin-1_2022-10-08_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 08:35:17 DEBUG : 2022-10-08/cm-prod-transcoder-newman_2022-10-08_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 08:35:17 DEBUG : 2022-10-15/cm-prod-api-albert_2022-10-15_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 08:35:17 DEBUG : 2022-10-15/cm-prod-app-joan-1_2022-10-15_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 08:35:17 DEBUG : 2022-10-15/cm-prod-cache-corona_2022-10-15_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 08:35:17 DEBUG : 2022-10-15/cm-prod-db-alexandria_2022-10-15_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 08:35:17 DEBUG : 2022-10-15/cm-prod-fs-martin-1_2022-10-15_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 08:35:17 DEBUG : 2022-10-15/cm-prod-transcoder-newman_2022-10-15_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 08:35:17 DEBUG : 2022-10-22/cm-prod-api-albert_2022-10-22_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 08:35:18 DEBUG : 2022-10-22/cm-prod-app-joan-1_2022-10-22_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 08:35:18 DEBUG : 2022-10-22/cm-prod-cache-corona_2022-10-22_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 08:35:18 DEBUG : 2022-10-22/cm-prod-db-alexandria_2022-10-22_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 08:35:18 DEBUG : 2022-10-22/cm-prod-fs-martin-1_2022-10-22_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 08:35:18 DEBUG : 2022-10-22/cm-prod-transcoder-newman_2022-10-22_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 08:35:18 DEBUG : 2022-10-29/cm-prod-api-albert_2022-10-29_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 08:35:18 DEBUG : 2022-10-29/cm-prod-app-joan-1_2022-10-29_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 08:35:18 DEBUG : 2022-10-29/cm-prod-cache-corona_2022-10-29_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 08:35:18 DEBUG : 2022-10-29/cm-prod-db-alexandria_2022-10-29_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 08:35:18 DEBUG : 2022-10-29/cm-prod-fs-martin-1_2022-10-29_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 08:35:18 DEBUG : 2022-10-29/cm-prod-transcoder-newman_2022-10-29_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 08:35:18 DEBUG : 2022-11-05/cm-prod-api-albert_2022-11-05_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 08:35:18 DEBUG : 2022-11-05/cm-prod-app-joan-1_2022-11-05_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 08:35:18 DEBUG : 2022-11-05/cm-prod-cache-corona_2022-11-05_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 08:35:18 DEBUG : 2022-11-05/cm-prod-db-alexandria_2022-11-05_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 08:35:18 DEBUG : 2022-11-05/cm-prod-fs-martin-1_2022-11-05_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 08:35:18 DEBUG : 2022-11-05/cm-prod-transcoder-newman_2022-11-05_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 08:35:18 DEBUG : S3 bucket cmtv-backups path scheduled-for-deletion: Waiting for checks to finish
2022/11/07 08:35:18 DEBUG : S3 bucket cmtv-backups path scheduled-for-deletion: Waiting for transfers to finish
2022/11/07 08:35:18 DEBUG : 2022-08-06/cm-prod-fs-martin-1_2022-08-06_0246_backup.tar: Starting  multipart copy with 22 parts
2022/11/07 08:35:18 DEBUG : 2022-08-13/cm-prod-fs-martin-1_2022-08-13_0246_backup.tar: Starting  multipart copy with 22 parts

Animosity022 · November 7, 2022, 1:42pm

That is years old. Update to the latest and please re-test.

Thomas_Defilippis · November 7, 2022, 2:50pm

Thanks for the quick response. I updated to this version:

rclone v1.60.0
- os/version: ubuntu 22.04 (64 bit)
- os/kernel: 5.15.0-52-generic (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.19.2
- go/linking: static
- go/tags: none

Then, I ran the command again, and after 30 minutes still no movement on the 100GB files. Same log files appeared. I can include again if necessary. If I move files to my local, then it shows movement although very slow.

Animosity022 · November 7, 2022, 2:53pm

If you could snag the logs, that would be great.

Thomas_Defilippis · November 7, 2022, 3:07pm

2022/11/07 09:49:27 DEBUG : --min-age 2.8M to 2022-08-15 10:49:27.98604183 -0400 EDT m=-7257599.927050817
2022/11/07 09:49:27 DEBUG : rclone: Version "v1.60.0" starting with parameters ["rclone" "--min-age" "12w" "-P" "move" "wasabi:cmtv-backups/data/" "wasabi:cmtv-backups/scheduled-for-deletion/" "--checkers" "16" "--fast-list" "--timeout" "1m" "--s3-chunk-size" "16M" "--s3-upload-concurrency" "16" "--log-file=/var/log/rclone.log" "--ignore-checksum" "-vv"]
2022/11/07 09:49:27 DEBUG : Creating backend with remote "wasabi:cmtv-backups/data/"
2022/11/07 09:49:27 DEBUG : Using config file from "/home/patmos/.config/rclone/rclone.conf"
2022/11/07 09:49:27 DEBUG : wasabi: detected overridden config - adding "{ZHuaN}" suffix to name
2022/11/07 09:49:27 NOTICE: s3: s3 provider "" not known - please set correctly
2022/11/07 09:49:27 DEBUG : fs cache: renaming cache item "wasabi:cmtv-backups/data/" to be canonical "wasabi{ZHuaN}:cmtv-backups/data"
2022/11/07 09:49:27 DEBUG : Creating backend with remote "wasabi:cmtv-backups/scheduled-for-deletion/"
2022/11/07 09:49:27 DEBUG : wasabi: detected overridden config - adding "{ZHuaN}" suffix to name
2022/11/07 09:49:27 NOTICE: s3: s3 provider "" not known - please set correctly
2022/11/07 09:49:27 DEBUG : fs cache: renaming cache item "wasabi:cmtv-backups/scheduled-for-deletion/" to be canonical "wasabi{ZHuaN}:cmtv-backups/scheduled-for-deletion"
2022/11/07 09:49:28 DEBUG : 2022-08-20/cm-prod-api-albert_2022-08-20_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 09:49:28 DEBUG : 2022-08-20/cm-prod-app-joan-1_2022-08-20_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 09:49:28 DEBUG : 2022-08-20/cm-prod-cache-corona_2022-08-20_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 09:49:28 DEBUG : 2022-08-20/cm-prod-db-alexandria_2022-08-20_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 09:49:28 DEBUG : 2022-08-20/cm-prod-fs-martin-1_2022-08-20_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 09:49:28 DEBUG : 2022-08-20/cm-prod-transcoder-newman_2022-08-20_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 09:49:28 DEBUG : 2022-08-27/cm-prod-api-albert_2022-08-27_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 09:49:28 DEBUG : 2022-08-27/cm-prod-app-joan-1_2022-08-27_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 09:49:28 DEBUG : 2022-08-27/cm-prod-cache-corona_2022-08-27_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 09:49:28 DEBUG : 2022-08-27/cm-prod-db-alexandria_2022-08-27_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 09:49:28 DEBUG : 2022-08-27/cm-prod-fs-martin-1_2022-08-27_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 09:49:29 DEBUG : 2022-08-27/cm-prod-transcoder-newman_2022-08-27_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 09:49:29 DEBUG : 2022-09-03/cm-prod-api-albert_2022-09-03_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 09:49:29 DEBUG : 2022-09-03/cm-prod-app-joan-1_2022-09-03_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 09:49:29 DEBUG : 2022-09-03/cm-prod-cache-corona_2022-09-03_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 09:49:29 DEBUG : 2022-09-03/cm-prod-db-alexandria_2022-09-03_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 09:49:29 DEBUG : 2022-09-03/cm-prod-fs-martin-1_2022-09-03_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 09:49:29 DEBUG : 2022-09-03/cm-prod-transcoder-newman_2022-09-03_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 09:49:29 DEBUG : 2022-09-10/cm-prod-api-albert_2022-09-10_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 09:49:29 DEBUG : 2022-09-10/cm-prod-app-joan-1_2022-09-10_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 09:49:29 DEBUG : 2022-09-10/cm-prod-cache-corona_2022-09-10_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 09:49:29 DEBUG : 2022-09-10/cm-prod-db-alexandria_2022-09-10_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 09:49:29 DEBUG : 2022-09-10/cm-prod-fs-martin-1_2022-09-10_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 09:49:29 DEBUG : 2022-09-10/cm-prod-transcoder-newman_2022-09-10_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 09:49:29 DEBUG : 2022-09-17/cm-prod-api-albert_2022-09-17_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 09:49:29 DEBUG : 2022-09-17/cm-prod-app-joan-1_2022-09-17_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 09:49:29 DEBUG : 2022-09-17/cm-prod-cache-corona_2022-09-17_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 09:49:30 DEBUG : 2022-09-17/cm-prod-db-alexandria_2022-09-17_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 09:49:30 DEBUG : 2022-09-17/cm-prod-fs-martin-1_2022-09-17_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 09:49:30 DEBUG : 2022-09-17/cm-prod-transcoder-newman_2022-09-17_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 09:49:30 DEBUG : 2022-09-24/cm-prod-api-albert_2022-09-24_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 09:49:30 DEBUG : 2022-09-24/cm-prod-app-joan-1_2022-09-24_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 09:49:30 DEBUG : 2022-09-24/cm-prod-cache-corona_2022-09-24_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 09:49:30 DEBUG : 2022-09-24/cm-prod-db-alexandria_2022-09-24_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 09:49:30 DEBUG : 2022-09-24/cm-prod-fs-martin-1_2022-09-24_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 09:49:30 DEBUG : 2022-09-24/cm-prod-transcoder-newman_2022-09-24_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 09:49:30 DEBUG : 2022-10-01/cm-prod-api-albert_2022-10-01_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 09:49:30 DEBUG : 2022-10-01/cm-prod-app-joan-1_2022-10-01_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 09:49:30 DEBUG : 2022-10-01/cm-prod-cache-corona_2022-10-01_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 09:49:30 DEBUG : 2022-10-01/cm-prod-db-alexandria_2022-10-01_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 09:49:30 DEBUG : 2022-10-01/cm-prod-fs-martin-1_2022-10-01_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 09:49:30 DEBUG : 2022-10-01/cm-prod-transcoder-newman_2022-10-01_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 09:49:30 DEBUG : 2022-10-08/cm-prod-api-albert_2022-10-08_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 09:49:30 DEBUG : 2022-10-08/cm-prod-app-joan-1_2022-10-08_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 09:49:30 DEBUG : 2022-10-08/cm-prod-cache-corona_2022-10-08_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 09:49:30 DEBUG : 2022-10-08/cm-prod-db-alexandria_2022-10-08_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 09:49:30 DEBUG : 2022-10-08/cm-prod-fs-martin-1_2022-10-08_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 09:49:30 DEBUG : 2022-10-08/cm-prod-transcoder-newman_2022-10-08_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 09:49:30 DEBUG : 2022-10-15/cm-prod-api-albert_2022-10-15_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 09:49:30 DEBUG : 2022-10-15/cm-prod-app-joan-1_2022-10-15_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 09:49:31 DEBUG : 2022-10-15/cm-prod-cache-corona_2022-10-15_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 09:49:31 DEBUG : 2022-10-15/cm-prod-db-alexandria_2022-10-15_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 09:49:31 DEBUG : 2022-10-15/cm-prod-fs-martin-1_2022-10-15_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 09:49:31 DEBUG : 2022-10-15/cm-prod-transcoder-newman_2022-10-15_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 09:49:31 DEBUG : 2022-10-22/cm-prod-api-albert_2022-10-22_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 09:49:31 DEBUG : 2022-10-22/cm-prod-app-joan-1_2022-10-22_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 09:49:31 DEBUG : 2022-10-22/cm-prod-cache-corona_2022-10-22_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 09:49:31 DEBUG : 2022-10-22/cm-prod-db-alexandria_2022-10-22_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 09:49:31 DEBUG : 2022-10-22/cm-prod-fs-martin-1_2022-10-22_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 09:49:31 DEBUG : 2022-10-22/cm-prod-transcoder-newman_2022-10-22_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 09:49:31 DEBUG : 2022-10-29/cm-prod-api-albert_2022-10-29_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 09:49:31 DEBUG : 2022-10-29/cm-prod-app-joan-1_2022-10-29_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 09:49:31 DEBUG : 2022-10-29/cm-prod-cache-corona_2022-10-29_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 09:49:31 DEBUG : 2022-10-29/cm-prod-db-alexandria_2022-10-29_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 09:49:31 DEBUG : 2022-10-29/cm-prod-fs-martin-1_2022-10-29_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 09:49:31 DEBUG : 2022-10-29/cm-prod-transcoder-newman_2022-10-29_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 09:49:31 DEBUG : 2022-11-05/cm-prod-api-albert_2022-11-05_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 09:49:31 DEBUG : 2022-11-05/cm-prod-app-joan-1_2022-11-05_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 09:49:31 DEBUG : 2022-11-05/cm-prod-cache-corona_2022-11-05_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 09:49:31 DEBUG : 2022-11-05/cm-prod-db-alexandria_2022-11-05_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 09:49:31 DEBUG : 2022-11-05/cm-prod-fs-martin-1_2022-11-05_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 09:49:31 DEBUG : 2022-11-05/cm-prod-transcoder-newman_2022-11-05_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 09:49:31 DEBUG : S3 bucket cmtv-backups path scheduled-for-deletion: Waiting for checks to finish
2022/11/07 09:49:31 DEBUG : S3 bucket cmtv-backups path scheduled-for-deletion: Waiting for transfers to finish
2022/11/07 09:49:32 DEBUG : 2022-08-06/cm-prod-fs-martin-1_2022-08-06_0246_backup.tar: Starting  multipart copy with 22 parts
2022/11/07 09:49:32 DEBUG : 2022-08-13/cm-prod-fs-martin-1_2022-08-13_0246_backup.tar: Starting  multipart copy with 22 parts
2022/11/07 10:01:18 DEBUG : pacer: low level retry 1/2 (error RequestError: send request failed
caused by: Put "https://s3.us-west-1.wasabisys.com/cmtv-backups/scheduled-for-deletion/2022-08-13/cm-prod-fs-martin-1_2022-08-13_0246_backup.tar?partNumber=1&uploadId=evC8iXQZcEhMMU5_fDDQkryhV_o9vSnOHKFpxMLKLZVH-mbz_puPixWf_lA5fggoHhmBwY5p6B0JnM8oTB8krBsCJPuvBO80T3OlUWpOuXzKmwCvkIYKE5QFWbMJmDj9": net/http: timeout awaiting response headers)
2022/11/07 10:01:18 DEBUG : pacer: Rate limited, increasing sleep to 10ms
2022/11/07 10:01:20 DEBUG : pacer: low level retry 1/2 (error RequestError: send request failed
caused by: Put "https://s3.us-west-1.wasabisys.com/cmtv-backups/scheduled-for-deletion/2022-08-06/cm-prod-fs-martin-1_2022-08-06_0246_backup.tar?partNumber=1&uploadId=VgJriRPv00AJHD2XFvO19Aq34E3Y9AER_VnDbRZjzM22jIOpoQwZ1nHMW_5kKdc3i6t6EMiGQl-g7nDpQmUGfZ7POaFb_HVfbSwzjPnhUFomskvNQ1gPOzZ0BxFIug1j": net/http: timeout awaiting response headers)
2022/11/07 10:01:20 DEBUG : pacer: Rate limited, increasing sleep to 20ms

Thomas_Defilippis · November 7, 2022, 3:57pm

Been running the command again for over an 1h and no movement. New logs though. Looks like it keeps timing out.

caused by: Put "https://s3.us-west-1.wasabisys.com/cmtv-backups/scheduled-for-deletion/2022-08-06/cm-prod-fs-martin-1_2022-08-06_0246_backup.tar?partNumber=1&uploadId=q9PBVxArLmMPhXCzwOhjEhQIrR05X9nW9pOS_WIGiyMfSc0ckZf2cYtFFwippdDBO2IvUjbfGhJyfjjqiCTIJF1RHNxkCysy_YmBPlV5Y66DIyVB_Qz0qisDJkifp9UB": net/http: timeout awaiting response headers)
2022/11/07 10:24:44 DEBUG : pacer: Rate limited, increasing sleep to 50.625ms
2022/11/07 10:24:58 DEBUG : pacer: low level retry 1/2 (error RequestError: send request failed
caused by: Put "https://s3.us-west-1.wasabisys.com/cmtv-backups/scheduled-for-deletion/2022-08-13/cm-prod-fs-martin-1_2022-08-13_0246_backup.tar?partNumber=1&uploadId=8VbvYhLQXomLpSKdVNE3LDHQuBsOXsQ2E8SGj4e8cOs90oIzW1gDbzXz3O4RQQRL2RdMQe7ScOHIbjgQJRGcYBrmZ8tJAvZzuxJo-AuuAax87rDPt87nKqfdIWEDLTg2": net/http: timeout awaiting response headers)
2022/11/07 10:24:58 DEBUG : pacer: Rate limited, increasing sleep to 101.25ms
2022/11/07 10:36:30 DEBUG : pacer: low level retry 2/2 (error RequestError: send request failed
caused by: Put "https://s3.us-west-1.wasabisys.com/cmtv-backups/scheduled-for-deletion/2022-08-06/cm-prod-fs-martin-1_2022-08-06_0246_backup.tar?partNumber=1&uploadId=q9PBVxArLmMPhXCzwOhjEhQIrR05X9nW9pOS_WIGiyMfSc0ckZf2cYtFFwippdDBO2IvUjbfGhJyfjjqiCTIJF1RHNxkCysy_YmBPlV5Y66DIyVB_Qz0qisDJkifp9UB": net/http: timeout awaiting response headers)
2022/11/07 10:36:30 DEBUG : pacer: Rate limited, increasing sleep to 202.5ms
2022/11/07 10:36:30 DEBUG : 2022-08-06/cm-prod-fs-martin-1_2022-08-06_0246_backup.tar: Cancelling multipart copy
2022/11/07 10:36:31 DEBUG : pacer: Reducing sleep to 151.875ms
2022/11/07 10:36:31 DEBUG : 2022-08-06/cm-prod-fs-martin-1_2022-08-06_0246_backup.tar: Received error: RequestError: send request failed
caused by: Put "https://s3.us-west-1.wasabisys.com/cmtv-backups/scheduled-for-deletion/2022-08-06/cm-prod-fs-martin-1_2022-08-06_0246_backup.tar?partNumber=1&uploadId=q9PBVxArLmMPhXCzwOhjEhQIrR05X9nW9pOS_WIGiyMfSc0ckZf2cYtFFwippdDBO2IvUjbfGhJyfjjqiCTIJF1RHNxkCysy_YmBPlV5Y66DIyVB_Qz0qisDJkifp9UB": net/http: timeout awaiting response headers - low level retry 2/10
2022/11/07 10:36:31 DEBUG : pacer: Reducing sleep to 113.90625ms
2022/11/07 10:36:31 DEBUG : 2022-08-06/cm-prod-fs-martin-1_2022-08-06_0246_backup.tar: Starting  multipart copy with 22 parts
2022/11/07 10:36:44 DEBUG : pacer: low level retry 2/2 (error RequestError: send request failed
caused by: Put "https://s3.us-west-1.wasabisys.com/cmtv-backups/scheduled-for-deletion/2022-08-13/cm-prod-fs-martin-1_2022-08-13_0246_backup.tar?partNumber=1&uploadId=8VbvYhLQXomLpSKdVNE3LDHQuBsOXsQ2E8SGj4e8cOs90oIzW1gDbzXz3O4RQQRL2RdMQe7ScOHIbjgQJRGcYBrmZ8tJAvZzuxJo-AuuAax87rDPt87nKqfdIWEDLTg2": net/http: timeout awaiting response headers)
2022/11/07 10:36:44 DEBUG : pacer: Rate limited, increasing sleep to 227.8125ms
2022/11/07 10:36:44 DEBUG : 2022-08-13/cm-prod-fs-martin-1_2022-08-13_0246_backup.tar: Cancelling multipart copy
2022/11/07 10:36:44 DEBUG : pacer: Reducing sleep to 170.859375ms
2022/11/07 10:36:44 DEBUG : 2022-08-13/cm-prod-fs-martin-1_2022-08-13_0246_backup.tar: Received error: RequestError: send request failed
caused by: Put "https://s3.us-west-1.wasabisys.com/cmtv-backups/scheduled-for-deletion/2022-08-13/cm-prod-fs-martin-1_2022-08-13_0246_backup.tar?partNumber=1&uploadId=8VbvYhLQXomLpSKdVNE3LDHQuBsOXsQ2E8SGj4e8cOs90oIzW1gDbzXz3O4RQQRL2RdMQe7ScOHIbjgQJRGcYBrmZ8tJAvZzuxJo-AuuAax87rDPt87nKqfdIWEDLTg2": net/http: timeout awaiting response headers - low level retry 2/10
2022/11/07 10:36:44 DEBUG : pacer: Reducing sleep to 128.144531ms
2022/11/07 10:36:44 DEBUG : 2022-08-13/cm-prod-fs-martin-1_2022-08-13_0246_backup.tar: Starting  multipart copy with 22 parts
2022/11/07 10:48:14 DEBUG : pacer: low level retry 1/2 (error RequestError: send request failed
caused by: Put "https://s3.us-west-1.wasabisys.com/cmtv-backups/scheduled-for-deletion/2022-08-06/cm-prod-fs-martin-1_2022-08-06_0246_backup.tar?partNumber=1&uploadId=xVbBo2M1pKBOY3GIAmn8NWuwe-1oPvCbJ-SDC-xmn597XeKFoCGjc4bdz86w1cEZeRYSJm7WOJj-r6GVx_nwi0Ll1iwFAuYJFNYX9ggrVoqV_nwGEDdISpIP2sKCHYv6": net/http: timeout awaiting response headers)
2022/11/07 10:48:14 DEBUG : pacer: Rate limited, increasing sleep to 256.289062ms
2022/11/07 10:48:29 DEBUG : pacer: low level retry 1/2 (error RequestError: send request failed
caused by: Put "https://s3.us-west-1.wasabisys.com/cmtv-backups/scheduled-for-deletion/2022-08-13/cm-prod-fs-martin-1_2022-08-13_0246_backup.tar?partNumber=1&uploadId=NvCMpLn0SQoCbCXumFnH_C8xZgntpFpJvjWA53f0ChCNLHIPB_tHQLpdEbhoG76C3KKynLk-c6iNJd5e17M8c43aDtbINh4WJSm7hB--5G97Ljajm9iaTeg0UAlheM_l": net/http: timeout awaiting response headers)
2022/11/07 10:48:29 DEBUG : pacer: Rate limited, increasing sleep to 512.578124ms

Animosity022 · November 7, 2022, 4:09pm

Why did you set these?

Thomas_Defilippis · November 7, 2022, 4:29pm

I was trying to run this command without those flags and getting similar results, and in trying to troubleshoot, I tried to see if setting these valuse --s3-chunk-size and --s3-upload-concurrency had any effect. Honestly, got the idea from a forum post that may have been unrelated, so I am not entirely sure of all their implications and effects. I am running the command again without these flags to verify my result.

Animosity022 · November 7, 2022, 4:31pm

Gotcha.

I'm not quite sure why it wouldn't server side move? Is that due to the server side encryption I'd imagine?

Ole · November 7, 2022, 4:36pm

Not expert here, but could this cause rclone to ommit some Wasabi specific handling:

Thomas_Defilippis · November 7, 2022, 5:00pm

Good catch. Didn't spot that. So it turns out that I didn't have the provider set to Wasabi in the config. Guess that part got lost in the setup. So, I added this and that log went away. Here are the new logs:

2022/11/07 11:55:56 DEBUG : rclone: Version "v1.60.0" starting with parameters ["rclone" "--min-age" "12w" "-P" "move" "wasabi:cmtv-backups/data/" "wasabi:cmtv-backups/scheduled-for-deletion/" "--checkers" "16" "--fast-list" "--timeout" "1m" "--log-file=/var/log/rclone.log" "-vv"]
2022/11/07 11:55:56 DEBUG : Creating backend with remote "wasabi:cmtv-backups/data/"
2022/11/07 11:55:56 DEBUG : Using config file from "/home/patmos/.config/rclone/rclone.conf"
2022/11/07 11:55:56 DEBUG : fs cache: renaming cache item "wasabi:cmtv-backups/data/" to be canonical "wasabi:cmtv-backups/data"
2022/11/07 11:55:56 DEBUG : Creating backend with remote "wasabi:cmtv-backups/scheduled-for-deletion/"
2022/11/07 11:55:56 DEBUG : fs cache: renaming cache item "wasabi:cmtv-backups/scheduled-for-deletion/" to be canonical "wasabi:cmtv-backups/scheduled-for-deletion"
2022/11/07 11:55:57 DEBUG : 2022-08-20/cm-prod-api-albert_2022-08-20_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 11:55:57 DEBUG : 2022-08-20/cm-prod-app-joan-1_2022-08-20_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 11:55:57 DEBUG : 2022-08-20/cm-prod-cache-corona_2022-08-20_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 11:55:57 DEBUG : 2022-08-20/cm-prod-db-alexandria_2022-08-20_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 11:55:57 DEBUG : 2022-08-20/cm-prod-fs-martin-1_2022-08-20_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 11:55:57 DEBUG : 2022-08-20/cm-prod-transcoder-newman_2022-08-20_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 11:55:57 DEBUG : 2022-08-27/cm-prod-api-albert_2022-08-27_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 11:55:57 DEBUG : 2022-08-27/cm-prod-app-joan-1_2022-08-27_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 11:55:57 DEBUG : 2022-08-27/cm-prod-cache-corona_2022-08-27_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 11:55:57 DEBUG : 2022-08-27/cm-prod-db-alexandria_2022-08-27_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 11:55:57 DEBUG : 2022-08-27/cm-prod-fs-martin-1_2022-08-27_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 11:55:57 DEBUG : 2022-08-27/cm-prod-transcoder-newman_2022-08-27_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 11:55:57 DEBUG : 2022-09-03/cm-prod-api-albert_2022-09-03_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 11:55:57 DEBUG : 2022-09-03/cm-prod-app-joan-1_2022-09-03_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 11:55:57 DEBUG : 2022-09-03/cm-prod-cache-corona_2022-09-03_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 11:55:57 DEBUG : 2022-09-03/cm-prod-db-alexandria_2022-09-03_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 11:55:57 DEBUG : 2022-09-03/cm-prod-fs-martin-1_2022-09-03_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 11:55:57 DEBUG : 2022-09-03/cm-prod-transcoder-newman_2022-09-03_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 11:55:57 DEBUG : 2022-09-10/cm-prod-api-albert_2022-09-10_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 11:55:57 DEBUG : 2022-09-10/cm-prod-app-joan-1_2022-09-10_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 11:55:57 DEBUG : 2022-09-10/cm-prod-cache-corona_2022-09-10_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 11:55:57 DEBUG : 2022-09-10/cm-prod-db-alexandria_2022-09-10_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 11:55:58 DEBUG : 2022-09-10/cm-prod-fs-martin-1_2022-09-10_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 11:55:58 DEBUG : 2022-09-10/cm-prod-transcoder-newman_2022-09-10_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 11:55:58 DEBUG : 2022-09-17/cm-prod-api-albert_2022-09-17_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 11:55:58 DEBUG : 2022-09-17/cm-prod-app-joan-1_2022-09-17_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 11:55:58 DEBUG : 2022-09-17/cm-prod-cache-corona_2022-09-17_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 11:55:58 DEBUG : 2022-09-17/cm-prod-db-alexandria_2022-09-17_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 11:55:58 DEBUG : 2022-09-17/cm-prod-fs-martin-1_2022-09-17_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 11:55:58 DEBUG : 2022-09-17/cm-prod-transcoder-newman_2022-09-17_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 11:55:58 DEBUG : 2022-09-24/cm-prod-api-albert_2022-09-24_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 11:55:58 DEBUG : 2022-09-24/cm-prod-app-joan-1_2022-09-24_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 11:55:58 DEBUG : 2022-09-24/cm-prod-cache-corona_2022-09-24_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 11:55:58 DEBUG : 2022-09-24/cm-prod-db-alexandria_2022-09-24_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 11:55:58 DEBUG : 2022-09-24/cm-prod-fs-martin-1_2022-09-24_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 11:55:58 DEBUG : 2022-09-24/cm-prod-transcoder-newman_2022-09-24_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 11:55:59 DEBUG : 2022-10-01/cm-prod-api-albert_2022-10-01_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 11:55:59 DEBUG : 2022-10-01/cm-prod-app-joan-1_2022-10-01_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 11:55:59 DEBUG : 2022-10-01/cm-prod-cache-corona_2022-10-01_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 11:55:59 DEBUG : 2022-10-01/cm-prod-db-alexandria_2022-10-01_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 11:55:59 DEBUG : 2022-10-01/cm-prod-fs-martin-1_2022-10-01_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 11:55:59 DEBUG : 2022-10-01/cm-prod-transcoder-newman_2022-10-01_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 11:55:59 DEBUG : 2022-10-08/cm-prod-api-albert_2022-10-08_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 11:55:59 DEBUG : 2022-10-08/cm-prod-app-joan-1_2022-10-08_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 11:55:59 DEBUG : 2022-10-08/cm-prod-cache-corona_2022-10-08_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 11:55:59 DEBUG : 2022-10-08/cm-prod-db-alexandria_2022-10-08_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 11:55:59 DEBUG : 2022-10-08/cm-prod-fs-martin-1_2022-10-08_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 11:55:59 DEBUG : 2022-10-08/cm-prod-transcoder-newman_2022-10-08_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 11:55:59 DEBUG : 2022-10-15/cm-prod-api-albert_2022-10-15_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 11:55:59 DEBUG : 2022-10-15/cm-prod-app-joan-1_2022-10-15_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 11:55:59 DEBUG : 2022-10-15/cm-prod-cache-corona_2022-10-15_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 11:55:59 DEBUG : 2022-10-15/cm-prod-db-alexandria_2022-10-15_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 11:55:59 DEBUG : 2022-10-15/cm-prod-fs-martin-1_2022-10-15_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 11:55:59 DEBUG : 2022-10-15/cm-prod-transcoder-newman_2022-10-15_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 11:55:59 DEBUG : 2022-10-22/cm-prod-api-albert_2022-10-22_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 11:55:59 DEBUG : 2022-10-22/cm-prod-app-joan-1_2022-10-22_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 11:55:59 DEBUG : 2022-10-22/cm-prod-cache-corona_2022-10-22_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 11:56:00 DEBUG : 2022-10-22/cm-prod-db-alexandria_2022-10-22_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 11:56:00 DEBUG : 2022-10-22/cm-prod-fs-martin-1_2022-10-22_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 11:56:00 DEBUG : 2022-10-22/cm-prod-transcoder-newman_2022-10-22_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 11:56:00 DEBUG : 2022-10-29/cm-prod-api-albert_2022-10-29_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 11:56:00 DEBUG : 2022-10-29/cm-prod-app-joan-1_2022-10-29_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 11:56:00 DEBUG : 2022-10-29/cm-prod-cache-corona_2022-10-29_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 11:56:00 DEBUG : 2022-10-29/cm-prod-db-alexandria_2022-10-29_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 11:56:00 DEBUG : 2022-10-29/cm-prod-fs-martin-1_2022-10-29_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 11:56:00 DEBUG : 2022-10-29/cm-prod-transcoder-newman_2022-10-29_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 11:56:00 DEBUG : 2022-11-05/cm-prod-api-albert_2022-11-05_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 11:56:00 DEBUG : 2022-11-05/cm-prod-app-joan-1_2022-11-05_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 11:56:00 DEBUG : 2022-11-05/cm-prod-cache-corona_2022-11-05_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 11:56:00 DEBUG : 2022-11-05/cm-prod-db-alexandria_2022-11-05_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 11:56:00 DEBUG : 2022-11-05/cm-prod-fs-martin-1_2022-11-05_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 11:56:00 DEBUG : 2022-11-05/cm-prod-transcoder-newman_2022-11-05_0246_backup.tar: Excluded from sync (and deletion)
2022/11/07 11:56:00 DEBUG : S3 bucket cmtv-backups path scheduled-for-deletion: Waiting for checks to finish
2022/11/07 11:56:00 DEBUG : S3 bucket cmtv-backups path scheduled-for-deletion: Waiting for transfers to finish
2022/11/07 11:56:00 DEBUG : 2022-08-06/cm-prod-fs-martin-1_2022-08-06_0246_backup.tar: Starting  multipart copy with 22 parts
2022/11/07 11:56:01 DEBUG : 2022-08-13/cm-prod-fs-martin-1_2022-08-13_0246_backup.tar: Starting  multipart copy with 22 parts

I am running the command again to see if I get any changes.

Thomas_Defilippis · November 7, 2022, 5:01pm

It could be caused by server-side encryption, but why are smaller files successfully moving with this command. I am only experiencing this issue with two large .tar files that are both 100GB in size.

Thomas_Defilippis · November 7, 2022, 5:09pm

Yeah, so its been running for 12 minutes and I am getting these logs:

2022/11/07 12:07:46 DEBUG : pacer: low level retry 1/2 (error RequestError: send request failed
caused by: Put "https://cmtv-backups.s3.us-west-1.wasabisys.com/scheduled-for-deletion/2022-08-13/cm-prod-fs-martin-1_2022-08-13_0246_backup.tar?partNumber=1&uploadId=LMrRcnaXXfNrBNgRD9gogfeNLZd-97-uXriP9EDMQ5vWDYrJX1sLfLci_62sOwGNsHzyqAt0d0MoXAJ8nMW3818my3F_uPzZwVZFXnoSVp_0Clp16kAPtzVxm-hvUGK3": net/http: timeout awaiting response headers)
2022/11/07 12:07:46 DEBUG : pacer: Rate limited, increasing sleep to 10ms
2022/11/07 12:07:58 DEBUG : pacer: low level retry 1/2 (error RequestError: send request failed
caused by: Put "https://cmtv-backups.s3.us-west-1.wasabisys.com/scheduled-for-deletion/2022-08-06/cm-prod-fs-martin-1_2022-08-06_0246_backup.tar?partNumber=1&uploadId=xdacCZjMGHS-m97PDUUOLKYG1FA-bqd-ER-kVB3nSJsmFd9vWEKOpMQGGf2F2ya9dv9EyVDRcn5UEgnVP1XUdDzmr04MZLLkpGMOMXJyTbeeNm_8RQ1HsTu-mx_IBDW-": net/http: timeout awaiting response headers)
2022/11/07 12:07:58 DEBUG : pacer: Rate limited, increasing sleep to 20ms

asdffdsa · November 7, 2022, 5:10pm

hi,
not expecting to get pacer messages with wasabi.
what was the exact command you ran, as the top lines of the debug log are missing?

Thomas_Defilippis · November 7, 2022, 5:12pm

rclone --min-age 12w -P move wasabi:cmtv-backups/data/ wasabi:cmtv-backups/scheduled-for-deletion/ --checkers 16 --fast-list --timeout 1m --log-file=/var/log/rclone.log -vv

I am removing all flags to see if it changes anything like this:
rclone --min-age 12w -P move wasabi:cmtv-backups/data/ wasabi:cmtv-backups/scheduled-for-deletion/ --log-file=/var/log/rclone.log -vv

asdffdsa · November 7, 2022, 5:13pm

why did you add that?

edit: i see that you re-edited your last post about removing that flag.

Thomas_Defilippis · November 7, 2022, 6:56pm

Well, I added it naively from another command that one of our developers used in a different command. The timeout flag seemed to be the issue. It took an hour to move all the files, but they succeeded now. In the future, I will be more careful with the flags that I use.

So, to recap. I added provider = Wasabi in my config, and I ran this command:

rclone --min-age 12w -P move wasabi:cmtv-backups/data/ wasabi:cmtv-backups/scheduled-for-deletion/ --log-file=/var/log/rclone.log -vv

and that worked.

Here are the logs:

2022/11/07 12:12:50 DEBUG : 2022-08-13/cm-prod-fs-martin-1_2022-08-13_0246_backup.tar: Starting  multipart copy with 22 parts
2022/11/07 12:12:50 DEBUG : 2022-10-29/cm-prod-transcoder-newman_2022-10-29_0246_backup.tar: Excluded
2022/11/07 12:12:50 DEBUG : S3 bucket cmtv-backups path scheduled-for-deletion: Waiting for checks to finish
2022/11/07 12:12:50 DEBUG : S3 bucket cmtv-backups path scheduled-for-deletion: Waiting for transfers to finish
2022/11/07 13:20:50 DEBUG : 2022-08-13/cm-prod-fs-martin-1_2022-08-13_0246_backup.tar: md5 = 89326dd0e74f916a090869cc0022ab47 OK
2022/11/07 13:20:50 INFO  : 2022-08-13/cm-prod-fs-martin-1_2022-08-13_0246_backup.tar: Copied (server-side copy)
2022/11/07 13:20:50 INFO  : 2022-08-13/cm-prod-fs-martin-1_2022-08-13_0246_backup.tar: Deleted
2022/11/07 13:25:47 DEBUG : 2022-08-06/cm-prod-fs-martin-1_2022-08-06_0246_backup.tar: md5 = 3df54c008aab0ab5c4a0ade0f43e00fd OK
2022/11/07 13:25:47 INFO  : 2022-08-06/cm-prod-fs-martin-1_2022-08-06_0246_backup.tar: Copied (server-side copy)
2022/11/07 13:25:47 INFO  : 2022-08-06/cm-prod-fs-martin-1_2022-08-06_0246_backup.tar: Deleted
2022/11/07 13:25:47 INFO  : 
Transferred:   	  201.143 GiB / 201.143 GiB, 100%, 6.272 GiB/s, ETA 0s
Checks:                 4 / 4, 100%
Deleted:                2 (files), 0 (dirs)
Renamed:                2
Transferred:            2 / 2, 100%
Elapsed time:   1h12m58.4s

2022/11/07 13:25:47 DEBUG : 9 go routines active

Thanks for the help.

asdffdsa · November 7, 2022, 7:00pm

sure, and wanted to be sure there was not issue with wasabi, as i am heavily dependent on it.

fwiw, using rclone defaults, can easily saturate with a 1Gbps internet to wasabi.

system · December 7, 2022, 7:00pm

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.