Rclone copying failing for a Compressed, Chunked Remote

jxmese · May 29, 2024, 1:42pm

What is the problem you are having with rclone?

Copying failing for a compressed remote.

Run the command 'rclone version' and share the full output of the command.

rclone v1.66.0
- os/version: Microsoft Windows Server 2022 Standard 21H2 (64 bit)
- os/kernel: 10.0.20348.2402 (x86_64)
- os/type: windows
- os/arch: amd64
- go/version: go1.22.1
- go/linking: static
- go/tags: cmount

Which cloud storage system are you using? (eg Google Drive)

Blomp

The command you were trying to run (eg `rclone copy /tmp remote:tmp`)

rclone copy HAE:"July-DD-0/Incident.mp4" blompc-gzip:To-delete -P -vv

Please run 'rclone config redacted' and share the full output. If you get command not found, please make sure to update rclone.

[HA]
type = drive
client_id = XXX
client_secret = XXX
scope = drive
token = XXX
root_folder_id = XXX

[HAE]
type = crypt
remote = HA:E
filename_encryption = standard
directory_name_encryption = true
password = XXX
password2 = XXX

[blomp]
type = swift
user = XXX
key = XXX
auth = https://authenticate.ain.net
tenant = XXX
auth_version = 2
endpoint_type = public
leave_parts_on_error = true
chunk_size = 1P
no_chunk = false

[blompc]
type = crypt
remote = blomp:x@gmail.com/E
password = XXX
password2 = XXX

[blompc-chunker]
type = chunker
remote = blompc:Google-archive
chunk_size = 4Gi

[blompc-gzip]
type = compress
remote = blompc-chunker:
level = 9

A log from the command that you were trying to run with the `-vv` flag

rclone copy HAE:"July-DD-0/Incident.mp4" blompc-gzip:To-delete -P -vv
2024/05/29 16:34:17 DEBUG : rclone: Version "v1.66.0" starting with parameters ["C:\\ProgramData\\chocolatey\\lib\\rclone.portable\\tools\\rclone-v1.66.0-windows-amd64\\rclone.exe" "copy" "HAE:July-DD-0/Incident.mp4" "blompc-gzip:To-delete" "-P" "-vv"]
2024/05/29 16:34:17 DEBUG : Creating backend with remote "HAE:July-DD-0/Incident.mp4"
2024/05/29 16:34:17 DEBUG : Using config file from "C:\\Users\\Administrator\\AppData\\Roaming\\rclone\\rclone.conf"
2024/05/29 16:34:17 DEBUG : Creating backend with remote "HA:E/3lq8375g5ek9tvpd5rirfvudhg/792tc1de92pkpoidc2vkousd1og5t6dpngad26ev7cejn996fa37j0qbe6pi5erlbnaqhjpjlf2im2770mn0h24egt9uiovuj3gq3ho"
2024/05/29 16:34:19 DEBUG : fs cache: adding new entry for parent of "HA:E/3lq8375g5ek9tvpd5rirfvudhg/792tc1de92pkpoidc2vkousd1og5t6dpngad26ev7cejn996fa37j0qbe6pi5erlbnaqhjpjlf2im2770mn0h24egt9uiovuj3gq3ho", "HA:E/3lq8375g5ek9tvpd5rirfvudhg"
2024/05/29 16:34:19 DEBUG : fs cache: adding new entry for parent of "HAE:July-DD-0/Incident.mp4", "HAE:July-DD-0"
2024/05/29 16:34:19 DEBUG : Creating backend with remote "blompc-gzip:To-delete"
2024/05/29 16:34:19 DEBUG : Creating backend with remote "blompc:Google-archive/To-delete.json"
2024/05/29 16:34:19 DEBUG : Creating backend with remote "blomp:x@gmail.com/E/di2uds15b0t1hbr8dojh865qe4/7ngijcba77d1q2niieo54g3a7o"
2024/05/29 16:34:21 DEBUG : Reset feature "ListR"
2024/05/29 16:34:21 DEBUG : Creating backend with remote "blompc:Google-archive/To-delete"
2024/05/29 16:34:21 DEBUG : Creating backend with remote "blomp:x@gmail.com/E/di2uds15b0t1hbr8dojh865qe4/mmqtmcqfttm8hurikhvfjg5p08"
2024/05/29 16:34:22 DEBUG : Reset feature "ListR"
2024/05/29 16:34:23 DEBUG : Incident.mp4: Need to transfer - File not found at Destination
2024/05/29 16:34:52 ERROR : Incident.mp4: Failed to copy: Operation forbidden
2024/05/29 16:34:52 ERROR : Attempt 1/3 failed with 1 errors and: Operation forbidden
2024/05/29 16:34:53 DEBUG : Incident.mp4: Need to transfer - File not found at Destination
2024/05/29 16:34:59 ERROR : Incident.mp4: Failed to copy: Operation forbidden
2024/05/29 16:34:59 ERROR : Attempt 2/3 failed with 1 errors and: Operation forbidden
2024/05/29 16:35:01 DEBUG : Incident.mp4: Need to transfer - File not found at Destination
2024/05/29 16:35:06 ERROR : Incident.mp4: Failed to copy: Operation forbidden
2024/05/29 16:35:06 ERROR : Attempt 3/3 failed with 1 errors and: Operation forbidden
Transferred:           60 MiB / 60 MiB, 100%, 3.315 MiB/s, ETA 0s
Errors:                 1 (retrying may help)
Elapsed time:        48.5s
2024/05/29 16:35:06 INFO  :
Transferred:           60 MiB / 60 MiB, 100%, 3.315 MiB/s, ETA 0s
Errors:                 1 (retrying may help)
Elapsed time:        48.5s


2024/05/29 16:35:06 DEBUG : 11 go routines active
2024/05/29 16:35:06 Failed to copy: Operation forbidden

Thnaks in advance.

asdffdsa · May 29, 2024, 1:49pm

the compress remote is experimental since it was released, it might have bugs.

can you test the underlying wrapped remotes

copy that file to blompc:Google-archive
copy that file to blomp:x@gmail.com/E

note: to keep the logs small, use --retries=1

jxmese · May 29, 2024, 1:55pm

@asdffdsa, thanks for replying.

Here's the output for them:

rclone copy HAE:"July-DD-0/Incident.mp4" blompc:Google-archive/to-delete -P -vv --retries 1
2024/05/29 16:51:30 DEBUG : rclone: Version "v1.66.0" starting with parameters ["C:\\ProgramData\\chocolatey\\lib\\rclone.portable\\tools\\rclone-v1.66.0-windows-amd64\\rclone.exe" "copy" "HAE:July-DD-0/Incident.mp4" "blompc:Google-archive/to-delete" "-P" "-vv" "--retries" "1"]
2024/05/29 16:51:30 DEBUG : Creating backend with remote "HAE:July-DD-0/Incident.mp4"
2024/05/29 16:51:30 DEBUG : Using config file from "C:\\Users\\Administrator\\AppData\\Roaming\\rclone\\rclone.conf"
2024/05/29 16:51:30 DEBUG : Creating backend with remote "HA:E/3lq8375g5ek9tvpd5rirfvudhg/792tc1de92pkpoidc2vkousd1og5t6dpngad26ev7cejn996fa37j0qbe6pi5erlbnaqhjpjlf2im2770mn0h24egt9uiovuj3gq3ho"
2024/05/29 16:51:32 DEBUG : fs cache: adding new entry for parent of "HA:E/3lq8375g5ek9tvpd5rirfvudhg/792tc1de92pkpoidc2vkousd1og5t6dpngad26ev7cejn996fa37j0qbe6pi5erlbnaqhjpjlf2im2770mn0h24egt9uiovuj3gq3ho", "HA:E/3lq8375g5ek9tvpd5rirfvudhg"
2024/05/29 16:51:32 DEBUG : fs cache: adding new entry for parent of "HAE:July-DD-0/Incident.mp4", "HAE:July-DD-0"
2024/05/29 16:51:32 DEBUG : Creating backend with remote "blompc:Google-archive/to-delete"
2024/05/29 16:51:32 DEBUG : Creating backend with remote "blomp:x@gmail.com/E/di2uds15b0t1hbr8dojh865qe4/88t0sb40dilp7gpi6u3n238m08"
2024/05/29 16:51:36 DEBUG : Incident.mp4: Need to transfer - File not found at Destination
2024/05/29 16:52:08 DEBUG : pacer: low level retry 1/1 (error HTTP Error: 503: 503 Service Unavailable)
2024/05/29 16:52:08 DEBUG : pacer: Rate limited, increasing sleep to 10ms
2024/05/29 16:52:08 DEBUG : Incident.mp4: Received error: HTTP Error: 503: 503 Service Unavailable - low level retry 0/10
2024/05/29 16:52:34 DEBUG : pacer: Reducing sleep to 0s
2024/05/29 16:52:34 DEBUG : Incident.mp4: md5 = f60d9527bb84ea04c6ca8cea590604df OK
2024/05/29 16:52:34 INFO  : Incident.mp4: Copied (new)
Transferred:      422.911 MiB / 422.911 MiB, 100%, 8.828 MiB/s, ETA 0s
Transferred:            1 / 1, 100%
Elapsed time:       1m4.4s
2024/05/29 16:52:34 INFO  :
Transferred:      422.911 MiB / 422.911 MiB, 100%, 8.828 MiB/s, ETA 0s
Transferred:            1 / 1, 100%
Elapsed time:       1m4.4s

2024/05/29 16:52:34 DEBUG : 7 go routines active

And

PS C:\Users\Administrator> rclone copy HAE:"July-DD-0/Incident.mp4" blompc:to-delete -P -vv --retries 1
2024/05/29 16:52:47 DEBUG : rclone: Version "v1.66.0" starting with parameters ["C:\\ProgramData\\chocolatey\\lib\\rclone.portable\\tools\\rclone-v1.66.0-windows-amd64\\rclone.exe" "copy" "HAE:July-DD-0/Incident.mp4" "blompc:to-delete" "-P" "-vv" "--retries" "1"]
2024/05/29 16:52:48 DEBUG : Creating backend with remote "HAE:July-DD-0/Incident.mp4"
2024/05/29 16:52:48 DEBUG : Using config file from "C:\\Users\\Administrator\\AppData\\Roaming\\rclone\\rclone.conf"
2024/05/29 16:52:48 DEBUG : Creating backend with remote "HA:E/3lq8375g5ek9tvpd5rirfvudhg/792tc1de92pkpoidc2vkousd1og5t6dpngad26ev7cejn996fa37j0qbe6pi5erlbnaqhjpjlf2im2770mn0h24egt9uiovuj3gq3ho"
2024/05/29 16:52:49 DEBUG : fs cache: adding new entry for parent of "HA:E/3lq8375g5ek9tvpd5rirfvudhg/792tc1de92pkpoidc2vkousd1og5t6dpngad26ev7cejn996fa37j0qbe6pi5erlbnaqhjpjlf2im2770mn0h24egt9uiovuj3gq3ho", "HA:E/3lq8375g5ek9tvpd5rirfvudhg"
2024/05/29 16:52:49 DEBUG : fs cache: adding new entry for parent of "HAE:July-DD-0/Incident.mp4", "HAE:July-DD-0"
2024/05/29 16:52:49 DEBUG : Creating backend with remote "blompc:to-delete"
2024/05/29 16:52:49 DEBUG : Creating backend with remote "blomp:x@gmail.com/E/88t0sb40dilp7gpi6u3n238m08"
2024/05/29 16:52:54 DEBUG : Incident.mp4: Need to transfer - File not found at Destination
Transferred:       96.961 MiB / 238.103 MiB, 41%, 17.664 MiB/s, ETA 7s
Transferred:            0 / 1, 0%
Elapsed time:        11.9s
Transferring:
 * Incident.mp4: 40% /238.103Mi, 18.984Mi/s, 7s
PS C:\Users\Administrator> ^C

Seems to work fine.

asdffdsa · May 29, 2024, 2:09pm

this is confusing, you created two topics about the same basic issue??
best not to post here again, ok?
post at the first topic and @kapitainsky can help you, ok?

jxmese · May 29, 2024, 2:13pm

Sure, I just think they're two separate issues. Only is focusing on the config and rearranging the chunker and compression, understanding if the compression compression comes first or the chunking. Whereas this issue is troubleshooting why the file doesn't upload regardless of chunking or compression.

asdffdsa · May 29, 2024, 2:14pm

ok, i am going to drop off now, good luck.

system · June 28, 2024, 2:14pm

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.