iDrive rclone mount copy failing on certain files

paulz · January 19, 2024, 12:01pm

What is the problem you are having with rclone?

Certain files fail to copy via rclone mount regardless of the mount settings but work fine with 'rclone copy' command. most files work but some, seemingly larger ones fail repeatedly. Have tried different filenames etc

Run the command 'rclone version' and share the full output of the command.

rclone version
rclone v1.65.1

os/version: debian 11.6 (64 bit)
os/kernel: 6.2.16-8-pve (x86_64)
os/type: linux
os/arch: amd64
go/version: go1.21.5
go/linking: static
go/tags: none

Which cloud storage system are you using? (eg Google Drive)

iDrive e2

The command you were trying to run (eg `rclone copy /tmp remote:tmp`)

cp /home/docker/sabnzbd/downloads/complete/Tom.Clancys.Jack.Ryan.\(2018\).S04.\(2160p.AMZN.WEB-DL.H265.DV.DDP.Atmos.5.1.English.-.HONE\)/Tom.Clancys.Jack.Ryan.S04E01.2160p.AMZN.WEB-DL.H265.DV.DDP.Atmos.5.1.English.mkv /mnt/idrivetest/Media/TV/TV\ 4K/Tom\ Clancy\'s\ Jack\ Ryan/Season\ 4/

Please run 'rclone config redacted' and share the full output. If you get command not found, please make sure to update rclone.

[crypt-e2]
type = crypt
remote = idrivee2:data1
password = XXX
filename_encoding = base32768

[idrivee2]
type = s3
provider = IDrive
access_key_id = XXX
secret_access_key = XXX
endpoint = y6w4.la.idrivee2-18.com
server_side_encryption = aws:kms

A log from the command that you were trying to run with the `-vv` flag

kapitainsky · January 19, 2024, 12:13pm

You should not be using base32768 encoding with iDrive.

$ rclone test info --check-length --check-base32768 iDrive:test

// iDrive
maxFileLength = 255 // for 1 byte unicode characters
maxFileLength = 127 // for 2 byte unicode characters
maxFileLength = 85 // for 3 byte unicode characters
maxFileLength = 63 // for 4 byte unicode characters
base32768isOK = true // make sure maxFileLength for 2 byte unicode chars is the same as for 1 byte characters

by using base32768 you are using 2 byte unicode characters which shorten your max file name length which I guess was not your intent.

For any S3 remote the most appropriate is base64

In terms of upload failures,

this is your mount command:

2024/01/19 11:49:03 DEBUG : rclone: Version "v1.65.1" starting with parameters ["/usr/bin/rclone" "mount" "--config=/root/.config/rclone/rclone.conf" "--log-file=/tmp/idrive_test.log" "--log-level" "DEBUG" "--dir-cache-time" "1m" "crypt-e2:" "/mnt/idrivetest"]

I suggest you add --vfs-cache-mode writes or even --vfs-cache-mode full with optional --vfs-cache-max-size to limit cache size used.

Default --vfs-cache-mode off most likely makes multi part uploads impossible (for large files). It would explain why rclone copy works and mount not.

paulz · January 19, 2024, 12:43pm

Thanks for the speedy response!

How do I go about changing filename_encoding, I assume changing it now would render all the current files as useless?

Thanks for the tip on --vfs-cache-mode writes I did have this before and had the same issue so stripped it out but will retry. For vfs-cache-max-size is there a recommended size?

kapitainsky · January 19, 2024, 12:58pm

What you do is set up a new crypt remote by copying the old one in the config file, renaming it and adding filename_encoding = base64. Choosing a different destination directory is a good idea here too so the old and the new don't overlap.

You can then do a server side move something like

[crypt-e2]
type = crypt
remote = idrivee2:data1
password = XXX
filename_encoding = base32768

[crypt-e2-base64]
type = crypt
remote = idrivee2:data2
password = XXX
filename_encoding = base64

rclone move --server-side-across-configs -vv crypt-e2: crypt-e2-base64:

Try with dry-run and/or small dataset first.

nope. Think about how big files you work with and set it higher than max.

Let's see how it goes. Maybe there is something more to deal with.

paulz · January 19, 2024, 1:13pm

So I am trying this mount command, and now the file copy returns as successful and it appears in the vfs cache, but the upload is still failing:

        /usr/bin/rclone mount \
        --umask 0 \
        --allow-other \
        --config=/root/.config/rclone/rclone.conf \
        --allow-non-empty \
        --vfs-read-chunk-size 32M \
        --vfs-read-chunk-size-limit 128M \
        --vfs-cache-mode full \
        --vfs-cache-max-size 100G \
        --multi-thread-streams 8 \
        --s3-upload-concurrency 8 \
        --s3-chunk-size 16M \
        --log-file=/tmp/idrive.log \
        --log-level DEBUG \
        --dir-cache-time 1m crypt-e2: /mnt/idrive

grep of ERROR in the log

root@dl-vm:~# tail -n 100000 /tmp/idrive.log | grep ERROR
2024/01/19 13:11:27 ERROR : Media/TV/TV 4K/Tom Clancy's Jack Ryan/Season 4/Tom.Clancys.Jack.Ryan.(2018).S04E05.(2160p.AMZN.WEB-DL.H265.DV.DDP.Atmos.5.1.English.-.HONE).mkv: Failed to copy: failed to upload chunk 1 with 16777216 bytes: NoSuchUpload: The specified multipart upload does not exist. The upload ID may be invalid, or the upload may have been aborted or completed.
2024/01/19 13:11:27 ERROR : Media/TV/TV 4K/Tom Clancy's Jack Ryan/Season 4/Tom.Clancys.Jack.Ryan.(2018).S04E05.(2160p.AMZN.WEB-DL.H265.DV.DDP.Atmos.5.1.English.-.HONE).mkv: vfs cache: failed to upload try #1, will retry in 10s: vfs cache: failed to transfer file from cache to remote: failed to upload chunk 1 with 16777216 bytes: NoSuchUpload: The specified multipart upload does not exist. The upload ID may be invalid, or the upload may have been aborted or completed.
2024/01/19 13:11:33 ERROR : Media/TV/TV 4K/Tom Clancy's Jack Ryan/Season 4/Tom.Clancys.Jack.Ryan.(2018).S04E04.(2160p.AMZN.WEB-DL.H265.DV.DDP.Atmos.5.1.English.-.HONE).mkv: Failed to copy: failed to upload chunk 1 with 16777216 bytes: NoSuchUpload: The specified multipart upload does not exist. The upload ID may be invalid, or the upload may have been aborted or completed.
2024/01/19 13:11:33 ERROR : Media/TV/TV 4K/Tom Clancy's Jack Ryan/Season 4/Tom.Clancys.Jack.Ryan.(2018).S04E04.(2160p.AMZN.WEB-DL.H265.DV.DDP.Atmos.5.1.English.-.HONE).mkv: vfs cache: failed to upload try #3, will retry in 40s: vfs cache: failed to transfer file from cache to remote: failed to upload chunk 1 with 16777216 bytes: NoSuchUpload: The specified multipart upload does not exist. The upload ID may be invalid, or the upload may have been aborted or completed.
2024/01/19 13:11:47 ERROR : Media/TV/TV 4K/Tom Clancy's Jack Ryan/Season 4/Tom.Clancys.Jack.Ryan.(2018).S04E03.(2160p.AMZN.WEB-DL.H265.DV.DDP.Atmos.5.1.English.-.HONE).mkv: Failed to copy: failed to upload chunk 1 with 16777216 bytes: NoSuchUpload: The specified multipart upload does not exist. The upload ID may be invalid, or the upload may have been aborted or completed.
2024/01/19 13:11:47 ERROR : Media/TV/TV 4K/Tom Clancy's Jack Ryan/Season 4/Tom.Clancys.Jack.Ryan.(2018).S04E03.(2160p.AMZN.WEB-DL.H265.DV.DDP.Atmos.5.1.English.-.HONE).mkv: vfs cache: failed to upload try #4, will retry in 1m20s: vfs cache: failed to transfer file from cache to remote: failed to upload chunk 1 with 16777216 bytes: NoSuchUpload: The specified multipart upload does not exist. The upload ID may be invalid, or the upload may have been aborted or completed.

paulz · January 19, 2024, 1:26pm

Back to the test config and just adding --vfs-cache-mode writes and still no jov.
Mount command:

/usr/bin/rclone mount         --config=/root/.config/rclone/rclone.conf         --log-file=/tmp/idrive_test.log         --log-level DEBUG     --vfs-cache-mode writes    --dir-cache-time 1m crypt-e2: /mnt/idrivetest

2024/01/19 13:21:52 DEBUG : vfs cache RemoveNotInUse (maxAge=3600000000000, emptyOnly=false): item Media/TV/TV 4K/Tom Clancy's Jack Ryan/Season 4/testing123.mkv not removed, freed 0 bytes
2024/01/19 13:21:52 INFO  : vfs cache: cleaned: objects 2 (was 2) in use 2, to upload 0, uploading 2, total size 16.127Gi (was 16.127Gi)
2024/01/19 13:22:31 DEBUG : ä™‚å‚è‚é ç““ã—…Ýºê„æŸ/á—‹ã¸Œè¯»å¬°éŒ™åµ›è›€é€´é™Ÿ/æ˜œæµºâš–Ý½éŠ‡å¶žé¬èžªé«Ÿ/ç´’á˜©è²‰ä¡µä°§ã‡âžè’‡á‡°çˆ¸åã¿¾è†è”šäŒ’æ¢¯æ™žÊŸ/é°æ²¯è¥…å›šêƒ†è™‘äµé‚‘ä´¿/æ¯Žè©’æ‡¦è£šä¼¢èŠžæ¬€ã‘”â–Ÿ: open chunk writer: started multipart upload: 02bbe6d8-d62c-4291-8389-af9f2a828603
2024/01/19 13:22:31 DEBUG : Media/TV/TV 4K/Tom Clancy's Jack Ryan/Season 4/testing123.mkv: multipart upload: starting chunk 0 size 5Mi offset 0/8.066Gi
2024/01/19 13:22:31 DEBUG : Media/TV/TV 4K/Tom Clancy's Jack Ryan/Season 4/testing123.mkv: multipart upload: starting chunk 1 size 5Mi offset 5Mi/8.066Gi
2024/01/19 13:22:31 DEBUG : Media/TV/TV 4K/Tom Clancy's Jack Ryan/Season 4/testing123.mkv: multipart upload: starting chunk 2 size 5Mi offset 10Mi/8.066Gi
2024/01/19 13:22:31 DEBUG : Media/TV/TV 4K/Tom Clancy's Jack Ryan/Season 4/testing123.mkv: multipart upload: starting chunk 3 size 5Mi offset 15Mi/8.066Gi
2024/01/19 13:22:31 DEBUG : Media/TV/TV 4K/Tom Clancy's Jack Ryan/Season 4/testing123.mkv: Cancelling multipart upload
2024/01/19 13:22:31 DEBUG : Media/TV/TV 4K/Tom Clancy's Jack Ryan/Season 4/testing123.mkv: Failed to cancel multipart upload: failed to abort multipart upload "02bbe6d8-d62c-4291-8389-af9f2a828603": NoSuchUpload: The specified multipart upload does not exist. The upload ID may be invalid, or the upload may have been aborted or completed.
        status code: 404, request id: 17ABC2547064EE84, host id: 
2024/01/19 13:22:31 ERROR : Media/TV/TV 4K/Tom Clancy's Jack Ryan/Season 4/testing123.mkv: Failed to copy: failed to upload chunk 1 with 5242880 bytes: NoSuchUpload: The specified multipart upload does not exist. The upload ID may be invalid, or the upload may have been aborted or completed.
        status code: 404, request id: 17ABC25460DA82F3, host id: 
2024/01/19 13:22:31 ERROR : Media/TV/TV 4K/Tom Clancy's Jack Ryan/Season 4/testing123.mkv: vfs cache: failed to upload try #1, will retry in 10s: vfs cache: failed to transfer file from cache to remote: failed to upload chunk 1 with 5242880 bytes: NoSuchUpload: The specified multipart upload does not exist. The upload ID may be invalid, or the upload may have been aborted or completed.
        status code: 404, request id: 17ABC25460DA82F3, host id: 
2024/01/19 13:22:41 DEBUG : Media/TV/TV 4K/Tom Clancy's Jack Ryan/Season 4/testing123.mkv: vfs cache: starting upload
2024/01/19 13:22:41 DEBUG : Media/TV/TV 4K/Tom Clancy's Jack Ryan/Season 4/testing123.mkv: Computing md5 hash of encrypted source
2024/01/19 13:22:52 DEBUG : vfs cache RemoveNotInUse (maxAge=3600000000000, emptyOnly=false): item Media/TV/TV 4K/Tom Clancy's Jack Ryan/Season 4/testing123.mkv not removed, freed 0 bytes
2024/01/19 13:22:52 INFO  : vfs cache: cleaned: objects 2 (was 2) in use 2, to upload 0, uploading 2, total size 16.127Gi (was 16.127Gi)

kapitainsky · January 19, 2024, 1:34pm

Create new crypt with base64 and try transferring the same file. I wonder if base32768 characters have something to do with it.

paulz · January 19, 2024, 1:39pm

Ok that's a job for tomorrow, thanks again for the assistance

paulz · January 19, 2024, 11:03pm

So just tested with a new crypt with base64 and uploaded first time. Still can't figure out why rclone copy works, but I guess now I will have a crack at moving everything from base32768 to base64.

My own fault, I reused the crypt settings from my failed attempt to use box, should have worked out what was best for my specific provider.

Cheers again.

kapitainsky · January 20, 2024, 6:37am

Do you mean that mount upload is still failing vs copy working?

paulz · January 20, 2024, 6:53am

No sorry I was not very clear:

crypt-e2 (base32768)
rclone copy - works
rclone mount and cp xx - doesn't work

crypt-e2-base64 (base64)
rclone copy - works
rclone mount and cp xx - works (at least for the 1 file I tested that was previously failing)

What confuses me is that if base32768 is the problem I would expect rclone copy to also fail as that is common across both the rclone copy and rclone mount procedures.

Admittedly I only tested copying the one file on crypt-e2-base64 and it worked, the real test will be how it behaves going forward. The move of files from base32768 to base64 is very slowly progressing.

2024/01/20 06:51:21 INFO  : 
Transferred:        3.327 TiB / 12.168 TiB, 27%, 0 B/s, ETA -
Checks:              1026 / 1026, 100%
Deleted:             1026 (files), 0 (dirs)
Renamed:             1026
Transferred:         1026 / 4675, 22%
Server Side Copies:  1026 @ 3.327 TiB
Elapsed time:     5h6m0.1s

kapitainsky · January 20, 2024, 6:56am

Problem might be more nuance than simple name length. I am a bit puzzled by this issue (especially that rclone copy works). Let's see how things stand with base64 when you start using it.

kapitainsky · January 20, 2024, 6:57am

Indeed it is slow. But still much faster than re-uploading:)

paulz · January 20, 2024, 7:27am

The cool thing is I have unioned crypt-e2 and crypt-e2-base64 together in an rclone mount on my content server, so downtime for the copy should actually be minimal

paulz · January 23, 2024, 2:40am

Just to give an update, I haven't had any failed uploads since the change to base64 yet.

system · February 22, 2024, 2:40am

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.