Rclone to iDrive e2 S3 - Error on copy attempt

Help and Support
1

What is the problem you are having with rclone?

iDrive e2 S3 Mount returning the following on copy attempts:

Failed to copy: InvalidParameter: 1 validation error(s) found.

  • minimum field size of 1, HeadObjectInput.Key.

iDrive bucket is encrpyted, and there seems to be a line missing in the config "server_side_encryption = aws:kms
", however, I'm not sure how to add this and am not sure if it is causing the issue.

Run the command 'rclone version' and share the full output of the command.

rclone v1.62.2
- os/version: ubuntu 22.04 (64 bit)
- os/kernel: 4.14.15-1.el7.elrepo.x86_64 (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.20.2
- go/linking: static
- go/tags: none

Are you on the latest version of rclone? You can validate by checking the version listed here: Rclone downloads
-->

Version 1.62.2, however, server helpdesk has also tried the latest version of 1.64.0 and received the same error.

Which cloud storage system are you using? (eg Google Drive)

iDrive e2 S3

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone --config=/home/user/.apps/rclone2/rclone.conf copy /home/user/files/Media/Movies/Shaft\ \(2019\)/Shaft.2019.NF.1080p.WEB-DL.H264.DDP5.1-EVO.mkv remote:/ -vvP

Please run 'rclone config redacted' and share the full output. If you get command not found, please make sure to update rclone.

[remote]
type = s3
provider = IDrive
access_key_id = *hidden*
secret_access_key = *hidden*
acl = private
endpoint = p6f1.ie.idrivee2-24.com

A log from the command that you were trying to run with the -vv flag

rclone --config=/home/user/.apps/rclone2/rclone.conf copy /home/user/files/Media/Movies/Shaft\ \(2019\)/Shaft.2019.NF.1080p.WEB-DL.H264.DDP5.1-EVO.mkv remote:/ -vvP
2023/09/14 17:59:25 DEBUG : rclone: Version "v1.62.2" starting with parameters ["rclone" "--config=/home/user/.apps/rclone2/rclone.conf" "copy" "/home/user/files/Media/Movies/Shaft (2019)/Shaft.2019.NF.1080p.WEB-DL.H264.DDP5.1-EVO.mkv" "remote:/" "-vvP"]
2023/09/14 17:59:25 DEBUG : Creating backend with remote "/home/user/files/Media/Movies/Shaft (2019)/Shaft.2019.NF.1080p.WEB-DL.H264.DDP5.1-EVO.mkv"
2023/09/14 17:59:25 DEBUG : Using config file from "/home/user/.apps/rclone2/rclone.conf"
2023/09/14 17:59:25 DEBUG : fs cache: adding new entry for parent of "/home/user/files/Media/Movies/Shaft (2019)/Shaft.2019.NF.1080p.WEB-DL.H264.DDP5.1-EVO.mkv", "/home/user/files/Media/Movies/Shaft (2019)"
2023/09/14 17:59:25 DEBUG : Creating backend with remote "remote:/"
2023/09/14 17:59:25 DEBUG : Resolving service "s3" region "us-east-1"
2023/09/14 17:59:25 DEBUG : fs cache: renaming cache item "remote:/" to be canonical "remote:"
2023-09-14 17:59:25 ERROR : Attempt 1/3 failed with 1 errors and: InvalidParameter: 1 validation error(s) found.
- minimum field size of 1, HeadObjectInput.Key.
2023-09-14 17:59:25 ERROR : Attempt 2/3 failed with 1 errors and: InvalidParameter: 1 validation error(s) found.
- minimum field size of 1, HeadObjectInput.Key.
2023-09-14 17:59:25 ERROR : Attempt 3/3 failed with 1 errors and: InvalidParameter: 1 validation error(s) found.
- minimum field size of 1, HeadObjectInput.Key.
Transferred:              0 B / 0 B, -, 0 B/s, ETA -
Errors:                 1 (retrying may help)
Elapsed time:         0.0s
2023/09/14 17:59:25 INFO  :
Transferred:              0 B / 0 B, -, 0 B/s, ETA -
Errors:                 1 (retrying may help)
Elapsed time:         0.0s

2023/09/14 17:59:25 DEBUG : 2 go routines active
2023/09/14 17:59:25 Failed to copy: InvalidParameter: 1 validation error(s) found.
- minimum field size of 1, HeadObjectInput.Key.
2

welcome to the forum,

i do not think server_side_encryption = aws:kms is the problem, unless you are not using the default key.

if you have not encrypted the rclone config file, then it is plain text in .ini format.
so can just copy/paste the text into the config file.

[remote]
type = s3
provider = IDrive
access_key_id = 
secret_access_key = 
server_side_encryption = aws:kms
endpoint = bucket.o0e2.va12.idrivee2-9.com
2 Likes
3

  1. Update your rclone to the latest version - it is v1.64.0 now

  2. As you already mentioned server_side_encryption = aws:kms is missing from your rclone config file. This is correct as per iDrive rclone examples. Open file in your favourite text editor and add this to your remote configuretion

[remote]
type = s3
provider = IDrive
access_key_id = *hidden*
secret_access_key = *hidden*
acl = private
endpoint = p6f1.ie.idrivee2-24.com
server_side_encryption = aws:kms
  1. Is your bucket in Ireland region?
1 Like
4

if server_side_encryption = aws:kms is not added to config, then i get the expected error.
corrupted on transfer: md5 hash differ

2 Likes
5

I have tried myself and encrypted bucket without server_side_encryption = aws:kms in config file results with the following error:

2023/09/14 19:56:24 DEBUG : file.test: md5 = b6e87fd695973abce0545bacba6b9b88 (Local file system at /Users/Desktop)
2023/09/14 19:56:24 DEBUG :.file.test: md5 = 458e9a692cc41d4b87f9b15f383723d3 (S3 bucket test-enc)
2023/09/14 19:56:24 ERROR :.file.test: corrupted on transfer: md5 hash differ "b6e87fd695973abce0545bacba6b9b88" vs "458e9a692cc41d4b87f9b15f383723d3"
2023/09/14 19:56:24 INFO  :file.test: Removing failed copy

this is in line with rclone documentation:

Key Management System (KMS)

If you are using server-side encryption with KMS then you must make sure rclone is configured with server_side_encryption = aws:kms otherwise you will find you can't transfer small objects - these will create checksum errors.

after adding server_side_encryption = aws:kms rclone copy works

your error is different though... so there is more issues.

First fix the basics and we will see.

1 Like
6

must use a bucket name, so try remote:p6f1

1 Like
7

BINGO! I think you nailed the root cause.

This is my test result:

rclone copy . iDriveSG:/
2023/09/14 20:24:05 ERROR : file.test: Failed to copy: InvalidParameter: 1 validation error(s) found.
- minimum field size of 1, PutObjectInput.Key.
1 Like
8

You guys are absolutely incredible! Thank you so very much! I attempted to download the config via FTP using Filezilla, but it gives me an unrelated error and stopped me from being able to open the ini or reupload. However, you guys were spot on in the server_side_encrypton not being the root cause.

Attempted to use remote:p6f1 and this worked like an absolute charm with 0 errors!

Thank you again so much for your prompt responses, and for helping me with this!

9

welcome.

fwiw, this is from rclone github pages.
"That's a cryptic error for trying to move something into the root!"

1 Like
10

Thank you again! I tried searching through GitHub using iDrive as the search as I thought it may have been a compatibility issue. Then, I somehow directed the attention to searching for the error through Google and on the rclone forum. I think ncw provided a response/solution similar on a post pertaining to a Tencent server, but I think I got lost with all the alias related stuff and seen something about not requiring the bucket name on other posts, so I was pretty much just chasing my tail on something I have a very beginner level understanding of lol. I can't thank you guys enough for the help, and for sharing that github page as well.

11 
[p6f1]
type = alias
remote = remote:p6f1

rclone ls p6f1:
12

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.