iDrive e2 listing error: directory not found - very strange

Help and Support
1

What is the problem you are having with rclone?

iDrive S3 bucket file listing returns "directory not found" when called from PowerShell, but works fine when called from Lazarus IDE application.
I cannot see any difference in parameters between these two calling styles. I made sure the config file, remote name, bucket name and subdirectory are identical.
Bucket name is "lap" - I hope I have used it correctly, this is my first try with something other than FTP with rclone.
Subdirectory in the bucket is "zdjecia/current" - what's more strange is that the files have uploaded without any issue, and can be listed with the Lazarus version, but I cannot see any file in the iDrive web interface. The bucket info reads "0 files 0 KB". I will ask iDrive about this, but perhaps it is connected somehow to the issue.

Run the command 'rclone version' and share the full output of the command.

rclone v1.63.1
- os/version: Microsoft Windows 10 Home 22H2 (64 bit)
- os/kernel: 10.0.19045.3208 (x86_64)
- os/type: windows
- os/arch: amd64
- go/version: go1.20.6
- go/linking: static
- go/tags: cmount

Also tried latest beta, without success.

Which cloud storage system are you using? (eg Google Drive)

iDrive e2

The command you were trying to run (eg rclone copy /tmp remote:tmp)

The command that doesn't work (PowerShell) is this:

C:\PProg\Dysk\rclone\rclone.exe lsf idrive_crypt:lap/zdjecia/current `
  --config="$PSScriptRoot\rclone.lapasus.idrive.config" `
  --password-command="$password_cmd" `
  --format="tsp" `
  --separator="*" `
  -vv

The command that works is impossible to type, because what I do in Lazarus is create a TProcess object and then supply the executable path, and each parameter separately. But the parameters are almost the same and in the same order. The only difference is in the password command.

The rclone config contents with secrets removed.

[idrive_e2]
type = s3
provider = idrive
access_key_id = <removed>
secret_access_key = <removed>
acl = private

[idrive_crypt]
type = crypt
remote = idrive_e2
password = <removed>

A log from the command with the -vv flag

Not working log (PowerShell):

2023/08/10 20:39:24 DEBUG : rclone: Version "v1.63.1" starting with parameters ["C:\\PProg\\Dysk\\rclone\\rclone.exe" "lsf" "idrive_crypt:lap/zdjecia/current" "--config=D:\\BackupScript\\rclone\\rclone.lapasus.idrive.config" "--password-command=powershell -NoProfile -File D:\\BackupScript\\rclone\\rclone_pass.ps1" "--format=tsp" "--separator=*" "-vv"]
2023/08/10 20:39:24 DEBUG : Creating backend with remote "idrive_crypt:lap/zdjecia/current"
2023/08/10 20:39:25 DEBUG : Using config file from "D:\\BackupScript\\rclone\\rclone.lapasus.idrive.config"
2023/08/10 20:39:25 DEBUG : Creating backend with remote "idrive_e2/<removed>"
2023/08/10 20:39:25 DEBUG : fs cache: renaming cache item "idrive_e2/<removed>" to be canonical "//?/D:/BackupScript/rclone/idrive_e2/<removed>"
2023/08/10 20:39:25 DEBUG : fs cache: switching user supplied name "idrive_e2/<removed>" for canonical name "//?/D:/BackupScript/rclone/idrive_e2/<removed>"
2023/08/10 20:39:25 ERROR : : error listing: directory not found
2023/08/10 20:39:25 DEBUG : 2 go routines active
2023/08/10 20:39:25 Failed to lsf with 2 errors: last error was: error in ListJSON: directory not found

Working log (Lazarus):

2023/08/10 20:43:46 DEBUG : rclone: Version "v1.63.1" starting with parameters ["C:\\PProg\\Dysk\\rclone\\rclone.exe" "lsf" "idrive_crypt:lap/zdjecia/current/" "--config=D:\\BackupScript\\rclone\\rclone.lapasus.idrive.config" "--password-command=D:\\BackupScript\\rcGUI_devel\\Release\\rcGUIpass.exe" "--format=tsp" "--separator=*" "-vv"]
2023/08/10 20:43:46 DEBUG : Creating backend with remote "idrive_crypt:lap/zdjecia/current/"
2023/08/10 20:43:46 DEBUG : Using config file from "D:\\BackupScript\\rclone\\rclone.lapasus.idrive.config"
2023/08/10 20:43:46 DEBUG : Creating backend with remote "idrive_e2/<removed>"
2023/08/10 20:43:46 DEBUG : fs cache: renaming cache item "idrive_e2/<removed>" to be canonical "//?/D:/BackupScript/rcGUI_devel/Release/idrive_e2/<removed>"
2023/08/10 20:43:46 DEBUG : fs cache: switching user supplied name "idrive_e2/<removed>" for canonical name "//?/D:/BackupScript/rcGUI_devel/Release/idrive_e2/<removed>"
2023-08-10 16:39:54*-1*2004-08-04_Pielgrzymka_do_Czestochowy/
2023-08-10 16:39:54*-1*2004-09-21_Bieszczady/
2023-08-10 16:40:10*-1*2006-07-30_Turniej_Janowiec_Nikoniarze/
2023-08-10 16:40:11*-1*2006-08-18_Po_pielgrzymce_do_Chestochowy/
<cut the listing>
2009-05-18 13:10:42*41794*rzeznik_lcduf.png
2022-01-15 16:39:11*14495*tapeta_drugi_monitor_czarna.jpg
2017-06-10 18:02:26*6220922*tapetka_dell.bmp
2022-03-08 19:50:37*2061192*twinsen_1920x1024.png
2023/08/10 20:43:47 DEBUG : 2 go routines active
2

hi,

remotes need to have a :, perhaps what you want is
remote = idrive_e2:

from rclone, idrive_e2 is a local subfolder. this can be seen in the output:
renaming cache item "idrive_e2/<removed>" to be canonical "//?/D:/BackupScript/rclone/idrive_e2/<removed>"

1 Like
3

you have not uploaded any files to the iDrive. You copied them somewhere locally to folder called idrive_e2

If you want your crypt to point to remote in the cloud then use:

remote = idrive_e2:

Please also note that for S3 style remotes you can not have any files in root - in idrive_e2: in your case.
Data is organized by buckets.

your files are in:

D:/BackupScript/rcGUI_devel/Release/idrive_e2/

Powershell probably starts from different directory and tries to list D:/BackupScript/rclone/idrive_e2/

1 Like
4

ohh sorry have not seen you were already replying. But I see we have the same conclusions:) so it must be 100% right:)

5

For your crypt I would recommend using filename_encoding = base64. iDrive (and all S3 remotes) is case sensitive and using default base32 unnecessary limits your files' names max length.

For S3 storage max file name is 255 bytes, when you add crypt and default base32 coding it is only ≈140. With base64 ≈170. I do not remember exact numbers now. You can google if you need it.

[idrive_crypt]
type = crypt
remote = idrive_e2:
password = <removed>
password2=XXX
filename_encoding = base64

and for better security provide salt "password2" as well.

To make recommended changes the easiest would be to recreate your crypt remote from the scratch. Do not change any parameters if you have any data already uploaded.

1 Like
6

Thank you, that was it. I have read the similar topics prior to posting this, but the problem in all of them seemed to be missing colon in the command line, not in the config file. I didn't know it was required in the config file too. That's a bit counter-intuitive for me. The parameter is named "remote" (i.e. remote name from the same config, that is to be encrypted). Why would it be interpreted as something else, like local directory?

Is there any advantage of a salted password over longer password without salt? My password is over 40 characters. I don't think I need a salt for that, but I may be wrong.

7

Short answer is yes. It is better. And These are two different things.

Encryption password is used to encrypt your data - longer password is more difficult to crack.

Salt is used to per-mutate encryption keys - every time you upload the same file its encrypted content will be different. It protects from certain classes of cryptographic attacks. If you do not provide your own salt rclone will use some default anyway - but then it is the same for you and another million users:)

You can read more here:

When you use encryption do it right:)

8

that is the way rclone works. as for the logic.
imho, just like most command line tools, default to local, nice and simple.

if you want something above that, then need the complexity and time of creating a remote.
and as you have found out, creating a remote is not always a trivial issue, despite the docs and examples.

that is up to you, search for it , as much discussed in the forum

9

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.