Sync creating imaginary folders/unable to write

[rolas2k (rolas)]

What is the problem you are having with rclone?

Hello, I am trying to copy the backup from wasabi back to my PC.
There are about 700 folders on wasabi, when I start the sync everything goes well, then I leave it overnight, and in the morning I come back to 1400 folders created on my PC ( the folder names are just increments of the number that's in folder name).
I tried changing the command with some suggestions from the forum, but it did not help, and this is the second day this happens.

Run the command 'rclone version' and share the full output of the command.

• rclone.exe version
  -rclone v1.64.2
• os/version: Microsoft Windows 10 Pro 22H2 (64 bit)
• os/kernel: 10.0.19045.2965 (x86_64)
• os/type: windows
• os/arch: amd64
• go/version: go1.21.3
• go/linking: static
• go/tags: cmount

Using wasabi S3 storage

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone.exe sync wasabi:visiVMaiZD3 E:\LDPlayer\LDPlayer9\vms -P --fast-list --transfers=2 --multi-thread-streams=16

Please run 'rclone config redacted' and share the full output. If you get command not found, please make sure to update rclone.

[wasabi]
type = s3
provider = Wasabi
access_key_id = XXX
secret_access_key = XXX
endpoint = s3.wasabisys.com
acl = public-read
### 

A log from the command that you were trying to run with the -vv flag

2023/11/10 19:57:44 ERROR : leidian230/sdcard.vmdk.jijikix6.partial: partial file rename failed: rename \\?\E:\LDPlayer\LDPlayer9\vms\leidian230\sdcard.vmdk.jijikix6.partial \\?\E:\LDPlayer\LDPlayer9\vms\leidian230\sdcard.vmdk: Access is denied.
2023/11/10 19:58:34 ERROR : leidian230/data.vmdk.jiyohok0.partial: partial file rename failed: rename \\?\E:\LDPlayer\LDPlayer9\vms\leidian230\data.vmdk.jiyohok0.partial \\?\E:\LDPlayer\LDPlayer9\vms\leidian230\data.vmdk: Access is denied.
2023/11/10 20:00:03 ERROR : leidian230/system.vmdk.qebipub9.partial: partial file rename failed: rename \\?\E:\LDPlayer\LDPlayer9\vms\leidian230\system.vmdk.qebipub9.partial \\?\E:\LDPlayer\LDPlayer9\vms\leidian230\system.vmdk: Access is denied.
2023/11/10 20:00:51 ERROR : leidian232/sdcard.vmdk.rumanek5.partial: partial file rename failed: rename \\?\E:\LDPlayer\LDPlayer9\vms\leidian232\sdcard.vmdk.rumanek5.partial \\?\E:\LDPlayer\LDPlayer9\vms\leidian232\sdcard.vmdk: Access is denied.
2023/11/10 20:03:11 ERROR : leidian232/system.vmdk.vikixen4.partial: partial file rename failed: rename \\?\E:\LDPlayer\LDPlayer9\vms\leidian232\system.vmdk.vikixen4.partial \\?\E:\LDPlayer\LDPlayer9\vms\leidian232\system.vmdk: Access is denied.
2023/11/10 20:03:14 ERROR : leidian232/data.vmdk.yanilep4.partial: partial file rename failed: rename \\?\E:\LDPlayer\LDPlayer9\vms\leidian232\data.vmdk.yanilep4.partial \\?\E:\LDPlayer\LDPlayer9\vms\leidian232\data.vmdk: Access is denied.
2023/11/10 20:07:42 ERROR : leidian233/data.vmdk: Failed to copy: multi-thread copy: failed to find object after copy: object not found
2023/11/10 20:08:05 ERROR : leidian224/data.vmdk: Failed to copy: multi-thread copy: failed to find object after copy: object not found
2023/11/10 20:12:14 ERROR : leidian234/sdcard.vmdk.gorasom3.partial: partial file rename failed: rename \\?\E:\LDPlayer\LDPlayer9\vms\leidian234\sdcard.vmdk.gorasom3.partial \\?\E:\LDPlayer\LDPlayer9\vms\leidian234\sdcard.vmdk: Access is denied.
2023/11/10 20:13:30 ERROR : leidian234/data.vmdk.fojojot5.partial: partial file rename failed: rename \\?\E:\LDPlayer\LDPlayer9\vms\leidian234\data.vmdk.fojojot5.partial \\?\E:\LDPlayer\LDPlayer9\vms\leidian234\data.vmdk: Access is denied.
2023/11/10 20:14:39 ERROR : leidian234/system.vmdk.yafamef8.partial: partial file rename failed: rename \\?\E:\LDPlayer\LDPlayer9\vms\leidian234\system.vmdk.yafamef8.partial \\?\E:\LDPlayer\LDPlayer9\vms\leidian234\system.vmdk: Access is denied

First time this happened I thought maybe it was a permission issue, so I started cmd as administrator yesterday, but that did not help.

Thank you, and if anyone has suggestions how to get better download speeds from wasabi, I'm all ears, as right now it's at 200Mb/s (files being downloaded are 2-4GB)

[asdffdsa (jojothehumanmonkey)]

@rolas2k

looks like rclone can write to dest. but not able to rename
to disable the rename, try --inplace

what is the file system of E: ?

@rolas2k

Access is denied

that should be an issue with windows, not rclone.
check user and file permissions on E: drive

@rolas2k

Thank you, and if anyone has suggestions how to get better download speeds from wasabi, I'm all ears

what is the result of a speedtest. that would help establish a baseline.
baased on that, we can tweak rclone flags

[rolas2k (rolas)]

Thank you for the response, the file system is NTFS

I think it's also worth mentioning that rclone.exe is located on the C drive.

And this is the permissions for the folder that we are downloading to, nothing was changed here, not sure if something is lacking

And this is the speedtest result-

Now that I'm thinking, maybe need to put rclone in the E folder

[kapitainsky]

I wonder if anything to do with a fact that you are trying to transfer 3.6 TiB of data to a disk with 339 GB free space ....

[rolas2k (rolas)]

That's the thing, this disk is almost empty, at least 4.5TB of free space before starting the sync then it starts syncing and creates these random folders

[asdffdsa (jojothehumanmonkey)]

@rolas2k

Now that I'm thinking, maybe need to put rclone in the E folder

no, should not be an issue, as you mentioned, rclone is run with admin privileges
as a test, might try running as system user.

did you try --inplace ?

[kapitainsky]

Ok. Very weird. Never have seen anything like this. Do not you have some "clever" antivirus software maybe? If yes could you turn it off and try copy again.

[rolas2k (rolas)]

Just a thought, but let's clarify, the downloaded files get the 'modified date' as today.
And the files on wasabi are modified a week ago.

Could it possibly be that rclone detects that local files have different modified date and tries to re-download them?

No antivirus, this is basically a fresh windows 10 install, I backed everything up before reinstalling it and want to get my things back now.

did you try --inplace ?

No I haven't, will give it a try, the annoying thing is, that everything begins correctly, and I only find this mess after it's running for 10+ hours

I have now enabled 'full permissions' for all users

[asdffdsa (jojothehumanmonkey)]

@rolas2k

Could it possibly be that rclone detects that local files have different modified date and tries to re-download them?

no need to guess,
use -vv for debug output, rclone will let you know the reason for the download.
or
run the command with -vv --dry-run and rclone will simulate the command without actually downloading files.

can log to a file, with or without --dry-run
--dry-run --log-level=DEBUG --log-file=c:\path\to\rclone.log

[rolas2k (rolas)]

Aaaah damnit...
I just realized that some of the files might have actually been in use by another program, of course rclone was not able to write them..

Thank you for help, and pardon, it was a silly mistake

[asdffdsa (jojothehumanmonkey)]

@rolas2k

in use by another program

ok, that explains of unable to write

now, that does not explain the other issue
Sync creating imaginary folders

[rolas2k (rolas)]

I was actually a very strange combination of the other program running at the same time, won't get too much into details because this is completely not rclones fault.