Rclone can't reuse filenames

Help and Support
1

What is the problem you are having with rclone?

The sequence of actions I am trying to do is:
Map Rclone drive on windows 2012r2 server running as system.
Copy file in to rclone drive
File replicates to s3 bucket correctly
Delete file from S3 bucket
Deletion is successfully replicated back to server
Copying another file of the same name in to the mounted drive gets the error message
"Could not find this item. This is no longer located in this PC. Verify items location and try again"

What is your rclone version (output from rclone version)

C:\Program Files\rclone>rclone version
rclone v1.54.0-beta.4887.226018dc2.fix-4446-vfs-virtual

  • os/arch: windows/amd64
  • go version: go1.15.3

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Windows 2012r2 server

Which cloud storage system are you using? (eg Google Drive)

AWS s3

The command you were trying to run (eg rclone copy /tmp remote:tmp)

cd "C:\Program Files\rclone"
.\rclone.exe mount
AWS-sftp:/sftp-olmapps-co-uk-thc-live/bobi/nthcfs1/ X:
--allow-other --allow-non-empty
--dir-cache-time 40s --poll-interval 30s
--config "C:\Program Files\rclone\rclone.conf" --vfs-cache-mode writes
--log-file ".\run-log" -vv
-o "uid=197108,gid=197121,umask=0000"

The rclone config contents with secrets removed.

[AWS-sftp]
type = s3
provider = AWS
env_auth = false
secret_access_key = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
region = eu-west-2
location_constraint = eu-west-2
acl = private
server_side_encryption = AES256
access_key_id = xxxxxxxxxxxxxxxxxxxxxxxxxx

A log from the command with the -vv flag

Debug log was too long to include.

2

I can't immediately replicate that...

Can you post a link to the debug log (share it on drive/dropbox/pastebin)?

I suspect it is something to do with the cache file not being removed - you could verify that by deleting it by hand from the cache directory after Deletion is successfully replicated back to server but before Copying another file of the same name

3

Yep, I am facing the same issue on Linux too. There it fails with this error:

cp: cannot create regular file 'foo': File exists

Relevant debug logs:

2020/11/12 11:07:49 DEBUG : /: Create: name="foo"
2020/11/12 11:07:49 DEBUG : foo: Open: flags=O_WRONLY|O_CREATE|O_EXCL
2020/11/12 11:07:49 DEBUG : foo: newRWFileHandle:
2020/11/12 11:07:49 DEBUG : foo: >newRWFileHandle: err=file already exists
2020/11/12 11:07:49 DEBUG : foo: File.openRW failed: file already exists
2020/11/12 11:07:49 DEBUG : foo: >Open: fd=<nil *RWFileHandle>, err=file already exists
2020/11/12 11:07:49 DEBUG : /: >Create: node=<nil>, handle=<nil>, err=file exists
4

Can you give me a recipe to recreate?

5

2 mounts with the following parameters:

./rclone -vvP mount TD: test-1 --poll-interval=30s --dir-cache-time=24h --rc --rc-no-auth --rc-addr=127.0.0.1:5573 --vfs-cache-mode=full --vfs-cache-max-age=12h --cache-dir="/home/darthshadow/cache/test-1"

./rclone -vvP mount TD: test-2 --poll-interval=30s --dir-cache-time=24h --rc --rc-no-auth --rc-addr=127.0.0.1:5574 --vfs-cache-mode=full --vfs-cache-max-age=12h --cache-dir="/home/darthshadow/cache/test-2"
  • Copy a file foo in test-1
  • Wait for it to appear in test-2
  • Delete foo from test-2
  • Wait for it to disappear from test-1 (and still remain in cache)
  • Try to copy another file in test-1 to the same name to get the above error
6

How are you copying the files? I'm trying with cp but I can't replicate the problem.

7

Same cp.

Your diagnosis appears correct and manually removing the file from cache fixes the issue.

8

It is to do with the cache file not being deleted - I've found the files remaining in the cache.

This doesn't happen on all of my machines, some of them with older versions run correctly so it may be a configuration issue or something to do with the cache folder.

9

OK I've had a go at a fix - can you try this?

v1.54.0-beta.4890.deb729220.fix-vfs-reuse on branch fix-vfs-reuse (uploaded in 15-30 mins)

10

It's working fine with this build.

11

I've merged this to master now which means it will be in the latest beta in 15-30 mins and released in v1.54

Thanks for testing :slight_smile:

12

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.