Google Server Side file copy bug

What is the problem you are having with rclone?

I have a bash script that bulk moves files into subdirectories. I use it on both local and rclone-mounted remote directories, and it worked perfectly until today.

I have 2 machines that both connect to the same GDrive backend. Machine 1 executes the script through an rclone-gdrive mount and the copy performs as normal. When completed, I can browse the directory from Machine 1 and everything is there as it should be. I can open, copy and delete the files as normal.

When I tried to access the same directory using a command-line rclone ls from Machine 2, the directory appears empty. All the files and associated subdirectories are gone.

I initially thought Machine 1 borked the copy and was pulling them from VFS cache, so I performed a rclone rc vfs/refresh on Machine 1 and every appeared to be fine. I could open and manipulate the files as normal. I then stopped and restarted the rclone mount on Machine 1 and the files disappeared.

So two things: How was Machine 1 able to access the files if they are, in fact, gone and two: why were the files gone. I've literally run this script hundreds of times across three different machines/OSs and rclone mount configurations with no issue.

What is your rclone version (output from rclone version)

v1.55 amd64 on Machine 1
v1.51 arm64 on Machine 2 (raspberry pi)

Which OS you are using and how many bits (eg Windows 7, 64 bit)

x64 Ubuntu Linux on Machine 1
arm64 Ubuntu Linux on Machine 2 (raspberry pi)

Which cloud storage system are you using? (eg Google Drive)

Google Drive

The command you were trying to run (eg rclone copy /tmp remote:tmp)

I was running mkdir and mv from the OS over an rclone mount.

The rclone config contents with secrets removed.

[gdrive]
type = drive
client_id = [REDACTED]
client_secret = [REDACTED]
token = [REDACTED]
root_folder_id = 0AO9MZRXNwt_jUk9PVA

[crypt_gdrive]
type = crypt
remote = gdrive:crypt
filename_encryption = standard
password = [REDACTED]
password2 = [REDACTED]

A log from the command with the -vv flag

sample log file, filenames have been changed to protect the innocent.

2021/06/08 18:29:42 INFO  : dir1/1998/file1.txt: Moved (server-side) to: dir1/1998/file1/file1.txt
2021/06/08 18:29:42 INFO  : dir1/1998/file1.txt: vfs cache: renamed in cache to "dir1/1998/file1/file1.txt"
2021/06/08 18:29:43 INFO  : dir1/1998/file2.txt: Moved (server-side) to: dir1/1998/file2/file2.txt
2021/06/08 18:29:43 INFO  : dir1/1998/file2.txt: vfs cache: renamed in cache to "dir1/1998/file2/file2.txt"
2021/06/08 18:29:44 INFO  : dir1/1998/file3.txt: Moved (server-side) to: dir1/1998/file3/file3.txt
2021/06/08 18:29:44 INFO  : dir1/1998/file3.txt: vfs cache: renamed in cache to "dir1/1998/file3/file3.txt"
2021/06/08 18:30:30 INFO  : vfs cache: cleaned: objects 7716 (was 7716) in use 0, to upload 0, uploading 0, total size 48.707G (was 48.707G)

hi,

before calling this a bug, perhaps update rclone on machine 2 and test again.

also, please post the rclone mount commands and full debug logs.

The files are already deleted, so I can't delete them again, and, as I said before, I've run this script hundreds of times with no issues, so it's probably not immediately reproducible.

I would LIKE to say it was user error, but it wasn't. Machine 1 could see, open and manipulate files in the directory and Machine 2 couldn't.

I will try to reproduce the issue, but I'm not sure I can.

I have (practically) no experience with rclone mount, but may have a few ideas for your troubleshooting:

  • What do you see in your Gdrive? (using a browser)
  • What do you see in your Gdrive trash folder? (using a browser)
  • What do you see in your historic backups? (are you able to pin-point when the files disappeared)

A wild guess: Crypt will only show encrypted files, so the situation could occur if Machine 1 was mounted without crypt, machine2 mounted with crypt and Machine 1 remounted with crypt.

There was some cache related issues in the old version which might be causing your issue as that was mentioned to get updated, I would do that.

You'd have to recreate the issue and share a debug log to figure it out as anything else would just be guessing and not a good use of your time.

Unfortunately, I performed the server-side moves from Machine 1 running v1.55. So maybe the bug still exists?

I only used the command-line from Machine 2.

I have validated that rclone was properly configured to access the encrypted files on Google Drive from both machines.

I have been running this mount configuration for many years.

Could be anything without a log / reproduction so I avoid guessing.

Understood. I will try to reproduce the error with proper logging in place, but it appears to be intermittent at best, so it may be awhile.