Google Server Side file copy bug

Suspected Bug
1

What is the problem you are having with rclone?

I have a bash script that bulk moves files into subdirectories. I use it on both local and rclone-mounted remote directories, and it worked perfectly until today.

I have 2 machines that both connect to the same GDrive backend. Machine 1 executes the script through an rclone-gdrive mount and the copy performs as normal. When completed, I can browse the directory from Machine 1 and everything is there as it should be. I can open, copy and delete the files as normal.

When I tried to access the same directory using a command-line rclone ls from Machine 2, the directory appears empty. All the files and associated subdirectories are gone.

I initially thought Machine 1 borked the copy and was pulling them from VFS cache, so I performed a rclone rc vfs/refresh on Machine 1 and every appeared to be fine. I could open and manipulate the files as normal. I then stopped and restarted the rclone mount on Machine 1 and the files disappeared.

So two things: How was Machine 1 able to access the files if they are, in fact, gone and two: why were the files gone. I've literally run this script hundreds of times across three different machines/OSs and rclone mount configurations with no issue.

What is your rclone version (output from rclone version)

v1.55 amd64 on Machine 1
v1.51 arm64 on Machine 2 (raspberry pi)

Which OS you are using and how many bits (eg Windows 7, 64 bit)

x64 Ubuntu Linux on Machine 1
arm64 Ubuntu Linux on Machine 2 (raspberry pi)

Which cloud storage system are you using? (eg Google Drive)

Google Drive

The command you were trying to run (eg rclone copy /tmp remote:tmp)

I was running mkdir and mv from the OS over an rclone mount.

The rclone config contents with secrets removed.

[gdrive]
type = drive
client_id = [REDACTED]
client_secret = [REDACTED]
token = [REDACTED]
root_folder_id = 0AO9MZRXNwt_jUk9PVA

[crypt_gdrive]
type = crypt
remote = gdrive:crypt
filename_encryption = standard
password = [REDACTED]
password2 = [REDACTED]

A log from the command with the -vv flag

sample log file, filenames have been changed to protect the innocent.

2021/06/08 18:29:42 INFO  : dir1/1998/file1.txt: Moved (server-side) to: dir1/1998/file1/file1.txt
2021/06/08 18:29:42 INFO  : dir1/1998/file1.txt: vfs cache: renamed in cache to "dir1/1998/file1/file1.txt"
2021/06/08 18:29:43 INFO  : dir1/1998/file2.txt: Moved (server-side) to: dir1/1998/file2/file2.txt
2021/06/08 18:29:43 INFO  : dir1/1998/file2.txt: vfs cache: renamed in cache to "dir1/1998/file2/file2.txt"
2021/06/08 18:29:44 INFO  : dir1/1998/file3.txt: Moved (server-side) to: dir1/1998/file3/file3.txt
2021/06/08 18:29:44 INFO  : dir1/1998/file3.txt: vfs cache: renamed in cache to "dir1/1998/file3/file3.txt"
2021/06/08 18:30:30 INFO  : vfs cache: cleaned: objects 7716 (was 7716) in use 0, to upload 0, uploading 0, total size 48.707G (was 48.707G)
2

hi,

before calling this a bug, perhaps update rclone on machine 2 and test again.

also, please post the rclone mount commands and full debug logs.

3

The files are already deleted, so I can't delete them again, and, as I said before, I've run this script hundreds of times with no issues, so it's probably not immediately reproducible.

I would LIKE to say it was user error, but it wasn't. Machine 1 could see, open and manipulate files in the directory and Machine 2 couldn't.

I will try to reproduce the issue, but I'm not sure I can.

4

I have (practically) no experience with rclone mount, but may have a few ideas for your troubleshooting:

  • What do you see in your Gdrive? (using a browser)
  • What do you see in your Gdrive trash folder? (using a browser)
  • What do you see in your historic backups? (are you able to pin-point when the files disappeared)

A wild guess: Crypt will only show encrypted files, so the situation could occur if Machine 1 was mounted without crypt, machine2 mounted with crypt and Machine 1 remounted with crypt.

5

There was some cache related issues in the old version which might be causing your issue as that was mentioned to get updated, I would do that.

You'd have to recreate the issue and share a debug log to figure it out as anything else would just be guessing and not a good use of your time.

6

Unfortunately, I performed the server-side moves from Machine 1 running v1.55. So maybe the bug still exists?

I only used the command-line from Machine 2.

7

I have validated that rclone was properly configured to access the encrypted files on Google Drive from both machines.

I have been running this mount configuration for many years.

8

Could be anything without a log / reproduction so I avoid guessing.

9

Understood. I will try to reproduce the error with proper logging in place, but it appears to be intermittent at best, so it may be awhile.

10

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.