Drive duplicate files reported by rclone lsl

What is the problem you are having with rclone?

rclone lsl of Google Drive lists file revisions (or duplicates) in some circumstances

In checking the rclone.org forum I see history of duplicate problems with Google drive, but not seeing anything specifically with rclone lsl. Should an rclone lsl drive: ever report more than one version of any given file?

Some rclonesync users report that when syncing to Drive that they get duplicate file warnings (an error check within rclonesync). I cannot reproduce this in my drive setup, and I do have revisions of some files. rclonesync does an rclone lsl <drive_remote>: to a file then parses this lsl file. If additional occurrence of a file are found it logs a duplicate warning (and keeps the most recent version). Here's a snip of one such lsl showing several different files with the same name, same directory, and differing dates and sizes. It seems that the lsl output has file revisions listed.

Duplicates found in Google drive rclone lsl output (thru rclonesync):

  • Issue#59 shows some debug output showing a few different versions of file OP/-OP-000.jpeg. This was logged in August 2020. I don't know which rclone version the user was running.
	Path2 versions (Google drive)
	   254331 2020-08-19 10:58:25.996000000 OP/-OP-000.jpeg
	    44288 2020-07-27 17:05:43.329000000 OP/-OP-000.jpeg
	    44288 2020-07-27 17:05:07.634000000 OP/-OP-000.jpeg
	   193826 2020-07-27 17:03:46.784000000 OP/-OP-000.jpeg
	   193826 2020-07-27 17:00:13.550000000 OP/-OP-000.jpeg
	The Path1 version is (local disk)
	   254331 2020-08-19 10:58:25.996000000 OP/-OP-000.jpeg
  • Issue#73 was submitted on March 17, 2021 with duplicates being found in the drive LSL file.
   997317 2018-07-26 17:35:42.806000000 redacted/redacted/Spots/mystique new.mp3
   997317 2018-07-26 17:23:42.476000000 redacted/redacted/Spots/mystique new.mp3

Are there any known sensitivities for having duplicates/revisions show up, or not? Anything in the config file? Anything about the Google drive account or config, or phase of the moon?

What is your rclone version (output from rclone version)

rclone v1.54.0 (and earlier - not a new problem)

os/arch: linux/amd64
go version: go1.15.7

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Linux, 64 bit

Which cloud storage system are you using? (eg Google Drive)

Google Drive

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone lsl drive:

The rclone config contents with secrets removed.

(available if needed)

A log from the command with the -vv flag

(available if needed)

If you have duplicates, you should run rclone dedupe and clean them up.

Thanks.

Are Google Drive "duplicates" the same thing as "versions" (rather than "revisions" as noted above)? Restated: Will an rclone reported duplicate be seen in as a file version in the Google Drive web interface?

I only see the newest version in the rclone lsl output, but other users are seeing what appears to be versions. Do we know what the difference is?

Google Drive allows duplicates.

If you have the same file there more than one time, you have a duplicate.

Most operating systems don't like duplicates as it breaks them so rclone would show them in a lsl and most like break things like copy / sync as it's not expecting duplicates.

felix@gemini:~$ rclone lsl GD: --max-depth 1
      221 2021-03-24 12:41:56.337000000 hosts
      221 2021-01-23 11:49:37.053000000 hosts

I'm not sure what rclonesync is as I've never made a duplicate other than on purpose through the WebUI.

rclone dedupe would be the tool to clean up duplicates.

Dedupe output looks like:

felix@gemini:~$ rclone dedupe GD: -vv
2021/03/24 12:44:04 DEBUG : rclone: Version "v1.54.1" starting with parameters ["rclone" "dedupe" "GD:" "-vv"]
2021/03/24 12:44:04 DEBUG : Using config file from "/opt/rclone/rclone.conf"
2021/03/24 12:44:04 DEBUG : Creating backend with remote "GD:"
2021/03/24 12:44:04 INFO  : Google drive root '': Looking for duplicate names using interactive mode.
2021/03/24 12:45:49 NOTICE: hosts: Found 2 files with duplicate names
2021/03/24 12:45:49 NOTICE: hosts: Deleting 1/2 identical duplicates (MD5 e9b49c993fe22326c398ecea2fd9b219)
2021/03/24 12:45:49 INFO  : hosts: Deleted
2021/03/24 12:45:49 NOTICE: hosts: All duplicates removed
2021/03/24 12:45:49 DEBUG : 18 go routines active

No, duplicates are not revisions. They are just two objects with the same name. They will appear in the google web interface.

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.