Force rclone not to check for duplicates

Help and Support
1

hello
I want to know wether I can disable duplicate detection on rclone. on rclone move or copy (to a remote), if a directory has several thousand files, it really slows down the process, and I'm sure I won't produce equal filenames before uploading.
is it possible?

2

Can you share what you are running? How would disable duplicate detection work as it would reupload everything if you used --ignore-existing?

3

overwritting everything?

4

Well. I know what you mean. Goole Drive can store multiple files/folders with the exact same name in the same area. Example:

  • folder/text.txt
  • folder/text.txt
  • folder/text.txt
  • folder/text.txt
  • folder/text.txt

This is because Drive isn't working like a normal file system. Each folder and file got an ID. The filename ist just a name like in a contact list in your phone. There you can still see two different people with the exact same name like "Marvin", but your phone still knows who is who.

Some Clouds are not Case Sensetiv. Means:

  • folder/text.txt
  • folder/Text.txt
  • folder/tExt.txt
  • folder/teXt.txt
  • folder/texT.txt

As you can see there are two same files with the same name but this time one has a bigger T. Those filesystems see them as different files. Windows doesn't care so it sees both as the same file.

Take a look here: https://rclone.org/overview/
Look at the "Case Insensitive" and "Duplicate Files" section.

Means, generally files are getting overwritten or skipped. This is how a file system works.

1 Like
5

--ignore-existing still checks for existing... my goal is for it to not even check, as my script generates unique filenames every time...duplicates isn't the worry, the worry is that it takes several minutes before it even starts uploading.

6

What command are you running?
What backend are you connecting to?

7

rclone move srcdir drive:destdir -P
to google drive

8

Did you make your own API key/Client ID?

https://rclone.org/drive/#making-your-own-client-id

You can add --fast-list and that should speed it up as well.

If you can run the command with -vv on it, you can share the output and we can see what's slow to start.

9

I added my own API key, yes.
and I'll try --fast-list and give you output.

10

https://pastebin.com/raw/xWyrmddp here is the output, I'll try with --fast-list now

11

--fast-list has no improvement here either, besides spending more memory, it actually seems to be considerably slower.

12

Can you share the output with the -vv?

13

Try adding --no-traverse.

1 Like
14

I just did.
@calisro I'll try later and give feedback

15

that was the solution after all, now the times are null. thanks!

1 Like
16

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.