Rclone move keeps trying to move same file

#1

move command:
rclone copy --tpslimit 6 --drive-chunk-size 32M --transfers 6 --checkers 6 --log-file=/home/xxx/logs/move.log --log-level INFO --stats 30s /home/xxx/Media/ gdrive:/Media

from the log file
<filename...> Duplicate directory found in destination - ignoring

Source path folder is empty so dont understand why it keeps popping up with that "error"

I think it have resulted in me constantly getting 403 errors
ReadFileHandle.Read error: low level retry 8/10: open file failed: googleapi: Error 403: The download quota for this file has been exceeded., downloadQuotaExceeded

#2

What does your rclone version show?

#3

rclone v1.48.0

  • os/arch: linux/amd64
  • go version: go1.12.6

I've tried to stop the service, edit move.sh to copy instead and reload the service. Still the same

#4

I suggest you fix that with rclone dedupe

1 Like
#5

Are the files in question being handled by some third-party program, like a torrent downloader?
In my experience, programs which "construct" files, like torrents, rendering in some cases ect. can have some pretty bad behaviour if the software likes to open and close the file-access many times in the process. This is because rclone (using mount) will assume the file is done as soon as it is no longer being accessed. If the file is not done at this point - it gets uploaded. Then it may get re-downloaded very soon after to continue working on it - and this cycle can create massive bandwidth load, not to mention slowing down the whole process immensely.

For programs that have this issue (like some torrent software) I find the best solution is to use a local "work folder" for files until they are properly done before upload. Many programs actually support this in options and will take care of moving the files for you after they are done so the process can be pretty transparent to you.

Not sure if this is your issue since you don't specify the use-case, but I feel it's worth mentioning since it's not uncommon.

#6

This solved the duplicate message in my move log.
Thank you

1 Like