Rclone making duplicate directory tree for the same path name

What is the problem you are having with rclone?

I'm moving files from one google drive account to another. But I ended up with multiple folders with the same path duplicated multiple times in the remote I moved my files to instead of having them be moved to the same path.

for example when running the commands:

rclone move "GoogleDrive:foo" "GoogleDriveN:GoogleDrive/foo"

rclone move "GoogleDrive:bar" "GoogleDriveN:GoogleDrive/bar"

it will create two folders with the name GoogleDrive instead of one folder that contains both foo and bar.

Also, when foo fails and I re-run the command, a new GoogleDrive folder is created with partial data, so the data is split into multiple path with the same path name

Here's what I ended up with: https://i.imgur.com/4b88ud8.png

How can I merge the folders into the same parent directories?

Run the command 'rclone version' and share the full output of the command.

rclone v1.58.0
- os/version: ubuntu 20.04 (64 bit)
- os/kernel: 5.4.0-42-generic (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.17.8
- go/linking: static
- go/tags: none

Which cloud storage system are you using? (eg Google Drive)

  • Google Drive

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone move "GoogleDrive:foo" "GoogleDriveN:GoogleDrive/foo" -P --stats-one-line

The rclone config contents with secrets removed.

[GoogleDrive]
type = drive
scope = drive
token = {"access_token":"REDACTED","token_type":"Bearer","refresh_token":"REDACTED","expiry":"REDACTED"}

[GoogleDriveN]
type = drive
scope = drive
token = {"access_token":"REDACTED","token_type":"Bearer","refresh_token":"REDACTED","expiry":"REDACTED"}

A log from the command with the -vv flag

  • Since I used the command rclone move I can't reproduce it with -vv because the files are already moved.

Rclone doesn't do what is in that screenshot so something isn't right.

You'd have to share some logs to see what did that.

I ran out of quota, but here's the logs from an example transfer:

➜ rclone move "GoogleDrive:Useful images" "GoogleDriveN:GoogleDrive/Useful images" -P --stats-one-line --drive-stop-on-upload-limit -vv

2022/04/08 01:37:36 DEBUG : rclone: Version "v1.57.0" starting with parameters ["rclone" "move" "GoogleDrive:Useful images" "GoogleDriveN:GoogleDrive/Useful images" "-P" "--stats-one-line" "--drive-stop-on-upload-limit" "-vv"]
2022/04/08 01:37:36 DEBUG : Creating backend with remote "GoogleDrive:Useful images"
2022/04/08 01:37:36 DEBUG : Using config file from "/Users/me/.config/rclone/rclone.conf"
2022/04/08 01:37:36 DEBUG : GoogleDrive: detected overridden config - adding "{-LSY5}" suffix to name
2022/04/08 01:37:36 DEBUG : Google drive root 'Useful images': 'root_folder_id = [REDACTED]' - save this in the config to speed up startup
2022/04/08 01:37:36 DEBUG : fs cache: renaming cache item "GoogleDrive:Useful images" to be canonical "GoogleDrive{-LSY5}:Useful images"
2022/04/08 01:37:36 DEBUG : Creating backend with remote "GoogleDriveN:GoogleDrive/Useful images"
2022/04/08 01:37:36 DEBUG : GoogleDriveN: detected overridden config - adding "{-LSY5}" suffix to name
2022/04/08 01:37:36 DEBUG : Google drive root 'GoogleDrive/Useful images': 'root_folder_id = [REDACTED]' - save this in the config to speed up startup
2022/04/08 01:37:37 DEBUG : fs cache: renaming cache item "GoogleDriveN:GoogleDrive/Useful images" to be canonical "GoogleDriveN{-LSY5}:GoogleDrive/Useful images"
2022-04-08 01:37:41 DEBUG : Google drive root 'GoogleDrive/Useful images': Waiting for checks to finish
2022-04-08 01:37:41 DEBUG : Google drive root 'GoogleDrive/Useful images': Waiting for transfers to finish
2022-04-08 01:37:43 ERROR : Google drive root 'GoogleDrive/Useful images': Received upload limit error: googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded
2022-04-08 01:37:43 ERROR : HQ Cats Photos Full Set/682.jpg: Failed to copy: googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded
2022-04-08 01:37:43 ERROR : HQ Cats Photos Full Set/682.jpg: Not deleting source as copy failed: googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded
2022-04-08 01:37:43 ERROR : Cancelling sync due to fatal error: googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded
2022-04-08 01:37:43 ERROR : HQ Cats Photos Full Set/681.jpg: Failed to copy: context canceled
2022-04-08 01:37:43 ERROR : HQ Cats Photos Full Set/68.jpg: Failed to copy: context canceled
2022-04-08 01:37:43 ERROR : HQ Cats Photos Full Set/68.jpg: Not deleting source as copy failed: context canceled
2022-04-08 01:37:43 DEBUG : HQ Cats Photos Full Set/683.jpg: Reopening on read failure after 733796 bytes: retry 1/10: context canceled
2022-04-08 01:37:43 ERROR : HQ Cats Photos Full Set/681.jpg: Not deleting source as copy failed: context canceled
2022-04-08 01:37:43 DEBUG : HQ Cats Photos Full Set/683.jpg: Reopen failed after 733796 bytes read: open file failed: Get "https://www.googleapis.com/drive/v3/files/[REDACTED]": context canceled
2022-04-08 01:37:43 ERROR : HQ Cats Photos Full Set/683.jpg: Failed to copy: context canceled
2022-04-08 01:37:43 ERROR : HQ Cats Photos Full Set/683.jpg: Not deleting source as copy failed: context canceled
2022-04-08 01:37:43 ERROR : Fatal error received - not attempting retries
2.323 MiB / 989.841 MiB, 0%, 0 B/s, ETA - (xfr#0/351)
2022/04/08 01:37:43 INFO  :     2.323 MiB / 989.841 MiB, 0%, 0 B/s, ETA - (xfr#0/351)
2022/04/08 01:37:43 DEBUG : 9 go routines active
2022/04/08 01:37:43 Failed to move with 4 errors: last error was: googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded

That doesn't tell me much as nothing was copied or done.

If you have duplicate folders, you may want to clean them up first as well.

rclone dedupe can provide help with this.

i would think to use --drive-server-side-across-configs

1 Like

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.