Dest location much larger

Help and Support
1

What is the problem you are having with rclone?

I am trying to sync a Google Drive shared folder to local.

A 6.68GB folder on a google drive shared is syncing to become over 100+GB.

I tried with a different shared folder, one that was 40GB, left it syncing and it became 700GB+ on my local disk.

Run the command 'rclone version' and share the full output of the command.

rclone v1.67.0

  • os/version: darwin 10.15.7 (64 bit)

  • os/kernel: 19.6.0 (x86_64)

  • os/type: darwin

  • os/arch: amd64

  • go/version: go1.22.4

  • go/linking: dynamic

  • go/tags: cmount

Which cloud storage system are you using? (eg Google Drive)

Google Drive

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone copy SF-Mgt-Drive: /Users/macserver/GoogleDriveMgt

Please run 'rclone config redacted' and share the full output. If you get command not found, please make sure to update rclone.

[SF-Directors-Drive]

type = drive

scope = drive

token = XXX

team_drive = XXX

root_folder_id =

[SF-Mgt-Drive]

type = drive

scope = drive

token = XXX

team_drive = XXX

root_folder_id =

[SF-Team-Drive]

type = drive

scope = drive

token = XXX

team_drive = XXX

root_folder_id =

Double check the config for sensitive info before posting publicly

A log from the command that you were trying to run with the -vv flag

2024/08/21 19:29:37 NOTICE: Dangling shortcut "Temp Doc: Kat PDP 2023" detected

2024/08/21 19:29:37 NOTICE: Dangling shortcut "Temp doc: JT’s PDP Responses 2023" detected

2024/08/21 19:29:38 NOTICE: All Hands/2024-05 All Hands/Final session/IMG_2850.MOV: Duplicate object found in source - ignoring

2024/08/21 19:29:38 NOTICE: All Hands/2024-05 All Hands/Final session/IMG_2851.MOV: Duplicate object found in source - ignoring

2024/08/21 19:29:38 NOTICE: All Hands/2024-05 All Hands/Final session/IMG_2856.MOV: Duplicate object found in source - ignoring

2024/08/21 19:29:38 NOTICE: Dangling shortcut "Growth" detected

2024/08/21 19:30:03 ERROR : Team/Line Management/JT/Temp doc: JT’s PDP Responses 2023: Failed to copy: failed to open source object: can't read dangling shortcut

2024/08/21 19:30:09 ERROR : Team/Line Management/KW/Temp Doc: Kat PDP 2023: Failed to copy: failed to open source object: can't read dangling shortcut

2024/08/21 19:33:13 ERROR : Misc/Lobo Archive/Lobo Creative/Brand/Growth: Failed to copy: failed to open source object: can't read dangling shortcut

2024/08/21 19:33:31 ERROR : Misc/Lobo Archive/Lobo Creative/Brand/Website/Design 2.0: error reading source directory: couldn't list directory: googleapi: Error 403: Quota exceeded for quota metric 'Queries' and limit 'Queries per minute' of service 'drive.googleapis.com' for consumer 'project_number:202264815644'.

Thanks,

2

Looks like the source has duplicates so you'd want to dedupe that first.

The snippet of the log really doesn't show much other than duplicates and some shortcuts. Odds are, that there is something recursive or a shortcut somewhere else which is making up the space.

Full debug log would shed light on the issue.

3

Thank you.

Would it be helpful to run it dry, and then send over the full log?

Would dedupe affect the source? That would scare me a little!

Thanks

4

If the source isn't deduped, you'd have no idea what rclone would do to copy files.

Google allows for duplicate files and folder. Mostly everything else does not. If you want to get a clean copy to any OS, you'd have to dedupe it first.

5

Here's a sample of the log; macserver@macservers-Mac-mini ~ % rclone copy SF-Mgt-Drive: /Users/macserver/Goo - Pastebin.com

6

That log looks much better.

I'm guessing you deduped it? I don't see any dupe errors which is good thing.

It's been awhile since I used Google Drive, but aren't these shortcuts that point to nothing?

2024/08/23 20:06:37 NOTICE: Dangling shortcut "Temp doc: JT’s PDP Responses 2023" detected

I never used any shortcuts in my drive when I was using it.

I think you want to use:

--drive-skip-dangling-shortcuts

To avoid any errors for those that actually don't exist.

If you did dedupde, I would give a copy test to a clean destination and get a full log and that would be the best.

You could have some recursive shortcut or something that's looping around as the log file would show you that as well.

7

Oddly, even with the flag, it still logs;

macserver@macservers-Mac-mini ~ % rclone copy SF-Mgt-Drive: /Users/macserver/GoogleDriveMgt --drive-skip-dangling-shortcuts

2024/08/24 08:45:01 NOTICE: All Hands/2024-05 All Hands/Final session/IMG_2850.MOV: Duplicate object found in source - ignoring

2024/08/24 08:45:01 NOTICE: All Hands/2024-05 All Hands/Final session/IMG_2851.MOV: Duplicate object found in source - ignoring

2024/08/24 08:45:01 NOTICE: All Hands/2024-05 All Hands/Final session/IMG_2856.MOV: Duplicate object found in source - ignoring

2024/08/24 08:45:01 NOTICE: Dangling shortcut "Temp Doc: Kat PDP 2023" detected

2024/08/24 08:45:01 NOTICE: Dangling shortcut "Temp doc: JT’s PDP Responses 2023" detected

2024/08/24 08:45:02 NOTICE: Dangling shortcut "Growth" detected

8

Did you rclone dedupe the source?

9

Hi,

Not yet - the drive we're talking about is quite critical, and the idea of running a cmd that has the power to delete concerns me.

I think i'll look for a way to backup the drives before deduping incase of lost data

Thanks,

10

You can rename instead of delete.

rclone dedupe

By sure to run dry-run to be 100% sure it is going to do what you want.

11

Smart!

Here's what i've written;

rclone --dry-run dedupe --dedupe-mode rename SF-Mgt-Drive:

It doesn't seem to run anything

12

You'd have to share some output as I can't see your screen :slight_smile:

Run it with -vv and share the output.

13

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.