macOS duplicate folder with accent created

haldir · May 12, 2021, 1:01pm

What is the problem you are having with rclone?

Hello,

I have existing folders on GDrive with accent in the name. Folder can be browse from macOS, Windows or Linux without issue and content can be opened.
However, when using rclone move command to move folders and files from local directory to google, a duplicate folder is created for all local folders with an accent in the name.

macOS doesn't display both folders but using web access or netdrive on Windows, I can see duplicate folder but finder will display only the oldest.

Using rclone command, I can see both folder:
rclone lsd plex-gdrive:Plex
-1 2021-05-12 14:44:12 -1 Séries TV
-1 2018-01-01 22:22:44 -1 Séries TV

Thanks

What is your rclone version (output from `rclone version`)

rclone v1.56.0-beta.5481.6ef7178ee

os/version: darwin 11.3.1 (64 bit)
os/kernel: 20.4.0 (x86_64)
os/type: darwin
os/arch: amd64
go/version: go1.16.3
go/linking: dynamic
go/tags: cmount

Which OS you are using and how many bits (eg Windows 7, 64 bit)

macOS 11.3.1

Which cloud storage system are you using? (eg Google Drive)

Google Drive

The command you were trying to run (eg `rclone copy /tmp remote:tmp`)

rclone move /Users/haldir/local plex-gdrive:/Plex --config=/Users/haldir/.config/rclone/rclone.conf --drive-chunk-size=64M --verbose=1 --user-agent='Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.131 Safari/537.36' --stats=60s --skip-links --checkers=16 --transfers=8 --exclude='partial~' --exclude='_HIDDEN~' --exclude='.unionfs/' --exclude='.unionfs-fuse/' --exclude='.DS_Store'```

The rclone config contents with secrets removed.

[plex-gdrive]
type = drive
client_id = 
client_secret = 
scope = drive
token =

A log from the command with the `-vv` flag

2021/05/12 14:44:08 DEBUG : rclone: Version "v1.56.0-beta.5481.6ef7178ee" starting with parameters ["rclone" "move" "/Users/haldir/local" "plex-gdrive:/Plex" "--config=/Users/haldir/.config/rclone/rclone.conf" "--drive-chunk-size=64M" "--verbose=1" "--user-agent=Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.131 Safari/537.36" "--stats=60s" "--skip-links" "--checkers=16" "--transfers=8" "--exclude=partial~" "--exclude=_HIDDEN~" "--exclude=.unionfs/" "--exclude=.unionfs-fuse/" "--exclude=.DS_Store" "--log-file=/Users/haldir/Desktop/rclone.log" "-vv"]
2021/05/12 14:44:08 DEBUG : Creating backend with remote "/Users/haldir/local"
2021/05/12 14:44:08 DEBUG : local: detected overridden config - adding "{HK82T}" suffix to name
2021/05/12 14:44:08 DEBUG : Using config file from "/Users/haldir/.config/rclone/rclone.conf"
2021/05/12 14:44:08 DEBUG : fs cache: renaming cache item "/Users/haldir/local" to be canonical "local{HK82T}:/Users/haldir/local"
2021/05/12 14:44:08 DEBUG : Creating backend with remote "plex-gdrive:/Plex"
2021/05/12 14:44:08 DEBUG : plex-gdrive: detected overridden config - adding "{84KcY}" suffix to name
2021/05/12 14:44:08 DEBUG : Google drive root 'Plex': root_folder_id = "redacted" - save this in the config to speed up startup
2021/05/12 14:44:08 DEBUG : fs cache: renaming cache item "plex-gdrive:/Plex" to be canonical "plex-gdrive{84KcY}:Plex"
2021/05/12 14:44:08 DEBUG : .DS_Store: Excluded
2021/05/12 14:44:09 DEBUG : Séries TV VOST/.DS_Store: Excluded
2021/05/12 14:44:11 DEBUG : Google drive root 'Plex': Waiting for checks to finish
2021/05/12 14:44:11 DEBUG : Google drive root 'Plex': Waiting for transfers to finish
2021/05/12 14:44:13 DEBUG : Séries TV/autoscan_v1.2.0_darwin_amd64: Sending chunk 0 length 12009344
2021/05/12 14:44:15 DEBUG : Séries TV/autoscan_v1.2.0_darwin_amd64: MD5 = 7d79446d0f0b40c5dae65f307d19d6a8 OK
2021/05/12 14:44:15 INFO  : Séries TV/autoscan_v1.2.0_darwin_amd64: Copied (new)
2021/05/12 14:44:15 INFO  : Séries TV/autoscan_v1.2.0_darwin_amd64: Deleted
2021/05/12 14:44:15 INFO  : 
Transferred:   	   11.453Mi / 11.453 MiByte, 100%, 2.943 MiByte/s, ETA 0s
Checks:                 2 / 2, 100%
Deleted:                1 (files), 0 (dirs)
Renamed:                1
Transferred:            1 / 1, 100%
Elapsed time:         7.0s

2021/05/12 14:44:15 DEBUG : 8 go routines active

Animosity022 · May 12, 2021, 1:08pm

You need to run rclone dedupe on the destination.

asdffdsa · May 12, 2021, 1:24pm

hi,
i thought that rclone lsd on gdrive could not display duplicate folders?

Animosity022 · May 12, 2021, 1:29pm

rclone lsd will show duplicates.

Last login: Tue May 11 10:42:43 2021 from 192.168.1.135
felix@gemini:~$ rclone lsd GD:
          -1 2020-12-27 15:33:11        -1 backups
          -1 2019-09-29 21:57:58        -1 crypt
          -1 2021-05-12 09:28:17        -1 test
          -1 2021-04-30 08:37:12        -1 test
felix@gemini:~$

Things on file systems like mounts and certain commands like copy/move/etc may have problems.

asdffdsa · May 12, 2021, 1:30pm

ok. good to know.

haldir · May 12, 2021, 1:33pm

but why the folders are not merged ?

Animosity022 · May 12, 2021, 1:38pm

You need to run rclone dedupe.

https://rclone.org/commands/rclone_dedupe/

haldir · May 12, 2021, 1:53pm

Some duplicate files have been removed but both folders are still displayed in GDrive

Animosity022 · May 12, 2021, 1:54pm

You'd have to run the command with -vv and share the full debug log so we can help.

haldir · May 12, 2021, 2:10pm

have to wait, I reached API limit :
low level retry 2/10 (error googleapi: Error 403: User Rate Limit Exceeded. Rate of requests for user exceed configured project quota. You may consider re-evaluating expected per-user traffic to the API and adjust project quota limits accordingly. You may monitor aggregate quota usage and adjust limits in the API Console: https://console.developers.google.com/apis/api/drive.googleapis.com/quotas?project=8677, userRateLimitExceeded)

Animosity022 · May 12, 2021, 2:13pm

If you can run the command and share the full log with -vv, that's what is needed.

You don't need to wait, that's just rclone being fast and it'll retry without an issue.

haldir · May 12, 2021, 2:18pm

I run the following command: rclone dedupe plex-gdrive:Plex -vv --log-file=/Users/haldir/Desktop/dedup.log

2021/05/12 16:01:05 DEBUG : rclone: Version "v1.56.0-beta.5481.6ef7178ee" starting with parameters ["rclone" "dedupe" "plex-gdrive:Plex" "-vv" "--log-file=/Users/haldir/Desktop/dedup.log"]
2021/05/12 16:01:05 DEBUG : Creating backend with remote "plex-gdrive:Plex"
2021/05/12 16:01:05 DEBUG : Using config file from "/Users/haldir/.config/rclone/rclone.conf"
2021/05/12 16:01:05 DEBUG : Google drive root 'Plex': root_folder_id = "0AMaDGyAC4KXLUk9PVA" - save this in the config to speed up startup
2021/05/12 16:01:06 INFO  : Google drive root 'Plex': Looking for duplicate names using interactive mode.
2021/05/12 16:01:07 DEBUG : pacer: low level retry 1/10 (error googleapi: Error 403: User Rate Limit Exceeded. Rate of requests for user exceed configured project quota. You may consider re-evaluating expected per-user traffic to the API and adjust project quota limits accordingly. You may monitor aggregate quota usage and adjust limits in the API Console: https://console.developers.google.com/apis/api/drive.googleapis.com/quotas?project=864134864277, userRateLimitExceeded)
2021/05/12 16:01:07 DEBUG : pacer: Rate limited, increasing sleep to 1.177745378s
2021/05/12 16:01:07 DEBUG : pacer: Reducing sleep to 0s
2021/05/12 16:01:07 DEBUG : pacer: low level retry 2/10 (error googleapi: Error 403: User Rate Limit Exceeded. Rate of requests for user exceed configured project quota. You may consider re-evaluating expected per-user traffic to the API and adjust project quota limits accordingly. You may monitor aggregate quota usage and adjust limits in the API Console: https://console.developers.google.com/apis/api/drive.googleapis.com/quotas?project=864134864277, userRateLimitExceeded)
2021/05/12 16:01:07 DEBUG : pacer: Rate limited, increasing sleep to 1.359468142s
2021/05/12 16:01:09 DEBUG : pacer: Reducing sleep to 0s
2021/05/12 16:01:10 DEBUG : pacer: low level retry 1/10 (error googleapi: Error 403: User Rate Limit Exceeded. Rate of requests for user exceed configured project quota. You may consider re-evaluating expected per-user traffic to the API and adjust project quota limits accordingly. You may monitor aggregate quota usage and adjust limits in the API Console: https://console.developers.google.com/apis/api/drive.googleapis.com/quotas?project=864134864277, userRateLimitExceeded)
2021/05/12 16:01:10 DEBUG : pacer: Rate limited, increasing sleep to 1.096126334s
2021/05/12 16:01:10 DEBUG : pacer: low level retry 1/10 (error googleapi: Error 403: User Rate Limit Exceeded. Rate of requests for user exceed configured project quota. You may consider re-evaluating expected per-user traffic to the API and adjust project quota limits accordingly. You may monitor aggregate quota usage and adjust limits in the API Console: https://console.developers.google.com/apis/api/drive.googleapis.com/quotas?project=864134864277, userRateLimitExceeded)
2021/05/12 16:01:10 DEBUG : pacer: Rate limited, increasing sleep to 2.347210404s
2021/05/12 16:01:10 DEBUG : pacer: low level retry 1/10 (error googleapi: Error 403: User Rate Limit Exceeded. Rate of requests for user exceed configured project quota. You may consider re-evaluating expected per-user traffic to the API and adjust project quota limits accordingly. You may monitor aggregate quota usage and adjust limits in the API Console: https://console.developers.google.com/apis/api/drive.googleapis.com/quotas?project=864134864277, userRateLimitExceeded)
2021/05/12 16:01:10 DEBUG : pacer: Rate limited, increasing sleep to 4.000512704s
2021/05/12 16:01:10 DEBUG : pacer: low level retry 2/10 (error googleapi: Error 403: User Rate Limit Exceeded. Rate of requests for user exceed configured project quota. You may consider re-evaluating expected per-user traffic to the API and adjust project quota limits accordingly. You may monitor aggregate quota usage and adjust limits in the API Console: https://console.developers.google.com/apis/api/drive.googleapis.com/quotas?project=864134864277, userRateLimitExceeded)
2021/05/12 16:01:10 DEBUG : pacer: Rate limited, increasing sleep to 8.050176845s
2021/05/12 16:01:11 DEBUG : pacer: Reducing sleep to 0s
2021/05/12 16:01:22 DEBUG : pacer: low level retry 1/10 (error googleapi: Error 403: User Rate Limit Exceeded. Rate of requests for user exceed configured project quota. You may consider re-evaluating expected per-user traffic to the API and adjust project quota limits accordingly. You may monitor aggregate quota usage and adjust limits in the API Console: https://console.developers.google.com/apis/api/drive.googleapis.com/quotas?project=864134864277, userRateLimitExceeded)
2021/05/12 16:01:22 DEBUG : pacer: Rate limited, increasing sleep to 1.78603676s
2021/05/12 16:01:23 DEBUG : pacer: Reducing sleep to 0s
2021/05/12 16:01:36 DEBUG : pacer: low level retry 1/10 (error googleapi: Error 403: User Rate Limit Exceeded. Rate of requests for user exceed configured project quota. You may consider re-evaluating expected per-user traffic to the API and adjust project quota limits accordingly. You may monitor aggregate quota usage and adjust limits in the API Console: https://console.developers.google.com/apis/api/drive.googleapis.com/quotas?project=864134864277, userRateLimitExceeded)
2021/05/12 16:01:36 DEBUG : pacer: Rate limited, increasing sleep to 1.120419343s
2021/05/12 16:01:36 DEBUG : pacer: low level retry 1/10 (error googleapi: Error 403: User Rate Limit Exceeded. Rate of requests for user exceed configured project quota. You may consider re-evaluating expected per-user traffic to the API and adjust project quota limits accordingly. You may monitor aggregate quota usage and adjust limits in the API Console: https://console.developers.google.com/apis/api/drive.googleapis.com/quotas?project=864134864277, userRateLimitExceeded)
2021/05/12 16:01:36 DEBUG : pacer: Rate limited, increasing sleep to 2.479510555s
2021/05/12 16:01:36 DEBUG : pacer: low level retry 1/10 (error googleapi: Error 403: User Rate Limit Exceeded. Rate of requests for user exceed configured project quota. You may consider re-evaluating expected per-user traffic to the API and adjust project quota limits accordingly. You may monitor aggregate quota usage and adjust limits in the API Console: https://console.developers.google.com/apis/api/drive.googleapis.com/quotas?project=864134864277, userRateLimitExceeded)
2021/05/12 16:01:36 DEBUG : pacer: Rate limited, increasing sleep to 4.267678867s
2021/05/12 16:01:36 DEBUG : pacer: low level retry 2/10 (error googleapi: Error 403: User Rate Limit Exceeded. Rate of requests for user exceed configured project quota. You may consider re-evaluating expected per-user traffic to the API and adjust project quota limits accordingly. You may monitor aggregate quota usage and adjust limits in the API Console: https://console.developers.google.com/apis/api/drive.googleapis.com/quotas?project=864134864277, userRateLimitExceeded)
2021/05/12 16:01:36 DEBUG : pacer: Rate limited, increasing sleep to 8.41084469s
2021/05/12 16:01:36 DEBUG : pacer: Reducing sleep to 0s
2021/05/12 16:01:38 DEBUG : pacer: low level retry 1/10 (error googleapi: Error 403: User Rate Limit Exceeded. Rate of requests for user exceed configured project quota. You may consider re-evaluating expected per-user traffic to the API and adjust project quota limits accordingly. You may monitor aggregate quota usage and adjust limits in the API Console: https://console.developers.google.com/apis/api/drive.googleapis.com/quotas?project=864134864277, userRateLimitExceeded)
2021/05/12 16:01:38 DEBUG : pacer: Rate limited, increasing sleep to 1.826601893s
2021/05/12 16:01:38 DEBUG : pacer: low level retry 1/10 (error googleapi: Error 403: User Rate Limit Exceeded. Rate of requests for user exceed configured project quota. You may consider re-evaluating expected per-user traffic to the API and adjust project quota limits accordingly. You may monitor aggregate quota usage and adjust limits in the API Console: https://console.developers.google.com/apis/api/drive.googleapis.com/quotas?project=864134864277, userRateLimitExceeded)
2021/05/12 16:01:38 DEBUG : pacer: Rate limited, increasing sleep to 2.578595375s
2021/05/12 16:01:38 DEBUG : pacer: low level retry 2/10 (error googleapi: Error 403: User Rate Limit Exceeded. Rate of requests for user exceed configured project quota. You may consider re-evaluating expected per-user traffic to the API and adjust project quota limits accordingly. You may monitor aggregate quota usage and adjust limits in the API Console: https://console.developers.google.com/apis/api/drive.googleapis.com/quotas?project=864134864277, userRateLimitExceeded)
2021/05/12 16:01:38 DEBUG : pacer: Rate limited, increasing sleep to 4.371199385s
2021/05/12 16:01:39 DEBUG : pacer: Reducing sleep to 0s
2021/05/12 16:01:42 DEBUG : pacer: low level retry 1/10 (error googleapi: Error 403: User Rate Limit Exceeded. Rate of requests for user exceed configured project quota. You may consider re-evaluating expected per-user traffic to the API and adjust project quota limits accordingly. You may monitor aggregate quota usage and adjust limits in the API Console: https://console.developers.google.com/apis/api/drive.googleapis.com/quotas?project=864134864277, userRateLimitExceeded)
2021/05/12 16:01:42 DEBUG : pacer: Rate limited, increasing sleep to 1.253441562s
2021/05/12 16:01:42 DEBUG : pacer: Reducing sleep to 0s
2021/05/12 16:01:47 DEBUG : pacer: low level retry 1/10 (error googleapi: Error 403: User Rate Limit Exceeded. Rate of requests for user exceed configured project quota. You may consider re-evaluating expected per-user traffic to the API and adjust project quota limits accordingly. You may monitor aggregate quota usage and adjust limits in the API Console: https://console.developers.google.com/apis/api/drive.googleapis.com/quotas?project=864134864277, userRateLimitExceeded)
2021/05/12 16:01:47 DEBUG : pacer: Rate limited, increasing sleep to 1.716111201s
2021/05/12 16:01:47 DEBUG : pacer: Reducing sleep to 0s
2021/05/12 16:01:47 DEBUG : pacer: low level retry 2/10 (error googleapi: Error 403: User Rate Limit Exceeded. Rate of requests for user exceed configured project quota. You may consider re-evaluating expected per-user traffic to the API and adjust project quota limits accordingly. You may monitor aggregate quota usage and adjust limits in the API Console: https://console.developers.google.com/apis/api/drive.googleapis.com/quotas?project=864134864277, userRateLimitExceeded)
2021/05/12 16:01:47 DEBUG : pacer: Rate limited, increasing sleep to 1.420345471s
2021/05/12 16:01:48 DEBUG : pacer: Reducing sleep to 0s
2021/05/12 16:01:50 DEBUG : pacer: low level retry 1/10 (error googleapi: Error 403: User Rate Limit Exceeded. Rate of requests for user exceed configured project quota. You may consider re-evaluating expected per-user traffic to the API and adjust project quota limits accordingly. You may monitor aggregate quota usage and adjust limits in the API Console: https://console.developers.google.com/apis/api/drive.googleapis.com/quotas?project=864134864277, userRateLimitExceeded)
2021/05/12 16:01:50 DEBUG : pacer: Rate limited, increasing sleep to 1.981821767s
2021/05/12 16:01:51 DEBUG : pacer: Reducing sleep to 0s
2021/05/12 16:02:04 DEBUG : pacer: low level retry 1/10 (error googleapi: Error 403: User Rate Limit Exceeded. Rate of requests for user exceed configured project quota. You may consider re-evaluating expected per-user traffic to the API and adjust project quota limits accordingly. You may monitor aggregate quota usage and adjust limits in the API Console: https://console.developers.google.com/apis/api/drive.googleapis.com/quotas?project=864134864277, userRateLimitExceeded)
2021/05/12 16:02:04 DEBUG : pacer: Rate limited, increasing sleep to 1.72929307s
2021/05/12 16:02:04 DEBUG : pacer: low level retry 2/10 (error googleapi: Error 403: User Rate Limit Exceeded. Rate of requests for user exceed configured project quota. You may consider re-evaluating expected per-user traffic to the API and adjust project quota limits accordingly. You may monitor aggregate quota usage and adjust limits in the API Console: https://console.developers.google.com/apis/api/drive.googleapis.com/quotas?project=864134864277, userRateLimitExceeded)
2021/05/12 16:02:04 DEBUG : pacer: Rate limited, increasing sleep to 2.57895355s
2021/05/12 16:02:05 DEBUG : pacer: Reducing sleep to 0s
2021/05/12 16:02:10 DEBUG : 20 go routines active

Animosity022 · May 12, 2021, 2:51pm

That looks like it cleaned up dupes. Can you redo the rclone lsd and share the output?

haldir · May 12, 2021, 3:26pm

still the quota error and always both folders from lsd

Animosity022 · May 12, 2021, 3:28pm

I can't see your screen so with no output, it's tough man.

haldir · May 12, 2021, 8:16pm

I did a test by uploading a file in a folder without accent and it has been merged in the correct folder.

system · July 12, 2021, 4:17pm

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.