What is the problem you are having with rclone?
I've been using rclone for about 2 years by now (really love it!), but I have never been able to successfully copy all of my files from a Team Drive to another Team Drive.
I kept missing about a hundred of files, which I don't even know what files are missing in the transfer.
I've been repeating the exact same command for more than 5 times and still missing some files, despite rclone saying that I have everything copied over to the destination drive.
Run the command 'rclone version' and share the full output of the command.
rclone v1.58.1
- os/version: Microsoft Windows 10 Pro 21H2 (64 bit)
- os/kernel: 10.0.19044.1766 (x86_64)
- os/type: windows
- os/arch: amd64
- go/version: go1.17.9
- go/linking: dynamic
- go/tags: cmount
Which cloud storage system are you using? (eg Google Drive)
Google Drive (a Team Drive to another Team Drive)
The command you were trying to run (eg rclone copy /tmp remote:tmp)
I wrote my own simple Python script for rclone to be able to rotate between some service acounts, but the command is basically as below
rclone copy Temporary: PrivateTD1: -P --check-first --fast-list --transfers 4 --checkers 6 --max-transfer 745G --create-empty-src-dirs --drive-acknowledge-abuse --drive-stop-on-upload-limit=true --drive-server-side-across-configs --log-file=log.txt --log-level DEBUG --drive-service-account-file=path_to_service_accounts.json
The rclone config contents with secrets removed.
(Should I include the Team Drive ID and the Folder ID as well?)
[PrivateTD1]
type = drive
client_id = *****
client_secret = *****
scope = drive
token = *****
team_drive = *****
root_folder_id =
service_account_file =
server_side_across_configs = true
[PrivateTD2]
type = drive
client_id = *****
client_secret = *****
scope = drive
token = *****
team_drive = *****
root_folder_id =
service_account_file =
server_side_across_configs = true
[Temporary]
type = drive
client_id = *****
client_secret = *****
scope = drive
token = *****
team_drive = *****
root_folder_id = *****
service_account_file =
server_side_across_configs = true
A log from the command with the -vv flag
I have removed the transfers log because it contain filenames that may not be safe to be shown.
complete log in my github gist
2022/07/05 18:43:25 DEBUG : rclone: Version "v1.58.1" starting with parameters ["D:\\Utility\\rclone\\rclone" "copy" "Temporary:" "PrivateTD1:" "-P" "--check-first" "--fast-list" "--transfers" "4" "--checkers" "6" "--max-transfer" "745G" "--create-empty-src-dirs" "--drive-acknowledge-abuse" "--drive-stop-on-upload-limit=true" "--drive-server-side-across-configs" "--log-file=log.txt" "--log-level" "DEBUG" "--drive-service-account-file=path_to_service_accounts.json"]
2022/07/05 18:43:25 DEBUG : Creating backend with remote "Temporary:"
2022/07/05 18:43:25 DEBUG : Using config file from "D:\\Utility\\rclone\\rclone.conf"
2022/07/05 18:43:25 DEBUG : Temporary: detected overridden config - adding "{ZJse-}" suffix to name
2022/07/05 18:43:25 DEBUG : fs cache: renaming cache item "Temporary:" to be canonical "Temporary{ZJse-}:"
2022/07/05 18:43:25 DEBUG : Creating backend with remote "PrivateTD1:"
2022/07/05 18:43:25 DEBUG : PrivateTD1: detected overridden config - adding "{ZJse-}" suffix to name
2022/07/05 18:43:25 DEBUG : fs cache: renaming cache item "PrivateTD1:" to be canonical "PrivateTD1{ZJse-}:"
2022/07/05 18:43:25 INFO : Google drive root '': Running all checks before starting transfers
2022/07/05 18:43:32 DEBUG : Google drive root '': Disabling ListR to work around bug in drive as multi listing (3) returned no entries
2022/07/05 18:43:32 DEBUG : Google drive root '': Recycled 3 entries
2022/07/05 18:43:33 DEBUG : Google drive root '': Re-enabling ListR as previous detection was in error
----- removed transfer logs -----
2022/07/05 18:43:43 INFO : Google drive root '': Checks finished, now starting transfers
2022/07/05 18:43:43 DEBUG : Google drive root '': Waiting for transfers to finish
2022/07/05 18:43:43 INFO : There was nothing to transfer
2022/07/05 18:43:43 INFO :
Transferred: 0 B / 0 B, -, 0 B/s, ETA -
Checks: 9836 / 9836, 100%
Elapsed time: 18.8s
2022/07/05 18:43:43 DEBUG : 34 go routines active
Total files on each drives
Temporary (source)
----------
Total objects: 9.933k (9933)
Total size: 2.984 TiB (3281416834376 Byte)
PrivateTD1 (destination)
----------
Total objects: 9.837k (9837)
Total size: 2.985 TiB (3282195126567 Byte)
PrivateTD2 (destination)
----------
Total objects: 9.837k (9837)
Total size: 2.985 TiB (3282195126567 Byte)
Check result
rclone check Temporary: PrivateTD1: -P --check-first --fast-list --one-way --log-file=rclone_check.txt --log-level DEBUG
Dedupe result (--dry-run)
rclone dedupe Temporary: --dry-run -P --check-first --fast-list --log-file=rclone_dedupe.txt --log-level DEBUG