Google Drive migration failing

What is the problem you are having with rclone?

Trying to copy tens of thousands of files from one GDrive account to another. Running copy multiple times and it keeps repeating some transfers. Results of check are not consistent.

FWIW – the 'source' here is one of machines in the "Computers" tanb in Drive that was backed up from a Mac and configured via root_folder_id

E.g., here are 3 consecutive runs of the command – every time it 'finds' more files to transfer, some of which are repeats:

$ d="Documents/AJG"; rclone copy gdnyu-alexmbp:"$d" gd0-shelly:fromnyu/COMPUTERS/alexmbp/"$d" -P --checkers 32 --transfers 30 --stats-file-name-length 180 --drive-chunk-size 64M --fast-list
Transferred:   	  113.160 MiB / 113.160 MiB, 100%, 3.929 MiB/s, ETA 0s
Checks:              1569 / 1569, 100%
Transferred:          109 / 109, 100%
Elapsed time:        56.5s

$ d="Documents/AJG"; rclone copy gdnyu-alexmbp:"$d" gd0-shelly:fromnyu/COMPUTERS/alexmbp/"$d" -P --checkers 32 --transfers 30 --stats-file-name-length 180 --drive-chunk-size 64M --fast-list
Transferred:   	    1.300 GiB / 1.300 GiB, 100%, 5.576 MiB/s, ETA 0s
Checks:               578 / 578, 100%
Transferred:         1100 / 1100, 100%
Elapsed time:      3m53.5s

$ d="Documents/AJG"; rclone copy gdnyu-alexmbp:"$d" gd0-shelly:fromnyu/COMPUTERS/alexmbp/"$d" -P --checkers 32 --transfers 30 --stats-file-name-length 180 --drive-chunk-size 64M --fast-list
Transferred:   	   84.655 MiB / 84.655 MiB, 100%, 3.945 MiB/s, ETA 0s
Checks:              1570 / 1570, 100%
Transferred:          108 / 108, 100%
Elapsed time:        30.7s

Is this because of fast-list ?

Run the command 'rclone version' and share the full output of the command.

rclone v1.66.0
- os/version: ubuntu 22.04 (64 bit)
- os/kernel: 6.5.0-1019-oracle (aarch64)
- os/type: linux
- os/arch: arm64 (ARMv8 compatible)
- go/version: go1.22.1
- go/linking: static
- go/tags: none

Which cloud storage system are you using? (eg Google Drive)

Google Drive

The command you were trying to run (eg rclone copy /tmp remote:tmp)

d="Documents/AJG"; rclone copy gdnyu-alexmbp:"$d" gd0-shelly:fromnyu/COMPUTERS/alexmbp/"$d" -P --checkers 32 --transfers 30 --stats-file-name-length 180 --drive-chunk-size 64M --fast-list

Please run 'rclone config redacted' and share the full output. If you get command not found, please make sure to update rclone.

[gd0-shelly]
type = drive
client_id = XXX
client_secret = XXX
scope = drive
token = XXX
team_drive = XXX
root_folder_id =

[gdnyu-alex]
type = drive
client_id = XXX
client_secret = XXX
scope = drive
token = XXX
root_folder_id = XXX

[gdnyu-alexmbp]
type = drive
client_id = XXX
client_secret = XXX
scope = drive
token = XXX
root_folder_id = XXX

please post a debug log that shows those issues.

if you think that might the issue, then run the commands without --fast-list and see what happens.

might use --server-side-across-configs for server-side transfers

also, small issue, there is no root_folder_id for gd0-shelly:
tho that should not matter unless you need it.

1 Like

I tried without fast-list and now things are consistent.

I tried that earlier and every transfer was failing.

Seems that fast-list is the problem.

Log is massive, but here are some highlights

2024/03/21 15:19:57 DEBUG : rclone: Version "v1.66.0" starting with parameters ["rclone" "copy" "gdnyu-alexmbp:Documents/AJG" "gd0-shelly:fromnyu/COMPUTERS/alexmbp/Documents/AJG" "-P" "--checkers" "32" "--transfers" "30" "--stats-file-name-length" "180" "--drive-chunk-size" "64M" "--fast-list" "-vv" "--log-file=rclonelog.txt"]
2024/03/21 15:19:57 DEBUG : Creating backend with remote "gdnyu-alexmbp:Documents/AJG"
2024/03/21 15:19:57 DEBUG : Using config file from "/home/ubuntu/.config/rclone/rclone.conf"
2024/03/21 15:19:57 DEBUG : gdnyu-alexmbp: detected overridden config - adding "{84KcY}" suffix to name
2024/03/21 15:19:58 DEBUG : fs cache: renaming cache item "gdnyu-alexmbp:Documents/AJG" to be canonical "gdnyu-alexmbp{84KcY}:Documents/AJG"
2024/03/21 15:19:58 DEBUG : Creating backend with remote "gd0-shelly:fromnyu/COMPUTERS/alexmbp/Documents/AJG"
2024/03/21 15:19:58 DEBUG : gd0-shelly: detected overridden config - adding "{84KcY}" suffix to name
2024/03/21 15:19:59 DEBUG : fs cache: renaming cache item "gd0-shelly:fromnyu/COMPUTERS/alexmbp/Documents/AJG" to be canonical "gd0-shelly{84KcY}:fromnyu/COMPUTERS/alexmbp/Documents/AJG"
2024/03/21 15:20:00 DEBUG : Google drive root 'fromnyu/COMPUTERS/alexmbp/Documents/AJG': Disabling ListR to work around bug in drive as multi listing (3) returned no entries
2024/03/21 15:20:00 DEBUG : Google drive root 'fromnyu/COMPUTERS/alexmbp/Documents/AJG': Recycled 3 entries
2024/03/21 15:20:00 DEBUG : Google drive root 'fromnyu/COMPUTERS/alexmbp/Documents/AJG': Recycled 2 entries

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.