Bulk moving from Google Drive to Box

Help and Support
1

What is the problem you are having with rclone?

I'm trying to move about 1.5 TB of data (compressed trained machine learning models) from Google Drive to Box, both are school accounts but it seems like the each file is too big so uploading is time out.

Run the command 'rclone version' and share the full output of the command.

rclone v1.60.1
- os/version: Microsoft Windows 10 Pro 21H2 (64 bit)
- os/kernel: 10.0.19044.2251 (x86_64)
- os/type: windows
- os/arch: amd64
- go/version: go1.19.3
- go/linking: static
- go/tags: cmount

Which cloud storage system are you using? (eg Google Drive)

From Google Drive to Box

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone.exe move drive_to_box:white-box-mi boxremote:white-box-mi  -vv

The rclone config contents with secrets removed.

[drive_to_box]
type = drive
client_id = 
client_secret =
scope = drive
token =
team_drive = 

[boxremote]
type = box
box_sub_type = enterprise
token =

A log from the command with the -vv flag

2022/11/24 17:20:25 DEBUG : rclone: Version "v1.60.1" starting with parameters ["rclone_file_path\\rclone.exe" "move" "drive_to_box:white-box-mi" "boxremote:white-box-mi" "-vv"]
2022/11/24 17:20:25 DEBUG : Creating backend with remote "drive_to_box:white-box-mi"
2022/11/24 17:20:25 DEBUG : Using config file from "[local_config_path]"
2022/11/24 17:20:25 DEBUG : Google drive root 'white-box-mi': 'root_folder_id = 0AGEP-Ruy1a71Uk9PVA' - save this in the config to speed up startup
2022/11/24 17:20:25 DEBUG : Creating backend with remote "boxremote:white-box-mi"
2022/11/24 17:20:27 DEBUG : box root 'white-box-mi': Waiting for checks to finish
2022/11/24 17:20:27 DEBUG : box root 'white-box-mi': Waiting for transfers to finish
2022/11/24 17:20:27 DEBUG : supervised_model_resnet50_data_cifar10_test.npz: Multipart upload session started for 634 parts of size 64Mi
2022/11/24 17:20:28 DEBUG : supervised_model_resnet50_data_cifar10_train_0.npz: Multipart upload session started for 634 parts of size 64Mi
2022/11/24 17:20:28 DEBUG : supervised_model_resnet50_data_cifar10_train_1.npz: Multipart upload session started for 634 parts of size 64Mi
2022/11/24 17:20:28 DEBUG : subset_train.pt: Multipart upload session started for 21 parts of size 16Mi
2022/11/24 17:20:28 DEBUG : subset_train.pt: Uploading part 1/21 offset 0/327.500Mi part size 16Mi
2022/11/24 17:20:28 DEBUG : subset_train.pt: Uploading part 2/21 offset 16Mi/327.500Mi part size 16Mi
2022/11/24 17:20:28 DEBUG : subset_train.pt: Uploading part 3/21 offset 32Mi/327.500Mi part size 16Mi
2022/11/24 17:20:29 DEBUG : subset_train.pt: Uploading part 4/21 offset 48Mi/327.500Mi part size 16Mi
2022/11/24 17:21:16 DEBUG : subset_train.pt: Uploading part 5/21 offset 64Mi/327.500Mi part size 16Mi
2022/11/24 17:21:16 DEBUG : supervised_model_resnet50_data_cifar10_train_0.npz: Uploading part 1/634 offset 0/39.582Gi part size 64Mi
2022/11/24 17:21:17 DEBUG : supervised_model_resnet50_data_cifar10_test.npz: Uploading part 1/634 offset 0/39.582Gi part size 64Mi
2022/11/24 17:21:17 DEBUG : supervised_model_resnet50_data_cifar10_train_1.npz: Uploading part 1/634 offset 0/39.582Gi part size 64Mi
2022/11/24 17:21:26 INFO  :
Transferred:       76.156 MiB / 1.465 TiB, 0%, 1.307 MiB/s, ETA 1w6d14h32m10s
Checks:                 0 / 4, 0%
Transferred:            0 / 140, 0%
Elapsed time:       1m0.9s
Checking:

Transferring:
 *                               subset_train.pt: 20% /327.500Mi, 927.817Ki/s, 4m46s
 * supervised_model_resne…_data_cifar10_test.npz:  0% /39.582Gi, 139.453Ki/s, 82h40m4s
 * supervised_model_resne…ta_cifar10_train_0.npz:  0% /39.582Gi, 146.069Ki/s, 78h55m22s
 * supervised_model_resne…ta_cifar10_train_1.npz:  0% /39.582Gi, 129.131Ki/s, 89h16m36s

2022/11/24 17:21:56 DEBUG : subset_train.pt: Uploading part 6/21 offset 80Mi/327.500Mi part size 16Mi
2022/11/24 17:21:56 DEBUG : supervised_model_resnet50_data_cifar10_train_0.npz: Uploading part 2/634 offset 64Mi/39.582Gi part size 64Mi
2022/11/24 17:21:56 DEBUG : supervised_model_resnet50_data_cifar10_train_1.npz: Uploading part 2/634 offset 64Mi/39.582Gi part size 64Mi
2022/11/24 17:21:56 DEBUG : pacer: low level retry 1/10 (error Put "https://upload.box.com/api/2.0/files/upload_sessions/FA1673C2DC49A304C4120E9F44FCE2C2": write tcp 192.168.1.6:59912->74.112.186.128:443: wsasend: An existing connection was forcibly closed by the remote host.)
2022/11/24 17:21:56 DEBUG : pacer: Rate limited, increasing sleep to 20ms
2022/11/24 17:22:26 INFO  :
Transferred:      130.797 MiB / 1.465 TiB, 0%, 1.197 MiB/s, ETA 2w20h28m3s
Checks:                 0 / 4, 0%
Transferred:            0 / 140, 0%
Elapsed time:       2m0.9s
Checking:

Transferring:
 *                               subset_train.pt: 24% /327.500Mi, 322.579Ki/s, 13m1s
 * supervised_model_resne…_data_cifar10_test.npz:  0% /39.582Gi, 11.817Ki/s, 975h25m27s
 * supervised_model_resne…ta_cifar10_train_0.npz:  0% /39.582Gi, 299.665Ki/s, 38h27m27s
 * supervised_model_resne…ta_cifar10_train_1.npz:  0% /39.582Gi, 593.387Ki/s, 19h25m1s

2022/11/24 17:32:06 INFO  :
Transferred:      300.500 MiB / 1.465 TiB, 0%, 0 B/s, ETA 5y37w3d10h16m38s77ms
Checks:                 0 / 4, 0%
Transferred:            0 / 140, 0%
Elapsed time:     11m41.8s
Checking:

Transferring:
 *                               subset_train.pt: 26% /327.500Mi, 0/s, 1712677h57m43.492460544s
 * supervised_model_resne…_data_cifar10_test.npz:  0% /39.582Gi, 0/s, 0s
 * supervised_model_resne…ta_cifar10_train_0.npz:  0% /39.582Gi, 0/s, 1556369h32m55.356645376s
 * supervised_model_resne…ta_cifar10_train_1.npz:  0% /39.582Gi, 0/s, 473289h17m32.896616448s

2022/11/24 17:22:45 DEBUG : pacer: Reducing sleep to 15ms
2022/11/24 17:32:06 DEBUG : pacer: Reducing sleep to 11.25ms
2022/11/24 17:32:06 DEBUG : supervised_model_resnet50_data_cifar10_test.npz: Uploading part 2/634 offset 64Mi/39.582Gi part size 64Mi
2022/11/24 17:32:06 DEBUG : supervised_model_resnet50_data_cifar10_train_1.npz: Uploading part 3/634 offset 128Mi/39.582Gi part size 64Mi
2022/11/24 17:32:06 DEBUG : subset_train.pt: Cancelling multipart upload: multipart upload failed to upload part: Put "https://upload.box.com/api/2.0/files/upload_sessions/DAC05CF15D9C2DA52B48598AACCCD29E": http2: client connection force closed via ClientConn.Close
2022/11/24 17:32:06 DEBUG : supervised_model_resnet50_data_cifar10_train_0.npz: Cancelling multipart upload: multipart upload failed to upload part: Put "https://upload.box.com/api/2.0/files/upload_sessions/8353941C601AFE79B5379230477BDBA6": http2: client connection force closed via ClientConn.Close
2022/11/24 17:32:06 DEBUG : pacer: Reducing sleep to 10ms
2022/11/24 17:32:07 DEBUG : supervised_model_resnet50_data_cifar10_train_1.npz: Reopening on read failure after 218102678 bytes: retry 1/10: read tcp 192.168.1.6:59910->142.251.215.234:443: i/o timeout
2022/11/24 17:32:07 ERROR : subset_train.pt: Failed to copy: multipart upload failed to upload part: Put "https://upload.box.com/api/2.0/files/upload_sessions/DAC05CF15D9C2DA52B48598AACCCD29E": http2: client connection force closed via ClientConn.Close
2022/11/24 17:32:07 ERROR : subset_train.pt: Not deleting source as copy failed: multipart upload failed to upload part: Put "https://upload.box.com/api/2.0/files/upload_sessions/DAC05CF15D9C2DA52B48598AACCCD29E": http2: client connection force closed via ClientConn.Close
2022/11/24 17:32:07 ERROR : supervised_model_resnet50_data_cifar10_train_0.npz: Failed to copy: multipart upload failed to upload part: Put "https://upload.box.com/api/2.0/files/upload_sessions/8353941C601AFE79B5379230477BDBA6": http2: client connection force closed via ClientConn.Close
2022/11/24 17:32:07 ERROR : supervised_model_resnet50_data_cifar10_train_0.npz: Not deleting source as copy failed: multipart upload failed to upload part: Put "https://upload.box.com/api/2.0/files/upload_sessions/8353941C601AFE79B5379230477BDBA6": http2: client connection force closed via ClientConn.Close
2022/11/24 17:32:09 DEBUG : supervised_model_resnet50_data_cifar10_train_2.npz: Multipart upload session started for 634 parts of size 64Mi
2022/11/24 17:32:10 DEBUG : supervised_model_resnet50_data_cifar10_train_3.npz: Multipart upload session started for 634 parts of size 64Mi
2022/11/24 17:32:21 DEBUG : supervised_model_resnet50_data_cifar10_train_2.npz: Uploading part 1/634 offset 0/39.582Gi part size 64Mi
2022/11/24 17:32:24 DEBUG : supervised_model_resnet50_data_cifar10_train_1.npz: Uploading part 4/634 offset 192Mi/39.582Gi part size 64Mi
2022/11/24 17:32:26 INFO  :
Transferred:      318.953 MiB / 1.426 TiB, 0%, 748.773 KiB/s, ETA 3w2d15h57m34s
Errors:                 2 (retrying may help)
Checks:                 2 / 6, 33%
Transferred:            0 / 138, 0%
Elapsed time:      12m0.9s
2

That seems to be the underlying error.

Not sure what that means!

Try

  --disable-http2   Disable HTTP/2 in the global transport

And see if that helps.

3

Hi Nick,

Thank you for your help! From my understanding, it looks like the file is too big, so the box servers that I'm transfering the file to time out.

Here's the output from .\rclone.exe move drive_to_box:white-box-mi boxremote:white-box-mi -vv --disable-http2

2022/11/26 21:52:39 DEBUG : rclone: Version "v1.60.1" starting with parameters ["rclone path"]
2022/11/26 21:52:39 DEBUG : Creating backend with remote "drive_to_box:white-box-mi"
2022/11/26 21:52:39 DEBUG : Using config file from "rclone config path"
2022/11/26 21:52:40 DEBUG : Google drive root 'white-box-mi': 'root_folder_id = 0AGEP-Ruy1a71Uk9PVA' - save this in the config to speed up startup
2022/11/26 21:52:40 DEBUG : Creating backend with remote "boxremote:white-box-mi"
2022/11/26 21:52:42 DEBUG : box root 'white-box-mi': Waiting for checks to finish
2022/11/26 21:52:42 DEBUG : box root 'white-box-mi': Waiting for transfers to finish
2022/11/26 21:52:42 DEBUG : supervised_model_resnet50_data_cifar10_train_0.npz: Multipart upload session started for 634 parts of size 64Mi
2022/11/26 21:52:42 DEBUG : supervised_model_resnet50_data_cifar10_train_1.npz: Multipart upload session started for 634 parts of size 64Mi
2022/11/26 21:52:42 DEBUG : supervised_model_resnet50_data_cifar10_test.npz: Multipart upload session started for 634 parts of size 64Mi
2022/11/26 21:52:43 DEBUG : subset_train.pt: Multipart upload session started for 21 parts of size 16Mi
2022/11/26 21:52:43 DEBUG : subset_train.pt: Uploading part 1/21 offset 0/327.500Mi part size 16Mi
2022/11/26 21:52:44 DEBUG : subset_train.pt: Uploading part 2/21 offset 16Mi/327.500Mi part size 16Mi
2022/11/26 21:52:45 DEBUG : subset_train.pt: Uploading part 3/21 offset 32Mi/327.500Mi part size 16Mi
2022/11/26 21:52:46 DEBUG : subset_train.pt: Uploading part 4/21 offset 48Mi/327.500Mi part size 16Mi
2022/11/26 21:53:28 DEBUG : subset_train.pt: Uploading part 5/21 offset 64Mi/327.500Mi part size 16Mi
2022/11/26 21:53:30 DEBUG : supervised_model_resnet50_data_cifar10_train_0.npz: Uploading part 1/634 offset 0/39.582Gi part size 64Mi
2022/11/26 21:53:40 INFO  :
Transferred:       70.438 MiB / 1.465 TiB, 0%, 1.308 MiB/s, ETA 1w6d14h18m18s
Checks:                 0 / 4, 0%
Transferred:            0 / 140, 0%
Elapsed time:       1m1.0s
Checking:

Transferring:
 *                               subset_train.pt: 20% /327.500Mi, 1.152Mi/s, 3m45s
 * supervised_model_resne…_data_cifar10_test.npz:  0% /39.582Gi, 0/s, -
 * supervised_model_resne…ta_cifar10_train_0.npz:  0% /39.582Gi, 160.169Ki/s, 71h58m30s
 * supervised_model_resne…ta_cifar10_train_1.npz:  0% /39.582Gi, 0/s, -

2022/11/26 21:53:43 DEBUG : supervised_model_resnet50_data_cifar10_train_1.npz: Uploading part 1/634 offset 0/39.582Gi part size 64Mi
2022/11/26 21:53:43 DEBUG : supervised_model_resnet50_data_cifar10_test.npz: Uploading part 1/634 offset 0/39.582Gi part size 64Mi
2022/11/26 21:54:08 DEBUG : subset_train.pt: Uploading part 6/21 offset 80Mi/327.500Mi part size 16Mi
2022/11/26 21:54:21 DEBUG : pacer: low level retry 1/10 (error Put "https://upload.box.com/api/2.0/files/upload_sessions/A0ED6CA41A63C1D86ED946F0E87DDC0C": write tcp 192.168.1.6:50230->74.112.186.128:443: wsasend: An existing connection was forcibly closed by the remote host.)
2022/11/26 21:54:21 DEBUG : pacer: Rate limited, increasing sleep to 20ms
2022/11/26 21:54:21 DEBUG : pacer: low level retry 1/10 (error Put "https://upload.box.com/api/2.0/files/upload_sessions/8E19CF60BCA2C2BE867E82325E6CBD6C": write tcp 192.168.1.6:50229->74.112.186.128:443: wsasend: An existing connection was forcibly closed by the remote host.)
2022/11/26 21:54:21 DEBUG : pacer: Rate limited, increasing sleep to 40ms
2022/11/26 21:54:40 INFO  :
Transferred:      150.781 MiB / 1.465 TiB, 0%, 1.371 MiB/s, ETA 1w5d23h14m59s
Checks:                 0 / 4, 0%
Transferred:            0 / 140, 0%
Elapsed time:       2m1.0s
Checking:

Transferring:
 *                               subset_train.pt: 27% /327.500Mi, 206.918Ki/s, 19m35s
 * supervised_model_resne…_data_cifar10_test.npz:  0% /39.582Gi, 340.012Ki/s, 33h53m45s
 * supervised_model_resne…ta_cifar10_train_0.npz:  0% /39.582Gi, 278.419Ki/s, 41h23m35s
 * supervised_model_resne…ta_cifar10_train_1.npz:  0% /39.582Gi, 578.553Ki/s, 19h54m45s

2022/11/26 21:54:41 DEBUG : pacer: low level retry 1/10 (error Put "https://upload.box.com/api/2.0/files/upload_sessions/D3E276110E2D5A5395593B4177E2358E": write tcp 192.168.1.6:50228->74.112.186.128:443: wsasend: An existing connection was forcibly closed by the remote host.)
2022/11/26 21:54:41 DEBUG : pacer: Rate limited, increasing sleep to 80ms
2022/11/26 21:55:01 DEBUG : pacer: low level retry 1/10 (error Put "https://upload.box.com/api/2.0/files/upload_sessions/A0CDDB3603C7F7E6BB5AF0773A7CC4A4": write tcp 192.168.1.6:50227->74.112.186.128:443: wsasend: An existing connection was forcibly closed by the remote host.)
2022/11/26 21:55:01 DEBUG : pacer: Rate limited, increasing sleep to 160ms
2022/11/26 21:55:26 DEBUG : pacer: Reducing sleep to 120ms
2022/11/26 21:55:26 DEBUG : supervised_model_resnet50_data_cifar10_train_0.npz: Uploading part 2/634 offset 64Mi/39.582Gi part size 64Mi
2022/11/26 21:55:40 INFO  :
Transferred:      231.469 MiB / 1.465 TiB, 0%, 1.356 MiB/s, ETA 1w6d2h38m42s
Checks:                 0 / 4, 0%
Transferred:            0 / 140, 0%
Elapsed time:       3m1.0s
Checking:

Transferring:
 *                               subset_train.pt: 32% /327.500Mi, 132.146Ki/s, 28m35s
 * supervised_model_resne…_data_cifar10_test.npz:  0% /39.582Gi, 375.261Ki/s, 30h41m35s
 * supervised_model_resne…ta_cifar10_train_0.npz:  0% /39.582Gi, 560.388Ki/s, 20h33m6s
 * supervised_model_resne…ta_cifar10_train_1.npz:  0% /39.582Gi, 314.316Ki/s, 36h38m27s

2022/11/26 21:56:01 DEBUG : pacer: low level retry 2/10 (error Put "https://upload.box.com/api/2.0/files/upload_sessions/A0CDDB3603C7F7E6BB5AF0773A7CC4A4": write tcp 192.168.1.6:50290->74.112.186.128:443: wsasend: An existing connection was forcibly closed by the remote host.)
2022/11/26 21:56:01 DEBUG : pacer: Rate limited, increasing sleep to 240ms
2022/11/26 21:56:01 DEBUG : pacer: low level retry 2/10 (error Put "https://upload.box.com/api/2.0/files/upload_sessions/8E19CF60BCA2C2BE867E82325E6CBD6C": write tcp 192.168.1.6:50266->74.112.186.128:443: wsasend: An existing connection was forcibly closed by the remote host.)
2022/11/26 21:56:01 DEBUG : pacer: low level retry 2/10 (error Put "https://upload.box.com/api/2.0/files/upload_sessions/A0ED6CA41A63C1D86ED946F0E87DDC0C": write tcp 192.168.1.6:50265->74.112.186.128:443: wsasend: An existing connection was forcibly closed by the remote host.)
2022/11/26 21:56:01 DEBUG : pacer: Rate limited, increasing sleep to 480ms
2022/11/26 21:56:01 DEBUG : pacer: Rate limited, increasing sleep to 960ms

I have contacted Box to see if there's anything Box-related they can change but I'm not sure if I know how to proceed from here on my end. Many thanks!

4

Check what Box plan you got. Box has very strict filesize limits, some account types as low as 250MB (free tier).

You might be interested in using Chunker to work around the limitation.

5

Hi Jose,

Thank you! Even with smaller files, it still happens. I will look into chunkier.

6

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.