UPDATE: Two hours later, I suspect this is not a Google Drive server side issue but possibly a macOS/USB issue.
I have stopped the rclone copy and will report back after further experiments.
What is the problem you are having with rclone?
For the past few years I have been able to move disk image files to Google Drive as a single copy as long as file was less than 5tb. Today, for the first time, this seems to be hanging. Note, I have not rerun the command with -vv due to my being 26 hours into the copy. A few years back I had experimented with various ways of copying large files. I arrived at single file disk images being the most efficient. Note, these are never incremental backups. They are an archival snapshot. My past experience was that a single file copy would always complete regardless of exceeding quota during the operation. In the past I had noticed some throttling, that is lower MiB copy rates once above the 750gb daily limit. However, I had never experienced the copy rate going to zero. I'm guessing that this is most likely a change with Google's management of the api.
The command is being run as a user of it's own from which there is no other Google Drive requests and with its own api key so no other activity in the account should be the cause of the issue.
My rclone copy command was not run with -vv nor creating a log file. I will include in future attempts.
UPDATE: after stopping process, I realised I should have run another rclone command with same User and API so as to see if it generated any error messages.
(1) Any knowledge as to Google restrictions which might be causing this behaviour? Has there been a change?
(2) Any view as to whether rclone copy will complete gracefully, or should I cancel and rerun with better logging?
(3) Are there any "rc" commands which might help diagnose the issue?
(4) Any suggestions?
What is your rclone version (output from rclone version)
> rclone rc --rc-addr=:52222 core/version 09:48:10
{
"arch": "amd64",
"decomposed": [
1,
57,
0
],
"goTags": "cmount",
"goVersion": "go1.17.2",
"isBeta": false,
"isGit": false,
"linking": "dynamic",
"os": "darwin",
"version": "v1.57.0"
}
Which cloud storage system are you using? (eg Google Drive)
Google Drive via Workspace
The command you were trying to run (eg rclone copy /tmp remote:tmp)
rclone copy -P --rc --rc-addr :52222 --buffer-size 4G --drive-chunk-size 1G /Volumes/Raid0/temp/UserHDasof20220228.sparseimage cryptb:_backups/from_UserHD_20220228/```
The rclone config contents with secrets removed.
[_bulkload_]
[gd_hal]
type = drive
client_id = 276[intentionallly obfuscated]846rk35.apps.googleusercontent.com
client_secret = [intentionallly obfuscated]
scope = drive
root_folder_id = 1[intentionallly obfuscated]
token = {"access_token":"[intentionallly obfuscated]1tuTHRnHITf0MOUiJdg2TI-TV3Eh[intentionallly obfuscated]N0mvmSFswOy5PPlK3RTy9ybzP8hrHf-[intentionallly obfuscated]eRG-UQX802d2lS_WEsI0-L93ufhrUY","token_type":"Bearer","refresh_token":"1//03w_2Bn5[intentionallly obfuscated]wF-L9IriP2n[intentionallly obfuscated]URKE0NJ38jk","expiry":"2022-03-23T02:59:45.897636Z"}
[cryptb]
type = crypt
remote = gd_hal:cryptb
filename_encryption = standard
directory_name_encryption = true
password = [intentionallly obfuscated]
password2 = [intentionallly obfuscated]
A log from the command with the -vv flag
The log is not available without restarting, which I'm hoping to avoid since I'm 28 hours in on the copy.
Here's what I have from rc core/stats:
> rclone rc --rc-addr=:52222 core/stats 09:37:03
{
"bytes": 2694752454928,
"checks": 0,
"deletedDirs": 0,
"deletes": 0,
"elapsedTime": 86826.551987478,
"errors": 0,
"eta": 0,
"fatalError": false,
"renames": 0,
"retryError": false,
"speed": 4e-323,
"totalBytes": 3240720547840,
"totalChecks": 0,
"totalTransfers": 1,
"transferTime": 0,
"transferring": [
{
"bytes": 2694752454928,
"eta": 0,
"group": "global_stats",
"name": "UserHDasof20220228.sparseimage",
"percentage": 83,
"size": 3240720547840,
"speed": 31036848.68616719,
"speedAvg": 4e-323
}
],
"transfers": 0
}
And no change two hours later:
> rclone rc --rc-addr=:52222 core/stats 11:52:00
{
"bytes": 2694752454928,
"checks": 0,
"deletedDirs": 0,
"deletes": 0,
"elapsedTime": 94263.088388973,
"errors": 0,
"eta": 0,
"fatalError": false,
"renames": 0,
"retryError": false,
"speed": 4e-323,
"totalBytes": 3240720547840,
"totalChecks": 0,
"totalTransfers": 1,
"transferTime": 0,
"transferring": [
{
"bytes": 2694752454928,
"eta": 0,
"group": "global_stats",
"name": "UserHDasof20220228.sparseimage",
"percentage": 83,
"size": 3240720547840,
"speed": 28588253.063853227,
"speedAvg": 4e-323
}
],
"transfers": 0
}
Here's the terminal output from the rclone copy command:
Here's the most recent command output progress:
| 31 => rclone copy -P --rc --rc-addr :52222 --buffer-size 4G --drive-chunk-size 1G /Volumes/Raid0/temp/UserHDasof20220228.sparseimage cryptb:_backups/from_UserHD_20220228/
2022/03/22 09:41:01 NOTICE: Serving remote control on http://[::]:52222/
Transferred: 2.451 TiB / 2.947 TiB, 83%, 0 B/s, ETA 0s
Transferred: 0 / 1, 0%
Elapsed time: 25h15m54.5s
Transferring:
* UserHDasof20220228.sparseimage: 83% /2.947Ti, 0/s, 0s
And some earlier snapshots of progress of command:
| 31 => rclone copy -P --rc --rc-addr :52222 --buffer-size 4G --drive-chunk-size 1G /Volumes/Raid0/test/UserHDasof20220228.sparseimage cryptb:_backups/from_UserHD_20220228/
2022/03/22 09:41:01 NOTICE: Serving remote control on http://[::]:52222/
Transferred: 37.462 GiB / 2.947 TiB, 1%, 51.606 MiB/s, ETA 16h25m44s
Transferred: 0 / 1, 0%
Elapsed time: 13m26.0s
Transferring:
* UserHDasof20220228.sparseimage: 1% /2.947Ti, 51.607Mi/s, 16h25m44s
| 31 => rclone copy -P --rc --rc-addr :52222 --buffer-size 4G --drive-chunk-size 1G /Volumes/Raid0/test/UserHDasof20220228.sparseimage cryptb:_backups/from_UserHD_20220228/
2022/03/22 09:41:01 NOTICE: Serving remote control on http://[::]:52222/
Transferred: 529.510 GiB / 2.947 TiB, 18%, 47.799 MiB/s, ETA 14h48m34s
Transferred: 0 / 1, 0%
Elapsed time: 3h12m41.0s
Transferring:
* UserHDasof20220228.sparseimage: 17% /2.947Ti, 47.799Mi/s, 14h48m34s
| 31 => rclone copy -P --rc --rc-addr :52222 --buffer-size 4G --drive-chunk-size 1G /Volumes/Raid0/test/UserHDasof20220228.sparseimage cryptb:_backups/from_UserHD_20220228/
2022/03/22 09:41:01 NOTICE: Serving remote control on http://[::]:52222/
Transferred: 753.339 GiB / 2.947 TiB, 25%, 43.709 MiB/s, ETA 14h44m18s
Transferred: 0 / 1, 0%
Elapsed time: 4h39m11.5s
Transferring:
* UserHDasof20220228.sparseimage: 24% /2.947Ti, 43.709Mi/s, 14h44m18s
/Volumes/Raid0/temp/UserHDasof20220228.sparseimage cryptb:_backups/from_UserHD_20220228/
2022/03/22 09:41:01 NOTICE: Serving remote control on http://[::]:52222/
Transferred: 1.444 TiB / 2.947 TiB, 49%, 41.523 MiB/s, ETA 10h32m39s
Transferred: 0 / 1, 0%
Elapsed time: 9h12m13.0s
Transferring:
* UserHDasof20220228.sparseimage: 49% /2.947Ti, 41.524Mi/s, 10h32m39s
| 31 => rclone copy -P --rc --rc-addr :52222 --buffer-size 4G --drive-chunk-size 1G /Volumes/Raid0/test/UserHDasof20220228.sparseimage cryptb:_backups/from_UserHD_20220228/
2022/03/22 09:41:01 NOTICE: Serving remote control on http://[::]:52222/
Transferred: 2.260 TiB / 2.947 TiB, 77%, 43.357 MiB/s, ETA 4h36m56s
Transferred: 0 / 1, 0%
Elapsed time: 14h58m44.0s
Transferring:
* UserHDasof20220228.sparseimage: 76% /2.947Ti, 43.357Mi/s, 4h36m56s
| 31 => rclone copy -P --rc --rc-addr :52222 --buffer-size 4G --drive-chunk-size 1G /Volumes/Raid0/test/UserHDasof20220228.sparseimage cryptb:_backups/from_UserHD_20220228/
2022/03/22 09:41:01 NOTICE: Serving remote control on http://[::]:52222/
Transferred: 2.414 TiB / 2.947 TiB, 82%, 23.934 MiB/s, ETA 6h29m14s
Transferred: 0 / 1, 0%
Elapsed time: 16h7m3.0s
Transferring:
* UserHDasof20220228.sparseimage: 81% /2.947Ti, 23.934Mi/s, 6h29m14s
