Union (Google Drive) not working correctly

What is the problem you are having with rclone?

I started copying 60gb files to Union consisting of 100 google drives (1.45tb), all these drives are service accounts. After filling up the first disk (14.9 GB), I started getting errors about exceeding the quota. As far as I understand, the used account for uploading does not switch to the next, empty disk. Individually, these disks fill up successfully.

Run the command 'rclone version' and share the full output of the command.

rclone v1.61.1
- os/version: Microsoft Windows 10 Pro 22H2 (64 bit)
- os/kernel: 10.0.19045.2364 (x86_64)
- os/type: windows
- os/arch: amd64
- go/version: go1.19.4
- go/linking: static
- go/tags: cmount

Which cloud storage system are you using? (eg Google Drive)

Google Drive service accounts (100 pcs)

The command you were trying to run (eg rclone copy /tmp remote:tmp)

I'm on Windows and just copied and pasted folders and files totaling 60gb.

The rclone config contents with secrets removed.

[google_1]
type = drive
scope = drive
service_account_file = C:\RClone\accounts\f881cf5a57492e9d9be45817c63e2ec218cb1638.json

*** 2 to 99 ***

[google_100]
type = drive
scope = drive
service_account_file = C:\RClone\accounts\fbe989c22978a07de81a572bec98d1768a42ea72.json

[google_unlimited]
type = union
upstreams = google_1:/ google_2:/ google_3:/ google_4:/ google_5:/ google_6:/ google_7:/ google_8:/ google_9:/ google_10:/ google_11:/ google_12:/ google_13:/ google_14:/ google_15:/ google_16:/ google_17:/ google_18:/ google_19:/ google_20:/ google_21:/ google_22:/ google_23:/ google_24:/ google_25:/ google_26:/ google_27:/ google_28:/ google_29:/ google_30:/ google_31:/ google_32:/ google_33:/ google_34:/ google_35:/ google_36:/ google_37:/ google_38:/ google_39:/ google_40:/ google_41:/ google_42:/ google_43:/ google_44:/ google_45:/ google_46:/ google_47:/ google_48:/ google_49:/ google_50:/ google_51:/ google_52:/ google_53:/ google_54:/ google_55:/ google_56:/ google_57:/ google_58:/ google_59:/ google_60:/ google_61:/ google_62:/ google_63:/ google_64:/ google_65:/ google_66:/ google_67:/ google_68:/ google_69:/ google_70:/ google_71:/ google_72:/ google_73:/ google_74:/ google_75:/ google_76:/ google_77:/ google_78:/ google_79:/ google_80:/ google_81:/ google_82:/ google_83:/ google_84:/ google_85:/ google_86:/ google_87:/ google_88:/ google_89:/ google_90:/ google_91:/ google_92:/ google_93:/ google_94:/ google_95:/ google_96:/ google_97:/ google_98:/ google_99:/ google_100:/

A log from the command with the -vv flag

I did it only once and I see no reason to repeat it. Here is my log entry without --vv. It shows how I get the error:

WriteFileHandle: Truncate: Can't change size without --vfs-cache-mode >= writes"

for each file transferred (I think this is normal), and then for the first time I see an error:

WriteFileHandle.Flush error: googleapi: Error 403: The user's Drive storage quota has been exceeded., storageQuotaExceeded
2023/01/06 16:25:31 ERROR : IO error: googleapi: Error 403: The user's Drive storage quota has been exceeded., storageQuotaExceeded

Further this error repeats for each file which I tried to copy. The first disk was successfully filled to 99%. However, the file explorer kept showing me new files in the correct size folder, so I didn't realize right away that these files were just empty files that couldn't be opened. Then I restarted rclone and saw that these files were gone, leaving only empty folders.

2023/01/06 16:24:48 ERROR : Unreal Engine 5 C++ The Ultimate Game Developer Course/unreal-engine-5-the-ultimate-game-developer-course/13. Weapon Mechanics/8. Root Motion Animations.mp4: WriteFileHandle: Truncate: Can't change size without --vfs-cache-mode >= writes
2023/01/06 16:25:29 ERROR : Unreal Engine 5 C++ The Ultimate Game Developer Course/unreal-engine-5-the-ultimate-game-developer-course/13. Weapon Mechanics/8. Root Motion Animations.srt: WriteFileHandle: Truncate: Can't change size without --vfs-cache-mode >= writes
2023/01/06 16:25:29 ERROR : Unreal Engine 5 C++ The Ultimate Game Developer Course/unreal-engine-5-the-ultimate-game-developer-course/13. Weapon Mechanics/8.1 Discord for Students.html: WriteFileHandle: Truncate: Can't change size without --vfs-cache-mode >= writes
2023/01/06 16:25:29 ERROR : Unreal Engine 5 C++ The Ultimate Game Developer Course/unreal-engine-5-the-ultimate-game-developer-course/13. Weapon Mechanics/8.2 Mixamo.html: WriteFileHandle: Truncate: Can't change size without --vfs-cache-mode >= writes
2023/01/06 16:25:29 ERROR : Unreal Engine 5 C++ The Ultimate Game Developer Course/unreal-engine-5-the-ultimate-game-developer-course/13. Weapon Mechanics/8.3 Mixamo Addon for Blender 2.78+.html: WriteFileHandle: Truncate: Can't change size without --vfs-cache-mode >= writes
2023/01/06 16:25:29 ERROR : Unreal Engine 5 C++ The Ultimate Game Developer Course/unreal-engine-5-the-ultimate-game-developer-course/13. Weapon Mechanics/8.4 Mixamo Addon for Blender 3.0+.html: WriteFileHandle: Truncate: Can't change size without --vfs-cache-mode >= writes
2023/01/06 16:25:29 ERROR : Unreal Engine 5 C++ The Ultimate Game Developer Course/unreal-engine-5-the-ultimate-game-developer-course/13. Weapon Mechanics/9. Implementing Interfaces.mp4: WriteFileHandle: Truncate: Can't change size without --vfs-cache-mode >= writes
2023/01/06 16:25:31 ERROR : Unreal Engine 5 C++ The Ultimate Game Developer Course/unreal-engine-5-the-ultimate-game-developer-course/13. Weapon Mechanics/8. Root Motion Animations.mp4: WriteFileHandle.New Rcat failed: googleapi: Error 403: The user's Drive storage quota has been exceeded., storageQuotaExceeded
2023/01/06 16:25:31 ERROR : Unreal Engine 5 C++ The Ultimate Game Developer Course/unreal-engine-5-the-ultimate-game-developer-course/13. Weapon Mechanics/8. Root Motion Animations.mp4: WriteFileHandle.Flush error: googleapi: Error 403: The user's Drive storage quota has been exceeded., storageQuotaExceeded
2023/01/06 16:25:31 ERROR : IO error: googleapi: Error 403: The user's Drive storage quota has been exceeded., storageQuotaExceeded

I forgot to paste the command:

./rclone mount google_unlimited:/ X: --network-mode

Hi AcTiViSioN,

Looks like you may be violating the Google terms of usage, how do you pay Google for the 1.45TB?

1 Like

Through service accounts, I pay them exactly $0 because they allow it (for non-commercial purposes). Again, my question was that Union is not working as I expected. Apparently, the functionality is designed to read files, not write, which disappointed me and I implemented my project this week (just plain C# api v3), which suits me now (files are spread across all drives like butter on bread for instant multi-upload/download). I just wanted to leave this bug report here, because I think that the functionality could be better, so good luck with your project :slight_smile: