Zipping big amounts of data between Google Shared Drive tips

Help and Support
1

Hello there rclone community

I think this goes in support rather than How to guides, apologies in advance if that's not the case.

Which cloud storage system are you using? (eg Google Drive)

Exlusively Google Drive (Shared Drives)

The command you were trying to run (eg rclone copy /tmp remote:tmp)

Not relevant here

Please run 'rclone config redacted' and share the full output. If you get command not found, please make sure to update rclone.

Not relevant here

A log from the command that you were trying to run with the -vv flag

Not relevant here

Rclone version and OS:

rclone v1.74.3
- os/version: debian 13.5 (64 bit)
- os/kernel: 6.12.90+deb13.1-cloud-amd64 (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.26.4
- go/linking: static
- go/tags: none

Hey community

So I have a use case, where I need to zip certain folders on certain Google Shared Drives [ORIG], then put the resulting zip file in another destination Shared Drive [DEST].

Since there are different ways to do this with rclone, I'd like to get some feedback on what you think is the most reasonable safety/performance option. I see 3 ways:

  • Do a rclone copy from ORIG to a local disk, compress the folder locally, rclone copy the zips to DEST. Easy, but it will need lots of local disk space. Can be done in a cloud VM though.
  • Do a rclone mount of ORIG and compress the folder there, so the resulting zip file is already there. Then move it to DEST from drive web interface. Not sure if this could cause bottlenecks. It is possible also that I hit upload/download quota, and not sure if rclone mount+zip will handle this well. I know rclone copy does it well.
  • Do a rclone mount of both ORIG and DEST, an zip from ORIG with destination to DEST. This could ease up bottlenecks, but not sure if it will be appropiate.

Does anyone have some hints? It is difficult for me to do pre-benchmarks of all this, since the use cases vary greatly. Sometimes it is literally 200k files totally a dozen TBs. Sometimes it is few files but very big ones. And so on.

Many thanks in advance.

Cheers,

Marc

2

For a zip, all bytes have to be read client-side anyway, so mounting both drives does not really avoid the expensive part. I would not write the zip back into the mounted source either; with lots of small files that is usually the most fragile option. If you can, run this on a VM close to Google, write archives to local/attached disk first, then use rclone copy to send the finished zips to DEST. For dozen-TB cases I would also split per top-level folder or date range, so one failure does not mean rebuilding one huge zip.

3

Hi Lucas. Much appreciated answers. Thanks!

The idea is to use a Google Cloud VM indeed! :slight_smile:

Actually, I think I will use rar with splitting in smaller .part files. It is really not feasible for me to micromanage the archiving of all those sub-folders. This is for an organization, not for personal use.

4

If you go with rar split parts on a VM, I would still write the archive to local/attached scratch disk first, then use `rclone copy` for the finished parts to DEST. Writing archive parts directly through a mount can make retries and partial files more annoying if the mount stalls. I would also do one small folder end-to-end and test extraction before starting the large set. For Shared Drives, clear names plus a small text log of the source folder next to the archive can save a lot of guessing if one chunk has to be rerun.

5

hi,

of course, that would also apply to rclone copy.

a major possible advantage of using mount is that after each part is written to the mount, rclone can upload/move that part in real-time. so the total local storage requirements would be greatly reduced.

and if 7zip needs to access the headers/footers of the individual parts, using --vfs-cache-mode=full would greatly reduce the total local space required. tho i would treat certain --vfs flags to reduce the size of each chunk that is download from gdrive.

tho, all that needs to be tested using a debug log