How to Resume Copy Process

What is the problem you are having with rclone?
Hi, I have a problem about how to resume the copy process and make the rest of files stay in the original folder.
Here is what I've done it before: There is 20TBs folders and files in "Shared-with-Me" and I would like to copy in My Drive in same Google Drive account with Server-Side-Copy way.**

Because the Google Drive API Transfer Limited (750GB of 24Hours), I had to make the Copy Process Divided into few days

Day One: I run the command:"rclone copy --drive-shared-with-me drive1: drive1: --ignore-existing". When Day Two, I try to resume copy process, used the same command of yesterday, thought the Rclone will auto ignore exist file and continue copy the rest of file.
But the result was: Rclone auto make a new folder name (same as yesterday) and copy 750GB file (same as yesterday),so now I have two same copy folder and files. :joy::joy::joy::joy:
Now here is my question:
How can I set the Rclone that will not create a new same name folder and ingore exist file, continue copy the rest of file when i try to resume copy process in the second day.
Or Should I use "sync" command in the beginning not copy command? Tried "copyto", same issue.

Hope you can help me solve this problem.

What is your rclone version (output from rclone version)
rclone v1.51.0

Which OS you are using and how many bits (eg Windows 7, 64 bit)
Windows 10 64bit

Which cloud storage system are you using? (eg Google Drive)
Google Drive

It seems that this is known issue - --drive-shared-with-me problem

Copy will ignore existing (identical copies) by default, something is wrong then.

The solution is here: --drive-shared-with-me problem

Rclone will normally handle this with completely default settings (no need to use --ignore existing as rclone will just skip existing files if they are unchanged by default).

But there may be a bug or limitation in effect here as you have linked...

Aside from that what was suggested in that thread about transferring to a different folder each time (with seems like a very inconvenient way), I have a couple of suggestions for how to work around this.

Firstly - if you are a Gsuite user (and you probably are if you are transfering more than 750GB at a time) then it is much much better to use teamdrive/shared drives for this kind of cross-account transfers. A unlimited Gsuite user can just spawn these at-will. Teamdrives are much easier to manage when it comes to multi-user access and don't require any janky workarounds like you may have to deal with using the standard "share" function on a personal drive. There is also very little you can't do on a teamdrive that you could on a personal.

Secondly - if you don't have this option then assuming you control both accounts or can request this from the other user - try setting up a second "reversed" share. For example, if the current location is a share from A to B, try having B share one with A. For transfers that go in the same direction as your test I suspect this is not going to have the limitation. (of course you may face the same issue on the new share if transferring the other way, but that's why you probably would want to keep both shares depending on the way the data is going).

I have not tested the last tip here - but from my understanding of how the Google Drive system works I think it would have a fair change of working better.

1 Like

Tq for replying.

But I really cannot get what u try to say.

I'm not a G-Suite user. This is EDU account and I'm a student.

I will see if I can run a test for you between 2 personal accounts - and suggest a good workaround for you. I am pretty sure I have had this sort of setup working well before, but normally I use only teamdrives so I have to refresh my memory.

Maybe tomorrow - but if I you don't hear from me in a few days then please remind me @thestigma :slight_smile:

1 Like

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.