Does copying locally use internet data?

Curious if rclone eats up internet data when copying locally/server side. This new ISP server caps me out at 700gb a month, whereas my old ISP was unlimited (just terrible speeds). So I never really had to pay attention to it before.

Using the commands:
"rclone copy teamgdrive:folder gdrive2:folder" --drive-server-side-across-configs"
rclone copy gdrive:folder gdrive2:folder"

These commands shouldn't use any data, correct?
Nearly 100gb was taken up yesterday, the only thing I've ran were my daily commands, so that's why I'm curious.
I guess the part I'm confused on is if it is eating up data, how it's not eating up even more.
The regular rclone copy command, I'm using on 200+ gdrive accounts with 750gb daily cloned over to each account.

Any insight would be helpful. Thanks.

When you posted, the template actually helps us answer that question as we don't know what version you are running and there are no logs files to look at.

If you are server side copying, it uses no data.
If it is not server side copying, it would use data.

My apologies. I assumed basic commands would be enough to determine if data is being used. I will certainly remember that next time though.

I'm using the latest beta.
To break it down a bit more, and hopefully this makes sense:

Rclone copy from a team drive to my main Gsuite:

"rclone copy - Pv teamgdrive:folder maingsuite:folder" --drive-server-side-across-configs --checksum"

Afterwards, I add the folder that was cloned to my main Gsuite to each of the accounts, with the Add folder to Drive option. Then clone from there.

"rclone copy -Pv gdrive1:folder gdrive1:folder2"
"rclone copy -Pv gdrive2:folder gdrive2:folder2"
"rclone copy -Pv gdrive3:folder gdrive3:folder2"

There are many, many betas and it's more helpful to use the template and just type rclone version and share the output.

Instead run that with -vv and share the log and we can help answer your question.

as an example of why it's needed

imagine your remote is a crypt on a team drive. any copy/move operation off of a crypt is going to use data (even from one crypt to another where they use the same keys). At least that has been my experience (and fundamentally makes sense due to the design of rclone)

now this might not be your case, but no one can tell, as have no idea what your config is. there could be other things in your config that also get in the way

@ncw Is it possible to make rclone check and see if the encryption keys are identical on i.e two crypt remotes, and do the transfer serverside (i.e that you can issue a command using cleartext paths, but rclone is smart enough to "translate back" to encrypted and move)?

that is a feature i have dreamed about.

to have two mounts, each a different encrypted remote, using same crypt settings
and then use a local file manger to move around the files - server-side.

Actually you can use --crypt-server-side-across-configs as well as --drive-server-side-across-configs and that should do what you wnat...

I think that is only in the beta

1 Like

--crypt-server-side-across-configs <- New? :+1:

Curious: Is that looking at the underlying remote for the crypt, then copying the underlying encrypted files? Have done this manually a few times, but this would be much cleaner.

hi @ncw , i am trying to test the beta flag --crypt-server-side-across-configs but my command fails.

here are three commands, the first and second work, the third does not.
the only difference between the second and the third, is the addition of --crypt-server-side-across-configs

  1. rclone-1.52.3.exe copy gcrypt-a1b2-01: gcrypt-jojo-01: --log-file=.\\log.cryptserverside.txt --log-level=DEBUG --drive-server-side-across-configs --progress

  2. rclone-1.52.3-326-g45afe97e-beta.exe copy gcrypt-a1b2-01: gcrypt-jojo-01: --log-file=.\\log.cryptserverside.txt --log-level=DEBUG --drive-server-side-across-configs --progress

  3. rclone-1.52.3-326-g45afe97e-beta.exe copy gcrypt-a1b2-01: gcrypt-jojo-01: --log-file=.\\log.cryptserverside.txt --log-level=DEBUG --crypt-server-side-across-configs --drive-server-side-across-configs

and here is the log for the third command. lightly edited.

2020/08/23 13:04:59 DEBUG : rclone: Version "v1.52.3-326-g45afe97e-beta" starting with parameters ["c:\\data\\rclone\\versions\\rclone-beta-latest-windows-amd64\\rclone-1.52.3-326-g45afe97e-beta.exe" "copy" "gcrypt-a1b2-01:" "gcrypt-jojo-01:" "--log-file=.\\\\log.cryptserverside.txt" "--log-level=DEBUG" "--crypt-server-side-across-configs" "--drive-server-side-across-configs"]
2020/08/23 13:05:00 DEBUG : Encrypted drive 'gcrypt-jojo-01:': Waiting for checks to finish
2020/08/23 13:05:00 DEBUG : Encrypted drive 'gcrypt-jojo-01:': Waiting for transfers to finish
2020/08/23 13:05:00 ERROR : ati.2020.25700.ur.linux.iso: Failed to copy: googleapi: Error 404: File not found: 1NMHz9-3mRAlKqO5qNknlJL2inkqu-mo_., notFound

404 file not found means that the drive server side copy failed. So the user for the destination couldn't read the source files.

You can fix this by sharing the source with the dest user.

i see my confusion now,
for command 2, since source and dest are both crypt, the flag --drive-server-side-across-configs does nothing, so there is no 404 permission issue.

as per the log file, --crypt-server-side-across-configs works with rclone copy
this is almost worthy of being called --magic, almost :upside_down_face:

but --crypt-server-side-across-configs does not seem to work with rclone mount
i mounted both crypted remotes and copied a file between the remotes.
i could not found server side copy in the log files but the log file is full of

source log file
2020/08/23 17:07:28 DEBUG : test/ati.2020.25700.ur.linux.iso: ChunkedReader.Read at 15724544 length 1048576 chunkOffset 0 chunkSize 134217728

dest log file
2020/08/23 16:57:35 DEBUG : /ati.2020.25700.ur.linux.iso: Write: ofst=10485760, fh=0x0

Server side copy only works with rclone copy and sync. Between two mounts or even inside one mount the OS doesn't tell rclone it is copying files, it just opens files and reads and writes data.

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.