--drive-service-account-file and --max-transfer

‘rclone copy --max-transfer’ works beautifully in normal cases, in that it stops rclone after hitting the max.

However --max-transfer does not work with ‘rclone copy --drive-service-account-file’. For example, with a normal (non server-side) copy the progress tracker shows:

Transferred: 0.100 / 1.100 TBytes, 0%, 0 Bytes/s, ETA -
Transferred: 0.200/ 1.100 TBytes, 0%, 0 Bytes/s, ETA -

and so on until it hits whatever amount you have put in --max-transfer=

But with --drive-service-account-file the progress tracker shows:

Transferred: 0 / 1.100 TBytes, 0%, 0 Bytes/s, ETA -
Transferred: 0 / 1.000 TBytes, 0%, 0 Bytes/s, ETA -
Transferred: 0 / 0.900 TBytes, 0%, 0 Bytes/s, ETA -

and so on. If you hit the maximum allocation for a specific service account (e.g. 750GB with GDrive) then rclone stalls, sometimes for quite a while.

Is there any way to have rclone stop when using --drive-service-account-file when it stalls? I tried --timeout but it seems there is something that keeps the I/O timeout from being triggered.

All suggestions welcomed. The rclone command is a part of a script, so the intent is to let it stop/fail gracefully and then move on to the next line.

Thank you as always for the help.

There is an issue about this already here:

I just posted a beta to test on that ticket - would appreciate feedback!

Hi @ncw. Thank you.

EDIT: Ignore my previous (now deleted) post. I installed the last rclone version in the branch folder rather than the specific one that you linked. Big apology.

Net result: --max-transfer worked nicely using this version https://beta.rclone.org/branch/v1.46.0-103-gc3aab713-fix-3033-copy-transfer-beta/
and --drive-service-account-file

A million thanks.

EDIT2: Found an anomaly with some further testing. In most cases this seems to be working, with rclone transitioning smoothly to the next line in the script immediately upon hitting the max transfer limit.

However in some cases even after hitting the --max-transfer limit and displaying the correct error [[ 2019-03-21 05:09:20 ERROR : Cancelling sync due to fatal error: Max transfer limit reached as set by --max-transfer ]] rclone is still hanging. I tried a number of combinations, including adding/removing --fast-list, changing --drive-chunk-size and a few other flags. None of these seemed to impact the hanging state.

The one adjustment that seemed to cause/solve the hanging state was --transfers. After playing with both large and small files my best guess (truly, a guess) is that when the # of transfers times the size of the queued files is larger than the gap between --max-transfers and the natural upload allocation (e.g. 750G in GDrive) then rclone hangs. For example:

Average file size is 20G. --transfers 4 --max-transfer 720G. Then rclone hangs.
Average file size is 20G. --transfers 1 --max-transfer 720G. Then rclone does not hang.
Average file size is 20G. --transfers 4 --max-transfer 400G. Then rclone does not hang.

I could be off base. But I tried these combinations several times and got the same results. I only found this error with especially large file transfers. When I was copying smaller files whose aggregate did not exceed the --max-transfer to upload limit ‘gap’ then everything went smoothly.

The command used is in this format, with variables adjusted for testing:
rclone copy --max-transfer 720G --transfers 4 -v -P --stats=5s --drive-service-account-file /jsons/1.json

Glad it works!

The hanging is a problem that I saw before when developing --max-transfers for the first time.

Can you make rclone hang then press CTRL-\ or kill the rclone process with SIGQUIT. This will make a backtrace of everything rclone is doing. Can you post that?

I have a question/observation then it hit max-transfer it outputs the following message

2019/03/21 21:34:43 ERROR : Cancelling sync due to fatal error: Max transfer limit reached as set by --max-transfer
2019/03/21 21:34:44 ERROR : Fatal error received - not attempting retries
2019/03/21 21:34:44 Failed to copy: Max transfer limit reached as set by --max-transfer

but generally speaking this is not an ERROR and not a Fatal error, should this not be an INFO message or is that just me

It is an ERROR since it didn’t complete the sync and it needs to be a Fatal error to stop rclone doing any further retries.

Thank you for the info, I will change my monitoring and add those to expected behavior, I just wanted to be sure that it was not going to change.

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.