How to really limit rclone copy by a file max-size?

What is the problem you are having with rclone?

Rclone copy --max-size parameter has no effect

What is your rclone version (output from rclone version)

rclone v1.50.1

  • os/arch: linux/amd64
  • go version: go1.13.4

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Ubuntu 18.04.3 LTS
Linux localhost 4.15.0-66-generic #75-Ubuntu SMP Tue Oct 1 05:24:09 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Which cloud storage system are you using? (eg Google Drive)

The command you were trying to run (eg rclone copy /tmp remote:tmp)

root 5256 12.0 0.0 136356 29212 ? Sl 12:47 0:00 _ /usr/bin/rclone copy --progress --max-size 250M --max-transfer 250M --modify-window 60s --retries 1 /var/path/a-huge-file box:/path

ls -la var/path/a-huge-file
-rw-r--r-- 1 user users 9406738432 Nov 11 12:02 /var/path/a-huge-file

A log from the command with the -vv flag (eg output from rclone -vv copy /tmp remote:tmp)

2019/11/11 13:05:21 DEBUG : rclone: Version "v1.50.1" starting with parameters ["/usr/bin/rclone" "-vv" "copy" "--progress" "--max-size" "250M" "--max-transfer" "250M" "--modify-window" "60s" "--retries" "1" "/var/path/a-huge-file" "box:/path"]
2019/11/11 13:05:21 DEBUG : Using config file from "/root/.config/rclone/rclone.conf"
2019-11-11 13:05:24 DEBUG : a-huge-file: Modification times differ by -5h22m3.39613788s: 2019-11-11 11:54:10.39613788 +0300 MSK, 2019-11-10 19:32:07 -0800 -0800
Transferred: 0 / 0 Bytes, -, 0 Bytes/s, ETA -
Errors: 0
Checks: 0 / 0, -
Transferred: 0 / 0, -
Elapsed time: 0s

(hung at this moment instead of just skip the file by its size)

2019-11-11 13:10:42 DEBUG : a-huge-file: SHA-1 = 45fa547af21510bfa379f3f4542dd0ef79a87487 (Local file system at /var/path/a-huge-file)
2019-11-11 13:10:42 DEBUG : a-huge-file: SHA-1 = 9e93c6b25cea47a28036c4844b1f4ab0c8553576 (box root 'path')
2019-11-11 13:10:42 DEBUG : a-huge-file: SHA-1 differ
2019-11-11 13:10:43 DEBUG : a-huge-file: Multipart upload session started for 141 parts of size 64M
2019-11-11 13:10:47 DEBUG : a-huge-file: Uploading part 1/141 offset 0/8.761G part size 64M
2019-11-11 13:10:51 DEBUG : a-huge-file: Uploading part 2/141 offset 64M/8.761G part size 64M
2019-11-11 13:10:56 DEBUG : a-huge-file: Uploading part 3/141 offset 128M/8.761G part size 64M
2019-11-11 13:10:59 DEBUG : a-huge-file: Uploading part 4/141 offset 192M/8.761G part size 64M
2019-11-11 13:11:04 DEBUG : a-huge-file: Uploading part 5/141 offset 256M/8.761G part size 64M
2019-11-11 13:11:08 DEBUG : a-huge-file: Uploading part 6/141 offset 320M/8.761G part size 64M
2019-11-11 13:11:13 DEBUG : a-huge-file: Uploading part 7/141 offset 384M/8.761G part size 64M
2019-11-11 13:11:13 DEBUG : a-huge-file: Cancelling multipart upload: multipart upload failed to upload part: Put h-t-t-ps://upload.box.com/api/2.0/files/upload_sessions/xxxxxxxxx: Max transfer limit reached as set by --max-transfer
2019-11-11 13:11:13 ERROR : a-huge-file: Failed to copy: multipart upload failed to upload part: Put h-t-t-ps://upload.box.com/api/2.0/files/upload_sessions/xxxxxxx: Max transfer limit reached as set by --max-transfer
2019-11-11 13:11:13 ERROR : Fatal error received - not attempting retries
Transferred: 250M / 250 MBytes, 100%, 8.064 MBytes/s, ETA 0s
Errors: 3 (fatal error encountered)
Checks: 0 / 0, -
Transferred: 0 / 0, -
Elapsed time: 31s
2019/11/11 13:11:13 Failed to copy with 3 errors: last error was: multipart upload failed to upload part: Put upload.box.com/api/2.0/files/upload_sessions/xxxxxx: Max transfer limit reached as set by --max-transfer

Ah, I see what is going on...

This is because you've named the actual file in the command line that rclone is not applying the filters...

You could workaround the problem by using something like

rclone copy --progress --max-size 250M --max-transfer 250M --modify-window 60s --retries 1 /var/path --include /a-huge-file box:/path

If you think this needs fixing properly then please please make a new issue on github and we'll see what we can do!

1 Like

Clear. Thank you for the quick response!

That's pity a bit. As the whole dir tree is quite big, the use case is to monitor file changes by inotifywait and sync/copy on per-file basis.
No problem in fact, will act per subdir a file belongs to. Thank you for the clarification!

Kind regards,

This is fixable if you want to make an issue :slight_smile:

1 Like

Thanks, I have zero experience in opening github issues, so afraid of its quality, but will try to manage that and file asap!
Regards,

1 Like

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.