Started to see multipart upload starting chunk in logs

Help and Support
1

What is the problem you are having with rclone?

i noticed multipart upload starting chunk size 5M in the logs.
whereas i never saw this in any logs ever in the past.

What is your rclone version (output from rclone version)

v1.51.0

Which OS you are using and how many bits (eg Windows 7, 64 bit)

w10.64bit 1909

Which cloud storage system are you using? (eg Google Drive)

wasabi

The command you were trying to run (eg rclone copy /tmp remote:tmp)

C:\data\rclone\scripts\rclone.exe sync "b:\vabj.homeserver01.iso_20200218.174709\veeambackups\abj_homeserver01_iso - homeserver01" wasabieast2:vserver03\vabj.homeserver01.iso\rclone\backup\ --stats=0 --progress --backup-dir=wasabieast2:vserver03\vabj.homeserver01.iso\rclone\archive\20200218.174709\ --log-level DEBUG --log-file=C:\data\rclone\logs\vabj.homeserver01.iso\20200218.174709\vabj.homeserver01.iso_20200218.174709_rclone.log

A log from the command with the -vv flag (eg output from rclone -vv copy /tmp remote:tmp)

2020/02/18 17:47:13 DEBUG : rclone: Version "v1.51.0" starting with parameters ["C:\data\rclone\scripts\rclone.exe" "sync" "b:\vabj.homeserver01.iso_20200218.174709\veeambackups\abj_homeserver01_iso - homeserver01" "wasabieast2:vserver03\vabj.homeserver01.iso\rclone\backup\" "--stats=0" "--progress" "--backup-dir=wasabieast2:vserver03\vabj.homeserver01.iso\rclone\archive\20200218.174709\" "--log-level" "DEBUG" "--log-file=C:\data\rclone\logs\vabj.homeserver01.iso\20200218.174709\vabj.homeserver01.iso_20200218.174709_rclone.log"]
2020/02/18 17:47:13 DEBUG : Using RCLONE_CONFIG_PASS password.
2020/02/18 17:47:13 DEBUG : Using config file from "C:\data\rclone\scripts\rclone.conf"
2020/02/18 17:47:13 INFO : S3 bucket vserver03 path vabj.homeserver01.iso/rclone/backup: Waiting for checks to finish
2020/02/18 17:47:13 DEBUG : ABJ_HOMESERVER01_ISO - HOMESERVER01.vbm: Sizes differ (src 2041373 vs dst 2008233)
2020/02/18 17:47:40 DEBUG : ABJ_HOMESERVER01_ISO - HOMESERVER01D2020-02-18T174031_3CA6.vib: multipart upload starting chunk 1 size 5M offset 0/4.341G
2020/02/18 17:47:40 DEBUG : ABJ_HOMESERVER01_ISO - HOMESERVER01D2020-02-18T174031_3CA6.vib: multipart upload starting chunk 2 size 5M offset 5M/4.341G
2020/02/18 17:47:40 DEBUG : ABJ_HOMESERVER01_ISO - HOMESERVER01D2020-02-18T174031_3CA6.vib: multipart upload starting chunk 3 size 5M offset 10M/4.341G

2

Are you facing any issue with it? It was added back pretty recently. The specific commit: https://github.com/rclone/rclone/commit/7e6fac8b1eaeccbf48e197e2c1f12627c3e69ca5

3

i am not getting any errors

the issue is that until the last few days, i have NEVER seen multipart upload in the log.

4

As mentioned, it was added back very recently i.e. in December so this would have been the first stable release with it included.

5

ok. thanks,

i recently updated rclone on my server to v1.51.0.

6

I re-wrote the multipart upload for 1.51 and as part of that I put some debugs in to see when the chunks are being uploaded. 1.50 didn't have these debugs.

I can take them out if they are annoying, but they are sometimes useful for troubleshooting and shouldn't affect anything.

7

well, thanks for re-writing the code.

the problem is the size of the logs becomes large.
for example, i had a log with over 19,000 lines and a size of 2.4MB

perhaps a flag, to enable, or to disable the multipart upload starting chunk in the debug file.

8

I'd be happy for that line to be commented out. Want to send a pull request?

9

sure, but the green button is grayed out...

image

10

I think you should be doing that on your fork of the repo?

11

i do not have a fork, this process is too complex.
there should be an easy way to make a suggestion concerning rclone.
thanks

12

You'd make a feature request on github if you wanted to make a suggestion or a request for an enhancement.

A pull request is if you want to adjust the code and submit a request to get it updated. It may seem daunting at first, but it really is not that complex or hard to do as it just requires a little reading and understanding of how github works.

The more folks are taught how to submit pulls, the less Nick has to do and more he can code :slight_smile:

13

thanks i will make a feature request,

for a pull request that would mean i have to:

  1. know go language which seems to have complex syntax like c.
  2. know rclone source code
  3. pinpoint all the files that need to change.
  4. make the changes

i program in python

14

Try using the online editing method: https://help.github.com/en/github/managing-files-in-a-repository/editing-files-in-another-users-repository

There is just that one line which needs commenting out with //

That should hopefully be easier.

15

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.