What is the problem you are having with rclone?
RClone sync progress reports incorrect file size transferred statistic in the "Press" compression beta. The --progress reports ~2x the amount of data that was actually transferred to the remote.
Command Output (Uncompressed v. Compressed):
root@6f8918600349:/# rclone sync /src GSuite:unRAIDArray/unpress --progress
Transferred: 13.559M / 13.559 MBytes, 100%, 646.020 kBytes/s, ETA 0s
Transferred: 5 / 5, 100%
Elapsed time: 21.4s
root@6f8918600349:/# rclone sync /src GSuitePress:press --progress
Transferred: 27.106M / 27.106 MBytes, 100%, 1.149 MBytes/s, ETA 0s
Transferred: 10 / 10, 100%
Elapsed time: 23.5s
root@6f8918600349:/#
The doubled number of files is expected as the press backend uploads a .meta file for each compressed file. However, looking at the remote I can see that the extra files are only ~190 bytes which doesn't account for the extra ~13MB reported.
Remote File Sizes:
root@6f8918600349:/# rclone ls GSuite:unRAIDArray/press
12675919 rclone.debce5ec10000000000.gz
224 rclone.deb.meta
333761 2560-schematic.png5c19050000000000.gz
190 2560-schematic.png.meta
191 arduino-2560-schematic-arduino-mega-2560-schematic-diagram-data-schema(1).png.meta
336503 arduino-2560-schematic-arduino-mega-2560-schematic-diagram-data-schema(1).pngc724050000000000.gz
189 arduino-2560-schematic-arduino-mega-2560-schematic-diagram-data-schema.png.meta
421005 arduino-2560-schematic-arduino-mega-2560-schematic-diagram-data-schema.png0e6c060000000000.gz
198 wallhaven-0pdqoj.jpg.meta
443947 wallhaven-0pdqoj.jpgf2e8060000000000.gz
I didn't know where exactly to bring this up so I'm hoping this is the right place! I tried the latest beta and the Press backend was not included so I don't know if this has already been fixed in another version.
What is your rclone version (output from rclone version
)
root@6f8918600349:/# rclone --version
rclone v1.51.0-089-g07e2cea1-press-beta
- os/arch: linux/amd64
- go version: go1.14
Which OS you are using and how many bits (eg Windows 7, 64 bit)
Ubuntu 20.04 LTS x64
Which cloud storage system are you using? (eg Google Drive)
Google Drive
The command you were trying to run (eg rclone copy /tmp remote:tmp
)
rclone sync /src GSuitePress:press --progress
The rclone config contents with secrets removed.
root@6f8918600349:/# cat /root/.config/rclone/rclone.conf
[GSuite]
type = drive
scope = drive
service_account_file = /root/.config/rclone/rclone-sa.json
impersonate = me@domain.com
chunk_size = 16M
root_folder_id = 0AMHUIBWTjbU3Uk9PVA
[GSuitePress]
type = press
remote = GSuite:unRAIDArray
compression_mode = gzip
A log from the command with the -vv
flag
root@6f8918600349:/# rclone sync /src GSuitePress:press --progress -vv
2020/08/20 03:25:36 DEBUG : rclone: Version "v1.51.0-089-g07e2cea1-press-beta" starting with parameters ["rclone" "sync" "/src" "GSuitePress:press" "--progress" "-vv"]
2020/08/20 03:25:36 DEBUG : Using config file from "/root/.config/rclone/rclone.conf"
2020-08-20 03:25:39 DEBUG : Compressed: GSuitePress:press: Waiting for checks to finish
2020-08-20 03:25:39 DEBUG : Compressed: GSuitePress:press: Waiting for transfers to finish
2020-08-20 03:25:41 DEBUG : 2560-schematic.png5c19050000000000.gz: Sending chunk 0 length 333761
2020-08-20 03:25:42 DEBUG : rclone.debce5ec10000000000.gz: Sending chunk 0 length 12675919
2020-08-20 03:25:42 DEBUG : arduino-2560-schematic-arduino-mega-2560-schematic-diagram-data-schema(1).pngc724050000000000.gz: Sending chunk 0 length 336503
2020-08-20 03:25:42 DEBUG : arduino-2560-schematic-arduino-mega-2560-schematic-diagram-data-schema.png0e6c060000000000.gz: Sending chunk 0 length 421005
2020-08-20 03:25:43 DEBUG : 2560-schematic.png5c19050000000000.gz: Size and modification time the same (differ by -678µs, within tolerance 1ms)
2020-08-20 03:25:45 DEBUG : 2560-schematic.png: MD5 = 9e4352ae0976385d19a4a06ca308c504 OK
2020-08-20 03:25:45 INFO : 2560-schematic.png: Copied (new)
2020-08-20 03:25:45 DEBUG : arduino-2560-schematic-arduino-mega-2560-schematic-diagram-data-schema(1).pngc724050000000000.gz: Size and modification time the same (differ by -785µs, within tolerance 1ms)
2020-08-20 03:25:45 DEBUG : wallhaven-0pdqoj.jpgf2e8060000000000.gz: Sending chunk 0 length 443947
2020-08-20 03:25:45 DEBUG : arduino-2560-schematic-arduino-mega-2560-schematic-diagram-data-schema.png0e6c060000000000.gz: Size and modification time the same (differ by -944µs, within tolerance 1ms)
2020-08-20 03:25:46 DEBUG : arduino-2560-schematic-arduino-mega-2560-schematic-diagram-data-schema(1).png: MD5 = b1e10662d400739751d0939c49b1ec65 OK
2020-08-20 03:25:46 INFO : arduino-2560-schematic-arduino-mega-2560-schematic-diagram-data-schema(1).png: Copied (new)
2020-08-20 03:25:47 DEBUG : arduino-2560-schematic-arduino-mega-2560-schematic-diagram-data-schema.png: MD5 = 253925b03e76815e618096c4f267d123 OK
2020-08-20 03:25:47 INFO : arduino-2560-schematic-arduino-mega-2560-schematic-diagram-data-schema.png: Copied (new)
2020-08-20 03:25:48 DEBUG : wallhaven-0pdqoj.jpgf2e8060000000000.gz: Size and modification time the same (differ by -250µs, within tolerance 1ms)
2020-08-20 03:25:49 DEBUG : wallhaven-0pdqoj.jpg: MD5 = 8681ea5d4d73a4c8b2a80ca375cb0c24 OK
2020-08-20 03:25:49 INFO : wallhaven-0pdqoj.jpg: Copied (new)
2020-08-20 03:26:01 DEBUG : rclone.debce5ec10000000000.gz: Size and modification time the same (differ by -685µs, within tolerance 1ms)
2020-08-20 03:26:03 DEBUG : rclone.deb: MD5 = b975d05e46d18a5e95598a73d5d0a16d OK
2020-08-20 03:26:03 INFO : rclone.deb: Copied (new)
2020-08-20 03:26:03 DEBUG : Waiting for deletions to finish
Transferred: 27.106M / 27.106 MBytes, 100%, 1.154 MBytes/s, ETA 0s
Transferred: 10 / 10, 100%
Elapsed time: 23.4s
2020/08/20 03:26:03 INFO :
Transferred: 27.106M / 27.106 MBytes, 100%, 1.154 MBytes/s, ETA 0s
Transferred: 10 / 10, 100%
Elapsed time: 23.4s
2020/08/20 03:26:03 DEBUG : 17 go routines active