Press Beta Incorrect Progress Results

What is the problem you are having with rclone?

RClone sync progress reports incorrect file size transferred statistic in the "Press" compression beta. The --progress reports ~2x the amount of data that was actually transferred to the remote.

Command Output (Uncompressed v. Compressed):

root@6f8918600349:/# rclone sync /src GSuite:unRAIDArray/unpress --progress
Transferred:       13.559M / 13.559 MBytes, 100%, 646.020 kBytes/s, ETA 0s
Transferred:            5 / 5, 100%
Elapsed time:        21.4s
root@6f8918600349:/# rclone sync /src GSuitePress:press --progress
Transferred:       27.106M / 27.106 MBytes, 100%, 1.149 MBytes/s, ETA 0s
Transferred:           10 / 10, 100%
Elapsed time:        23.5s
root@6f8918600349:/# 

The doubled number of files is expected as the press backend uploads a .meta file for each compressed file. However, looking at the remote I can see that the extra files are only ~190 bytes which doesn't account for the extra ~13MB reported.

Remote File Sizes:

root@6f8918600349:/# rclone ls GSuite:unRAIDArray/press
 12675919 rclone.debce5ec10000000000.gz
      224 rclone.deb.meta
   333761 2560-schematic.png5c19050000000000.gz
      190 2560-schematic.png.meta
      191 arduino-2560-schematic-arduino-mega-2560-schematic-diagram-data-schema(1).png.meta
   336503 arduino-2560-schematic-arduino-mega-2560-schematic-diagram-data-schema(1).pngc724050000000000.gz
      189 arduino-2560-schematic-arduino-mega-2560-schematic-diagram-data-schema.png.meta
   421005 arduino-2560-schematic-arduino-mega-2560-schematic-diagram-data-schema.png0e6c060000000000.gz
      198 wallhaven-0pdqoj.jpg.meta
   443947 wallhaven-0pdqoj.jpgf2e8060000000000.gz

I didn't know where exactly to bring this up so I'm hoping this is the right place! I tried the latest beta and the Press backend was not included so I don't know if this has already been fixed in another version.

What is your rclone version (output from rclone version)

root@6f8918600349:/# rclone --version
rclone v1.51.0-089-g07e2cea1-press-beta
- os/arch: linux/amd64
- go version: go1.14

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Ubuntu 20.04 LTS x64

Which cloud storage system are you using? (eg Google Drive)

Google Drive

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone sync /src GSuitePress:press --progress

The rclone config contents with secrets removed.

root@6f8918600349:/# cat /root/.config/rclone/rclone.conf 
[GSuite]
type = drive
scope = drive
service_account_file = /root/.config/rclone/rclone-sa.json
impersonate = me@domain.com
chunk_size = 16M
root_folder_id = 0AMHUIBWTjbU3Uk9PVA

[GSuitePress]
type = press
remote = GSuite:unRAIDArray
compression_mode = gzip

A log from the command with the -vv flag

root@6f8918600349:/# rclone sync /src GSuitePress:press --progress -vv
2020/08/20 03:25:36 DEBUG : rclone: Version "v1.51.0-089-g07e2cea1-press-beta" starting with parameters ["rclone" "sync" "/src" "GSuitePress:press" "--progress" "-vv"]
2020/08/20 03:25:36 DEBUG : Using config file from "/root/.config/rclone/rclone.conf"
2020-08-20 03:25:39 DEBUG : Compressed: GSuitePress:press: Waiting for checks to finish
2020-08-20 03:25:39 DEBUG : Compressed: GSuitePress:press: Waiting for transfers to finish
2020-08-20 03:25:41 DEBUG : 2560-schematic.png5c19050000000000.gz: Sending chunk 0 length 333761
2020-08-20 03:25:42 DEBUG : rclone.debce5ec10000000000.gz: Sending chunk 0 length 12675919
2020-08-20 03:25:42 DEBUG : arduino-2560-schematic-arduino-mega-2560-schematic-diagram-data-schema(1).pngc724050000000000.gz: Sending chunk 0 length 336503
2020-08-20 03:25:42 DEBUG : arduino-2560-schematic-arduino-mega-2560-schematic-diagram-data-schema.png0e6c060000000000.gz: Sending chunk 0 length 421005
2020-08-20 03:25:43 DEBUG : 2560-schematic.png5c19050000000000.gz: Size and modification time the same (differ by -678µs, within tolerance 1ms)
2020-08-20 03:25:45 DEBUG : 2560-schematic.png: MD5 = 9e4352ae0976385d19a4a06ca308c504 OK
2020-08-20 03:25:45 INFO  : 2560-schematic.png: Copied (new)
2020-08-20 03:25:45 DEBUG : arduino-2560-schematic-arduino-mega-2560-schematic-diagram-data-schema(1).pngc724050000000000.gz: Size and modification time the same (differ by -785µs, within tolerance 1ms)
2020-08-20 03:25:45 DEBUG : wallhaven-0pdqoj.jpgf2e8060000000000.gz: Sending chunk 0 length 443947
2020-08-20 03:25:45 DEBUG : arduino-2560-schematic-arduino-mega-2560-schematic-diagram-data-schema.png0e6c060000000000.gz: Size and modification time the same (differ by -944µs, within tolerance 1ms)
2020-08-20 03:25:46 DEBUG : arduino-2560-schematic-arduino-mega-2560-schematic-diagram-data-schema(1).png: MD5 = b1e10662d400739751d0939c49b1ec65 OK
2020-08-20 03:25:46 INFO  : arduino-2560-schematic-arduino-mega-2560-schematic-diagram-data-schema(1).png: Copied (new)
2020-08-20 03:25:47 DEBUG : arduino-2560-schematic-arduino-mega-2560-schematic-diagram-data-schema.png: MD5 = 253925b03e76815e618096c4f267d123 OK
2020-08-20 03:25:47 INFO  : arduino-2560-schematic-arduino-mega-2560-schematic-diagram-data-schema.png: Copied (new)
2020-08-20 03:25:48 DEBUG : wallhaven-0pdqoj.jpgf2e8060000000000.gz: Size and modification time the same (differ by -250µs, within tolerance 1ms)
2020-08-20 03:25:49 DEBUG : wallhaven-0pdqoj.jpg: MD5 = 8681ea5d4d73a4c8b2a80ca375cb0c24 OK
2020-08-20 03:25:49 INFO  : wallhaven-0pdqoj.jpg: Copied (new)
2020-08-20 03:26:01 DEBUG : rclone.debce5ec10000000000.gz: Size and modification time the same (differ by -685µs, within tolerance 1ms)
2020-08-20 03:26:03 DEBUG : rclone.deb: MD5 = b975d05e46d18a5e95598a73d5d0a16d OK
2020-08-20 03:26:03 INFO  : rclone.deb: Copied (new)
2020-08-20 03:26:03 DEBUG : Waiting for deletions to finish
Transferred:       27.106M / 27.106 MBytes, 100%, 1.154 MBytes/s, ETA 0s
Transferred:           10 / 10, 100%
Elapsed time:        23.4s
2020/08/20 03:26:03 INFO  : 
Transferred:   	   27.106M / 27.106 MBytes, 100%, 1.154 MBytes/s, ETA 0s
Transferred:           10 / 10, 100%
Elapsed time:        23.4s

2020/08/20 03:26:03 DEBUG : 17 go routines active

That's a really old beta, can you try with the latest?

The latest beta doesn't seem to have the "Press" backend that I could find.

rclone v1.52.3-315-g85f9bd1a-beta
- os/arch: linux/amd64
- go version: go1.15
Type of storage to configure.
Enter a string value. Press Enter for the default ("").
Choose a number from below, or type in your own value
 1 / 1Fichier
   \ "fichier"
 2 / Alias for an existing remote
   \ "alias"
 3 / Amazon Drive
   \ "amazon cloud drive"
 4 / Amazon S3 Compliant Storage Provider (AWS, Alibaba, Ceph, Digital Ocean, Dreamhost, IBM COS, Minio, etc)
   \ "s3"
 5 / Backblaze B2
   \ "b2"
 6 / Box
   \ "box"
 7 / Cache a remote
   \ "cache"
 8 / Citrix Sharefile
   \ "sharefile"
 9 / Dropbox
   \ "dropbox"
10 / Encrypt/Decrypt a remote
   \ "crypt"
11 / FTP Connection
   \ "ftp"
12 / Google Cloud Storage (this is not Google Drive)
   \ "google cloud storage"
13 / Google Drive
   \ "drive"
14 / Google Photos
   \ "google photos"
15 / Hubic
   \ "hubic"
16 / In memory object storage system.
   \ "memory"
17 / Jottacloud
   \ "jottacloud"
18 / Koofr
   \ "koofr"
19 / Local Disk
   \ "local"
20 / Mail.ru Cloud
   \ "mailru"
21 / Mega
   \ "mega"
22 / Microsoft Azure Blob Storage
   \ "azureblob"
23 / Microsoft OneDrive
   \ "onedrive"
24 / OpenDrive
   \ "opendrive"
25 / OpenStack Swift (Rackspace Cloud Files, Memset Memstore, OVH)
   \ "swift"
26 / Pcloud
   \ "pcloud"
27 / Put.io
   \ "putio"
28 / QingCloud Object Storage
   \ "qingstor"
29 / SSH/SFTP Connection
   \ "sftp"
30 / Sugarsync
   \ "sugarsync"
31 / Tardigrade Decentralized Cloud Storage
   \ "tardigrade"
32 / Transparently chunk/split large files
   \ "chunker"
33 / Union merges the contents of several upstream fs
   \ "union"
34 / Webdav
   \ "webdav"
35 / Yandex Disk
   \ "yandex"
36 / http Connection
   \ "http"
37 / premiumize.me
   \ "premiumizeme"
38 / seafile
   \ "seafile"
Storage> 


We haven't merged the press backend yet - there are some technical problems with it we haven't fixed.

I would guess what you are seeing is one set of MB transferred which is rclone reading the files from the disk to compress the data then additionally you are seeing the uploaded file data, so a bit less than 2x!

That's totally fine! I just didn't know if this had been caught and I didn't want to try posting directly on the github pull request. It's still amazing work and I'm going to try using it anyway to backup my 14TB unRAID array.

1 Like

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.