Rclone 1.59.0 release

Releases
1

Rclone 1.59.0 has been released. Find it in the rclone downloads or use rclone selfupdate to upgrade.

Some highlights of the release:

Thank you to all the contributors to this release, those who contributed code or doc fixes (64 people) or made issues or answered questions in the forum - your help is much appreciated! Especial thanks to @albertony for a whopping 69 commits this release and their hard work fixing docs and staticcheck problems. Many thanks also to Manz Solutions for sponsoring the metadata framework.

If you (or your company) is enjoying rclone, please consider donating or sponsoring to keep the project sustainable, or hire me to add a feature you really want!

v1.59.0 - 2022-07-09

See commits

  • New backends
  • New commands
    • test makefile: Create a single file for testing (Nick Craig-Wood)
  • New Features
    • Metadata framework to read and write system and user metadata on backends (Nick Craig-Wood)
      • Implemented initially for local, s3 and internetarchive backends
      • --metadata/-M flag to control whether metadata is copied
      • --metadata-set flag to specify metadata for uploads
      • Thanks to Manz Solutions for sponsoring this work.
    • build
      • Update to go1.18 and make go1.16 the minimum required version (Nick Craig-Wood)
      • Update android go build to 1.18.x and NDK to 23.1.7779620 (Nick Craig-Wood)
      • All windows binaries now no longer CGO (Nick Craig-Wood)
      • Add linux/arm/v6 to docker images (Nick Craig-Wood)
      • A huge number of fixes found with staticcheck (albertony)
      • Configurable version suffix independent of version number (albertony)
    • check: Implement --no-traverse and --no-unicode-normalization (Nick Craig-Wood)
    • config: Readability improvements (albertony)
    • copyurl: Add --header-filename to honor the HTTP header filename directive (J-P Treen)
    • filter: Allow multiple --exclude-if-present flags (albertony)
    • fshttp: Add --disable-http-keep-alives to disable HTTP Keep Alives (Nick Craig-Wood)
    • install.sh
      • Set the modes on the files and/or directories on macOS (Michael C Tiernan - MIT-Research Computing Project)
      • Pre verify sudo authorization -v before calling curl. (Michael C Tiernan - MIT-Research Computing Project)
    • lib/encoder: Add Semicolon encoding (Nick Craig-Wood)
    • lsf: Add metadata support with M flag (Nick Craig-Wood)
    • lsjson: Add --metadata/-M flag (Nick Craig-Wood)
    • ncdu
      • Implement multi selection (CrossR)
      • Replace termbox with tcell's termbox wrapper (eNV25)
      • Display correct path in delete confirmation dialog (Roberto Ricci)
    • operations
      • Speed up hash checking by aborting the other hash if first returns nothing (Nick Craig-Wood)
      • Use correct src/dst in some log messages (zzr93)
    • rcat: Check checksums by default like copy does (Nick Craig-Wood)
    • selfupdate: Replace deprecated x/crypto/openpgp package with ProtonMail/go-crypto (albertony)
    • serve ftp: Check --passive-port arguments are correct (Nick Craig-Wood)
    • size: Warn about inaccurate results when objects with unknown size (albertony)
    • sync: Overlap check is now filter-sensitive so --backup-dir can be in the root provided it is filtered (Nick)
    • test info: Check file name lengths using 1,2,3,4 byte unicode characters (Nick Craig-Wood)
    • test makefile(s): --sparse, --zero, --pattern, --ascii, --chargen flags to control file contents (Nick Craig-Wood)
    • Make sure we call the Shutdown method on backends (Martin Czygan)
  • Bug Fixes
    • accounting: Fix unknown length file transfers counting 3 transfers each (buda)
    • ncdu: Fix issue where dir size is summed when file sizes are -1 (albertony)
    • sync/copy/move
      • Fix --fast-list --create-empty-src-dirs and --exclude (Nick Craig-Wood)
      • Fix --max-duration and --cutoff-mode soft (Nick Craig-Wood)
    • Fix fs cache unpin (Martin Czygan)
    • Set proper exit code for errors that are not low-level retried (e.g. size/timestamp changing) (albertony)
  • Mount
    • Support windows/arm64 (may still be problems - see #5828) (Nick Craig-Wood)
    • Log IO errors at ERROR level (Nick Craig-Wood)
    • Ignore _netdev mount argument (Hugal31)
  • VFS
    • Add --vfs-fast-fingerprint for less accurate but faster fingerprints (Nick Craig-Wood)
    • Add --vfs-disk-space-total-size option to manually set the total disk space (Claudio Maradonna)
    • vfscache: Fix fatal error: sync: unlock of unlocked mutex error (Nick Craig-Wood)
  • Local
    • Fix parsing of --local-nounc flag (Nick Craig-Wood)
    • Add Metadata support (Nick Craig-Wood)
  • Crypt
    • Support metadata (Nick Craig-Wood)
  • Azure Blob
    • Calculate Chunksize/blocksize to stay below maxUploadParts (Leroy van Logchem)
    • Use chunksize lib to determine chunksize dynamically (Derek Battams)
    • Case insensitive access tier (Rob Pickerill)
    • Allow remote emulator (azurite) (Lorenzo Maiorfi)
  • B2
    • Add --b2-version-at flag to show file versions at time specified (SwazRGB)
    • Use chunksize lib to determine chunksize dynamically (Derek Battams)
  • Chunker
    • Mark as not supporting metadata (Nick Craig-Wood)
  • Compress
    • Support metadata (Nick Craig-Wood)
  • Drive
    • Make backend config -o config add a combined AllDrives: remote (Nick Craig-Wood)
    • Make --drive-shared-with-me work with shared drives (Nick Craig-Wood)
    • Add --drive-resource-key for accessing link-shared files (Nick Craig-Wood)
    • Add backend commands exportformats and importformats for debugging (Nick Craig-Wood)
    • Fix 404 errors on copy/server side copy objects from public folder (Nick Craig-Wood)
    • Update Internal OAuth consent screen docs (Phil Shackleton)
    • Moved root_folder_id to advanced section (Abhiraj)
  • Dropbox
    • Migrate from deprecated api (m8rge)
    • Add logs to show when poll interval limits are exceeded (Nick Craig-Wood)
    • Fix nil pointer exception on dropbox impersonate user not found (Nick Craig-Wood)
  • Fichier
    • Parse api error codes and them accordingly (buengese)
  • FTP
    • Add support for disable_utf8 option (Jason Zheng)
    • Revert to upstream github.com/jlaffaye/ftp from our fork (Nick Craig-Wood)
  • Google Cloud Storage
    • Add --gcs-no-check-bucket to minimise transactions and perms (Nick Gooding)
    • Add --gcs-decompress flag to decompress gzip-encoded files (Nick Craig-Wood)
      • by default these will be downloaded compressed (which previously failed)
  • Hasher
    • Support metadata (Nick Craig-Wood)
  • HTTP
    • Fix missing response when using custom auth handler (albertony)
  • Jottacloud
    • Add support for upload to custom device and mountpoint (albertony)
    • Always store username in config and use it to avoid initial API request (albertony)
    • Fix issue with server-side copy when destination is in trash (albertony)
    • Fix listing output of remote with special characters (albertony)
  • Mailru
    • Fix timeout by using int instead of time.Duration for keeping number of seconds (albertony)
  • Mega
    • Document using MEGAcmd to help with login failures (Art M. Gallagher)
  • Onedrive
    • Implement --poll-interval for onedrive (Hugo Laloge)
    • Add access scopes option (Sven Gerber)
  • Opendrive
    • Resolve lag and truncate bugs (Scott Grimes)
  • Pcloud
    • Fix about with no free space left (buengese)
    • Fix cleanup (buengese)
  • S3
    • Use PUT Object instead of presigned URLs to upload single part objects (Nick Craig-Wood)
    • Backend restore command to skip non-GLACIER objects (Vincent Murphy)
    • Use chunksize lib to determine chunksize dynamically (Derek Battams)
    • Retry RequestTimeout errors (Nick Craig-Wood)
    • Implement reading and writing of metadata (Nick Craig-Wood)
  • SFTP
    • Add support for about and hashsum on windows server (albertony)
    • Use vendor-specific VFS statistics extension for about if available (albertony)
    • Add --sftp-chunk-size to control packets sizes for high latency links (Nick Craig-Wood)
    • Add --sftp-concurrency to improve high latency transfers (Nick Craig-Wood)
    • Add --sftp-set-env option to set environment variables (Nick Craig-Wood)
    • Add Hetzner Storage Boxes to supported sftp backends (Anthrazz)
  • Storj
    • Fix put which lead to the file being unreadable when using mount (Erik van Velzen)
  • Union
    • Add min_free_space option for lfs/eplfs policies (Nick Craig-Wood)
    • Fix uploading files to union of all bucket based remotes (Nick Craig-Wood)
    • Fix get free space for remotes which don't support it (Nick Craig-Wood)
    • Fix eplus policy to select correct entry for existing files (Nick Craig-Wood)
    • Support metadata (Nick Craig-Wood)
  • Uptobox
    • Fix root path handling (buengese)
  • WebDAV
    • Add SharePoint in other specific regions support (Noah Hsu)
  • Yandex
    • Handle api error on server-side move (albertony)
  • Zoho
    • Add Japan and China regions (buengese)
11 Likes
3

Using Elgiganten Cloud (Jottacloud) I only did a selfupdate, and below is what i get, even after trying a rclone config again to get a new token.
Reverted to 1.58.1 and it works again with no problems. Any ideas what might cause this?
I removed some names from the log.

  • 2022/07/10 09:05:25 DEBUG : rclone: Version "v1.59.0" starting with parameters ["C:\Windows\System32\rclone.exe" "mount" "Crypt:" "J:" "--volname=" "--config=C:\Users\****\.config\rclone\rclone.conf" "-o" "ThreadCount=16" "--transfers=16" "--vfs-cache-mode=full" "--file-perms=0777" "--jottacloud-hard-delete" "--cache-dir" "E:\Cache" "--vfs-cache-max-age=4h0m0s" "--vfs-write-back=1s" "--log-file=C:\rclone\Backup2022-07-10.log" "--log-level=DEBUG" "--no-console"]
  • 2022/07/10 09:05:25 DEBUG : Creating backend with remote "****Crypt:"
  • 2022/07/10 09:05:25 DEBUG : Using config file from "C:\Users\****\.config\rclone\rclone.conf"
  • 2022/07/10 09:05:25 DEBUG : Creating backend with remote ":"
  • 2022/07/10 09:05:26 DEBUG : ****: detected overridden config - adding "{jlU5h}" suffix to name
  • 2022/07/10 09:05:26 Failed to create file system for "Crypt:": failed to make remote ":****" to wrap: read metadata failed: error 401: org.springframework.security.authentication.BadCredentialsException: No token or token is malformed! (Unauthorized)
4

Try reconfiguring the backend (back up the config file first) and see if that helps.

5

I still get the exact same error. Copying over with 1.58.1 files again and it starts working again with no issues.
I redid the config, updated the token using legacy authentication, got my sms and everything seems to work except when mounting it complains with the error i posted.
I also did a new test mount and it also displayed exact same error and added suffix to name aswell.
I got no clue.

6

@Krakkan can you open a new issue on Github about that with as much detail as possible and we'll fix it up - thank you.

7

Hi Nick, do you have an ETA for bringing back multithread downloads when cache full is used? If you don't that's fine, obviously. :slight_smile:

8

Please follow the issue:

Support multi-threaded downloads when downloading a file to the cache · Issue #4760 · rclone/rclone (github.com)

9

Tried my best to make one :slight_smile:

1 Like
10

I am already. Nick stated in April ‘21 that the feature was gonna come back soon and that’s why I was asking if he had a time frame in mind. Thank you.

11

I had a sponsorship deal to implement multithread downloads into the vfs cache, but unfortunately it fell though at the last minute :frowning: I don't currently have a timescale for implementing it, but I do have quite a good plan which is a start!

2 Likes
12

Thanks for the reply. Not sure what you mean by "sponsorship deal" but I get that there have been setbacks. As always, you don't owe us anything and I was asking only due to Drive's problems with single threaded downloads... which are not your fault to begin with.

I''l wait. Thanks again! :slight_smile:

1 Like
13

@albertony, thanks

14

what is the difference between Union and Combine remotes?

15

Check out the docs:

Union (rclone.org)

Combine (rclone.org)

16

The docs are not exactly clear in what will be used when this is enabled, only size?

Also perhaps if you are really like me and just want maximum performance and have files that NEVER change, wouldn't be better to just disable all fingerprinting?

Then if I really update a file in the cache, i should be able to use a RC command to force fingerpriting check to refresh them or something like that

17

I did and it's still not clear? Union intertwines all files and combine only directories?

18

It depends on the backend. For local this will just use modtime and size when it's on and it will include hash when off.

For drive it will make no difference.

For S3 it will use hash and size when on and include modtime when off.

Setting this to on will speed things up slightly if you are using s3 but not at all if using drive.

With fast fingerprint, removing the check wouldn't speed things up as rclone doesn't have to do any network operations to check it.

19

Union intermingles files but combine attaches remotes to top level directories without intermingling files.

Here is the example from the docs which hopefully makes a bit more sense now.

The combine backend joins remotes together into a single directory tree.

For example you might have a remote for images on one provider:

$ rclone tree s3:imagesbucket
/
├── image1.jpg
└── image2.jpg

And a remote for files on another:

$ rclone tree drive:important/files
/
├── file1.txt
└── file2.txt

The combine backend can join these together into a synthetic directory structure like this:

$ rclone tree combined:
/
├── files
│ ├── file1.txt
│ └── file2.txt
└── images
├── image1.jpg
└── image2.jpg

20

That assumes your backend is local, but not actually a network mount mounted, so you'd still have to make external connections to retrieve size of the items.