Rclone v1.37 release


#1

rclone 1.37 is now released!

Thank you to everyone who tested the betas, reported bugs and sent in pull requests.

Especial thanks to Stefan Breunig who has agreed to help me maintain rclone and has made many contributions in the changelog below.

Headline features are FTP and HTTP backends, Google Team Drive support, Dropbox v2, rclone mount working on Windows, rclone ncdu (personal fave), --fast-list, --tpslimit (useful for rclone mount users) and lots of fixes!

Happy rclone-ing!

– Nick

  • v1.37 - 2017-07-22
    • New backends
      • FTP - thanks to Antonio Messina
      • HTTP - thanks to Vasiliy Tolstov
    • New commands
      • rclone ncdu - for exploring a remote with a text based user interface.
      • rclone lsjson - for listing with a machine readable output
      • rclone dbhashsum - to show Dropbox style hashes of files (local or Dropbox)
    • New Features
      • Implement --fast-list flag
        • This allows remotes to list recursively if they can
        • This uses less transactions (important if you pay for them)
        • This may or may not be quicker
        • This will user more memory as it has to hold the listing in memory
        • –old-sync-method deprecated - the remaining uses are covered by --fast-list
        • This involved a major re-write of all the listing code
      • Add --tpslimit and --tpslimit-burst to limit transactions per second
        • this is useful in conjuction with rclone mount to limit external apps
      • Add --stats-log-level so can see --stats without -v
      • Print password prompts to stderr - Hraban Luyat
      • Warn about duplicate files when syncing
      • Oauth improvements
        • allow auth_url and token_url to be set in the config file
        • Print redirection URI if using own credentials.
      • Don’t Mkdir at the start of sync to save transactions
    • Compile
      • Update build to go1.8.3
      • Require go1.6 for building rclone
      • Compile 386 builds with “GO386=387” for maximum compatibility
    • Bug Fixes
      • Fix menu selection when no remotes
      • Config saving reworked to not kill the file if disk gets full
      • Don’t delete remote if name does not change while renaming
      • moveto, copyto: report transfers and checks as per move and copy
    • Local
      • Add --local-no-unicode-normalization flag - Bob Potter
    • Mount
      • Now supported on Windows using cgofuse and WinFsp - thanks to Bill Zissimopoulos for much help
      • Compare checksums on upload/download via FUSE
      • Unmount when program ends with SIGINT (Ctrl+C) or SIGTERM - Jérôme Vizcaino
      • On read only open of file, make open pending until first read
      • Make --read-only reject modify operations
      • Implement ModTime via FUSE for remotes that support it
      • Allow modTime to be changed even before all writers are closed
      • Fix panic on renames
      • Fix hang on errored upload
    • Crypt
      • Report the name:root as specified by the user
      • Add an “obfuscate” option for filename encryption - Stephen Harris
    • Amazon Drive
      • Fix initialization order for token renewer
      • Remove revoked credentials, allow oauth proxy config and update docs
    • B2
      • Reduce minimum chunk size to 5MB
    • Drive
      • Add team drive support
      • Reduce bandwidth by adding fields for partial responses - Martin Kristensen
      • Implement --drive-shared-with-me flag to view shared with me files - Danny Tsai
      • Add --drive-trashed-only to read only the files in the trash
      • Remove obsolete --drive-full-list
      • Add missing seek to start on retries of chunked uploads
      • Fix stats accounting for upload
      • Convert / in names to a unicode equivalent (/)
      • Poll for Google Drive changes when mounted
    • OneDrive
      • Fix the uploading of files with spaces
      • Fix initialization order for token renewer
      • Display speeds accurately when uploading - Yoni Jah
      • Swap to using http://localhost:53682/ as redirect URL - Michael Ledin
      • Retry on token expired error, reset upload body on retry - Yoni Jah
    • Google Cloud Storage
      • Add ability to specify location and storage class via config and command line - thanks gdm85
      • Create container if necessary on server side copy
      • Increase directory listing chunk to 1000 to increase performance
      • Obtain a refresh token for GCS - Steven Lu
    • Yandex
      • Fix the name reported in log messages (was empty)
      • Correct error return for listing empty directory
    • Dropbox
      • Rewritten to use the v2 API
        • Now supports ModTime
          • Can only set by uploading the file again
          • If you uploaded with an old rclone, rclone may upload everything again
          • Use --size-only or --checksum to avoid this
        • Now supports the Dropbox content hashing scheme
        • Now supports low level retries
    • S3
      • Work around eventual consistency in bucket creation
      • Create container if necessary on server side copy
      • Add us-east-2 (Ohio) and eu-west-2 (London) S3 regions - Zahiar Ahmed
    • Swift, Hubic
      • Fix zero length directory markers showing in the subdirectory listing
        • this caused lots of duplicate transfers
      • Fix paged directory listings
        • this caused duplicate directory errors
      • Create container if necessary on server side copy
      • Increase directory listing chunk to 1000 to increase performance
      • Make sensible error if the user forgets the container
    • SFTP
      • Add support for using ssh key files
      • Fix under Windows
      • Fix ssh agent on Windows
      • Adapt to latest version of library - Igor Kharin

#2

Woah !
Many thanks @ncw for giving us the oportunity to use such a great peace of software !

windows mount !!! wow ! that’s the next killer feature !

Thank you so much !

P.S. : i think it is nessesary to install this software for windows mount -> https://github.com/billziss-gh/winfsp/releases/tag/v1.1


#3

Arg, I don’t believe it. I managed to miss that out of the mount docs!

I’ve added this to the docs

Installing on Windows

To run rclone mount on Windows, you will need to download and install WinFsp.

WinFsp is an open source Windows File System Proxy which makes it easy to write user space file systems for Windows. It provides a FUSE emulation layer which rclone uses combination with cgofuse. Both of these packages are by Bill Zissimopoulos who was very helpful during the implementation of rclone mount for Windows.


#4

Windows mount on Windows 10 doesn’t work for me … WinFSP is installed - i did a reboot after install …

Drive is not showing up …

Command in administrative CLI :
rclone mount gdrivecrypt: X:

Log :

2017/07/23 15:40:22 DEBUG : rclone: Version “v1.37” starting with parameters [“rclone” “mount” “gdrivecrypt:” “X:” “-vv”]
2017/07/23 15:40:26 INFO : Encrypted drive ‘gdrivecrypt:’: Modify window is 1ms
2017/07/23 15:40:26 DEBUG : Encrypted drive ‘gdrivecrypt:’: Mounting on “X:”
2017/07/23 15:40:26 DEBUG : Encrypted drive ‘gdrivecrypt:’: Mounting with options: ["-o" “fsname=gdrivecrypt:” “-o” “subtype=rclone” “-o” “max_readahead=131072” “-o” “uid=-1” “-o” “gid=-1” “–FileSystemName=rclone”]
2017/07/23 15:40:26 DEBUG : Encrypted drive ‘gdrivecrypt:’: Init:
2017/07/23 15:40:26 DEBUG : Encrypted drive ‘gdrivecrypt:’: >Init:
2017/07/23 15:40:26 DEBUG : /: Statfs:
2017/07/23 15:40:26 DEBUG : /: >Statfs: stat={Bsize:4096 Frsize:4096 Blocks:8796093022207 Bfree:8796093022207 Bavail:8796093022207 Files:1000000000 Ffree:1000000000 Favail:0 Fsid:0 Flag:0 Namemax:255}, errc=0
2017/07/23 15:40:26 DEBUG : /: Getattr: fh=0xFFFFFFFFFFFFFFFF
2017/07/23 15:40:26 DEBUG : /: >Getattr: errc=0
The service rclone has been started.
2017/07/23 15:40:26 DEBUG : /: Statfs:
2017/07/23 15:40:26 DEBUG : /: >Statfs: stat={Bsize:4096 Frsize:4096 Blocks:8796093022207 Bfree:8796093022207 Bavail:8796093022207 Files:1000000000 Ffree:1000000000 Favail:0 Fsid:0 Flag:0 Namemax:255}, errc=0
2017/07/23 15:40:26 DEBUG : /: Getattr: fh=0xFFFFFFFFFFFFFFFF
2017/07/23 15:40:26 DEBUG : /: >Getattr: errc=0
2017/07/23 15:40:26 DEBUG : /: Getattr: fh=0xFFFFFFFFFFFFFFFF
2017/07/23 15:40:26 DEBUG : /: >Getattr: errc=0
2017/07/23 15:40:26 DEBUG : /: Opendir:
2017/07/23 15:40:26 DEBUG : /: >Opendir: errc=0, fh=0x1
2017/07/23 15:40:26 DEBUG : /: Releasedir: fh=0x1
2017/07/23 15:40:26 DEBUG : /: >Releasedir: errc=0
2017/07/23 15:40:26 DEBUG : /: Getattr: fh=0xFFFFFFFFFFFFFFFF
2017/07/23 15:40:26 DEBUG : /: >Getattr: errc=0
2017/07/23 15:40:26 DEBUG : /: Getattr: fh=0xFFFFFFFFFFFFFFFF
2017/07/23 15:40:26 DEBUG : /: >Getattr: errc=0
2017/07/23 15:40:26 DEBUG : /: Opendir:
2017/07/23 15:40:26 DEBUG : /: >Opendir: errc=0, fh=0x1
2017/07/23 15:40:26 DEBUG : /: Releasedir: fh=0x1
2017/07/23 15:40:26 DEBUG : /: >Releasedir: errc=0
2017/07/23 15:40:26 DEBUG : /: Getattr: fh=0xFFFFFFFFFFFFFFFF
2017/07/23 15:40:26 DEBUG : /: >Getattr: errc=0
2017/07/23 15:40:26 DEBUG : /: Getattr: fh=0xFFFFFFFFFFFFFFFF
2017/07/23 15:40:26 DEBUG : /: >Getattr: errc=0
2017/07/23 15:40:26 DEBUG : /: Opendir:
2017/07/23 15:40:26 DEBUG : /: >Opendir: errc=0, fh=0x1
2017/07/23 15:40:26 DEBUG : /: Releasedir: fh=0x1
2017/07/23 15:40:26 DEBUG : /: >Releasedir: errc=0


#5

Strange… From the rclone log it looks like it is working.

Can you make please make a new issue on github with as many details as possible in please and I’ll take a look.

Thanks


#6

This is awesome! Thank you everyone who helped!


#7

Congratulations to @ncw for this release.

I agree with @ncw that it is best to report these issues on GitHub and cc me (billziss-gh) for faster resolution. However @FoGBaV 's problem is probably straightforward to resolve.

The issue appears to be the fact that drives created as Administrator are not visible by other accounts (including the account that was elevated as Administrator). So if you start a Windows drive from an Administrative Command Prompt and then try to access the same drive from Explorer (which does not run as Administrator), you will not be able to see the new drive.

The easiest way around this is to start the drive from a normal command prompt. It is also possible to start a drive from the SYSTEM account (using the WinFsp.Launcher infrastructure) which creates drives accessible from everyone on the system.

Additional information re: drive behavior on Windows:

https://msdn.microsoft.com/en-us/library/windows/desktop/aa363908(v=vs.85).aspx

Bill Zissimopoulos


#8

Oh my god ! Thank you very much … that really relsolved the problem.
I was not aware that adminstrator has not the right rights …8)

Sorry for not already posting the problem on GitHub - i still tried to figure out what is wrong with my system(s) …

So … thank you so much … starting CLI with the rights the explorer shell was running - the normal user - resolved the problem !!!

Greetings

FoGBaV


#9

Glad that this was easily resolved :slight_smile:

Bill


#10

Is there anyway of starting the mount without leaving a command prompt window open?


#11

Hi all,

Thanks for the update on the new release! The windows mount in particular is a great step forward , Well done!

Just wanted to quickly share my experience with it, As i was initially getting BSOD errors when attempting to access the mount point in Windows 10.

Found that AVAST antivirus has some kind of trigger in it that caused it to bring the whole system down - The error seemed to indicate a failure with “aswsnx.sys”.

Removing Avast AV has allowed me to proceed past this point but thought that there may be more like me in the same boat so am sharing it here!


#12

@CyberCrime this is possible using the WinFsp.Launcher infrastructure. Some instructions are here (although they require expertise with working on Windows): https://github.com/billziss-gh/winfsp/wiki/WinFsp-Service-Architecture

@zaldre please file a github issue, so that I can track this. I have never seen a BSOD related to WinFsp (at least since it reached release quality), so something like this, even if caused by a third party product is very important. I would like to investigate it and determine if it is the fault of WinFsp and fix it if it is.


#13

@zaldre any chance that you are able to produce a minidump or WinDbg report that demonstrates the issue?


#14

Sure thing @billziss-gh - I’ve got a minidump but it seems not-so-verbose. (indicates an NTOSKRNL.exe error which is fairly non-descript)

RE: Github, On the Rclone or Winfsp github?


#15

Please open an issue with WinFsp and attach the minidump. Please add all relevant information such as use of Avast AV, error message about aswsnx.sys, etc.

BTW, WinFsp includes public symbols in the directory:
\Program Files (x86)\WinFsp\sym

EDIT: if you feel uncomfortable attaching a minidump in a public forum (like GitHub) you can also try running analyze -v from WinDbg, although you would have to set up NTOS symbols properly for this to produce useful info.


#16

Sure thing. I’ll sort it out. Thanks.


#17

@billziss-gh As requested - https://github.com/billziss-gh/winfsp/issues/97


#18

Thank you. I am going on vacation in a couple of days, but I am nevertheless hoping to find some time to look at this soon.


Blue screen when trying to access gdrive rclone mount on windows 10
#19

Will the new —fast-list work on B2 mounts?


#20

Absolutely! Yes it will. B2 was one of the main reasons for the flag.