MacOS mount Tips & Tricks

Help and Support
1

What is the problem you are having with rclone?

Mount is very slow when copying lots of small files. Are there any tips and tricks for handling lots of smaller files? If I compress a directory and copy into the mount it is extremely fast. But copying a code repo, for example, takes a really long time. I know there are restrictions on Google API requests and perhaps that is the bottleneck? Is there any way to tell the mount perhaps to compress directories on the fly?

What is your rclone version (output from rclone version)

rclone v1.52.2-212-g80d2f381-beta
- os/arch: darwin/amd64
- go version: go1.14.4

Which OS you are using and how many bits (eg Windows 7, 64 bit)

MacOS 10.15.4 64bit

Which cloud storage system are you using? (eg Google Drive)

Google Drive

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone mount encrypted:/ ~/rclone --buffer-size 256M -vv

The rclone config contents with secrets removed.

[encrypted]
type = crypt
remote = unlimited:encrypted/share/
filename_encryption = standard
directory_name_encryption = true
password = 

[unlimited]
type = drive
scope = drive
service_account_file = 
team_drive = 
chunk_size = 128M

A log from the command with the -vv flag

N/A

2

I have also tested out cmount but it didn't look to improve the speed of transfering small files.

3

See:

https://rclone.org/drive/#limitations

As you can only copy a few files per second so small files are generally slow on Google Drive.

4

Yeah. Is there perhaps a way to archive using maybe tar if the content added to the mount is a directory? Similar to how we can encrypt files on the fly?

5

A transparent tar would be nice. We haven't got that yet though. You can mount an ISO on an rclone mount so that would be one way!

6

Oh clever! I had a look through the codebase but Goland crashed when opening. I guess it would work by uploading tard chunks. It would only be one directional as well I guess (i.e write only) and then you may as well just use copy/sync.

7

I'm having an experiment with a tar backend. I'll post it if it works out!

1 Like
8

Amazing let me know if you have a working branch :slight_smile: !

9

Any luck on this? :slight_smile:

10

I decided tar wasn't such a good format, so I switched to using zip.

I have made a zip backend and it does work, you can do

rclone copy /path/to/files :zip:drive:file.zip
rclone copy :zip:drive:file.zip /path/to/files

I'm not sure that is useful enough though - what do you think?

11

ahh how come tar wasn't?

Yes looks great! Did you notice any performance increase with a directory full of small files?

12

It works very well with a lot of small files as rclone caches the zip index then just reads the files it needs.

13

Great! Let me know when I can test!

14

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.