Rclone Performance Issues

Help and Support
1

What is the problem you are having with rclone?

We're using rclone to mount several S3 buckets within an AWS Appstream instance. Performance issues have started happening where accessing the mounted S3 buckets to save files will cause the process to hang for several minutes. Issues is worse when multiple people are accessing the S3 buckets through the mount.

We have improvement by changing the Rclone cache settings from 1s to 1m (only marginal when multiple users are involved). We have verified with AWS support that the Appstream/S3 services are working properly. Tests with individual EC2’s not running in Appstream have the same end results regarding performance.

We are looking for any suggestions or solutions for increasing performance/reliability.

What is your rclone version (output from rclone version)

rclone v1.55.01

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Windows Server 2012 R2 Standard 64bit

Which cloud storage system are you using? (eg Google Drive)

Amazon S3

The command you were trying to run (eg rclone copy /tmp remote:tmp)

Call C:\rclone\rclone.exe mount s3-bucket:s3-bucket/ X: --vfs-cache-mode full --ignore-checksum --dir-cache-time=1m

Call C:\rclone\rclone.exe mount s3-bucket2:s3-bucket2/ W: --vfs-cache-mode full --ignore-checksum --dir-cache-time=1m

The above commands are run using 2 separate cmd scripts.

The rclone config contents with secrets removed.

[bucket1]
type = s3
provider = AWS
env_auth = true
region = us-east-1
acl = private
server_side_encryption = aws:kms
sse_kms_key_id = secret
storage_class = STANDARD
upload_cutoff = 0

[bucket2]
type = s3
provider = AWS
env_auth = true
region = us-east-1
acl = private
server_side_encryption = aws:kms
sse_kms_key_id = secret
storage_class = STANDARD
upload_cutoff = 0

A log from the command with the -vv flag

https://gist.github.com/hwyatt789/67f0714ff53c947ed319d64693ec7350
2

hello and welcome to the forum,

what is the name of the process?

3

It's usually when trying to access the mount through File Explorer.

4

using windows explorer can be very slow.
i gave up on using that 10+ years ago.

read this about network mode
https://rclone.org/commands/rclone_mount/#mounting-modes-on-windows

5

I'm trying to include the --network-mode flag but when I run the command after adding it nothing happens. Also for more background the commands are being run as cmd scripts. So when the script runs it flashes for a second then immediately closes. The command looks like the following

Call C:\rclone\rclone.exe mount bucket:bucket/ V: --network-mode --vfs-cache-mode full --ignore-checksum --no-modtime --dir-cache-time=1m

I've also tried it without every flag except for --network-mode and get the same result.

6

thanks, but without using a rclone debug log, hard to know what is going on.

7

The command doesn't ever run long enough to collect logs.

I ran it in command line and the error that came up was that --network-mode is not a valid flag.

8

what ever command you ran on the command line, please post it.

enclose it with bakticks so it look like
rclone mount

9 
`rclone mount bucket:bucket/ V: --network-mode`
10

run the command with -vv and post the command with output...

enclose with three backticks so it looks like this.

rclone version -vv
2021/08/01 15:53:55 DEBUG : rclone: Version "v1.56.0" starting with parameters ["rclone" "version" "-vv"]
rclone v1.56.0
- os/version: Microsoft Windows 10 Pro 2009 (64 bit)
- os/kernel: 10.0.19043.1110 (x86_64)
- os/type: windows
- os/arch: amd64
- go/version: go1.16.5
- go/linking: dynamic
- go/tags: cmount
2021/08/01 15:53:55 DEBUG : rclone: Version "v1.56.0" finishing with parameters ["rclone" "version" "-vv"]
11 
PS C:\Users\ImageBuilderTest> C:\rclone\rclone.exe mount -vv bucket:bucket/ W: --network-mode

Error: unknown flag: --network-mode
12

that is strange, i really do not know why...

can you rclone version -vv from that same command prompt

13 
PS C:\Users\ImageBuilderTest> rclone -vv version
2021/08/01 16:33:12 DEBUG : rclone: Version "v1.53.1" starting with parameters ["C:\\rclone\\rclone.exe" "-vv" "version"
]
rclone v1.53.1
- os/arch: windows/amd64
- go version: go1.15
2021/08/01 16:33:12 DEBUG : rclone: Version "v1.53.1" finishing with parameters ["C:\\rclone\\rclone.exe" "-vv" "version
"]
14

when you first posted, you wrote that you were using v1.55.01?
in any event, update to latest stable, v1.56.0 and try again...

15

I honestly thought I was using 1.55, in any case I updated to 1.55 and it started working.

Thanks for the assistance.

16

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.