Rclone.exe sync (with swiftstack API) filtering all data through pagefile.sys


#1

We are attempting to solve a problem. We are using rclone.exe sync command with an API from our customers vendor swiftstack. We just upgraded the server on the backend storage to SSD drives, and we have 24 CPU’s and 64GB Of ram on a virtual server.

We have over 50TB of data with hundreds of millions of files and directories. I know we have a few bottlenecks with directory checking and validation between the local drive and the remote swiftstack appliance. At the current time we are running a version of 1.42 from the vendor. At the current time per the screenshot below you can see that the server is using C:\pagefile.sys to process and facilitate the transfers and not utilizing the physical ram. This is a VM on a new infrastructure.

IS there a way to have rclone.exe sync bypass loading into the pagefile.sys? Anyone have any suggestions? We could turn off the pagefile.sys though I think just letting windows manage it would be a better options.

Thoughts?

Screenshot of server load on disk and pagefile.sys being used with rclone.exe


#2

Why not the latest version?

I’m not seeing where it isn’t using memory as it an application would hit the pagefile once the memory is exhausted. Depending on your setup and how you have files, it will quite a bit of memory to get the file information depending on how you have the file structure setup.


#3

How many files do you have in a directory? Rclone has to load one directory worth of objects into memory at a time. Each object can take ~1k of memory.

Also try a more recent version from the latest release.

Which backend are you using? s3 or swift?

And what is your command line?

Thanks


#4

the company that makes swiftstack provided it. I would need to ask them.


#5

I would have to ask swiftstack, they say it’s built with their API baked in.


#6

We have 64GB of ram we are barely using 50% of it. So the pagefile.sys is rather large so I believe we need to go with a system managed file. We are speaking to Microsoft to see their reccomendation and assure understanding of how processes utilize the swap file.


#7

How much memory is rclone using? I can’t tell from your screenshot