Rclone serve vs plain SFTP for millions of small files

Help and Support
1

I'll migrate ~5TB of millions of small files from AWS EFS to a selfhosted server using rclone. Should I use rclone serve on the destination for better performance with small files or is plain SFTP good enough? Curious if anyone has experience / suggestion for this kind of workload.

Related: Does anyone already compare the performance of rclone vs SFTP in term of uploading speed?

rclone 1.75.0

Thanks.

2

hi, sorry, not sure what you need help with?

  1. how to perform the migration?
  2. how to rclone serve the files after the migration has completed?

imo, i would trust not rclone serve sftp in production.

plain sftp server is battle tested over many decades and many millions of machines.

one possible advantage of rclone serve sftp is pre-caching the names of files and directories in memory, assuming you have enough free memory
i have a summary of the two rclone vfs caches

3

That is a lot of small files...

I'm not sure I understand the architecture though.

You have files on EFS and you'll run an EC2 VM to access the files. These appear as an NFS mount, so effectively as local disk as far as rclone is concerned.

You also have a server not at AWS where you want the files to end up.

Assuming your destination server has ssh access then I'd run rclone on the AWS server and push the files over sftp to the dest server. No need to run rclone server .

You would then run rclone check as a final validation that everything transferred properly

4

Sorry, I complicate the question by mentioning EFS. I was trying to ask "is rclone serve sftp is more efficient then plain sftp". I'm planning to do my migration exactly as @ncw wrote. And by @asdffdsa 's answer I've learned that rclone serve sftp with dir-cache can help but needs testing on my side.

I'm long time rclone user but never noticed the serve sftp option so i was curious. It seems it's better to use plain sftp if we already have it.

Thanks for the both answers.

5

Alternatively, pack the files into a tar stream, send them over the network, and unpack them instantly at the destination without writing a intermediate archive file:

$ tar cf - source_dir/ | ssh user@remote 'tar xf - -C destination_dir/'

The overhead for millions of small files can be enormous - perhaps run a couple of tests with a million empty files, just to get a feeling of how your environment and setup handles the overhead.

$ mkdir transfer_test; for (( i = 0; i < 1000000; i++ )) do > transfer_test/file_$i; done

If you can't make ssh work, streaming over netcat could perhaps be another way.

# nc -l -p 8080 | tar -xvf - -C /destination/path/

# tar -cvf - /source/path/ | nc target-host-ip 8080