--files-from seems to filter "from" path instead of taking the paths as-is


#1

Hi there! I’m Alan.

First time posting here. I did a bit of searching in the forum and github issues but couldn’t figure this one out…

I’m using a command like this:

rclone sync -vvv --config /etc/rclone.conf --stats=30s --files-from /path/to/filepaths.txt --copy-links --fast-list --auto-confirm --transfers=100 / gcs:bucketname

/ is my “from” and gcs:bucketname points to a Google Cloud Storage bucket. I want the folder structure to be copied exactly as-is from / to the bucket but ONLY with the paths from the --files-from file.

When using --files-from I would expect rclone to go path by path and transfer those only. Instead, it appears to be scanning the “from” path and comparing it against the paths given by --files-from, as evident by the high-verbosity logs.

Is there a way I can stop it scanning and just accept the paths as-is? It’s super inefficient the way it’s acting right now. I explicitly give it paths to transfer but it’s still scanning every directory and filepath in the “from” path. That can’t be right, right??

What am I doing wrong? Do I need to --exclude=* before my --files-from or something?


#2

I noticed behaviour it seems to scan differently if the paths in --files-from are relative instead of absolute.

Is this normal?


#3

I suspect the --fast-list is causing that - try dropping it.

The paths should be relative to the transfer root. It shouldn’t matter if they start with a leading / or not.


#4

I did some more testing…

The behaviour doesn’t really change when the paths are relative or absolute.

I tried without --fast-list and it’s still scanning the entire filesystem and every single one of my network mounts and everything. :confused:

With verbosity of 2 (-vv), I know this is the case because I see “DEBUG : Excluded from sync (and deletion)” spammed over and over for files that are not in my filelist. If at least it stopped at the / root it’d be tolerable but it does a deep recursive scan to every subfolder.

Does --files-from work as expected for GCS for anybody?


#5

Actually, just checked my terminal output a bit closer and removing --fast-list does fix the issue.

So it seems there’s a bug with --fast-list scanning too much.


#6

I have opened a github issue: https://github.com/ncw/rclone/issues/2639