Just an interesting Bug. I have just uploaded a heap of data from my system to ACD with rclone copy. I used the --delete-excluded just to be safe, although the documents state copy doesn’t do any deleting.
Anyway, all my data done, if I now run the following command to see if anything new (dry run)
All good there, I want to schedule this take, I decided I should drop the --delete-excluded to simplify the command. But it appears there is a bug, the dry run tells me it is going to re-upload most of my content.
A side effect of using --delete-excluded when using copy will cause it to read all the files on the destination and not apply the filters.
Without using it, the filters are applied on the destination too.
I think what is happening is that because ACD doesn’t store modtime, in the second command all the files you’ve uploaded are appearing as younger than 14d and hence not being listed. If they aren’t being listed, then they aren’t appearing and need to be transferred according to rclone.
You can confirm this by doing
rclone ls --min-age 14d /mnt/user/Media
and comparing it to
rclone ls --min-age 14d secret:media
There should only be 25 files in the second listing.
As to whether this is a bug, I’m not sure. It certainly isn’t obvious that --delete-excluded behaves like that, nor that --min-age doesn’t work properly on a destination remote which doesn’t support mod times…
OK, so my initial copy didn’t include a --min-age, but I want to keep it on my cron run, to give local files time to stabilise before being uploaded.
You are correct, rclone ls --min-age 14d /mnt/user/Media results in only 25 files. I was thinking with a copy that it would only filter local files with the min-age.
My initial sync was only done over the last week, finishing a couple of days ago. Am I correct in assuming if I wait 14 days from my sync, I will then be able to drop the --delete-excluded and it will run as expected as remote files will be older than 14 days.
I don’t think that is an unreasonable expectation. I’m not sure I want to change the sync mechanism though as that does the right thing for remotes which support mod time.
I have been seeding a new folder, around 2Tb worth, I had it run for a few days and had about 1.2Tb uploaded, I stopped it to do a few things with my config and started again, note: this time no --delete-excluded being used at any point this time.
So I have been seeding for two more days and wondering why my total file count hasn’t been moving, it has been re-uploading the same files because of the --min-age 14d applying to the destination. It just doesn’t seem right, I think it will catch a heap of people out when initially seeding, It wouldn’t be an issue with a normal rsync as the destination files would have the same timestamp as source , but at list with ACD this is not the case.
As I see it, if I cron a task with min-age 14d, once the local file reaches 14d+ it will get uploaded, but it will also be continually uploaded after that date, as each day it runs it will re-upload as the remote will be filtered out by 14d, the timestamp will then get reset, and it will upload again the next day.
The problem is you are trying to do a copy/sync which relies on modification times to a remote which doesn’t support them.
That is the filter (--min-age) is applied to the destination directory traversal combined with ACD not supporting modification times causes the problem.
If ACD supported mod times then this wouldn’t be a problem.
However it isn’t obvious that the copy relies on modtime working on the destination, I agree.
I could produce a warning here which might not be a bad idea.
Here is an idea for a work-around
for a remote which doesn’t support modtime used as the destination of a sync, copy or move
and the user has specified age based filters (eg --min-age, --max-age)
disallow it as a destination for sync - that just isn’t going to work
for move and copy ignore the age based filters on the destination
(note that internally to rclone sync/copy/move are almost the same thing)
That sounds like a good solution to me, at the moment using --delete-excluded seems to work around the issue, but it would be nice if it played nicer with ACD.
I would like to have my local files delayed in uploading, giving them 14d to stabilise, they should then stay unchanged, I am using a copy rather than sync as I plan on then moving my local copies in say a year or two, to offline storage and want to keep the ACD versions as is.
And daily (once a day at midnight): rclone move /local/Media acd:Media -v --no-traverse --size-only --min-age 1d
Going through the logs I noticed that new files were successfully being copied over to my ACD as they came in. BUT when the nightly move command was called, it was re-transferring files that already existed on my ACD. The expected behavior was that it would detect that they existed on the remote and simply delete them locally.
After troubleshooting my setup for a few hours, I found this thread. Thank you @Wob76 for noting your observations and @ncw for providing a very thorough explanation and laying the groundwork for a workaround. I see that the bug has a milestone of v1.37, so I’ll simply stop using the --min-age option for now. Thanks!
Until @ncw implements the enhancement, there is a workaround for copy/move if you add the --delete-excluded option to your command(s). That forces rclone to iterate through all files on the remote (regardless of age) in the comparison with the local files. Make sure to first test using the --dry-run option to make sure the results are what you are expecting.
I have been using this option for months with the rclone copy command with both --min-age and --max-age options and it works great. I have not tested it with the move command but it should work the same.