If the log doesn’t show the problem then I don’t need to see it!
It is possible --fast-list masks the problem, so if you could try again without that would be very interesting. Feel free to gzip the log to make it smaller.
It is possible --fast-list masks the problem, so if you could try again without that would be very interesting. Feel free to gzip the log to make it smaller.
Ran without –fast-list and limited to just the Ian Dury directory. Again, this time it doesn’t want to re-transfer what it wanted to earlier.
http://www.wywh.com/rclonelogs/Aspire-R1600-2017-06-29_09-52-22.txt
There may be no point in pursuing this until it pops up again either for me or another user…I realize you can’t exactly track down a bug that I can’t consistently demonstrate. But I’m going to do another –dry-run without either of the other two switches just because without them is where I last saw the issue.
But I’m going to do another --dry-run without either of the other two switches just because without them is where I last saw the issue.
And without those switches, the problem is back!
http://www.wywh.com/rclonelogs/Aspire-R1600-2017-06-29_10-03-30.txt
And without those switches, the problem is back!
Unless I limit the backup to just the Ian Dury directory, in which case it all looks fine. Maddening!
Without -vv or –fast-list:
http://www.wywh.com/rclonelogs/Aspire-R1600-2017-06-29_10-24-32.txt
With -vv:
http://www.wywh.com/rclonelogs/Aspire-R1600-2017-06-29_10-28-33.txt
With -vv and –fast-list:
http://www.wywh.com/rclonelogs/Aspire-R1600-2017-06-29_10-32-08.txt
Hmm… It looks like it needs to be the complete sync to show the problem.
Can you keep trying the full sync with -vv, with or without --dry-run I don’t mind, until it shows the problem? The problem has happened if you see “Not moving into backup” in the log.
Often problems are intermittent - hopefully with a full log I can track it down.
I can only think of two reasons for this problem, either
- The local listing sometimes is missing files
- The remote listing sometimes has duplicated files.
A full log with -vv will differentiate those two cases.
You could run /mnt/WD6TBNAS01/ | sort > list 10 times say and see if any of the lists differ.
cd /tmp
for i in `seq 10`; do rclone ls /mnt/WD6TBNAS01/ | sort > list$i ; done
md5sum list*
and see if any of the md5sums differ. You can do the same thing for hubicrypt:WD6TBNAS01/. I’m not hopeful these will actually work, but worth a try!
Hello everybody,
I apologize for my lack of participation (too much happening in RL, and besides I’ve lost access to the setup I was using for the tests), but I’m thrilled to see other people stepping in and helping @ncw fix it for good.
In 2-3 weeks I hope I will be able to get back to uploading stuff w/ rclone, and I will report then following your latest developments.
Cheers,
Durval.
Can you keep trying the full sync with -vv, with or without --dry-run I don’t mind, until it shows the problem? The problem has happened if you see “Not moving into backup” in the log.
It happened again on the first try, once again with the Ian Dury directory:
http://www.wywh.com/rclonelogs/Aspire-R1600-2017-06-29_14-39-13.txt
The relevant messages are right at the bottom.
I ran your very clever check for differing MD5 sums locally and saw no issues. I submitted one against the hubiC remote, but unsurprisingly it’s taking a long time.
I should have mentioned right up-front that I’ve seen this behavio[u]r before but never made the time to report it. Interestingly, it seems to consistently affect a particular directory or set of directories…for a while. Today even over the course of several hours, it’s happening with the Ian Dury directory, but in the past I’ve seen it happen over and over with several others. As I said before, the fix was to run the sync with v1.35, though now that we’ve seen this work and fail and fail and work it’s probably not a safe assumption that the older version will always be a reliable workaround.
I’ve been running a very similar job locally using rsync to back up to two other NASes for years with no issues, but of course that doesn’t mean this problem is due to rclone, it may well be something on hubiC’s end.
When I run ls -LRla “/mnt/WD6TBNAS01/Squeezebox Music/NotDTS/Ian Dury” I get:
/mnt/WD6TBNAS01/Squeezebox Music/NotDTS/Ian Dury: total 0 drwxr-xr-x 2 administrator root 0 Jul 19 2011 . drwxr-xr-x 2 administrator root 0 Apr 30 07:04 .. drwxr-xr-x 2 administrator root 0 Jul 19 2011 1978-03-22 San Francisco FM broadcast from 1st generation cassette drwxr-xr-x 2 administrator root 0 Feb 11 14:59 Ian Dury & The Music Students drwxr-xr-x 2 administrator root 0 Nov 25 2016 Live 78 Speed Corrected drwxr-xr-x 2 administrator root 0 Jun 13 2016 Sex & Drugs & Rock 'n' Roll; The Best of Ian Dury and the Blockheads /mnt/WD6TBNAS01/Squeezebox Music/NotDTS/Ian Dury/1978-03-22 San Francisco FM broadcast from 1st generation cassette: total 256032 drwxr-xr-x 2 administrator root 0 Jul 19 2011 . drwxr-xr-x 2 administrator root 0 Jul 19 2011 .. -rwxr-xr-x 1 administrator root 28347752 Jul 19 2011 01 1978-03-22 Sex And Drugs And Rock And Roll.flac -rwxr-xr-x 1 administrator root 18611825 Jul 19 2011 02 1978-03-22 I'm Partial To Your Abacadabra.flac -rwxr-xr-x 1 administrator root 26876204 Jul 19 2011 03 1978-03-22 Wake Up And Make Love To Me.flac -rwxr-xr-x 1 administrator root 42167396 Jul 19 2011 04 1978-03-22 Clever Trevor.flac -rwxr-xr-x 1 administrator root 27290988 Jul 19 2011 05 1978-03-22 If I Was With A Woman.flac -rwxr-xr-x 1 administrator root 21904760 Jul 19 2011 06 1978-03-22 Billericay Dickie.flac -rwxr-xr-x 1 administrator root 28491385 Jul 19 2011 07 1978-03-22 Sweet Gene Vincent.flac -rwxr-xr-x 1 administrator root 68469550 Jul 19 2011 08 1978-03-22 I Made Mary Cry - Blockheads.flac -rwxr-xr-x 1 administrator root 724 Jul 19 2011 1978-03-22.ffp -rwxr-xr-x 1 administrator root 1471 Jul 19 2011 1978-03-22.txt drwxr-xr-x 2 administrator root 0 Jul 19 2011 scans /mnt/WD6TBNAS01/Squeezebox Music/NotDTS/Ian Dury/1978-03-22 San Francisco FM broadcast from 1st generation cassette/scans: total 6228 drwxr-xr-x 2 administrator root 0 Jul 19 2011 . drwxr-xr-x 2 administrator root 0 Jul 19 2011 .. -rwxr-xr-x 1 administrator root 346327 Jul 19 2011 1978-03-22 San Francisco.jpg -rwxr-xr-x 1 administrator root 644525 Jul 19 2011 1978-04-22 Passaic backstage pass.jpg -rwxr-xr-x 1 administrator root 2267225 Jul 19 2011 1978-06-02 Glasgow flyer.jpg -rwxr-xr-x 1 administrator root 432457 Jul 19 2011 1978-06-02 Glasgow.jpg -rwxr-xr-x 1 administrator root 683455 Jul 19 2011 1984-03-24 Ian Dury Backstage Pass Brixton Academy London.jpg -rwxr-xr-x 1 administrator root 942153 Jul 19 2011 Ian Dury.jpg -rwxr-xr-x 1 administrator root 1047599 Jul 19 2011 The Blockheads.jpg /mnt/WD6TBNAS01/Squeezebox Music/NotDTS/Ian Dury/Ian Dury & The Music Students: total 20472 drwxr-xr-x 2 administrator root 0 Feb 11 14:59 . drwxr-xr-x 2 administrator root 0 Jul 19 2011 .. -rwxr-xr-x 1 administrator root 337 Oct 18 2010 Ian Dury Noddy Notes.txt -rwxr-xr-x 1 administrator root 20958012 Feb 11 14:59 Noddy.flac /mnt/WD6TBNAS01/Squeezebox Music/NotDTS/Ian Dury/Live 78 Speed Corrected: total 331612 drwxr-xr-x 2 administrator root 0 Nov 25 2016 . drwxr-xr-x 2 administrator root 0 Jul 19 2011 .. -rwxr-xr-x 1 administrator root 22625812 Feb 26 2011 01.flac -rwxr-xr-x 1 administrator root 27733593 Apr 25 2011 02.flac -rwxr-xr-x 1 administrator root 18822178 Apr 25 2011 03.flac -rwxr-xr-x 1 administrator root 17563952 Apr 25 2011 04.flac -rwxr-xr-x 1 administrator root 25856518 Apr 25 2011 05.flac -rwxr-xr-x 1 administrator root 18186248 Apr 25 2011 06.flac -rwxr-xr-x 1 administrator root 23992512 Apr 25 2011 07.flac -rwxr-xr-x 1 administrator root 29296586 Apr 25 2011 08.flac -rwxr-xr-x 1 administrator root 27389123 Apr 25 2011 09.flac -rwxr-xr-x 1 administrator root 23573974 Apr 25 2011 10.flac -rwxr-xr-x 1 administrator root 35313616 Apr 25 2011 11.flac -rwxr-xr-x 1 administrator root 20533165 Apr 25 2011 12.flac -rwxr-xr-x 1 administrator root 21701924 Apr 25 2011 13.flac -rwxr-xr-x 1 administrator root 26931732 Apr 25 2011 14.flac -rwxr-xr-x 1 administrator root 14475 Feb 26 2011 Folder.jpg -rwxr-xr-x 1 administrator root 2085 Feb 26 2011 ID.txt /mnt/WD6TBNAS01/Squeezebox Music/NotDTS/Ian Dury/Sex & Drugs & Rock 'n' Roll; The Best of Ian Dury and the Blockheads: total 442368 drwxr-xr-x 2 administrator root 0 Jun 13 2016 . drwxr-xr-x 2 administrator root 0 Jul 19 2011 .. -rwxr-xr-x 1 administrator root 20833876 Nov 7 2009 01 - Sex & Drugs & Rock & Roll.flac -rwxr-xr-x 1 administrator root 19074794 Nov 7 2009 02 - Razzle in My Pocket (A True Story).flac -rwxr-xr-x 1 administrator root 29496743 Nov 7 2009 03 - Wake Up and Make Love With Me.flac -rwxr-xr-x 1 administrator root 24126719 Nov 7 2009 04 - Sweet Gene Vincent.flac -rwxr-xr-x 1 administrator root 27758488 Nov 7 2009 05 - Billericay Dickie.flac -rwxr-xr-x 1 administrator root 32097488 Nov 7 2009 06 - Clevor Trever.flac -rwxr-xr-x 1 administrator root 27759387 Nov 7 2009 07 - Blockheads.flac -rwxr-xr-x 1 administrator root 18239768 Nov 7 2009 08 - You're More Than Fair.flac -rwxr-xr-x 1 administrator root 22727411 Nov 7 2009 09 - What a Waste!.flac -rwxr-xr-x 1 administrator root 28430111 Nov 7 2009 10 - Hit Me With Your Rhythm Stick.flac -rwxr-xr-x 1 administrator root 19176760 Nov 7 2009 11 - There Ain't Half Been Some Clever Bastards.flac -rwxr-xr-x 1 administrator root 37397979 Nov 7 2009 12 - Inbetweenies.flac -rwxr-xr-x 1 administrator root 33728097 Nov 7 2009 13 - Reasons to Be Cheerful, Part 3.flac -rwxr-xr-x 1 administrator root 27979005 Nov 7 2009 14 - Common as Muck.flac -rwxr-xr-x 1 administrator root 24236475 Nov 7 2009 15 - I Want to Be Straight.flac -rwxr-xr-x 1 administrator root 16463907 Nov 7 2009 16 - That's Not All.flac -rwxr-xr-x 1 administrator root 21398425 Nov 7 2009 17 - Superman's Big Sister.flac -rwxr-xr-x 1 administrator root 21994744 Nov 7 2009 18 - You'll See Glimpses.flac -rwxr-xr-x 1 administrator root 18708 Nov 7 2009 Folder.jpg -rwxr-xr-x 1 administrator root 7034 Nov 7 2009 Ian Dury - Sex & Drugs & Rock 'n' Roll; The Best of Ian Dury and the Blockheads.txt
When I run rclone ls “hubicrypt:WD6TBNAS01/Squeezebox Music/NotDTS/Ian Dury” I get:
22625812 Live 78 Speed Corrected/01.flac
27733593 Live 78 Speed Corrected/02.flac
18822178 Live 78 Speed Corrected/03.flac
17563952 Live 78 Speed Corrected/04.flac
25856518 Live 78 Speed Corrected/05.flac
18186248 Live 78 Speed Corrected/06.flac
23992512 Live 78 Speed Corrected/07.flac
29296586 Live 78 Speed Corrected/08.flac
27389123 Live 78 Speed Corrected/09.flac
23573974 Live 78 Speed Corrected/10.flac
35313616 Live 78 Speed Corrected/11.flac
20533165 Live 78 Speed Corrected/12.flac
21701924 Live 78 Speed Corrected/13.flac
26931732 Live 78 Speed Corrected/14.flac
14475 Live 78 Speed Corrected/Folder.jpg
2085 Live 78 Speed Corrected/ID.txt
337 Ian Dury & The Music Students/Ian Dury Noddy Notes.txt
20958012 Ian Dury & The Music Students/Noddy.flac
28347752 1978-03-22 San Francisco FM broadcast from 1st generation cassette/01 1978-03-22 Sex And Drugs And Rock And Roll.flac
18611825 1978-03-22 San Francisco FM broadcast from 1st generation cassette/02 1978-03-22 I'm Partial To Your Abacadabra.flac
26876204 1978-03-22 San Francisco FM broadcast from 1st generation cassette/03 1978-03-22 Wake Up And Make Love To Me.flac
42167396 1978-03-22 San Francisco FM broadcast from 1st generation cassette/04 1978-03-22 Clever Trevor.flac
27290988 1978-03-22 San Francisco FM broadcast from 1st generation cassette/05 1978-03-22 If I Was With A Woman.flac
21904760 1978-03-22 San Francisco FM broadcast from 1st generation cassette/06 1978-03-22 Billericay Dickie.flac
28491385 1978-03-22 San Francisco FM broadcast from 1st generation cassette/07 1978-03-22 Sweet Gene Vincent.flac
68469550 1978-03-22 San Francisco FM broadcast from 1st generation cassette/08 1978-03-22 I Made Mary Cry - Blockheads.flac
724 1978-03-22 San Francisco FM broadcast from 1st generation cassette/1978-03-22.ffp
1471 1978-03-22 San Francisco FM broadcast from 1st generation cassette/1978-03-22.txt
20833876 Sex & Drugs & Rock 'n' Roll; The Best of Ian Dury and the Blockheads/01 - Sex & Drugs & Rock & Roll.flac
19074794 Sex & Drugs & Rock 'n' Roll; The Best of Ian Dury and the Blockheads/02 - Razzle in My Pocket (A True Story).flac
29496743 Sex & Drugs & Rock 'n' Roll; The Best of Ian Dury and the Blockheads/03 - Wake Up and Make Love With Me.flac
24126719 Sex & Drugs & Rock 'n' Roll; The Best of Ian Dury and the Blockheads/04 - Sweet Gene Vincent.flac
27758488 Sex & Drugs & Rock 'n' Roll; The Best of Ian Dury and the Blockheads/05 - Billericay Dickie.flac
32097488 Sex & Drugs & Rock 'n' Roll; The Best of Ian Dury and the Blockheads/06 - Clevor Trever.flac
27759387 Sex & Drugs & Rock 'n' Roll; The Best of Ian Dury and the Blockheads/07 - Blockheads.flac
18239768 Sex & Drugs & Rock 'n' Roll; The Best of Ian Dury and the Blockheads/08 - You're More Than Fair.flac
22727411 Sex & Drugs & Rock 'n' Roll; The Best of Ian Dury and the Blockheads/09 - What a Waste!.flac
28430111 Sex & Drugs & Rock 'n' Roll; The Best of Ian Dury and the Blockheads/10 - Hit Me With Your Rhythm Stick.flac
19176760 Sex & Drugs & Rock 'n' Roll; The Best of Ian Dury and the Blockheads/11 - There Ain't Half Been Some Clever Bastards.flac
37397979 Sex & Drugs & Rock 'n' Roll; The Best of Ian Dury and the Blockheads/12 - Inbetweenies.flac
33728097 Sex & Drugs & Rock 'n' Roll; The Best of Ian Dury and the Blockheads/13 - Reasons to Be Cheerful, Part 3.flac
27979005 Sex & Drugs & Rock 'n' Roll; The Best of Ian Dury and the Blockheads/14 - Common as Muck.flac
24236475 Sex & Drugs & Rock 'n' Roll; The Best of Ian Dury and the Blockheads/15 - I Want to Be Straight.flac
16463907 Sex & Drugs & Rock 'n' Roll; The Best of Ian Dury and the Blockheads/16 - That's Not All.flac
21398425 Sex & Drugs & Rock 'n' Roll; The Best of Ian Dury and the Blockheads/17 - Superman's Big Sister.flac
21994744 Sex & Drugs & Rock 'n' Roll; The Best of Ian Dury and the Blockheads/18 - You'll See Glimpses.flac
18708 Sex & Drugs & Rock 'n' Roll; The Best of Ian Dury and the Blockheads/Folder.jpg
7034 Sex & Drugs & Rock 'n' Roll; The Best of Ian Dury and the Blockheads/Ian Dury - Sex & Drugs & Rock 'n' Roll; The Best of Ian Dury and the Blockheads.txt
346327 1978-03-22 San Francisco FM broadcast from 1st generation cassette/scans/1978-03-22 San Francisco.jpg
644525 1978-03-22 San Francisco FM broadcast from 1st generation cassette/scans/1978-04-22 Passaic backstage pass.jpg
2267225 1978-03-22 San Francisco FM broadcast from 1st generation cassette/scans/1978-06-02 Glasgow flyer.jpg
432457 1978-03-22 San Francisco FM broadcast from 1st generation cassette/scans/1978-06-02 Glasgow.jpg
683455 1978-03-22 San Francisco FM broadcast from 1st generation cassette/scans/1984-03-24 Ian Dury Backstage Pass Brixton Academy London.jpg
942153 1978-03-22 San Francisco FM broadcast from 1st generation cassette/scans/Ian Dury.jpg
1047599 1978-03-22 San Francisco FM broadcast from 1st generation cassette/scans/The Blockheads.jpg
Hmmm…just to use one particular file as an example, here are the three references to it in the log.
First encounter seems to make it clear that it’s found locally and remotely:
2017/06/29 14:49:53 DEBUG : Squeezebox Music/NotDTS/Ian Dury/Sex & Drugs & Rock 'n' Roll; The Best of Ian Dury and the Blockheads/13 - Reasons to Be Cheerful, Part 3.flac: Size of src and dst objects identical 2017/06/29 14:49:53 DEBUG : Squeezebox Music/NotDTS/Ian Dury/Sex & Drugs & Rock 'n' Roll; The Best of Ian Dury and the Blockheads/13 - Reasons to Be Cheerful, Part 3.flac: Unchanged skipping
But then:
2017/06/29 14:58:10 NOTICE: Squeezebox Music/NotDTS/Ian Dury/Sex & Drugs & Rock 'n' Roll; The Best of Ian Dury and the Blockheads/13 - Reasons to Be Cheerful, Part 3.flac: Not moving into backup dir as --dry-run
Thanks for your logs - very useful!
What this looks most like is that there are duplicated files on the destination. This explains what @durval is seeing perfectly because there is a known problem with Google drive which duplicates files. However swift doesn’t normally duplicate files…
Here is an example of a run through with two identically named files on google drive (Note I used --size-only because the mod-times are different on the files). The logs have exactly the same pattern as what @atrocity is seeing.
$ rclone -vv --dry-run --size-only --delete-after --backup-dir drive:duplicate-backup sync src drive:duplicate
2017/06/30 15:35:55 DEBUG : rclone: Version "v1.36-230-gdc56ad98" starting with parameters ["rclone" "-vv" "--dry-run" "--size-only" "--delete-after" "--backup-dir" "drive:duplicate-backup" "sync" "src" "drive:duplicate"]
2017/06/30 15:35:56 INFO : Google drive root 'duplicate': Modify window is 1ms
2017/06/30 15:35:57 DEBUG : duplicate.txt: Sizes identical
2017/06/30 15:35:57 DEBUG : duplicate.txt: Unchanged skipping
2017/06/30 15:35:57 INFO : Google drive root 'duplicate': Waiting for checks to finish
2017/06/30 15:35:57 INFO : Google drive root 'duplicate': Waiting for transfers to finish
2017/06/30 15:35:57 INFO : Waiting for deletions to finish
2017/06/30 15:35:57 NOTICE: duplicate.txt: Not moving into backup dir as --dry-run
2017/06/30 15:35:57 INFO :
Transferred: 0 Bytes (0 Bytes/s)
Errors: 0
Checks: 2
Transferred: 0
Elapsed time: 1.7s
2017/06/30 15:35:57 DEBUG : Go routines at exit 6
2017/06/30 15:35:57 DEBUG : rclone: Version "v1.36-230-gdc56ad98" finishing with parameters ["rclone" "-vv" "--dry-run" "--size-only" "--delete-after" "--backup-dir" "drive:duplicate-backup" "sync" "src" "drive:duplicate"]
In Issue #1506 I’m going to put back the duplicate file detection code which will output a warning if it detects duplicate files.
In the mean time, @atrocity can you collect a log with -vv --dry-run --dump-bodies showing the error. This log really will be massive! Can you run it through
grep -v X-Auth-Token: dirty.log > clean.log
To remove the swift Auth tokens. However it will provide definitive proof that hubic is or isn’t duplicating files in the listing!
I’ve just run a job as you requested, but of course the error is not happening this time, even after two runs! I wonder…if the issue is on hubiC’s end, is it possible that I’m running into some conflict with some type of maintenance on their end? If so, I wonder if the error isn’t going to crop up again until I attempt to make an actual change.
I’ve just added a couple of cover art files and am running a live backup, after which I’ll try another super-verbose dry run. Today is going to be busy, though…can’t promise I’ll be swift.
Thank you for looking into this!
@atrocity thanks for your help tracking this down! Duplicate files or a bug in the core sync routines are my top candidates right now!
And of course, as soon as I did a live backup WITHOUT all the verbose text, the problem came up again!
http://www.wywh.com/rclonelogs/Aspire-R1600-2017-06-30_08-53-01.txt
Have just started another verbose dry run.
Here is a beta which will log duplicate files in the sync - it would be interesting if you could use that instead.
https://beta.rclone.org/v1.36-231-g575e779b/ (uploaded in 15-30 mins)
I’m repeatedly trying with the new beta and am almost convinced that turning on the additional logging data prevents the problem from occurring (or it’s not getting logged in a way I can read). After multiple attempts (live and dry run) generating big logs but no obvious errors I decided on a whim to try a dry run without the additional logging and I’m again seeing files rclone claims it isn’t copying only because it’s a dry run. It appears to really like to pick on British singers named “Ian” as this time it’s saying that “Ian Dury” (the entire directory?) is a duplicate while also saying it wants to re-copy all the Ian Hunter even though it doesn’t need to:
http://www.wywh.com/rclonelogs/Aspire-R1600-2017-06-30_14-37-03.txt
This is the output from rclone ls “hubicrypt:WD6TBNAS01/Squeezebox Music/NotDTS/Ian Hunter”:
http://www.wywh.com/rclonelogs/IanHunterDir.txt
And the ouptut from rclone ls “hubicrypt:WD6TBNAS01/Squeezebox Music/NotDTS/Ian Dury”:
http://www.wywh.com/rclonelogs/IanDuryDir.txt
Here’s the cleaned and gzipped log from a subsequent heavily-commented run which I don’t believe shows the problem:
http://www.wywh.com/rclonelogs/Aspire-R1600-2017-06-30_14-56-20_clean.txt.gzip
I’d say that the fact it said Squeezebox Music/NotDTS/Ian Dury: Duplicate file found in destination - ignoring is evidence that the directory listing from hubic was corrupt, or rclone corrupted it. What it looks like to me is that instead of getting “Ian Dury”, “Ian Hunter” in the listing, it somehow got “Ian Dury”, “Ian Dury” which a) explains the duplicate and b) explains why all of “Ian Hunter” was missing.
I couldn’t see anything wrong with the listings., however there was one problem in the verbose log
2017/06/30 15:05:47 NOTICE: Squeezebox Music/NotDTS/Randy Newman/everyone_youve_ever_heard_of_19780211/Folder.jpg: Not copying as --dry-run
I thought that would be the key to crack it, but I forgot the directory listings were encrypted! So if you could run
rclone ls "hubicrypt:WD6TBNAS01/Squeezebox Music/NotDTS/Randy Newman/everyone_youve_ever_heard_of_19780211/Folder.jpg" --crypt-show-mapping
And tell me what filename on Hubic “hubicrypt:WD6TBNAS01/Squeezebox Music/NotDTS/Randy Newman/everyone_youve_ever_heard_of_19780211/Folder.jpg” maps to, then we can have a look in the verbose logs and see whether it is in the listing from hubic or not. If it is in the listing then it is probably an rclone problem, if not then it is a hubic problem.
I note that the last experiment used --fast-list which may or may not be more likely to show the problem.
I may have made things more confusing than they need to be. In the case of Folder.jpg for Randy Newman, I had arbitrarily picked that file to “sacrifice” in order to give rclone something to either upload or, in the case of a dry run, believe it needed to upload. I was trying to see if the outcome was different for runs that did or did not need to copy files.
As it stands at the moment, Folder.jpg is still not in that particular directory because I never did a live copy. But I will right now.
Just did a live backup that needed to copy the Randy Newman Folder.jpg. I don’t see anything unusual in the verbose log, which is at http://www.wywh.com/rclonelogs/Clean2.txt.gzip
However, I immediately did another live backup, this time non-verbose, and it appears to have copied several Ian Hunter files, even though the prior log didn’t show them as moved (that I saw) or missing or whatever. The log from that run is at http://www.wywh.com/rclonelogs/Aspire-R1600-2017-07-01_09-43-06.txt
I’ve also included the crypt mapping output for all the Ian Hunter files at http://www.wywh.com/rclonelogs/IanHunterCrypt.txt
This morning’s cron-scheduled dry run is reporting a combination of files that legitimately need to be uploaded (the Godfathers and Traffic) as well as Ian Hunter and now the Squirrels files that haven’t changed locally:
http://www.wywh.com/rclonelogs/Aspire-R1600-2017-07-02_06-02-33.txt
@atrocity I’ve had a look through your logs, unfortunately none of them have quite got what I need 
What I need is a run done with -vv --dump-bodies but without --fast-list. I don’t mind whether it is --dry-run or not. And if it does show a problem then I need the crypt mappings for the directory it declares as duplicate.
That will let me nail down whether this is an rclone or a hubic problem.
http://www.wywh.com/rclonelogs/Clean3.gz is a log for a dry run with the requested parameters. It shows that it wants to copy the Ian Hunter files again, so I’ve placed the Ian Hunter crypt mappings at http://www.wywh.com/rclonelogs/crypt.txt.
Just in case it helps, http://www.wywh.com/rclonelogs/localdir.txt has the results of ls -LRla “/mnt/WD6TBNAS01/Squeezebox Music/NotDTS/Ian Hunter”, mainly just to show that these really are old files that shouldn’t need re-uploading.
Thank you again!