rClone appears to have ignored "--include-from" statement once

Help and Support
1

I've written a script for downloading items using SFTP over rclone, and it has worked pretty well so far. However, earlier today I ran a transfer and noticed that it was attempting to transfer files that were not on the download list. This is the command that was running:

rclone copy --include-from "$LOGFOLDER"/Current.txt "$REMOTE" "$DEST"

I noticed there was a problem because one of the files couldn't be transferred because it was no longer there. I checked the Current.txt file and the file that caused the error was not listed in that document, so it shouldn't have been downloading.

It seems like it's working okay now. This appears to have happened out of the blue which makes it even more frustrating to figure out why.

rclone v1.62.2
- os/version: slackware 15.0+ (64 bit)
- os/kernel: 5.19.17-Unraid (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.20.2
- go/linking: static
- go/tags: none

Which cloud storage system are you using? (eg Google Drive)

I'm using SFTP from a seedbox

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone copy --include-from "$LOGFOLDER"/Current.txt "$REMOTE" "$DEST"

The rclone config contents with secrets removed.

[USBX]
type = sftp
host = hostname
user = username
pass = password
shell_type = unix
md5sum_command = md5sum
sha1sum_command = sha1sum

A log from the command with the -vv flag

I'm not sure if it's feasible to do this here. Each download command typically takes at least 8 hours to complete and the output would be extremely long. I can try to do that if I need to. Here is a very small sample of the output when using -v which is currently running.

Download date is 2023-03-26T18:46:51+00:00, previous download was 2000-01-01
Previous download list found. Downloading only new files from Seedbox.
2023/03/26 14:47:59 INFO  : 
Transferred:   	  143.840 MiB / 74.363 GiB, 0%, 2.840 MiB/s, ETA 7h26m2s
Checks:                29 / 29, 100%
Transferred:            0 / 162, 0%
Elapsed time:       1m2.3s
Transferring:
 * Casserine_-_We_Gotta_Run_(ntsc_cda_astro).vob:  8% /288.982Mi, 474.066Ki/s, 9m34s
 *                            Fantaghirò 1.1.mkv:  1% /3.539Gi, 1.052Mi/s, 56m37s
 *      Cat Stevens - Wild world (PAL-DD2.0).VOB: 14% /190.781Mi, 333.586Ki/s, 8m22s
 *                            Fantaghirò 1.2.mkv:  1% /3.496Gi, 1019.802Ki/s, 59m11s

2023/03/26 14:48:59 INFO  : 
Transferred:   	  304.527 MiB / 81.066 GiB, 0%, 2.578 MiB/s, ETA 8h54m36s
Checks:                29 / 29, 100%
Transferred:            0 / 176, 0%
Elapsed time:       2m2.3s
Transferring:
 * Casserine_-_We_Gotta_Run_(ntsc_cda_astro).vob: 17% /288.982Mi, 465.763Ki/s, 8m43s
 *                            Fantaghirò 1.1.mkv:  2% /3.539Gi, 933.946Ki/s, 1h4m15s
 *      Cat Stevens - Wild world (PAL-DD2.0).VOB: 23% /190.781Mi, 254.716Ki/s, 9m50s
 *                            Fantaghirò 1.2.mkv:  2% /3.496Gi, 982.068Ki/s, 1h0m26s

2023/03/26 14:49:59 INFO  : 
Transferred:   	  449.309 MiB / 81.066 GiB, 1%, 2.043 MiB/s, ETA 11h13m34s
Checks:                29 / 29, 100%
Transferred:            0 / 176, 0%
Elapsed time:       3m2.3s
Transferring:
 * Casserine_-_We_Gotta_Run_(ntsc_cda_astro).vob: 27% /288.982Mi, 412.498Ki/s, 8m40s
 *                            Fantaghirò 1.1.mkv:  4% /3.539Gi, 709.346Ki/s, 1h23m23s
 *      Cat Stevens - Wild world (PAL-DD2.0).VOB: 30% /190.781Mi, 206.346Ki/s, 10m58s
 *                            Fantaghirò 1.2.mkv:  4% /3.496Gi, 761.902Ki/s, 1h16m44s

2023/03/26 14:50:59 INFO  : 
Transferred:   	  625.871 MiB / 86.509 GiB, 1%, 2.832 MiB/s, ETA 8h37m39s
Checks:                29 / 29, 100%
Transferred:            0 / 188, 0%
Elapsed time:       4m2.3s
Transferring:
 * Casserine_-_We_Gotta_Run_(ntsc_cda_astro).vob: 38% /288.982Mi, 523.479Ki/s, 5m46s
 *                            Fantaghirò 1.1.mkv:  6% /3.539Gi, 1.017Mi/s, 55m47s
 *      Cat Stevens - Wild world (PAL-DD2.0).VOB: 38% /190.781Mi, 272.455Ki/s, 7m18s
 *                            Fantaghirò 1.2.mkv:  6% /3.496Gi, 1.035Mi/s, 54m6s

2023/03/26 14:51:59 INFO  : 
Transferred:   	  800.590 MiB / 89.841 GiB, 1%, 2.927 MiB/s, ETA 8h39m14s
Checks:                29 / 29, 100%
Transferred:            0 / 214, 0%
Elapsed time:       5m2.3s
Transferring:
 * Casserine_-_We_Gotta_Run_(ntsc_cda_astro).vob: 49% /288.982Mi, 546.629Ki/s, 4m34s
 *                            Fantaghirò 1.1.mkv:  7% /3.539Gi, 1.072Mi/s, 51m55s
 *      Cat Stevens - Wild world (PAL-DD2.0).VOB: 48% /190.781Mi, 331.598Ki/s, 5m5s
 *                            Fantaghirò 1.2.mkv:  7% /3.496Gi, 1.057Mi/s, 51m58s

2023/03/26 14:52:59 INFO  : 
Transferred:   	  987.277 MiB / 93.152 GiB, 1%, 3.175 MiB/s, ETA 8h15m29s
Checks:                29 / 29, 100%
Transferred:            0 / 222, 0%
Elapsed time:       6m2.3s
Transferring:
 * Casserine_-_We_Gotta_Run_(ntsc_cda_astro).vob: 60% /288.982Mi, 551.896Ki/s, 3m31s
 *                            Fantaghirò 1.1.mkv:  9% /3.539Gi, 1.138Mi/s, 47m56s
 *      Cat Stevens - Wild world (PAL-DD2.0).VOB: 57% /190.781Mi, 311.469Ki/s, 4m28s
 *                            Fantaghirò 1.2.mkv:  9% /3.496Gi, 1.191Mi/s, 45m10s

2023/03/26 14:53:59 INFO  : 
Transferred:   	    1.147 GiB / 93.152 GiB, 1%, 3.145 MiB/s, ETA 8h19m18s
Checks:                29 / 29, 100%
Transferred:            0 / 222, 0%
Elapsed time:       7m2.3s
Transferring:
 * Casserine_-_We_Gotta_Run_(ntsc_cda_astro).vob: 71% /288.982Mi, 559.057Ki/s, 2m29s
 *                            Fantaghirò 1.1.mkv: 11% /3.539Gi, 1.158Mi/s, 46m5s
 *      Cat Stevens - Wild world (PAL-DD2.0).VOB: 66% /190.781Mi, 318.573Ki/s, 3m25s
 *                            Fantaghirò 1.2.mkv: 11% /3.496Gi, 1.128Mi/s, 46m42s

2023/03/26 14:54:59 INFO  : 
Transferred:   	    1.326 GiB / 93.152 GiB, 1%, 3.087 MiB/s, ETA 8h27m39s
Checks:                29 / 29, 100%
Transferred:            0 / 222, 0%
Elapsed time:       8m2.3s
Transferring:
 * Casserine_-_We_Gotta_Run_(ntsc_cda_astro).vob: 82% /288.982Mi, 552.864Ki/s, 1m31s
 *                            Fantaghirò 1.1.mkv: 13% /3.539Gi, 1.085Mi/s, 48m11s
 *      Cat Stevens - Wild world (PAL-DD2.0).VOB: 75% /190.781Mi, 325.612Ki/s, 2m27s
 *                            Fantaghirò 1.2.mkv: 13% /3.496Gi, 1.143Mi/s, 45m5s

2023/03/26 14:55:59 INFO  : 
Transferred:   	    1.504 GiB / 101.737 GiB, 1%, 3.027 MiB/s, ETA 9h25m3s
Checks:                29 / 29, 100%
Transferred:            0 / 233, 0%
Elapsed time:       9m2.3s
Transferring:
 * Casserine_-_We_Gotta_Run_(ntsc_cda_astro).vob: 93% /288.982Mi, 517.867Ki/s, 38s
 *                            Fantaghirò 1.1.mkv: 15% /3.539Gi, 1.131Mi/s, 45m13s
 *      Cat Stevens - Wild world (PAL-DD2.0).VOB: 84% /190.781Mi, 322.912Ki/s, 1m31s
 *                            Fantaghirò 1.2.mkv: 15% /3.496Gi, 1.075Mi/s, 46m54s
2

Can we see the contents of this file? And an example of a file which rclone transferred but shouldn't have done should be enough to work out the problem - thanks!

3

Here are the contents of current.txt.

One thing that I know it errantly attempted to download was a folder called "The Pink Panther Show".

So far it looks like it's not doing it again. It's been downloading since then and any time that I've checked, it's been downloading shows that are on the list.

Is there a way to look at the logs for what was happening when it went haywire?

I've got a script that makes a list of things that haven't been downloaded yet, which usually works well. If it accidentally downloads everything though then it takes days to complete and I end up with a bunch of extra files that need to be deleted.

05 The Flower Rebellion WebDL x264.mp4
1948-02-26 Eye Witness - The Story of Television.mkv
1986.00.00 - CBN - A.D. Part 3 - 1988.01.18 - 4th Annual Soap Opera Digest Awards - 1987.12.22 - USA Tonight - INN - WAVE - Jesus of Nazareth.mkv
1990-04-21 \[CBS WMAZ-13\] Pee-wee's Playhouse & Cartoon All Stars.mkv
1999.08.00 - MTV - TRL with guests 702 - TRL - End of Summer Countdown 1999 \[bd\] - August 1999/**
A Christmas Carol 1977.mkv
A Royal Commission Into The Australian Economy.mkv
ABC 1985 Saturday Morning Cartoons Turbo Teen Rubik WOC/**
Ain't Misbehavin/**
Amerikill \[1999\]/**
Art of Memory (1987).mkv
Astro.Boy.1980.US.DVDRip.x264-HANDJOB/**
Astro.Boy.S01.1963.DVDRIP.XviD-TVV/**
Astro.Boy.S02.1964.DVDRIP.XviD-TVV/**
Bakersfield, P.D.  1993 Season 1 Complete TVRip x264 \[i_c\]/**
Belle and Sebastian/**
Big Deal - Series 2 - Complete/**
Big Deal - Series 3 - Complete/**
Big.Girls.Blouse.S01.DVD.COMPLETE.PAL-BTN/**
Black.Neon.1991.VHSRip.mkv
Bleak House 1985 DVDRIP/**
Brewster.Place.S01.AMZN.WEB-DL.DD2.0.H.264/**
Bruno from Brazil Collection - Assorted TV Recordings - Discs 6926-6950/**
Bruno from Brazil Collection - Assorted TV Recordings - Discs 6951-6975/**
Bruno from Brazil Collection - Assorted TV Recordings - Discs 6976-7000/**
Can.Ellen.Be.Saved.1974.TVRip.dycaite(CG).mkv
Captain Gallant 106 & 128/**
Captain Gallant 221/**
Captain.Gallant.And.The.Foreign.Legion.S01E05,15,23,27,28,36.WEB.DD+2.0.x264/**
Captain.Gallant.And.The.Foreign.Legion.S02E11,21.WEB.DD+2.0.x264/**
Cartoon Cartoon Fridays 1 - Eustace Hosts (12-01-2000).mpg
Casserine_-_We_Gotta_Run_(ntsc_cda_astro).vob
Cat Stevens - Wild world (PAL-DD2.0).VOB
Dan August/**
Dan August 1. E2,3,5,6/**
Dan August Season 1 . E07-10/**
Dan August Season 1 . E11,13,15,18/**
Dan August Season 1 . E19-21/**
Dan August disc 1/**
Dan August disc 2/**
Dan August disc 3/**
Dan August disc 4/**
De Jurk/**
Dolly Parton Treasures (1996 TV Special)/**
Dragon.Flyz.S01.Unknown.H.264-NOGRP/**
El Cid La leyenda AKA El Cid The Legend \[2003\] (2003) DVDRip.mkv
Everythingsgonegreen.avi
Fables/**
Fables of the Green Forest - s01e30 - Coyote Is the Smartest.mkv
Fame - Season 1/**
Fame Season 3 TVCap/**
Fame Season 4 TVCap/**
Fame Season 5 TVCap/**
Fame Season 6 TVCap/**
Fantaghirò 1.1.mkv
Fantaghirò 1.2.mkv
Gavilan - 1x02 - Pirates.mp4
Gavilan Season 1/**
Girl Who Saved The World (1979).m4v
Give Us a Break/**
God, The Devil and Bob/**
God, The Devil and Bob - NBC WOC - Date From Hell - 3.21.00.mkv
Good Morning Miss Bliss/**
Great Expectations (1999)/**
Grimm Masterpiece Theatre (Complete Series)/**
Hee Haw S10E21.mp4
Hiroshima/**
Holocaust - Complete Mini-Series (1978)/**
Hot Pursuit Season 1/**
I Dream Of Jeannie - S01/**
I Dream Of Jeannie - S02/**
I Dream Of Jeannie - S03/**
I Dream Of Jeannie - S04/**
I Dream Of Jeannie - S05/**
I Dream of Jeannie... Fifteen Years Later (1985) - 1080p HDTVRip.x264 (Kingtut)-CG.mkv
I Still Dream of Jeannie (1991) - 1080p HDTVRip.x264 (Kingtut)-CG.mkv
In.A.Glass.Cage.1987.720p.BluRay.x264-KaKa.mkv
Jane Eyre 1983 DVDRIP/**
Jesus.of.Nazareth.1977.576p.BDRip.x264-HANDJOB/**
Kratts' Creatures/**
MTV - December 1985.mp4
Marine Boy S01 (27-52)/**
Marine Boy S01 (53-78)/**
Marine Boy Season 1/**
Monkey.Magic.Season.1.\[Eng.Dub\].DVDrip.\[BALA\]/**
Monkey.Magic.Season.2.\[Eng.Dub\].DVDrip.\[BALA\]/**
My.Old.Man.1979.mkv
NBC 1986-01-16.mp4
Neverwhere.S01.DVDRip.Plus.Commentary.x264-MaG/**
New Scotland Yard - Series 1/**
New Scotland Yard - Series 4 (1974)/**
New Scotland Yard 1972-73 Series 2/**
New Scotland Yard 1973 Series 3/**
Nick at Nite - Car 54 and more 1988-1995/**
Nickelodeon - September 1995 Mixtape.mp4
North.and.South.Book.01.1985.DVDRip.x264-HANDJOB/**
North.and.South.Book.03.1994.DVDRip.x264-HANDJOB/**
North.and.South.S02.DVDRip.x264/**
Oddities 2/**
Oliver Twist 1999/**
Once Upon A Time... Man - Season 01 - MultiAudio.DVDRip.XviD.mp3-mvgroup/**
Once.Upon.a.Time...Life.S01.DVDRip.x264-TVV/**
Operation.Ganymed.1977.DVDRip.x264.AC3-DEEP.mkv
Operation.Ganymed.Eng.1977.VHSRip.XViD-CG.avi
Papyrus S1/**
Papyrus.S02.TVRip.(ThV) \[s-mouche\]/**
Paula_Abdul_-_Straight_Up_(uncensored)_(ntsc_cda_astro).vob
Peter The Great/**
Phantom Agents- Operation Slave Labor___SB6x~TVV.mkv
Phantom Agents- Operaton Red Ghost____SB6x~TVV.mkv
Poldark Series 1 Unedited/**
Poldark Series 2 Unedited/**
Poldark TV Movie 1996.mkv
PowersofMath-Pilot.avi
Press Gang Series 1/**
Press Gang Series 2/**
Press Gang Series 3/**
Press Gang Series 4/**
Press Gang Series 5/**
Quo.Vadis.S01.DVDRip.AAC2.0.x264-TVV/**
Reluctant_Heroes.avi
Robo Story/**
Rock.&.Rule.1983.480p.Bluray.DD.5.1.x264-mintSD/**
Rock.and.Rule.1983.1080p.BluRay.x264-SPRiNTER/**
SNICK - 07-03-93.mp4
SNICK - 07-10-93.mp4
Saved by the Bell The College Years COMPLETE NTSC DVD DD2.0 x264 WiE/**
Saved.by.the.Bell.Hawaiian.Style.1992.DVDRip.x264-HANDJOB/**
Saved.by.the.Bell.The.New.Class.S01.DVDRip.x264-TVV/**
Saved.by.the.Bell.The.New.Class.S02.DVDRip.x264-TVV/**
Saved.by.the.Bell.The.New.Class.S03.DVDRip.x264-TVV/**
Saved.by.the.Bell.The.New.Class.S04.DVDRip.x264-TVV/**
Saved.by.the.Bell.The.New.Class.S05.DVDRip.x264-TVV/**
Saved.by.the.Bell.The.New.Class.S06.DVDRip.x264-TVV/**
Saved.by.the.Bell.The.New.Class.S07.DVDRip.x264-TVV/**
Saved.by.the.Bell.Wedding.in.Las.Vegas.1994.DVDRip.x264-HANDJOB/**
Season 1/**
Season 2/**
Sectaurs Mini Series (1986)/**
September Song S02 DVDRIP/**
September Song S03 DVDRIP/**
Series 11, Episode 05/**
Series 21, Episode 22.mp4
Series 4, Episode 13.mp4
Sharpe_s1/**
Sharpe_s2/**
Sharpe_s3/**
Sharpe_s4/**
Sheena_Easton_-_One_Man_Woman_(ntsc_cda_astro).vob
Silk.Stalkings.S01.AMZN.WEB-DL.DDP2.0.H.264-STARBUCKS/**
Silk.Stalkings.S01.DVD.RIP/**
Silk.Stalkings.S02.AMZN.WEB-DL.DDP2.0.H.264-STARBUCKS/**
Silk.Stalkings.S03.AMZN.WEB-DL.DDP2.0.H.264-STARBUCKS/**
Space Police (Pilot Episode).avi
Spectreman Series 1 - Space Apeman Gori/**
Spectreman Series 2 - Space Apeman Gori vs. Spectreman/**
Spectreman Series 3 - Spectreman/**
Spectreman-Pilot (raw).avi
Speed.Racer.Season1/**
Star Blazers - Series 1 - Quest For Iscandar/**
Star Blazers - Series 2 - The Comet Empire/**
Star Blazers - Series 3 - Bolar Wars/**
Style & Substance/**
TRACEY_TAKES_ON S03/**
TRACEY_TAKES_ON___S04/**
Tales From The Neverending Story/**
Tess of the D'Urbervilles.mkv
The Cinder Path Miniseries DVDRIP/**
The Fruitties (1987)/**
The Gambling Man (1995).mkv
The Last Musketeer.mkv
The Million Dollar Hijack (1984) \[cg\].mkv
The Mysterious Cities of Gold/**
The Night They Saved Christmas (1984) 1080p HDTV (Kingtut).mkv
The Pickwick Papers Miniseries DVDRIP/**
The Relcutant Agent aka Double Trouble.mkv
The.Adventures.of.Blake.and.Mortimer.S01.DVDRip.x264-TVV/**
The.Land.Before.Time.1988.1080p.BluRay.x264-SiNNERS/**
Theatre.Macabre.1971.576p.Bluray.AAC.1.0.x264-SaL/**
Too.Close.for.Comfort.S01.NTSC.DVDRip.DD2.0.x264-BTN/**
Too.Close.for.Comfort.S02.NTSC.DVDRip.DD2.0.x264-BTN/**
Too.Close.for.Comfort.S03.PDTV.DD2.0.x264-BTN/**
Too.Close.for.Comfort.S04.PDTV.DD2.0.x264-BTN/**
Too.Close.for.Comfort.S05.PDTV.DD2.0.x264-BTN/**
Too.Close.for.Comfort.S06.PDTV.DD2.0.x264-BTN/**
Tracey.Takes.On.S01.NTSC.DVD9/**
VHS Shit 126/**
VHS Shit 127/**
VHS Shit 128/**
VHS Shit 129/**
VHS Shit 130/**
Walking_The_Dead_x264_dvdrip_by_Franco153_for_CG.mkv
Wrestling Classics Vol 4/**
X.Bomber.S01.DVDRip.x264-TVV/**
\[R-Raws\] Grimm Meisaku Gekijou \[WEB-DL\]\[576p\]/**
\[R-Raws\] Shin Grimm Meisaku Gekijou \[WEB-DL\]\[576p\]/**
mousehunt1997_202001/**
sweet justice.avi
4

Are these things always in the root? If so put a / at the start of each line. That will stop rclone looking for them in other directories which will make the process more efficient.

For example if there is a folder called Season 1 anywhere in the source, rclone will copy it, but if you put /Season 1/** then it needs to be at the root.

If you use a debug log with -vv it tells you about included and excluded files.

5

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.