Move and Delete

Help and Support
41

If I could delete the files before or after the new file downloads that would be good.

For example if on my local computer the folder is:

E:\Files and in that folder it has multiple folders (John, Matt, Steven, Jade etc..). On the server there’s a new file added to Steven’s directory (Steven\Work\Videos\2022\Silverton\NewVideo2022-02-01.mp4) it would download that file to the same location on my computer and delete the other file in that location on my computer.

Hope I’ve explained that good enough.

42

maybe run a rclone sync for each folder.
rclone sync ftpserver:Steven e:\Steven

43

Would that still be able to delete the file from the server and any existing files in the specific folders?

44

perhaps do a
--- rclone sync which will delete the dest file
--- rclone move which will delete the source file

might need to write a script.
--- get a list of directories using rclone lsd --recursive
--- for each directory, do a rclone sync --max-depth=1

45

So does sync work by checking the destination and if it's not the same it removes the files that are in the folder?

I'll give that a test try and see how I get on with it.

46

I've just been testing this and I don't think sync will work as it removes all folders on destination that aren't on the source. I will have other folders on the destination that aren't on the server as new files/folders are added regularly.

Maybe what I'm asking for isn't possible with rclone?

47

you tried rclone sync --max-depth=1

48

I didn't but I will try it now

49

I just tested it but it didn't delete the files already in the destination folder unfortunately.

50

cannot see into your computer,
need to see a full debug log.

51

Changing the max depth to 2 downloads the all the files but it still doesn't delete file that are already on the destination.

This is the command I used.

rclone sync "ftpserver:test/" "I:\" --progress --transfers=1 --max-depth=1

I am using a USB for all the test as it doesn't matter if anything gets deleted then.

52

Here's the full debug log. The files are just test music files.

PS C:\Users\Matt> rclone sync "ftpserver:test/" "I:\" --progress --transfers=1 --max-depth=2 -vv
2022/02/08 17:22:24 DEBUG : rclone: Version "v1.57.0" starting with parameters ["C:\\rclone\\rclone.exe" "sync" "ftpserver:test/" "I:\\" "--progress" "--transfers=1" "--max-depth=2" "-vv"]
2022/02/08 17:22:24 DEBUG : Creating backend with remote "ftpserver:test/"
2022/02/08 17:22:24 DEBUG : Using config file from "C:\\Users\\Matt\\AppData\\Roaming\\rclone\\rclone.conf"
2022/02/08 17:22:25 DEBUG : sftp://matt3@iapetus.feralhosting.com:22/test/: New connection 192.168.0.2:64316->185.21.216.160:22 to "SSH-2.0-OpenSSH_7.4p1 Debian-10+deb9u7"
2022/02/08 17:22:25 DEBUG : sftp://matt3@iapetus.feralhosting.com:22/test/: Using absolute root directory "/media/sdh1/matt3/test"
2022/02/08 17:22:25 DEBUG : Creating backend with remote "I:\\"
2022/02/08 17:22:25 DEBUG : fs cache: renaming cache item "I:\\" to be canonical "//?/I:/"
2022-02-08 17:22:25 DEBUG : Alan Walker - Faded (Slushii Remix).mp3: Size and modification time the same (differ by 0s, within tolerance 1s)
2022-02-08 17:22:25 DEBUG : Alan Walker - Faded (Slushii Remix).mp3: Unchanged skipping
2022-02-08 17:22:25 DEBUG : sftp://matt3@iapetus.feralhosting.com:22/test/: New connection 192.168.0.2:64318->185.21.216.160:22 to "SSH-2.0-OpenSSH_7.4p1 Debian-10+deb9u7"
2022-02-08 17:22:25 DEBUG : sftp://matt3@iapetus.feralhosting.com:22/test/: New connection 192.168.0.2:64320->185.21.216.160:22 to "SSH-2.0-OpenSSH_7.4p1 Debian-10+deb9u7"
2022-02-08 17:22:25 DEBUG : sftp://matt3@iapetus.feralhosting.com:22/test/: New connection 192.168.0.2:64321->185.21.216.160:22 to "SSH-2.0-OpenSSH_7.4p1 Debian-10+deb9u7"
2022-02-08 17:22:25 DEBUG : sftp://matt3@iapetus.feralhosting.com:22/test/: New connection 192.168.0.2:64319->185.21.216.160:22 to "SSH-2.0-OpenSSH_7.4p1 Debian-10+deb9u7"
2022-02-08 17:22:25 DEBUG : sftp://matt3@iapetus.feralhosting.com:22/test/: New connection 192.168.0.2:64322->185.21.216.160:22 to "SSH-2.0-OpenSSH_7.4p1 Debian-10+deb9u7"
2022-02-08 17:22:25 DEBUG : sftp://matt3@iapetus.feralhosting.com:22/test/: New connection 192.168.0.2:64323->185.21.216.160:22 to "SSH-2.0-OpenSSH_7.4p1 Debian-10+deb9u7"
2022-02-08 17:22:26 DEBUG : Local file system at //?/I:/: Waiting for checks to finish
2022-02-08 17:22:26 DEBUG : Tony/Aero Chord - Surface (Original Mix).mp3: Modification times differ by 1s: 2022-02-08 17:17:47 +0000 GMT, 2022-02-08 17:17:48 +0000 GMT
2022-02-08 17:22:26 DEBUG : Tony/Instructions! .txt: Modification times differ by 1s: 2022-02-08 17:17:27 +0000 GMT, 2022-02-08 17:17:28 +0000 GMT
2022-02-08 17:22:26 DEBUG : sftp cmd = /media/sdh1/matt3/test/Tony/Aero\ Chord\ -\ Surface\ \(Original\ Mix\).mp3
2022-02-08 17:22:26 DEBUG : sftp cmd = /media/sdh1/matt3/test/Tony/Instructions\!\ .txt
2022-02-08 17:22:26 DEBUG : sftp output = "e5f044870d6ed0483666bdf16a61796a  /media/sdh1/matt3/test/Tony/Aero Chord - Surface (Original Mix).mp3\n"
2022-02-08 17:22:26 DEBUG : sftp output = "32cd92cb979ce78f52e6732131bea3a5  /media/sdh1/matt3/test/Tony/Instructions! .txt\n"
2022-02-08 17:22:26 DEBUG : sftp hash = "e5f044870d6ed0483666bdf16a61796a"
2022-02-08 17:22:26 DEBUG : sftp hash = "32cd92cb979ce78f52e6732131bea3a5"
2022-02-08 17:22:26 DEBUG : Tony/Aero Chord - Surface (Original Mix).mp3: md5 = e5f044870d6ed0483666bdf16a61796a OK
2022-02-08 17:22:26 DEBUG : Tony/Instructions! .txt: md5 = 32cd92cb979ce78f52e6732131bea3a5 OK
2022-02-08 17:22:26 INFO  : Tony/Aero Chord - Surface (Original Mix).mp3: Updated modification time in destination
2022-02-08 17:22:26 INFO  : Tony/Instructions! .txt: Updated modification time in destination
2022-02-08 17:22:26 DEBUG : Tony/Aero Chord - Surface (Original Mix).mp3: Unchanged skipping
2022-02-08 17:22:26 DEBUG : Tony/Instructions! .txt: Unchanged skipping
2022-02-08 17:22:26 DEBUG : Local file system at //?/I:/: Waiting for transfers to finish
2022-02-08 17:22:27 DEBUG : sftp cmd = /media/sdh1/matt3/test/Adam/3LAU\ \&\ Nom\ De\ Strip\ ft.\ Estelle\ -\ The\ Night\ \(LAXX\ Remix\).mp3
2022-02-08 17:22:27 DEBUG : sftp output = "2013ed761f661d906926df055e3f6081  /media/sdh1/matt3/test/Adam/3LAU & Nom De Strip ft. Estelle - The Night (LAXX Remix).mp3\n"
2022-02-08 17:22:27 DEBUG : sftp hash = "2013ed761f661d906926df055e3f6081"
2022-02-08 17:22:27 DEBUG : Adam/3LAU & Nom De Strip ft. Estelle - The Night (LAXX Remix).mp3: md5 = 2013ed761f661d906926df055e3f6081 OK
2022-02-08 17:22:27 INFO  : Adam/3LAU & Nom De Strip ft. Estelle - The Night (LAXX Remix).mp3: Copied (new)
2022-02-08 17:22:30 DEBUG : sftp cmd = /media/sdh1/matt3/test/Christopher/Bassjackers\ \&\ Apster\ -\ Klambu\ \(Original\ Mix\).mp3
2022-02-08 17:22:30 DEBUG : sftp output = "dbe93b2899235ed84388c41b7ec5d007  /media/sdh1/matt3/test/Christopher/Bassjackers & Apster - Klambu (Original Mix).mp3\n"
2022-02-08 17:22:30 DEBUG : sftp hash = "dbe93b2899235ed84388c41b7ec5d007"
2022-02-08 17:22:30 DEBUG : Christopher/Bassjackers & Apster - Klambu (Original Mix).mp3: md5 = dbe93b2899235ed84388c41b7ec5d007 OK
2022-02-08 17:22:30 INFO  : Christopher/Bassjackers & Apster - Klambu (Original Mix).mp3: Copied (new)
2022-02-08 17:22:32 DEBUG : sftp cmd = /media/sdh1/matt3/test/Aimee/3LAU\ vs.\ Justin\ Bieber\ -\ What\ Do\ You\ Mean\ Again\ \(3LAU\ Mashup\).mp3
2022-02-08 17:22:32 DEBUG : sftp output = "f9086daba2c0ee5b11fd32a512a1e870  /media/sdh1/matt3/test/Aimee/3LAU vs. Justin Bieber - What Do You Mean Again (3LAU Mashup).mp3\n"
2022-02-08 17:22:32 DEBUG : sftp hash = "f9086daba2c0ee5b11fd32a512a1e870"
2022-02-08 17:22:32 DEBUG : Aimee/3LAU vs. Justin Bieber - What Do You Mean Again (3LAU Mashup).mp3: md5 = f9086daba2c0ee5b11fd32a512a1e870 OK
2022-02-08 17:22:32 INFO  : Aimee/3LAU vs. Justin Bieber - What Do You Mean Again (3LAU Mashup).mp3: Copied (new)
2022-02-08 17:22:35 DEBUG : sftp cmd = /media/sdh1/matt3/test/Allison/Bassjackers\ \&\ Apster\ -\ Klambu\ \(Original\ Mix\).mp3
2022-02-08 17:22:35 DEBUG : sftp output = "dbe93b2899235ed84388c41b7ec5d007  /media/sdh1/matt3/test/Allison/Bassjackers & Apster - Klambu (Original Mix).mp3\n"
2022-02-08 17:22:35 DEBUG : sftp hash = "dbe93b2899235ed84388c41b7ec5d007"
2022-02-08 17:22:35 DEBUG : Allison/Bassjackers & Apster - Klambu (Original Mix).mp3: md5 = dbe93b2899235ed84388c41b7ec5d007 OK
2022-02-08 17:22:35 INFO  : Allison/Bassjackers & Apster - Klambu (Original Mix).mp3: Copied (new)
2022-02-08 17:22:40 DEBUG : sftp cmd = /media/sdh1/matt3/test/Allison/Cirez\ D\ -\ Glow\ \(In\ The\ Dark\ Dub\).mp3
2022-02-08 17:22:40 DEBUG : sftp output = "9e50833cd47daac2757a09a548b3be13  /media/sdh1/matt3/test/Allison/Cirez D - Glow (In The Dark Dub).mp3\n"
2022-02-08 17:22:40 DEBUG : sftp hash = "9e50833cd47daac2757a09a548b3be13"
2022-02-08 17:22:40 DEBUG : Allison/Cirez D - Glow (In The Dark Dub).mp3: md5 = 9e50833cd47daac2757a09a548b3be13 OK
2022-02-08 17:22:40 INFO  : Allison/Cirez D - Glow (In The Dark Dub).mp3: Copied (new)
2022-02-08 17:22:42 DEBUG : sftp cmd = /media/sdh1/matt3/test/Charlie/A-Trak\ \&\ Tommy\ Trash\ -\ Tuna\ Melt\ \(Grandtheft\ Remix\).mp3
2022-02-08 17:22:42 DEBUG : sftp output = "8c86f424d396c953464467fb6a69a732  /media/sdh1/matt3/test/Charlie/A-Trak & Tommy Trash - Tuna Melt (Grandtheft Remix).mp3\n"
2022-02-08 17:22:42 DEBUG : sftp hash = "8c86f424d396c953464467fb6a69a732"
2022-02-08 17:22:42 DEBUG : Charlie/A-Trak & Tommy Trash - Tuna Melt (Grandtheft Remix).mp3: md5 = 8c86f424d396c953464467fb6a69a732 OK
2022-02-08 17:22:42 INFO  : Charlie/A-Trak & Tommy Trash - Tuna Melt (Grandtheft Remix).mp3: Copied (new)
2022-02-08 17:22:45 DEBUG : sftp cmd = /media/sdh1/matt3/test/Sonia/Adventure\ Club\ ft.\ The\ Kite\ String\ Tangle\ -\ Wonder\ \(Andru\ Remix\).mp3
2022-02-08 17:22:45 DEBUG : sftp output = "373cadf6ec1032f92c532933c14e587d  /media/sdh1/matt3/test/Sonia/Adventure Club ft. The Kite String Tangle - Wonder (Andru Remix).mp3\n"
2022-02-08 17:22:45 DEBUG : sftp hash = "373cadf6ec1032f92c532933c14e587d"
2022-02-08 17:22:45 DEBUG : Sonia/Adventure Club ft. The Kite String Tangle - Wonder (Andru Remix).mp3: md5 = 373cadf6ec1032f92c532933c14e587d OK
2022-02-08 17:22:45 INFO  : Sonia/Adventure Club ft. The Kite String Tangle - Wonder (Andru Remix).mp3: Copied (new)
2022-02-08 17:22:45 DEBUG : Waiting for deletions to finish
Transferred:       79.693 MiB / 79.693 MiB, 100%, 3.973 MiB/s, ETA 0s
Checks:                 3 / 3, 100%
Transferred:            7 / 7, 100%
Elapsed time:        20.4s
2022/02/08 17:22:45 INFO  :
Transferred:       79.693 MiB / 79.693 MiB, 100%, 3.973 MiB/s, ETA 0s
Checks:                 3 / 3, 100%
Transferred:            7 / 7, 100%
Elapsed time:        20.5s

2022/02/08 17:22:45 DEBUG : 66 go routines active
53

perhaps try adding --delete-excluded

54

I added that but it still didn't delete existing files in the folder of the destination

55

Does anyone else know of any solutions or if it is possible with rclone?

56

imho, your use case is not well explained.
each log file uses different source and dest dir/files.
that makes it hard to follow

i thought and still think that this could work

might need to write a script.
--- get a list of directories using rclone lsd --recursive
--- for each directory, do a rclone sync --max-depth=1

not sure that you tested that?

at this point,

  1. create the simplest, most minimal set of source and dest directories.
  2. document the source and dest using rclone ls or rclone tree
  3. without explanations and commentary, document what the dest should be, using the same format as rclone ls/tree.
57

I'm so bad at explaining, sorry. Let me try and explain in detail and see if that helps with a clearer picture of what I'm trying to achieve.

I didn't try this as writing scripts isn't my strong point.

Anyway, onto the detailed explanation.

I have a server that has multiple files in folders that are always changing. New files and folders are added regularly (I don't know what the names of the folders or files will be before hand).

So, what I would like to be able to do is to download the files within a specific folder (with loads of other folders and files in) to my computer. For example, the folder structure could be...

  • Folder Name
  • Sub Folder 1
    • File 1
  • Sub Folder 2
    • File 2
  • Sub Folder 3
    • File 3
  • Sub Folder 4
    • File 4

and so on... When that has finished downloading, I would like to be able to delete any existing files on my computer within the folders (only delete the files and not the folders on my computer) and I would also like the source files and folders to be deleted from the server and if possible to have this run automatically.

Hopefully that has been explained a bit better, if there's still anything that needs explaining please let me know. I know I'm terrible at explaining things so just tell when I haven't explained anything properly :slight_smile:

Thanks

58

This little picture is very helpful, could you do examples showing "the server before" "the local computer before" "the server after" "the local computer after" to explain what you want.

59

Yes, sure...

Server before

  • Folder Name
    • Sub Folder 1
      • File 1
    • Sub Folder 2
      • File 2
    • Sub Folder 3
      • File 3
    • Sub Folder 4
      • File 4

Local Computer Before

  • Folder Name
    • Sub Folder 1
      • File 11
    • Sub Folder 2
      • File 23
    • Sub Folder 55
      • File 98
    • Sub Folder 65
      • File 74
    • Sub Folder 100
      • File 185
    • Sub Folder 33
      • File 478
    • Sub Folder 879
      • File 444
    • Sub Folder 15
      • File 667

Server After

  • Folder Name

Local Computer After

  • Folder Name
    • Sub Folder 1
      • File 1
    • Sub Folder 2
      • File 23
    • Sub Folder 55
      • File 98
    • Sub Folder 65
      • File 74
    • Sub Folder 100
      • File 185
    • Sub Folder 33
      • File 478
    • Sub Folder 879
      • File 444
    • Sub Folder 15
      • File 667

On the Local Computer After, it will always have more folders than the server as all files downloaded from the server should be deleted and any files within the sub folders before should be removed when downloading a new file.

Hope that helps.

Anything else that needs clarifying, just let me know.

Thanks

60

What happened to

On the server - they didn't get copied?

So I think what you want to do is for every folder on the server

  • remove the local folder of the same name
  • move the remove folder to the local disk

You could do this something like

for folder in $(rclone lsf --dirs-only "remote:Folder Name"); do
    rm -rf "/path/to/local/${folder}"
    # or maybe safer
    mv "/path/to/local/${folder}" "/path/to/backup/${folder}"
    rclone move "remote:Folder Name/${folder}" "/path/to/local/${folder}"
done

The rclone lsf --dirs-only "remote:Folder Name" makes a list of the folder names, the rm -rf "/path/to/local/${folder}" deletes the local one, and the rclone move "remote:Folder Name/${folder}" "/path/to/local/${folder}" moves the remote folder to the local server.

Don't try to use this script directly, it needs careful testing to see if it would delete your data but it is simple enough that hopefully you can see how it works.

Note also that it won't work if the directory names have spaces in

1 Like