Move and Delete

[matt3m (Matt)]

Thanks Nick

@ncw

What happened to

@matt3m

• • Sub Folder 3

* File 3

• Sub Folder 4
  • File 4

On the server - they didn't get copied?

Sorry, that's my mistake they were meant to be copied over as well.

@ncw

So I think what you want to do is for every folder on the server

• remove the local folder of the same name
• move the remove folder to the local disk

It would actually be:

• Remove the local file in the folder of the same name
• Move the file on the server within the same folder name as the one on the local disk and then remove the folder and file from the server.

@ncw

You could do this something like

for folder in $(rclone lsf --dirs-only "remote:Folder Name"); do
    rm -rf "/path/to/local/${folder}"
    # or maybe safer
    mv "/path/to/local/${folder}" "/path/to/backup/${folder}"
    rclone move "remote:Folder Name/${folder}" "/path/to/local/${folder}"
done

The rclone lsf --dirs-only "remote:Folder Name" makes a list of the folder names, the rm -rf "/path/to/local/${folder}" deletes the local one, and the rclone move "remote:Folder Name/${folder}" "/path/to/local/${folder}" moves the remote folder to the local server.

Don't try to use this script directly, it needs careful testing to see if it would delete your data but it is simple enough that hopefully you can see how it works.

Note also that it won't work if the directory names have spaces in

Some folders will have spaces in them, is it difficult to get it to work with spaces? Would that command just be copied and pasted into the command prompt?

Also, for it to run automatically is that done with Task Scheduler? I currently have a WinSCP script running that only copies files and that is done by having a .txt file with the commands and then creating a .bat file with the following in:

winscp.com /script=SyncToLocalScript.txt
exit

Would it be the same or does rclone have a different way?

Thanks

[asdffdsa (jojothehumanmonkey)]

@matt3m

is it difficult to get it to work with spaces?

just quote the paths, as @ncw has already done
"remote:Folder Name"
"/path/to/local/${folder}"

@matt3m

for it to run automatically is that done with Task Scheduler?

that script that @ncw shared is for linux, not windows.
but if it was for windows, could use task scheduler.

that script is written in bash for linux, will not work on windows.
so you at least two options

1. convert the script to powershell, python or your script language of choice.
2. run the script on WSL or cygwin and convert the paths to work on windows.
   for example, with WSL, E:\Test\" becomes /mnt/e/test/

[matt3m (Matt)]

@asdffdsa

just quote the paths, as @ncw has already done
"remote:Folder Name"
"/path/to/local/${folder}"

Great! Thanks for confirming. I asked @ncw said it wouldn’t work with spaces.

@asdffdsa

that script that @ncw shared is for linux, not windows.
but if it was for windows, could use task scheduler.

that script is written in bash for linux, will not work on windows.
so you at least two options

1. convert the script to powershell, python or your script language of choice.
2. run the script on WSL or cygwin and convert the paths to work on windows.
   for example, with WSL, E:\Test\" becomes /mnt/e/test/

Oh I see, the Linux one could be handy anyway as I’m possibly going to get a Synology NAS which is Linux. I wasn’t aware that the commands would be different on each OS.

Also, is it easy to convert to Powershell? Can I use an online service to do it?

Thanks

[asdffdsa (jojothehumanmonkey)]

@matt3m

I asked ask @ncw said it wouldn’t work with spaces.

ok, i guess i did your ask and did not see his answer.

rclone ls "d:\d i r\f i l e.txt"
        1 f i l e.txt

@matt3m

get a Synology NAS

i have access to a bunch of synboxes, rclone runs great on it.

@matt3m

is it easy to convert to Powershell?

it depends on your script skills or perhaps now is a great time to learn to script.
the bash script, with a few minor path tweaks, will run on WSL - WIndows Subsystem for Linux

[matt3m (Matt)]

@asdffdsa

ok, i guess i did your ask and did not see his answer.

rclone ls "d:\d i r\f i l e.txt"
        1 f i l e.txt

So, to confirm… I just need to use the quote symbol for spaces to be ok?

@asdffdsa

it depends on your script skills or perhaps now is a great time to learn to script.
the bash script, with a few minor path tweaks, will run on WSL - WIndows Subsystem for Linux

My skills are pretty much nonexistent. I know some ssh commands like ls but that’s only because I’ve been told to do it when needed.

I do actually have access to a book on Powershell as my brother was learning it so I might have to get it from him and read it but I imagine it’s quite a steep learning curve.

[ncw (Nick Craig-Wood)]

@ncw

for folder in $(rclone lsf --dirs-only "remote:Folder Name"); do

This is the bit which won't work with spaces in the file names.

@matt3m

I do actually have access to a book on Powershell as my brother was learning it so I might have to get it from him and read it but I imagine it’s quite a steep learning curve.

I'm afraid I don't know Powershell, though we do have some people on the forum who do...

Could you ask your brother to help?

The basic idea is to list the remote directories with this, which will list one per line

rclone lsf --dirs-only "remote:Folder Name"

Then iterate through them, deleting the local files and moving the remote ones with

rclone move "remote:Folder Name/${folder}" "/path/to/local/${folder}"

[matt3m (Matt)]

@ncw

@ncw

> for folder in $(rclone lsf --dirs-only "remote:Folder Name"); do
> ```

This is the bit which won't work with spaces in the file names.

Thanks for clarifying.

@ncw

I'm afraid I don't know Powershell, though we do have some people on the forum who do...

Could you ask your brother to help?

The basic idea is to list the remote directories with this, which will list one per line

rclone lsf --dirs-only "remote:Folder Name"

Then iterate through them, deleting the local files and moving the remote ones with

rclone move "remote:Folder Name/${folder}" "/path/to/local/${folder}"

I can see if he will be able to but I don't think he is great at it but I will see what he says. If not it would still be awesome if someone else who may know Powershell in the forum could help out.

Thanks

[albertony]

Quick rewrite to PowerShell.

You must change the value of the two (three in alternative 2) initial variables to match your configuration.

More or less untested, so please try out on testdata first.

Alternative 1: Permanent delete

$RemoteRoot = "remote:Folder Name"
$LocalRoot = "C:\path\to\local"
rclone lsf --dirs-only $RemoteRoot | ForEach-Object {
    $LocalPath = Join-Path -Path $LocalRoot -ChildPath $_
    Remove-Item -Recurse -Force -LiteralPath $LocalPath
    rclone move "${RemoteRoot}/${_}" $LocalPath
}

Alternative 2: Safer, move to backup location instead of delete.

$RemoteRoot = "remote:Folder Name"
$LocalRoot = "C:\path\to\local"
$BackupRoot = "C:\path\to\backup"
rclone lsf --dirs-only $RemoteRoot | ForEach-Object {
    $LocalPath = Join-Path -Path $LocalRoot -ChildPath $_
    $BackupPath = Join-Path -Path $BackupRoot -ChildPath $_
    Move-Item -LiteralPath $LocalRoot -Destination $BackupPath
    rclone move "${RemoteRoot}/${_}" $LocalPath
}

[matt3m (Matt)]

Thank you @albertony

I will try it on test data on a USB stick so it doesn't matter if the data is deleted.

How would I go about running it? Is it a simply copy and paste job into PowerShell?

Thanks

[albertony]

@matt3m

How would I go about running it? Is it a simply copy and paste job into PowerShell?

That is one option.

Another is to save it in a file, with extension .ps1. Then execute that file using something like this:

powershell.exe filename.ps1

Either from a command prompt, from Windows Run (Win+R), or you can create a Windows shortcut to run that.

• Beware of relative file paths, as current directory may vary with different methods - and especially if you run PowerShell as administrator as it will default to C:\Windows\system32 as current directory, which has great potential for wrecking your windows install if run the wrong command with relative paths...

Depending on your setup, you might have to enable the possibility to execute PowerShell scripts from file. This can be done by starting PowerShell as administrator and execute the following:

Set-ExecutionPolicy RemoteSigned

Do that once, and close the PowerShell instance, and it is permanently changed.

[Yamiraan (Marc Cuadras)]

This thread have same topic what im looking for, so with the author permission im gonna ask same question,
I need to move one folder that have many subfolder and and every subfolder have many files, so below cmd will be ok?

rclone move "/root/www/foldername" "remote:/root/www/box"

/root/www/box is remote path in my case.
Thanks

[albertony]

Probably ok, but there may be corner cases where it depends what you want, e.g. what you want to happen if destination already exists etc.

And perhaps you may want this option:

If you want to delete empty source directories after move, use the --delete-empty-src-dirs flag.

In any case, just try it out:

Important : Since this can cause data loss, test first with the --dry-run or the --interactive /-i flag.

(quotes are from the docs)

[matt3m (Matt)]

Thanks.

I tried copying and pasting it into PowerShell but it didn't work so I saved the file as SyncToLocal.ps1 and right clicked in chose to Run with PowerShell.

This is the command I used.

$RemoteRoot = "ftpserver:test/"
$LocalRoot = "I:\"
rclone lsf --dirs-only $RemoteRoot | ForEach-Object {
    $LocalPath = Join-Path -Path $LocalRoot -ChildPath $_
    Remove-Item -Recurse -Force -LiteralPath $LocalPath
    rclone move "${RemoteRoot}/${_}" $LocalPath
}

PowerShell opened and I see the following...

It does look like it worked though, however, one file in a folder was copied across but not deleted and how can I add to delete empty directories on the remote?

Thanks

[albertony]

Ah, yes, if the local directories may or may not exist you can add -ErrorAction Ignore to the Remove-Item command. And to avoid empty directories being left behind in remote, add --delete-empty-src-dirs to the rclone move command, like this:

$RemoteRoot = "ftpserver:test/"
$LocalRoot = "I:\"
rclone lsf --dirs-only $RemoteRoot | ForEach-Object {
    $LocalPath = Join-Path -Path $LocalRoot -ChildPath $_
    Remove-Item -Recurse -Force -LiteralPath $LocalPath -ErrorAction Ignore
    rclone move --delete-empty-src-dirs "${RemoteRoot}/${_}" $LocalPath
}

[matt3m (Matt)]

Awesome! I tried that but it still didn't delete the remote empty folders. Here's what I used... I added progress and 1 transfer at a time.

$RemoteRoot = "ftpserver:test/"
$LocalRoot = "I:\"
rclone lsf --dirs-only $RemoteRoot | ForEach-Object {
    $LocalPath = Join-Path -Path $LocalRoot -ChildPath $_
    Remove-Item -Recurse -Force -LiteralPath $LocalPath -ErrorAction Ignore
    rclone move --delete-empty-src-dirs --progress --transfers=1 "${RemoteRoot}/${_}" $LocalPath
}

[albertony]

@matt3m

I tried that but it still didn't delete the remote empty folders

I haven't used that option much myself, but seems it only deletes empty subdirectories (when the move is not server-side). To fix that you could add another command, after the rclone move:

rclone rmdir "${RemoteRoot}/${_}" 2>&1 | Out-Null

Which should be a safe thing to do, as it:

Will not remove the path if it has any objects in it, not even empty subdirectories

Note that the approach we have so far leaves some corner cases not being handled:

• rclone rmdir command above will hide any errors as a result of the directory existing but could not be deleted, e.g. because it was not empty after all. But if this were to happen, the previous rclone move command above should have reported errors, I think.
• The Remove-Item command with -ErrorAction Ignore is similar: If it fails for other reasons than the path not exists, e.g. you do not have permissions, you will not notice, and the script will continue with the rclone move into the destination that already exists. But if you had no permissions to the destination, this move operations should fail as well, and it will report errors..

These may or may not be relevant in your case, depending on things like paths used, permissions, admin privileges, your level of paranoia... The points mentioned above are not very difficult to improve, but it makes the script a bit more complex (i.e. hard to read).

[matt3m (Matt)]

@albertony

I haven't used that option much myself, but seems it only deletes empty subdirectories (when the move is not server-side). To fix that you could add another command, after the rclone move:

rclone rmdir "${RemoteRoot}/${_}" 2>&1 | Out-Null

Which should be a safe thing to do, as it:

Will not remove the path if it has any objects in it, not even empty subdirectories

Thanks, I tried adding that and it deleted everything from the server and local.

If I run the script again (the one below) and if the empty folders on the server are still there it deletes the folders on my local computer even if there's a file in the folder.

$RemoteRoot = "ftpserver:test/"
$LocalRoot = "I:\"
rclone lsf --dirs-only $RemoteRoot | ForEach-Object {
    $LocalPath = Join-Path -Path $LocalRoot -ChildPath $_
    Remove-Item -Recurse -Force -LiteralPath $LocalPath -ErrorAction Ignore
    rclone move "${RemoteRoot}/${_}" $LocalPath --progress --transfers=1 --delete-empty-src-dirs
}

[albertony]

@matt3m

Thanks, I tried adding that and it deleted everything from the server and local.

Well, that command alone did not do that. Did you start out with "clean" test case, no empty subdirs remaining from previous attempts etc..?

I haven't really followed this thread in detail from start, but migrated the suggested script to powershell, and think it should be equivalent as of now. If it does not do what you want, could you give some details of what it is doing and what you want it to do..

[matt3m (Matt)]

I did a clean test and then tested it again after by adding one new file into a folder on the server with the empty folders and it moved the file across to the relevant folder but deleted all the others on local.

This is what I'm hoping to achieve...

I have a server that has multiple files in folders that are always changing. New files and folders are added regularly (I don't know what the names of the folders or files will be before hand).

So, what I would like to be able to do is to download the files within a specific folder (with loads of other folders and files in) to my computer. For example, the folder structure could be...

• Folder Name
• Sub Folder 1
  • File 1
• Sub Folder 2
  • File 2
• Sub Folder 3
  • File 3
• Sub Folder 4
  • File 4

and so on... When that has finished downloading, I would like to be able to delete any existing files on my computer within the folders (only delete the files and not the folders on my computer) and I would also like the source files and folders to be deleted from the server and if possible to have this run automatically.

This is how the before and after folders would look like...

Server before

• Folder Name
  • Sub Folder 1
    • File 1
  • Sub Folder 2
    • File 2
  • Sub Folder 3
    • File 3
  • Sub Folder 4
    • File 4

Local Computer Before

• Folder Name
  • Sub Folder 1
    • File 11
  • Sub Folder 2
    • File 23
  • Sub Folder 55
    • File 98
  • Sub Folder 65
    • File 74
  • Sub Folder 100
    • File 185
  • Sub Folder 33
    • File 478
  • Sub Folder 879
    • File 444
  • Sub Folder 15
    • File 667

Server After

• Folder Name

Local Computer After

• Folder Name
  • Sub Folder 1
    • File 1
  • Sub Folder 2
    • File 23
  • Sub Folder 3
    • File 3
  • Sub Folder 4
    • File 4
  • Sub Folder 55
    • File 98
  • Sub Folder 65
    • File 74
  • Sub Folder 100
    • File 185
  • Sub Folder 33
    • File 478
  • Sub Folder 879
    • File 444
  • Sub Folder 15
    • File 667

On the Local Computer After, it will always have more folders than the server as all files downloaded from the server should be deleted and any files within the sub folders before should be removed when downloading a new file.

[albertony]

@matt3m

I did a clean test and then tested it again after by adding one new file into a folder on the server with the empty folders and it moved the file across to the relevant folder but deleted all the others on local.

If there were empty subdirs on remote left behind from first test, second test would delete local dirs with same name.

(I can test your example myself later, when I'm back at my pc)