Move and Delete

matt3m · February 11, 2022, 3:35pm

Yes, that's exactly what happened.

If you wouldn't mind that would be appreciated

albertony · February 11, 2022, 3:37pm

And what we want to make sure then, is to not leave empty subdirs behind, would you agree?

matt3m · February 11, 2022, 3:41pm

That's right. Once the files have been moved, delete the empty folders on the remote.

albertony · February 11, 2022, 6:09pm

Is that correct how you want it, or should it be:

Folder Name
- Sub Folder 2
  - File 2

albertony · February 11, 2022, 6:30pm

I just ran some local testing on this...

There was one issue regarding ending slash in paths, which is fixed by adding --dir-slash=false to the rclone lsf command.

Another important thing is to make sure RemoteRoot and LocalRoot point to the directory at correct level, must point to "Folder Name" in your example, and not its parent.

My complete test from PowerShell. Starting out in an empty local directory. Then creating one subdirectory "Server" and one subdirectory "Local" with files according to your description. Running the script to move between those paths.

Setup:

rclone touch "Server\Folder Name\Sub Folder 1\File 1"
rclone touch "Server\Folder Name\Sub Folder 2\File 2"
rclone touch "Server\Folder Name\Sub Folder 3\File 3"
rclone touch "Server\Folder Name\Sub Folder 4\File 4"
rclone touch "Local\Folder Name\Sub Folder 1\File 11"
rclone touch "Local\Folder Name\Sub Folder 2\File 23"
rclone touch "Local\Folder Name\Sub Folder 55\File 98"
rclone touch "Local\Folder Name\Sub Folder 65\File 74"
rclone touch "Local\Folder Name\Sub Folder 100\File 185"
rclone touch "Local\Folder Name\Sub Folder 33\File 478"
rclone touch "Local\Folder Name\Sub Folder 879\File 444"
rclone touch "Local\Folder Name\Sub Folder 15\File 667"

Server before:

rclone lsf Server --recursive | Sort-Object
Folder Name/
Folder Name/Sub Folder 1/
Folder Name/Sub Folder 1/File 1
Folder Name/Sub Folder 2/
Folder Name/Sub Folder 2/File 2
Folder Name/Sub Folder 3/
Folder Name/Sub Folder 3/File 3
Folder Name/Sub Folder 4/
Folder Name/Sub Folder 4/File 4

Local before:

rclone lsf Local --recursive | Sort-Object
Folder Name/
Folder Name/Sub Folder 1/
Folder Name/Sub Folder 1/File 11
Folder Name/Sub Folder 100/
Folder Name/Sub Folder 100/File 185
Folder Name/Sub Folder 15/
Folder Name/Sub Folder 15/File 667
Folder Name/Sub Folder 2/
Folder Name/Sub Folder 2/File 23
Folder Name/Sub Folder 33/
Folder Name/Sub Folder 33/File 478
Folder Name/Sub Folder 55/
Folder Name/Sub Folder 55/File 98
Folder Name/Sub Folder 65/
Folder Name/Sub Folder 65/File 74
Folder Name/Sub Folder 879/
Folder Name/Sub Folder 879/File 444

Process:

$RemoteRoot = "Server/Folder Name"
$LocalRoot = "Local/Folder Name"
rclone lsf --dirs-only --dir-slash=false $RemoteRoot | ForEach-Object {
    $LocalPath = Join-Path -Path $LocalRoot -ChildPath $_
    Remove-Item -Recurse -Force -LiteralPath $LocalPath -ErrorAction Ignore
    rclone move --delete-empty-src-dirs "${RemoteRoot}/${_}" $LocalPath
}

Server after:

rclone lsf Server --recursive | Sort-Object
Folder Name/

Local after:

rclone lsf Local --recursive | Sort-Object
Folder Name/
Folder Name/Sub Folder 1/
Folder Name/Sub Folder 1/File 1
Folder Name/Sub Folder 100/
Folder Name/Sub Folder 100/File 185
Folder Name/Sub Folder 15/
Folder Name/Sub Folder 15/File 667
Folder Name/Sub Folder 2/
Folder Name/Sub Folder 2/File 2
Folder Name/Sub Folder 3/
Folder Name/Sub Folder 3/File 3
Folder Name/Sub Folder 33/
Folder Name/Sub Folder 33/File 478
Folder Name/Sub Folder 4/
Folder Name/Sub Folder 4/File 4
Folder Name/Sub Folder 55/
Folder Name/Sub Folder 55/File 98
Folder Name/Sub Folder 65/
Folder Name/Sub Folder 65/File 74
Folder Name/Sub Folder 879/
Folder Name/Sub Folder 879/File 444

matt3m · February 11, 2022, 7:06pm

That was just an example, the number don't mean anything. The files would have actual names.

albertony:

$RemoteRoot = "Server/Folder Name"
$LocalRoot = "Local/Folder Name"
rclone lsf --dirs-only --dir-slash=false $RemoteRoot | ForEach-Object {
    $LocalPath = Join-Path -Path $LocalRoot -ChildPath $_
    Remove-Item -Recurse -Force -LiteralPath $LocalPath -ErrorAction Ignore
    rclone move --delete-empty-src-dirs "${RemoteRoot}/${_}" $LocalPath
}

I tried this (replacing the $RemoteRoot and $LocalRoot but it didn't delete the empty folders on the server. The new script has fixed the issue when it runs with empty folders on the server it doesn't delete the files on local.

The files in the sub folders will always have an extension such as .jpg, .docx, .mp4, .mkv

The path to my folder is /media/sdh1/matt3/test this will be pretty much the same, only different is test will be another name.

This is what I ran...

$RemoteRoot = "ftpserver:test/"
$LocalRoot = "I:\"
rclone lsf --dirs-only --dir-slash=false $RemoteRoot | ForEach-Object {
    $LocalPath = Join-Path -Path $LocalRoot -ChildPath $_
    Remove-Item -Recurse -Force -LiteralPath $LocalPath -ErrorAction Ignore
    rclone move --delete-empty-src-dirs --progress --transfers=1 "${RemoteRoot}/${_}" $LocalPath
}

albertony · February 11, 2022, 7:17pm

Do you agree that my test works exactly the way you want it to?

matt3m · February 11, 2022, 7:23pm

Yes it looks correct

albertony · February 11, 2022, 7:26pm

Then I would say the remaining issue is one of:

Your paths are wrong, that is: at a different level in the hierarchy. E.g. $RemoteRoot should point deeper, into "ftpserver:test/Folder Name".
The ftp backend behaves different.

matt3m · February 11, 2022, 7:37pm

Hmmm is there a command that allows me to check the remote path so I can see if it is correct. I'm guessing it is as it's downloading the files just not removing the empty sub folders.

matt3m · February 11, 2022, 7:51pm

I updated the RemoteRoot so it is $RemoteRoot = "ftpserver:/media/sdh1/matt3/test/" and it still didn't delete the empty folders within that location.

It was deleting them previously when I running the other commands others suggested. It's only not worked when converting to PowerShell.

Running this command rclone move "ftpserver:test/" "I:\" --delete-empty-src-dirs --progress worked and removed the empty dirs.

Screenshot 2022-02-11 195207

Also running rclone move "ftpserver:/media/sdh1/matt3/test/" "I:\" --delete-empty-src-dirs --progress deleted the empty dirs as well.

albertony · February 11, 2022, 8:52pm

Ok. I think I see what is happening. My test case was probably tainted by server-side moves.

When you run:

rclone move "ftpserver:test/" "I:\" --delete-empty-src-dirs

That works as you want it to, because it deletes any empty subdirectories below ftpserver:test/.

While the powershell script will instead perform multiple of these:

rclone move "ftpserver:test/subdir1" "I:\subdir1" --delete-empty-src-dirs
rclone move "ftpserver:test/subdir2" "I:\subdir2" --delete-empty-src-dirs

Which will delete empty subdirectories below ftpserver:test/subdir1 and ftpserver:test/subdir2, but not the subdir1, subdir2 etc directories themselves. These are probably the empty directories you see left behind.

What you can do, is to after all move operations are done, execute the following, which should cleanup any empty directories just like the first command:

rclone rmdirs ftpserver:test --leave-root

So, what about:

$RemoteRoot = "ftpserver:test/"
$LocalRoot = "I:\"
rclone lsf --dirs-only --dir-slash=false $RemoteRoot | ForEach-Object {
    $LocalPath = Join-Path -Path $LocalRoot -ChildPath $_
    Remove-Item -Recurse -Force -LiteralPath $LocalPath -ErrorAction Ignore
    rclone move --progress --transfers=1 "${RemoteRoot}/${_}" $LocalPath
}
rclone rmdirs $RemoteRoot --leave-root

(The --delete-empty-src-dirs on individual rclone move is no longer needed since the rmdirs at the end will delete all empties recursively)

matt3m · February 11, 2022, 8:58pm

Absolutely perfect! Exactly what I wanted, thank you so much

If I ever needed to do it on two folders, is that hard to do?

albertony · February 11, 2022, 9:03pm

No. But getting this one folder working was in principle not hard either, it still took 93 posts to get there...

Two folders on the ftp server, to same local folder? Or two entirely separate pairs of remote and local?

matt3m · February 11, 2022, 9:07pm

Haha that’s partly my fault it took so long. I’m not very good at explaining things.

One folder would be the one just done and the would be same server different location on server and local. For example it the remote could be “ftpserver:Another Folder” and that would go to “E:\Another Folder”

albertony · February 11, 2022, 9:15pm

This should do it:

function DownloadFolders($RemoteRoot, $LocalRoot) {
    rclone lsf --dirs-only --dir-slash=false $RemoteRoot | ForEach-Object {
        $LocalPath = Join-Path -Path $LocalRoot -ChildPath $_
        Remove-Item -Recurse -Force -LiteralPath $LocalPath -ErrorAction Ignore
        rclone move --progress --transfers=1 "${RemoteRoot}/${_}" $LocalPath
    }
    rclone rmdirs $RemoteRoot --leave-root
}
DownloadFolders "ftpserver:test/" "I:\"
DownloadFolders "ftpserver:Another Folder/" "E:\Another Folder"

Now just append as many DownloadFolders "remote" "local" lines as you like..

matt3m · February 11, 2022, 9:25pm

That's perfect! Thank you so much for all your help and everyone else who has helped.

You've saved me a lot of time!

I may be back soon if I eventually get a NAS to convert it for Linux.

Again, I appreciate all the help

system · February 14, 2022, 9:25pm

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.