Rclone move --delete-empty-src-dirs different if it's a server side vs non-server side?

What is the problem you are having with rclone?

rclone move --delete-empty-src-dirs acting differently (removing or not the directory) depending on being a server side move or not?

Use case: I want to rename some directories and some files. If I do rclone move for directories (using SharePoint) the modified time of the files in the directories gets updated if using the WebDav API to access SharePoint. We use the WebDav API for another software (not related to rclone).

I'm trying to rename many directories keeping the modified date of the files.

Some weeks ago to fix a problem with this (not related to rclone, it was using SharePoint Web) I fixed the modified time visible to WebDav with Python+requests (doing a PATCH to the files, kudos for rclone -vv --dump bodies so I could see what rclone was doing in similar cases to update the date modified date). Long story for another day (and 6 or 7 hours of processing...).

Now I'm trying to move some other directories in a way that feels less fragile.

What I've learned:

  • If I do rclone move it does a server side move (hurray!) and in my opinion Microsoft has a bug where the WebDav modified date for the files in the directory get updated. I could technically then fix it with my Python+requests+PATCH but I'm trying to avoid this
  • I thought of doing rclone "move" from a remote to another remote (from rclone point of view). Now I have an rclone-config with two remotes pointing to the same SharePoint library :slight_smile: to "force" rclone to NOT do a server side copy. In the examples below when I have "test-SharePoint-2" and without the "-2" is the same config in two sections. rclone does the move it but it doesn't delete the source directory (but yes the files).

I've see though: --delete-empty-src-dirs flag not working - #5 by thestigma that it says that it never deletes the root directory? But it does (user point of view, or renames) if it's a server side move.

I want to say that if I do:

rclone --delete-empty-src-dirs --ignore-size --ignore-checksum "test-SharePoint:test41/" "test-SharePoint:test42/"

The source directory disappears (well, it gets moved on the server side)

If I do:

rclone --delete-empty-src-dirs --ignore-size --ignore-checksum "test-SharePoint:test41/" "test-SharePoint-2:test42/"

I would expect the same semantics but slow (download+upload). But the test41 directory stays (empty).

For my case I guess that I can just add && rclone rmdir test-SharePoint:test41/ after the move (not tested).

But I wondered if the rclone move should behave the same for the server side and client side?

What is your rclone version (output from rclone version)

rclone v1.55.0
- os/type: linux
- os/arch: amd64
- go/version: go1.16.2
- go/linking: static
- go/tags: cmount

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Linux 64 bit (Debian)

Which cloud storage system are you using? (eg Google Drive)

Microsoft SharePoint

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone move --delete-empty-src-dirs --ignore-size --ignore-checksum "test-SharePoint:test39/" "test-SharePoint:test40/"

And:

rclone move --delete-empty-src-dirs --ignore-size --ignore-checksum "test-SharePoint:test39/" "test-SharePoint-2:test40/"

The rclone config contents with secrets removed.

[test-SharePoint-2]
type = onedrive
token = {the token}
drive_id = the drive_id
drive_type = documentLibrary

A log from the command with the -vv flag

carles@pinux:~$ rclone -vv move --delete-empty-src-dirs --ignore-size --ignore-checksum "test-SharePoint:test41/" "test-SharePoint-2:test42/"
2021/04/08 14:16:38 DEBUG : Using config file from "/home/carles/.rclone.conf"
2021/04/08 14:16:38 DEBUG : rclone: Version "v1.55.0" starting with parameters ["rclone" "-vv" "move" "--delete-empty-src-dirs" "--ignore-size" "--ignore-checksum" "test-SharePoint:test41/" "test-SharePoint-2:test42/"]
2021/04/08 14:16:38 DEBUG : Creating backend with remote "test-SharePoint:test41/"
2021/04/08 14:16:39 DEBUG : fs cache: renaming cache item "test-SharePoint:test41/" to be canonical "test-SharePoint:test41"
2021/04/08 14:16:39 DEBUG : Creating backend with remote "test-SharePoint-2:test42/"
2021/04/08 14:16:41 DEBUG : fs cache: renaming cache item "test-SharePoint-2:test42/" to be canonical "test-SharePoint-2:test42"
2021/04/08 14:16:42 DEBUG : One drive root 'test42': Waiting for checks to finish
2021/04/08 14:16:42 DEBUG : One drive root 'test42': Waiting for transfers to finish
2021/04/08 14:16:44 DEBUG : date5.txt: Starting multipart upload
2021/04/08 14:16:45 DEBUG : date5.txt: Uploading segment 0/5 size 5
2021/04/08 14:16:46 INFO  : date5.txt: Copied (new)
2021/04/08 14:16:47 INFO  : date5.txt: Deleted
2021/04/08 14:16:47 INFO  : 
Transferred:   	         5 / 5 Bytes, 100%, 1 Bytes/s, ETA 0s
Checks:                 2 / 2, 100%
Deleted:                1 (files), 0 (dirs)
Renamed:                1
Transferred:            1 / 1, 100%
Elapsed time:         8.9s

2021/04/08 14:16:47 DEBUG : 9 go routines active

This is a historical accident, and yes you are right it is inconsistent.

Lots of users use rclone move /path/to/ingest remote:ingest to move a whole load of files to a remote. Imagine a load of files in an ingest directory. Here you don't want the ingest directory removed as there might be more files arriving in there at any moment.

However for a server side move we are literally moving the directory so we don't leave the old directory behind.

I can't find the discussion now, but I attempted to make the two commands consistent but there was user backlash :wink:

rclone move could have a --remove-root flag which might be useful.

You can also use rclone rmdirs (note the extra s) to remove the directory and any empty subdirs.

Thanks Nick for the explanation! I thought that would be something like what you described :slight_smile: good to confirm. A flag is always useful, quite a few times i had a "hurray" moment finding an obscure flag that does what I need! :slight_smile:

For my case (testing needed) I've split my cases between:

  • rename a file: rclone copy between remotes && rclone deletefile of the first file
  • rename a directory: rclone copy between remotes && rclone purge of the source directory

So I think (next week more tests and pair planning) that I'm good to go! It's only 40 files and 15 directories but I want to avoid mistakes. The script generates rclone commands that I might do a copy-paste "go to the SharePoint and verify", do the next one, etc.

Have a good weekend!

If you want to be belt and braces insert an rclone check after the rclone copy.

Note that rclone move does essentially rclone copy + rclone delete if you don't want the extra assurances.

Avoiding mistakes is good when it comes to transferring data. That's rclone's philosophy too!

Reminder: I'm not doing rclone move for the problem with SharePoint that then the WebDav API reports the modified date of the files wrongly. Long story out of our control (I'd like to report this to Microsoft, I need to find where is the best place!).

Because it's SharePoint the file size and hash changes on the upload (well, I'll check if it happens in this particular case that we take one file from a remote and put it in the same remote... but I suspect that will change it well).

Would rclone check still work for SharePoint? I can check later (weekend or Monday) but you might have some interesting comment :slight_smile:

I would (the generated script) do only rclone deletefile / rclone purge if the rclone copy exit code is 0, that's for sure.

Thank you very much!

Ah OK.

:wink: Story of my life!

Hmm, not sure. If you can find a relevant GitHub repo, they seem responsive there.

OK

It should do but if the hashes change it will be less useful. If nothing else it will report that all the files are present which you may find useful.

:+1:

Hi Nick,

A follow-up, off-topic (in this thread title) question:

Feature request (or question about flags): I would like to be able to set remotes as read-only in the config file. When it's on an S3 backend I can generate read-only keys, but in some remotes this might not be possible (e.g. I think on FTP would not be possible, and as far as I know on SharePoint which is the case for me now is not possible). Even in S3 alike in one of the systems that I use creating read-only keys is not trivial.

Have you thought of this?

Thank you!

It is a new idea to me :slightly_smiling_face:

A backend flash would need implenting in all 30 backends and checking in 15 places in each one, so a lot of work.

It would be easier to make a read only alias for a remote, so like the alias backend but read only: Alias

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.