Rclone move command - Why is it renaming files and what files are they?

Help and Support
1

What is the problem you are having with rclone?

While running rclone move command, I noticed it shows "Renamed: xx ". I am migrating backup data from an East to West location, then out to cloud for offsite redundant copy and need to ensure I have an exact copy at every step so I am wondering why it is renaming files and what files those are.

What is your rclone version (output from rclone version)

rcloen 1.53.1

Which OS you are using and how many bits (eg Windows 7, 64 bit)

CentOS Linux release 7.8.2003

Which cloud storage system are you using? (eg Google Drive)

sftp - source is Centos 7.8 , Destination is RHEL 8.2

The command you were trying to run (eg rclone copy /tmp remote:tmp)

The rclone config contents with secrets removed.

Paste config here

[yb-sd-stor1]
type = sftp
host = x.x.x.x
user = rcloneserviceacct
disable_hashcheck = false
key_pem = -----BEGIN OPENSSH PRIVATE KEY----- OMITTED \n-----END OPENSSH PRIVATE KEY-----\n
md5sum_command = md5sum
sha1sum_command = sha1sum

A log from the command with the -vv flag

Transferred:       58.694M / 58.694 MBytes, 100%, 2.499 MBytes/s, ETA 0s
Checks:             20209 / 20209, 100%
Deleted:            10104
Renamed:               67
Transferred:           67 / 67, 100%
Elapsed time:      6m10.3s
2

hello and welcome to the forum,

you would need to post the command and debug log.

3

I'm guessing you've got --backup-dir set, but if you could post your command line we can work it out :slight_smile:

Nice to see people using brand new features :slight_smile:

4

I run the following copy command to keep the data flowing from East to West:

rclone copy ${bundle_source}${bundles} ${host_dest}${bundles} -P --transfers=100 --checkers=200 --create-empty-src-dirs --max-backlog=300000

I'll then create a new full backup aka "bundle" and then move the old bundle so it only exists in the west using the following command:

rclone move ${bundle_source}${bundles} ${host_dest}${bundles} -P --transfers=100 --checkers=200 --create-empty-src-dirs --delete-empty-src-dirs --max-backlog=300000

It's in this "move" command after a "copy" operation that I'll see the file rename issue. I'm guessing it's detecting a duplicate file or file name during the move and so it is renaming the file but I'm curious as to why it would even try to copy a file that already exists in the destination, thus the question about why it's renaming files and how to log the files it's renaming.

I have noticed if I do a sync on a per bundle basis, then a move the issue goes away so I've been using that as a workaround. I have lost data in the past using "sync" due to user error so I tend to be a little gun-shy using it and prefer the copy followed by move command. Excellent tool though, I use it to manage approx. 1.5PB of data and this is the only issue I've run into so far. :slightly_smiling_face:

5

if you post the debug log, then we can see the exact command and debug info.

add -vv to your command.

6

Depending on what the source and destination are an rclone move can either be a Copy+Delete or a server side move. It is these latter server side moves which will cause Renamed: stats.

So if ${bundle_source}${bundles} and ${host_dest}${bundles} were on the same machine then that would make sense of the Renamed: stats.

If the source is mostly the same as the dest then you'll only get the Renamed: stats when rclone is filling in a gap.

Yes rclone sync is a dangerous tool!

1 Like
7

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.