Copy issue copies files already copied

[Leonardo_Parisi (Leonardo)]

What is the problem you are having with rclone?

I launched the command yesterday and it was still running this morning. What I noticed was that it was recopying all the files instead of just copying the new ones.

How is this possible?

The old files have definitely not changed as they are just pictures. The only thing is that they are very large.

Run the command 'rclone version' and share the full output of the command.

rclone v1.66.0
- os/version: Microsoft Windows 11 Pro 23H2 (64 bit)
- os/kernel: 10.0.22631.3296 (x86_64)
- os/type: windows
- os/arch: amd64
- go/version: go1.22.1
- go/linking: static
- go/tags: cmount

Which cloud storage system are you using?

Synology

The command you were trying to run

copy D:\DATA Synology:Crono/POLICLINICO/RAW --sftp-ssh "ssh xxxxx@login.roma1.infn.it" --sftp-path-override /volume2/Crono/POLICLINICO/RAW

[kapitainsky]

@Leonardo_Parisi

What I noticed was that it was recopying all the files instead of just copying the new ones.

How is this possible?

Nobody can see your screen. If you want somebody to try to figure out what is happening please provide debug logfile.

[Leonardo_Parisi (Leonardo)]

you are right...as soon as it finishes making the copy I will try to debug it.

Anyway what I need to transfer are very large photos of about 100Mb each that are produced daily.

I saw the options:

--max-age 24h --no-traverse

but while of the first one I understood the meaning of the second one I didn't understand what it is for?

also i saw the topic Rclone copy problem - copies files already copied and i can't understand what is the purpose of the option

--ignore-existing

shouldn't the copy command already not copy identical files?

also is it possible that being large photos he wastes time calculating the MD5SUM?

[kapitainsky]

@Leonardo_Parisi

--max-age 24h --no-traverse

but while of the first one I understood the meaning of the second one I didn't understand what it is for?

--no-traverse saves time by not listing all destination. Effectively not checking whether files already exist or not. It makes sense when destination is very large (it takes time to list it) and you copy only few new files.

@Leonardo_Parisi

also i saw the topic Rclone copy problem - copies files already copied and i can't understand what is the purpose of the option

--ignore-existing

--ignore-existing

Using this option will make rclone unconditionally skip all files that exist on the destination, no matter the content of these files.

I am sure you do not need it for what you described.

[Leonardo_Parisi (Leonardo)]

Thank you very much for the info.

But I am a moment confused.

--no-traverse has the effect of not checking whether the file exists or not. so if it exists he copies it again anyway.

while

--ignore-existing I have the effect of not checking the "content" of the file. so if a file already exists in the destination it doesn't copy it no matter what.

the simple command

rclone copy source:sourcepath dest:destpath

should instead copy only files that are not in dst or that have been modified. To do this does he have to check the MD5SUM every time?

[kapitainsky]

@Leonardo_Parisi

should instead copy only files that are not in dst or that have been modified. To do this does he have to check the MD5SUM every time?

by default differences are detected by testing by size and modification time. To check by checksum you have to force it by providing --checksum flag (assuming that both source and destination support some common checksum mechanism).

run your copy command with -vv (debug output) flag and log will tell you what is happening..

[Leonardo_Parisi (Leonardo)]

So sorry for the silence.
Only recently have I been able to find time to analyze what happened.

Result:
I found out that my boss like a moron had changed the name of a folder.
This explains why Rclone was recopying everything.