Getting a lot of "can't copy - source file is being updated"

#1

Hi there, I’m getting quite a few errors like below. I’m trying to do a sync and it wont remove anything due to the errors - in a batch of around 3500 files, I’m getting 12 errors each time I run it, it seems to be the same source files having the issue each time. The source files are not being touched as far as I know (they’re old video files sitting on a network drive). I checked the last modified date on a few of them and they go back 3+ years on each.

The full line from the log is:

2018/04/29 17:22:16 ERROR : DD To 2014/DD To 2014.mp4: Failed to copy: Post https://www.googleapis.com/upload/drive/v3/files?alt=json&fields=id%2Cname%2Csize%2Cmd5Checksum%2Ctrashed%2CmodifiedTime%2CcreatedTime%2CmimeType&uploadType=resumable&upload_id=AEn2Uom-vROfZ3E809ET-w6Wt-mn_KKJbr7YEKtClzH7xwQnkyJm5tjngszZh-huHuerBFxud2OfpA-iqWv: can’t copy - source file is being updated

#2

Unix or Windows? If Unix, you can use lsof to see what’s accessing the file and if it’s really in use:

[root@gemini ~]# lsof /gmedia/
COMMAND    PID USER   FD   TYPE DEVICE    SIZE/OFF                NODE NAME
Plex\x20T 7688 plex    5r   REG   0,48 11738883541 5366970359800250097 /gmedia/Movies/Captain.America.Civil.War.(2016)/Captain.America.Civil.War.(2016).HDTV-1080p.mkv
#3

Windows unfortunately :confused: Thank you for the suggestion anyway

#4

Sorry that is not true at all. I’m running rclone on windows but the files are all stored on a freenas machine. I’m not too savvy with that stuff though. When I tried to run “lsof” I got:

“bash: lsof: command not found”

#5

Can you try this with 1.41 please? It will at least tell you why rclone thinks the file is changing.

It also has a new flag --local-no-check-updated to disable the check.

#6

Thanks for the help, you’re always so quick to reply to these… I did get a little bit more detail with 1.41 in the error but I think you can see by the error that it doesn’t really make sense… It’s change from some time in 2015 to another time in 2015. Something must be touching these files and making very minor changes to the mod time?

I’ve tried restarting the file server and the same files are getting the error.

Thanks for letting me know about the --local-no-check I can use that to get by for now but ideally it would still error (on a real error).

2018/04/30 14:26:40 ERROR : AWAT (2014)/AWAT.2014.mp4: Failed to copy: Post https://www.googleapis.com/upload/drive/v3/files?alt=json&fields=id%2Cname%2Csize%2Cmd5Checksum%2Ctrashed%2CmodifiedTime%2CcreatedTime%2CmimeType&uploadType=resumable&upload_id=AEB2Uocrdkraz-aOcePQvldZ85lFnx4IVAQ5VaPJny5SoON7KQpiu_UxjQIoe95f_z6y3oPKcgqRSf06WwcHbeAcJ: can’t copy - source file is being updated (mod time changed from 2015-01-01 20:05:46.884529 -0500 EST to 2015-01-01 20:05:46.8845294 -0500 EST)

#7

Oh whoops, I got this when trying the new flag:

2018/04/30 14:32:31 Fatal error: unknown flag: --local-no-check
C:\Users\jammi\Desktop\rclone\rclone-v1.41-windows-386>

#8

–local-no-check-updated

#9

So it looks like the modification time changed by 400nS!

That must be some OS oddity.

What file server are you using? Other users have seen things like that on GlusterFS

#10

Very weird. I’m running FreeNAS 9.2.1.5. It’s an older version now but has been running stable for some time so i’m scared to touch it :slight_smile:

Weird that its the same files over and over also.

Apologies on screwing up the flag! It’s running now and seems to be working.

#11

I think this is probably a FreeNAS bug then…

If you do ls -l --full-time /path/to/file over a period of time, you’ll probably see the modification time change there too.

Yes that is strange.

Well, the flag is a workaround for exactly your problem!

#12

Hi,

Having the same issue with sync from Windows NTFS to e-Open Data Storage version 7.0up57.
For example:
mod time changed from 2018-06-08 13:33:49.692955 +0200 CEST to 2018-06-08 13:33:49.6929556 +0200 CEST
I guess all these are decimal rounding discrepancies between (file) systems.
Is there any way to configure the time difference tolerance in rclone?

Dan

#13

That is a little strange… That is rclone asking for the modification time on a file twice and it coming back with different answers…

However it isn’t unheard of (eg with GlusterFS) so…

If you use a recent beta then you’ll find the

  --local-no-check-updated              Don't check to see if the files change during upload

Which will do what you want.

#14

I was aware of the --local-no-check-updated (found it on this forum page) and it works. I was just wondering if there is a way to configure the file stamp comparare tolerance.

#15

Oh sorry. No, rclone just checks for equality at the moment.

#16

Ok, thanks Nick. Maybe a feature to consider in a next version. --log-level Debug emits tolerance information, it would be really nice if one could configure it.

#17

Can you please make a new issue on github about this. It would be easy to do I think and would save a lot of bother!