File Locking during copy?

Help and Support
1

Hi - I'm using rclone to copy files from my remote to my local folder which gets backed up by googleDrive (formally backup&Sync) to google Photos.

I'm noticing an issue where I'm getting partially uploaded/corrupted files in google Photos. I'm thinking that drive is picking up and starting to sync the files before they have actually finished copying with rclone.

What type of file locking/blocking is in place when doing an rclone copy ? Does it lock the file so other processes can't "use" it until its done?

Log of one such malformed pic on the google side:

2022/01/08 20:13:07.542398 DEBUG : 4 go routines active
2022/01/08 20:13:07.640391 DEBUG : rclone: Version "v1.57.0" starting with parameters ["C:\Program Files (x86)\rclone\rclone.exe" "copy" "remote:My Pictures/Tony/Automatic Upload/Tony’s iPhone/2021-12-03 11-50-36.jpeg" "C:\Users\Tony\Pictures" "--no-traverse" "--log-file=P:\scripts\log_prod_copy.txt" "--log-level" "DEBUG" "--log-format" "date,time,microseconds"]
2022/01/08 20:13:07.640906 DEBUG : Creating backend with remote "remote:My Pictures/Tony/Automatic Upload/Tony’s iPhone/2021-12-03 11-50-36.jpeg"
2022/01/08 20:13:07.640906 DEBUG : Using config file from "C:\Users\Tony\.config\rclone\rclone.conf"
2022/01/08 20:13:10.812568 DEBUG : fs cache: adding new entry for parent of "remote:My Pictures/Tony/Automatic Upload/Tony’s iPhone/2021-12-03 11-50-36.jpeg", "remote:My Pictures/Tony/Automatic Upload/Tony’s iPhone"
2022/01/08 20:13:10.812568 DEBUG : Creating backend with remote "C:\Users\Tony\Pictures"
2022/01/08 20:13:10.813081 DEBUG : fs cache: renaming cache item "C:\Users\Tony\Pictures" to be canonical "//?/C:/Users/Tony/Pictures"
2022/01/08 20:13:11.268045 DEBUG : 2021-12-03 11-50-36.jpeg: Need to transfer - File not found at Destination
2022/01/08 20:13:11.874124 DEBUG : Couldn't parse Date: from server p-def8.pcloud.com: "Sun, 09 Jan 2022 01:13:12 +0000": parsing time "Sun, 09 Jan 2022 01:13:12 +0000" as "Mon Jan _2 15:04:05 2006": cannot parse ", 09 Jan 2022 01:13:12 +0000" as " "
2022/01/08 20:13:13.413082 DEBUG : 2021-12-03 11-50-36.jpeg: md5 = d717bf0f583a44777240e89f2f236d0a OK
2022/01/08 20:13:13.413082 INFO : 2021-12-03 11-50-36.jpeg: Copied (new)
2022/01/08 20:13:13.413082 INFO :
Transferred: 3.551 MiB / 3.551 MiB, 100%, 1.767 MiB/s, ETA 0s
Transferred: 1 / 1, 100%
Elapsed time: 5.8s

2

hi,
not sure what you are asking to lock, the source or the dest?

3

The Dest/local.

Remote > local >Gphotos via drive

like I said I think that drive is picking up the file before rclone finishes copying it locally...

4

are you using rclone to upload or the some google app?

5

using the google app to upload but rclone to copy from my remote to local

(the google api support for rclone uploads to their services is not good)

6

in that debug log, rclone successfully downloaded 2021-12-03 11-50-36.jpeg and the md5 hashes matched.

so what is the exact problem you are trying to solve?

7

The problem I'm trying to solve is that I don't think rclone is LOCKING the file while its copying data - hence google drive picks it up and starts uploading it before its complete. This results in a corrupted/partial file ending up in google photos.

8

i have never used that google app, not sure how it works?

the source folder for the gdrive app is C:\Users\Tony\Pictures?
the dest folder for rclone is C:\Users\Tony\Pictures?
is that correct?

9

yep that's correct.

10

if the files are already local, why have rclone try to re-download them?

so the next question, why try to do that?
--- to use the same directory as a source with gdrive sync progream.
--- to use the same directory as a dest with rclone
--- to allow both programs to sync at the same time

11

they aren't local they are only in my remote, I use rclone to copy them locally and then gdrive to sync them.

I guess I could change it so that rclone copies to one place and then when its done MOVE them all to the gsync folder but seems like I'm just adding a step.

12

imho, if the source and dest are the same between two sync apps running at the same time, then there has to be some extra step.

for example, in this way, no need to move files.

  1. pause the gdrive sync app
  2. do the rclone copy
  3. unpause the gdrive app.
13

Yeah ... I guess its between kill the service OR use something like the windows shadow copy service.

My only issue with killing the service is if its syncing something I'm not sure how their app handles that gracefully or not - so right now I'm leaning towards shadow...

14

i did not suggest to kill any service.

from what i have read, that google sync app has built-in support to pause syncing,
i find it hard to believe that the app cannot handle pausing gracefully.
or just quit that app using its user interface

imho, you have come to the correct person to discuss VSS.

i manage a bunch of win servers and desktops and run rclone on all of them.
every time i run rclone, my script uses VSS

please see this topic and link to my rclone wiki about VSS.
I want to share how to enable vss for rclone - #6 by asdffdsa

15

Problem being is that I want to automate it - I already have a powershell script that handles the rclone download of the right files etc, but if rclone doesn't properly lock files while they are being copied then I either need to kill the sync service or use something like VSS.

If we look at the VSS approach where would that go in the flow? Script starts its rclone copy - I need the sync app to be looking at the VSS instance - rclone finishes then I need to update the VSS instance to include all the new changes??

16

i thought that the gdrive drive app does two way sync, is that a solution?

my script uses the vss mountpoint, as a source.
not sure the logic is using vss as a dest.

17

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.