Copy and then sync the same folders

What is the problem you are having with rclone?

I am looking to have a non-destructive way to mirror a folder on remote and a local folder. I would like to know if the following code in cronjob will do what I am expecting:

*/10 * * * * flock -n /tmp/google_drv_sync.lock
/usr/bin/rclone copy "googledrive-folder:" "/my-local-folder" &&
/usr/bin/rclone copy "/my-local-folder" "googledrive-folder:" &&
/usr/bin/rclone sync "googledrive-folder:" "/my-local-folder" &&
/usr/bin/rclone sync "/my-local-folder" "googledrive-folder:"

(The above commands are in a single line. I used linebreaks here just for readability)

I tried the first two lines and the copy happens properly.
When I move a file to a different location, I am getting duplicate files since the copy command does not delete the source files.

I was curious to know:

  1. Can I run sync commands after copy commands, assuming the sync command to remove the duplicate file in the destination after the copy command has copied the moved file.
  2. if these four commands execute sequentially waiting for one command to complete the job and then run the next command. Or will they get executed simultaneously confusing the processes?
  3. will just one file lock be enough?

Run the command 'rclone version' and share the full output of the command.

rclone v1.58.1

  • os/version: raspbian 11.3
  • os/kernel: 5.15.32-v7l+ (armv7l)
  • os/type: linux
  • os/arch: arm
  • go/version: go1.17.9
  • go/linking: static
  • go/tags: none

Which cloud storage system are you using? (eg Google Drive)

Google Drive

The command you were trying to run (eg rclone copy /tmp remote:tmp)

*/10 * * * * flock -n /tmp/google_drv_sync.lock 
/usr/bin/rclone copy "googledrive-folder:" "/my-local-folder" &&  /usr/bin/rclone copy "/my-local-folder" "googledrive-folder:" && /usr/bin/rclone sync "googledrive-folder:" "/my-local-folder" && /usr/bin/rclone sync "/my-local-folder" "googledrive-folder:" 

The rclone config contents with secrets removed.

remote> 1
--------------------
[googledrive]
type = drive
client_id = ####
client_secret = ####
scope = drive
root_folder_id = ####
team_drive =
token = 

A log from the command with the -vv flag

Paste  log here

The copy command shouldn't be making duplicate files though they do happen sometimes for reasons unknown. You can get rid of them with rclone dedupe.

As you've written it, the sync commands won't do anything. The two copy commands effectively make "googledrive-folder:" and "/my-local-folder" identical already.

They will run one after the other

It would be better to write all the statements in a bash file with a single lock like this, and call the bash file off the crontab.

#!/bin/bash

LOCKFILE="/var/lock/`basename $0`"

(
    flock -n 9 || {
	echo "$0 already running"
	exit 1
    }

    /usr/bin/rclone copy "googledrive-folder:" "/my-local-folder"
    /usr/bin/rclone copy "/my-local-folder" "googledrive-folder:"
    /usr/bin/rclone sync "googledrive-folder:" "/my-local-folder"
    /usr/bin/rclone sync "/my-local-folder" "googledrive-folder:"
) 9>$LOCKFILE

Thank you so much for taking out your precious time to give a detailed reply to my question.

When I move a fileA, let's say from source/folderA to source/folderB, the copy command is keeping the fileA in the destination/folderA and creating a new fileA in destination/folderB (while the source and destination will be googledrive and raspberry pi in one command and in the next command the reverse)

So the two copy commands are not (at least in my case) making the two folders identical. Am I missing something?

(Also I just noticed that the second copy command is bringing back the fileA to source/folderA, so that the fileA is now in both the folders on source as well as destination)

I really appreciate the clear answer that you have given to my question

not sure exactly trying to do but looks like you might want bisync
https://rclone.org/commands/rclone_bisync/

Apologies for the confusing sentence.
First copy command creates duplicate file in the destination. so the googledrive and localfolder are not identical.
The second copy command is again recreating the moved file by copying it from the localfolder to googledrive. Now the googledrive and localfolder are identical but with duplicate files.
Logically speaking this the expected behavior of the copy operation, but I was trying to understand the answer

"the copy command shouldn't be making duplicate files"

How safe is bisync? I mean is it as reliable as the native mirroring done by googledrive apps on windows?

Thank you so much for your reply.

I was meaning a duplicate file with the same name in the same directory which is only possible on Google drive and not your local filesystem.

Why don't you give it a go and see what you think? It has extensive unit tests and no bugs reported about data loss. It is a relatively new feature.

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.