Rclone sync related

Help and Support
1

Hi All,

In my current setup I have created a remote for s3 bucket and running rclone sync command via cron tab every 5 mins. I am wondering how rclone sync internally works, I have couple of queries which i am looking an answer for.

a. For rsync are all the files scanned on source and destination every-time or there is some mechanism using which rclone keep track of files previously synced and they are skipped. I presume it keeps some record of files already synced somewhere, Kindly let me know where it keeps track of such information and how can we see and interpret it.
b. Do we need to do any special setting for syncing big files for e.g of 10GB or more.
c. Is there any better way of running it than scheduling it to run via cron.
d. If during a sync a file is deleted from source how it is handled. I give a use case here for better clarity.
Say there are 10 files marked for sync say 8 files of 10 GB and 2 files of 1MB. Sync started with 10 GB file but during this time 1 MB file got deleted how it will be handled during the sync.

Your replies with be highly appreciated.
Thanks,
Ali Sajjad

2

hello and welcome to the forum,

you should read this to understand how sync works
https://rclone.org/commands/rclone_sync/

4

Thanks for your response, Information you shared is really helpful.

5

Hi,

I observed a strange behavior, not sure if rclone provides a way to override this behavior.

If there is a process say A which is continuously writing to a file and then rclone sync starts , process A fails to write file any longer and throws an error message which says
"The process cannot access the file 'D:\1.txt' because it is being used by another process"

Kindly advise.

6
  • which process cannot access the file when rclone is running?
  • what do you mean 'throws' an error, what exactly happens?
  • post the rclone debug log.
1 Like
7

Your cron rclone processes are starting before the prior ones have finished.

Maybe just do a pgrep for rclone, and cancel in your script so that the instances don't conflict. Running less often might reduce AWS bills too.

8

He's not running Unix/Linux because he asked about:

The issue is you have a file in use and on Windows, you either need to use VSS as someone has a nice tutorial on that or close the file out before you copy it.

Or ditch Windows and move to a real OS :slight_smile:

9

This has been a linux day... Sorry.

10

if you have questions about VSS, i wrote that wiki.
let me know if you have questions.

11

Thanks mates for taking out time to answer my queries. It seems I missed few details which caused some confusion. Let me explain the scenario below, I wish to understand if we have any flag or workaround to override this behavior.

Test Case: If a file is continuously getting written/updated and rclone is triggered to sync this file to s3 bucket, write operations should continue without any impact.
Result= Failed.
Actual Behavior: Rcone locks the file and write operations fail.

Setup: We have a powershell script which is just appending lines to a file in a windows environment.
**PowerShell script:**This script is just updating a a file D:\alpha\Test-structure\1.txt

param([string]$src)
while($true)
{
$NumArray = (1..1)
ForEach ($number in $numArray)
{
if (Test-Path $src$number.txt)
{
Add-Content $src$number.txt " Hello"
write-host "$(Get-Date) : New Content appended to $src$number.txt file"
}
}
}

Rsync command:

.\rclone.exe sync D:\alpha\Test-structure s3cloneuser:cleardr-europe/mum-wip --log-level=DEBUG --log-file=rsynclog.txt

Error msg of powershell:

    Add-Content : The process cannot access the file 
    'D:\alpha\Test-structure\1.txt' because it is being used by another process.
    At E:\script\modify_present_file_in_D.ps1:9 char:2
    +     Add-Content $src\$number.txt " Charlie one 2 three testing 221144 'n 
    Charlie on ...
    +    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    ~~~~~~
    + CategoryInfo          : WriteError: (D:\alpha\Test-structure\1.txt:Strin 
    g) [Add-Content], IOException
    + FullyQualifiedErrorId : GetContentWriterIOError,Microsoft.PowerShell.Com 
    mands.AddContentCommand

Best Regards,
Ali Sajjad

12

As noted above, use VSS if you want to backup on Windows:

13

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.