Issues doing an rclone sync to a network server

Help and Support
1

Hi!,

I’m trying to set up rclone on my Windows 10 machine to sync several folders on scheduled intervals to my unRAID server in the network so I can keep an updated backup of important things like documents, scripts, and stuff like that (which the server will then upload to ACD) but I can’t seem to get it to transmit anything.

I’ve setup the destination share as an SFTP remote like this:

--------------------
[vault]
type = sftp
host = 192.168.1.2
user = root
port = 22
pass = *** ENCRYPTED ***
--------------------

And use command to start the sync:

rclone sync -v --stats=5s --filter-from "G:\generic-rclone-filter.txt" G:\documents vault:/mnt/user/backups/desktop/documents

The transfer process seems to start and I get the stats output but after 2 hours running this is what I keep getting:

2017/04/01 02:45:48 INFO  :
Transferred:      0 Bytes (0 Bytes/s)
Errors:                 0
Checks:                 0
Transferred:            0
Elapsed time:      2h2m0s

Transferring 1.6GB over local network shouldn’t take anywhere near that long haha.

The weird thing is that if I manually copy the documents folder to the destination then run sync it will delete/create any removed or new files as expected. It seems to only have the problem when doing the initial sync to an empty directory.

If the destination directory doesn’t exist then it doesn’t do anything and just gets stuck repeating the stats above. If I create the /documents/ folder in the destination it copies one random file then gets stuck listing a couple others but never transfers anything else.

2017/04/01 02:54:11 INFO  :
Transferred:      0 Bytes (0 Bytes/s)
Errors:                 0
Checks:                 0
Transferred:            1
Elapsed time:       2m55s
Transferring:
*                             3DMark/3DMark.log:  0% done, 0 Bytes/s, ETA: -
*                    StarCraft II/Variables.txt:  0% done, 0 Bytes/s, ETA: -
*                StarCraft II/Laucien_343@1.lnk:  0% done, 0 Bytes/s, ETA: -
*              StarCraft II/EditorVariables.txt:  0% done, 0 Bytes/s, ETA: -

Is there anything I’m missing? Not getting right? Or plain just screwing up somwhere?.

2

Bump. Any thoughts on this? It’s making using rclone to sync huge/many files from my Windows PC to my NAS completely impossible.

Tried with both the 32 and 64 version and no luck.

Right now I’m doing a workaround by mounting the network drive I want to sync files to and using ‘local’ type as the destination but would appreciate not having to mount every folder I want to copy to as a different drive haha.

3

Hmm that is strange!

Can you try with -vv and post the log.

Also try with the latest beta and see if that makes a difference.

4

OK, sorry for the delay but finally got around to check this again!.

Downloaded the latest beta version from Github:

PS C:\> rclone -V
rclone v1.36-22-gbc25190β

Windows version I’m using:

PS C:\> [System.Environment]::OSVersion.Version

Major  Minor  Build  Revision
-----  -----  -----  --------
10     0      15063  0

Here’s how the destination is set up:

--------------------
[vault]
type = sftp
host = 192.168.1.2
user = root
port = 22
pass = *** ENCRYPTED ***
--------------------
Edit remote
Value "host" = "192.168.1.2"
Edit? (y/n)>
y) Yes
n) No
y/n>

And the remote works as I can check with this:

PS C:\> rclone lsd vault:/
          -1 2017-04-10 21:59:15        -1 .config
          -1 2017-04-11 23:51:01        -1 tmp
          -1 2017-04-11 00:00:54        -1 etc
          -1 2017-04-10 21:58:13        -1 proc
          -1 2017-04-10 17:53:14        -1 usr
          -1 2017-04-10 21:59:08        -1 boot
          -1 2017-04-10 21:58:23        -1 sys
          -1 2017-04-10 21:58:46        -1 lib64
          -1 2017-04-10 21:58:58        -1 mnt
          -1 2017-04-10 21:58:53        -1 sbin
          -1 2017-03-30 17:54:03        -1 home
          -1 2017-03-30 17:54:07        -1 lib
          -1 2017-04-10 21:59:56        -1 run
          -1 2015-12-18 01:42:24        -1 bin
          -1 2017-02-09 15:42:23        -1 var
          -1 2017-04-11 04:40:01        -1 root
          -1 2017-04-10 21:59:00        -1 dev

Here are the logs for 2 different retries.
When the destination folder doesn’t exist:

PS C:\> rclone -vv sync G:\documents\ vault:/mnt/user/backups/documents2
2017/04/11 23:39:47 DEBUG : rclone: Version "v1.36-22-gbc25190β" starting with parameters ["C:\\dev_stuff\\rclone\\rclone.exe" "-vv" "sync" "G:\\documents\\" "vault:/mnt/user/backups/documents2"]
2017/04/11 23:39:47 INFO  : sftp://root@192.168.1.2:22//mnt/user/backups/documents2: Modify window is 1s
2017/04/11 23:40:47 INFO  :
Transferred:      0 Bytes (0 Bytes/s)
Errors:                 0
Checks:                 0
Transferred:            0
Elapsed time:        1m0s

It just gets stuck there and doesn’t copies anything.

And here’s what happens when the destination /documents2/ folder exists:

PS C:\> rclone -vv sync G:\documents\ vault:/mnt/user/backups/documents2
2017/04/11 23:42:32 DEBUG : rclone: Version "v1.36-22-gbc25190β" starting with parameters ["C:\\dev_stuff\\rclone\\rclone.exe" "-vv" "sync" "G:\\documents\\" "vault:/mnt/user/backups/documents2"]
2017/04/11 23:42:32 INFO  : sftp://root@192.168.1.2:22//mnt/user/backups/documents2: Modify window is 1s
2017/04/11 23:42:32 INFO  : desktop.ini: Copied (new)
2017/04/11 23:42:33 INFO  : Default.rdp: Copied (new)
2017/04/11 23:43:32 INFO  :
Transferred:    418 Bytes (6 Bytes/s)
Errors:                 0
Checks:                 0
Transferred:            2
Elapsed time:        1m0s
Transferring:
 *                             3DMark/3DMark.log:  0% done, 0 Bytes/s, ETA: -
 * ...mark-autosave-20160712111024.3dmark-result:  0% done, 0 Bytes/s, ETA: -
 * ...mark-autosave-20160712111829.3dmark-result:  0% done, 0 Bytes/s, ETA: -
 * ...mark-autosave-20160712113202.3dmark-result:  0% done, 0 Bytes/s, ETA: -

2017/04/11 23:44:32 INFO  :
Transferred:    418 Bytes (3 Bytes/s)
Errors:                 0
Checks:                 0
Transferred:            2
Elapsed time:        2m0s
Transferring:
 *                             3DMark/3DMark.log:  0% done, 0 Bytes/s, ETA: -
 * ...mark-autosave-20160712111024.3dmark-result:  0% done, 0 Bytes/s, ETA: -
 * ...mark-autosave-20160712111829.3dmark-result:  0% done, 0 Bytes/s, ETA: -
 * ...mark-autosave-20160712113202.3dmark-result:  0% done, 0 Bytes/s, ETA: -

It copied 2 random files but everything else got stuck with 0 Bytes transferred.

Sadly I have absolutely no idea what the issue could be =(. Are there any other logs or info I could provide you with?.

5

Bumping for when you have time to look into this :D.

6

Did you try to mount the remote folder and just copy one file manually ?

Can’t it be a right problem : The account used have read only right on destination folder ?

7

Yeah, it actually copies 2 files (always the same 2) before freezing and if I manually copy the entire folder and then just run syncs to keep it in syncs it actually works. It seems to have a problem when moving big amounts of data. I’ve been able to reproduce this behavior with 3 different folders/file sets and they all were at least 3 or 4 gigs.

I’m honestly discarding access right issues as I’m using the root account on the destination server to connect through SFTP.

8

Have you ever tried “local file system” instead of “SFTP” ?

9

It’s the workaround I mentioned on my second post yeah, but having to map every possible destination on that server I might end up wanting to rclone to isn’t the most optimal solution.

Using rclone with a local drive or a mapped network drive as the destination works great. It’s the SFTP option that, in this case at least, doesn’t seem to work properly.