Rclone does not copy some files during sync

outofphase · October 2, 2022, 12:12pm

What is the problem you are having with rclone?

I have 2 remotes:

an SFTP remote orchestra-jammy that connects to a local Linux container
a Swift remote memstore that connects to a cloud storage provider

I want to backup a file-tree from the first to the second, using rclone sync. However some files are not transferred.

Run the command 'rclone version' and share the full output of the command.

david@alpe:~/Bin/rclone-v1.59.2-linux-amd64$ ./rclone version
rclone v1.59.2

os/version: ubuntu 22.04 (64 bit)
os/kernel: 5.15.0-48-generic (x86_64)
os/type: linux
os/arch: amd64
go/version: go1.18.6
go/linking: static
go/tags: none

Which cloud storage system are you using? (eg Google Drive)

Memset Swift object storage ("Memstore").

The command you were trying to run (eg `rclone copy /tmp remote:tmp`)

The contents of a directory in the source remote:

david@alpe:~/Bin/rclone-v1.59.2-linux-amd64$ ./rclone lsd orchestra-jammy:github/hightouch-roe/.git
          -1 2022-08-16 21:25:45        -1 branches
          -1 2022-08-16 21:25:45        -1 hooks
          -1 2022-08-16 21:25:45        -1 info
          -1 2022-09-05 12:32:16        -1 logs
          -1 2022-08-16 21:31:13        -1 modules
          -1 2022-09-29 21:28:15        -1 objects
          -1 2022-09-29 17:14:22        -1 refs

The copy operation, note that "branches" is not in the log. Only head and tail shown as full log too long.

david@alpe:~/Bin/rclone-v1.59.2-linux-amd64$ ./rclone sync -vv orchestra-jammy:github/hightouch-roe/.git memstore:orchestra-jammy/github/hightouch-roe/.git
2022/10/02 12:57:05 DEBUG : rclone: Version "v1.59.2" starting with parameters ["./rclone" "sync" "-vv" "orchestra-jammy:github/hightouch-roe/.git" "memstore:orchestra-jammy/github/hightouch-roe/.git"]
2022/10/02 12:57:05 DEBUG : Creating backend with remote "orchestra-jammy:github/hightouch-roe/.git"
2022/10/02 12:57:05 DEBUG : Using config file from "/home/david/.config/rclone/rclone.conf"
2022/10/02 12:57:05 DEBUG : sftp://ubuntu@10.51.175.82:22/github/hightouch-roe/.git: New connection 10.51.175.1:50748->10.51.175.82:22 to "SSH-2.0-OpenSSH_8.9p1 Ubuntu-3"
2022/10/02 12:57:05 DEBUG : sftp://ubuntu@10.51.175.82:22/github/hightouch-roe/.git: Shell type "unix" from config
2022/10/02 12:57:05 DEBUG : sftp://ubuntu@10.51.175.82:22/github/hightouch-roe/.git: Relative path resolved to "/home/ubuntu/github/hightouch-roe/.git"
2022/10/02 12:57:05 DEBUG : sftp://ubuntu@10.51.175.82:22/github/hightouch-roe/.git: Using root directory "/home/ubuntu/github/hightouch-roe/.git"
2022/10/02 12:57:05 DEBUG : Creating backend with remote "memstore:orchestra-jammy/github/hightouch-roe/.git"
2022/10/02 12:57:06 DEBUG : sftp://ubuntu@10.51.175.82:22/github/hightouch-roe/.git: New connection 10.51.175.1:50760->10.51.175.82:22 to "SSH-2.0-OpenSSH_8.9p1 Ubuntu-3"
2022/10/02 12:57:06 DEBUG : ORIG_HEAD: Size and modification time the same (differ by 164.024855ms, within tolerance 1s)
2022/10/02 12:57:06 DEBUG : ORIG_HEAD: Unchanged skipping
2022/10/02 12:57:06 DEBUG : index: Size and modification time the same (differ by 479.542077ms, within tolerance 1s)
2022/10/02 12:57:06 DEBUG : index: Unchanged skipping
2022/10/02 12:57:06 DEBUG : HEAD: Size and modification time the same (differ by 239.858083ms, within tolerance 1s)
2022/10/02 12:57:06 DEBUG : HEAD: Unchanged skipping
2022/10/02 12:57:06 DEBUG : FETCH_HEAD: Size and modification time the same (differ by 539.177569ms, within tolerance 1s)
2022/10/02 12:57:06 DEBUG : FETCH_HEAD: Unchanged skipping
2022/10/02 12:57:06 DEBUG : COMMIT_EDITMSG: Size and modification time the same (differ by 528.112043ms, within tolerance 1s)
2022/10/02 12:57:06 DEBUG : COMMIT_EDITMSG: Unchanged skipping
2022/10/02 12:57:06 DEBUG : description: Size and modification time the same (differ by 292.224438ms, within tolerance 1s)
2022/10/02 12:57:06 DEBUG : description: Unchanged skipping
2022/10/02 12:57:06 DEBUG : config: Size and modification time the same (differ by 503.578855ms, within tolerance 1s)
2022/10/02 12:57:06 DEBUG : config: Unchanged skipping
2022/10/02 12:57:06 DEBUG : sftp://ubuntu@10.51.175.82:22/github/hightouch-roe/.git: New connection 10.51.175.1:50774->10.51.175.82:22 to "SSH-2.0-OpenSSH_8.9p1 Ubuntu-3"
2022/10/02 12:57:06 DEBUG : sftp://ubuntu@10.51.175.82:22/github/hightouch-roe/.git: New connection 10.51.175.1:50786->10.51.175.82:22 to "SSH-2.0-OpenSSH_8.9p1 Ubuntu-3"
2022/10/02 12:57:06 DEBUG : sftp://ubuntu@10.51.175.82:22/github/hightouch-roe/.git: New connection 10.51.175.1:50794->10.51.175.82:22 to "SSH-2.0-OpenSSH_8.9p1 Ubuntu-3"
2022/10/02 12:57:06 DEBUG : logs/HEAD: Size and modification time the same (differ by 239.858083ms, within tolerance 1s)
2022/10/02 12:57:06 DEBUG : logs/HEAD: Unchanged skipping
2022/10/02 12:57:06 DEBUG : sftp://ubuntu@10.51.175.82:22/github/hightouch-roe/.git: New connection 10.51.175.1:50806->10.51.175.82:22 to "SSH-2.0-OpenSSH_8.9p1 Ubuntu-3"
2022/10/02 12:57:06 DEBUG : sftp://ubuntu@10.51.175.82:22/github/hightouch-roe/.git: New connection 10.51.175.1:50808->10.51.175.82:22 to "SSH-2.0-OpenSSH_8.9p1 Ubuntu-3"
2022/10/02 12:57:06 DEBUG : hooks/pre-applypatch.sample: Size and modification time the same (differ by 292.224438ms, within tolerance 1s)
2022/10/02 12:57:06 DEBUG : hooks/fsmonitor-watchman.sample: Size and modification time the same (differ by 292.224438ms, within tolerance 1s)
2022/10/02 12:57:06 DEBUG : hooks/fsmonitor-watchman.sample: Unchanged skipping
2022/10/02 12:57:06 DEBUG : hooks/post-update.sample: Size and modification time the same (differ by 292.224438ms, within tolerance 1s)
2022/10/02 12:57:06 DEBUG : hooks/post-update.sample: Unchanged skipping
2022/10/02 12:57:06 DEBUG : hooks/applypatch-msg.sample: Size and modification time the same (differ by 292.224438ms, within tolerance 1s)
2022/10/02 12:57:06 DEBUG : hooks/applypatch-msg.sample: Unchanged skipping
2022/10/02 12:57:06 DEBUG : hooks/pre-applypatch.sample: Unchanged skipping
2022/10/02 12:57:06 DEBUG : hooks/commit-msg.sample: Size and modification time the same (differ by 292.224438ms, within tolerance 1s)
2022/10/02 12:57:06 DEBUG : hooks/commit-msg.sample: Unchanged skipping
2022/10/02 12:57:06 DEBUG : hooks/pre-merge-commit.sample: Size and modification time the same (differ by 292.224438ms, within tolerance 1s)
2022/10/02 12:57:06 DEBUG : hooks/pre-merge-commit.sample: Unchanged skipping
2022/10/02 12:57:06 DEBUG : hooks/pre-push.sample: Size and modification time the same (differ by 292.224438ms, within tolerance 1s)
2022/10/02 12:57:06 DEBUG : hooks/pre-push.sample: Unchanged skipping
2022/10/02 12:57:06 DEBUG : hooks/pre-rebase.sample: Size and modification time the same (differ by 292.224438ms, within tolerance 1s)
2022/10/02 12:57:06 DEBUG : hooks/pre-rebase.sample: Unchanged skipping
2022/10/02 12:57:06 DEBUG : hooks/pre-commit.sample: Size and modification time the same (differ by 292.224438ms, within tolerance 1s)
2022/10/02 12:57:06 DEBUG : hooks/pre-commit.sample: Unchanged skipping
2022/10/02 12:57:06 DEBUG : hooks/pre-receive.sample: Size and modification time the same (differ by 292.224438ms, within tolerance 1s)
2022/10/02 12:57:06 DEBUG : hooks/pre-receive.sample: Unchanged skipping
2022/10/02 12:57:06 DEBUG : hooks/prepare-commit-msg.sample: Size and modification time the same (differ by 292.224438ms, within tolerance 1s)
2022/10/02 12:57:06 DEBUG : hooks/prepare-commit-msg.sample: Unchanged skipping
2022/10/02 12:57:06 DEBUG : sftp://ubuntu@10.51.175.82:22/github/hightouch-roe/.git: New connection 10.51.175.1:50824->10.51.175.82:22 to "SSH-2.0-OpenSSH_8.9p1 Ubuntu-3"
2022/10/02 12:57:06 DEBUG : hooks/update.sample: Size and modification time the same (differ by 292.224438ms, within tolerance 1s)
2022/10/02 12:57:06 DEBUG : hooks/update.sample: Unchanged skipping
2022/10/02 12:57:06 DEBUG : hooks/push-to-checkout.sample: Size and modification time the same (differ by 292.224438ms, within tolerance 1s)
2022/10/02 12:57:06 DEBUG : hooks/push-to-checkout.sample: Unchanged skipping

[snip]

2022/10/02 12:57:32 DEBUG : Swift container orchestra-jammy path github/hightouch-roe/.git: Waiting for transfers to finish
2022/10/02 12:57:32 DEBUG : Waiting for deletions to finish
2022/10/02 12:57:32 INFO  : There was nothing to transfer
2022/10/02 12:57:32 INFO  : 
Transferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Checks:               540 / 540, 100%
Elapsed time:        27.9s

2022/10/02 12:57:32 DEBUG : 100 go routines active
2022/10/02 12:57:32 DEBUG : sftp://ubuntu@10.51.175.82:22/github/hightouch-roe/.git: Closing 8 unused connections

The contents of the destination after the copy, note branches is missing.

david@alpe:~/Bin/rclone-v1.59.2-linux-amd64$ ./rclone lsd memstore:orchestra-jammy/github/hightouch-roe/.git
           0 2022-10-02 13:00:04        -1 hooks
           0 2022-10-02 13:00:04        -1 info
           0 2022-10-02 13:00:04        -1 logs
           0 2022-10-02 13:00:04        -1 modules
           0 2022-10-02 13:00:04        -1 objects
           0 2022-10-02 13:00:04        -1 refs

FYI, the missing directory is a regular directory, not a symlink:

ubuntu@orchestra-jammy:~/github/hightouch-roe/.git$ ls -l
total 42
-rw-rw-r--   1 ubuntu ubuntu  357 Sep 29 20:28 COMMIT_EDITMSG
-rw-rw-r--   1 ubuntu ubuntu   92 Sep 26 22:16 FETCH_HEAD
-rw-rw-r--   1 ubuntu ubuntu   22 Sep 30 15:41 HEAD
-rw-rw-r--   1 ubuntu ubuntu   41 Sep 29 20:13 ORIG_HEAD
drwxrwxr-x   2 ubuntu ubuntu    2 Aug 16 20:25 branches
-rw-rw-r--   1 ubuntu ubuntu  566 Sep 28 20:37 config
-rw-rw-r--   1 ubuntu ubuntu   73 Aug 16 20:25 description
drwxrwxr-x   2 ubuntu ubuntu   15 Aug 16 20:25 hooks
-rw-rw-r--   1 ubuntu ubuntu 2048 Sep 30 17:48 index
drwxrwxr-x   2 ubuntu ubuntu    3 Aug 16 20:25 info
drwxrwxr-x   3 ubuntu ubuntu    4 Sep  5 11:32 logs
drwxrwxr-x   3 ubuntu ubuntu    3 Aug 16 20:31 modules
drwxrwxr-x 221 ubuntu ubuntu  221 Sep 29 20:28 objects
drwxrwxr-x   5 ubuntu ubuntu    5 Sep 29 16:14 refs

The rclone config contents with secrets removed.

david@alpe:~/Bin/rclone-v1.59.2-linux-amd64$ rclone config show orchestra-jammy:
--------------------
[orchestra-jammy]
type = sftp
host = 10.51.175.82
user = ubuntu
key_file = ~/.ssh/id_rsa
key_file_pass = *** ENCRYPTED ***
shell_type = unix
md5sum_command = /usr/bin/md5sum
sha1sum_command = /usr/bin/sha1sum
--------------------

david@alpe:~/Bin/rclone-v1.59.2-linux-amd64$ rclone config show memstore:
--------------------
[memstore]
type = swift
env_auth = false
user = admin
key = <hidden>
auth = https://auth.storage.memset.com/v2.0
tenant = <hidden>
--------------------

outofphase · October 2, 2022, 12:17pm

Full log of the sync at Pastebin

Ole · October 2, 2022, 1:28pm

Do you see the files in "branches" if you try:

rclone lsl orchestra-jammy:github/hightouch-roe/.git

or

rclone lsl orchestra-jammy:github/hightouch-roe/.git/branches

Are there any files in "branches" or is it empty?

If empty, then you probably want to add --create-empty-src-dirs

outofphase · October 2, 2022, 2:04pm

How embarrassing, you are right: the source directory is empty.

I assumed it would contain something, since that git repo does have branches, but in fact it doesn't.

ubuntu@orchestra-jammy:~/github/hightouch-roe/.git$ git branch
* FIX42
  FIX44
  bugged
  documentation
  extragroups
  main
ubuntu@orchestra-jammy:~/github/hightouch-roe/.git$ ls -la branches/
total 10
drwxrwxr-x 2 ubuntu ubuntu  2 Aug 16 20:25 .
drwxrwxr-x 9 ubuntu ubuntu 16 Sep 30 17:48 ..

Ole · October 2, 2022, 2:43pm

Not at all, we all make mistakes - especially when learning new things

system · October 5, 2022, 2:43pm

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.