Sync multiple directories

Help and Support
1

What is the problem you are having with rclone?

I try 2 sync 2 directories in 1 command

I looked for a solution in the manual but this part is a bit unclear in its description. Also on internet I found a very little info, and they all did not work.

What is your rclone version (output from rclone version)

rclone v1.57.0

  • os/version: darwin 12.1 (64 bit)
  • os/kernel: 21.2.0 (arm64)
  • os/type: darwin
  • os/arch: arm64
  • go/version: go1.17.2
  • go/linking: dynamic
  • go/tags: cmount

The command you were trying to run (eg rclone copy /tmp remote:tmp)

I tried several but this is the one that seems the closest:

rclone sync --delete-during -v -P --include='/Users/marcelloh/{stack,data}/**'  /Volumes/Kingston-2TB/** --modify-window 15s --transfers=25 --checkers=50 --copy-links --ignore-size

Normally I have 2 commandline commands (which both work), for example:

rclone sync --delete-during -v -P /Users/marcelloh/stack/$1 /Volumes/Kingston-2TB/stack/$1 --modify-window 15s --transfers=25 --checkers=50 --copy-links --ignore-size

The rclone config contents with secrets removed.

rclone works, it's only the 2 dirs that is a mistery

A log from the command with the -vv flag

2021/12/30 08:21:05 DEBUG : rclone: Version "v1.57.0" starting with parameters ["/Users/marcelloh/Applications/RClone/rclone" "sync" "--delete-during" "-vv" "-P" "--include=/Users/marcelloh/{stack,data}/**" "/Volumes/Kingston-2TB/Dummy" "/Volumes/Kingston-2TB/data" "/Volumes/Kingston-2TB/stack" "--modify-window" "15s" "--transfers=25" "--checkers=50" "--copy-links" "--ignore-size"]
Usage:
  rclone sync source:path dest:path [flags]

Flags:
      --create-empty-src-dirs   Create empty source dirs on destination after sync
  -h, --help                    help for sync

Use "rclone [command] --help" for more information about a command.
Use "rclone help flags" for to see the global flags.
Use "rclone help backends" for a list of supported services.
Command sync needs 2 arguments maximum: you provided 3 non flag arguments: ["/Volumes/Kingston-2TB/Dummy" "/Volumes/Kingston-2TB/data" "/Volumes/Kingston-2TB/stack"]
2

You need to use the root path and include the two directories you want to sync.

You can only pass one source so trying to pass more than that will always fail.

3

I suggest you first try this command:

rclone lsf --include='/Users/marcelloh/{stack,data}/**'

Then correct the missing source and add a simple filter like this:

rclone lsf /Users/marcelloh --include='/stack/**'

Then expand the filter filter like this:

rclone lsf /Users/marcelloh --include='/{stack,data}/**'

Then try a simple sync command like this:

rclone sync --dry-run /Users/marcelloh /Volumes/Kingston-2TB/**

Then correct the unexpected expansion of **:

rclone sync --dry-run /Users/marcelloh /Volumes/Kingston-2TB

Then add the filter approach tested with the lsf command:

rclone sync --dry-run /Users/marcelloh /Volumes/Kingston-2TB --include='/{stack,data}/**'

Then add the remaining parameters (one-by-one to catch errors and misunderstandings):

rclone sync --dry-run /Users/marcelloh /Volumes/Kingston-2TB --include='/{stack,data}/**' …

And finally run without --dry-run when everything looks good.

4

I have to do that with ./ in front (else permission denied)

This gives me the help screen:

 ./rclone lsf --include='/Users/marcelloh/{stack,data}/**'
Usage:
  rclone lsf remote:path [flags]

Flags:
      --absolute           Put a leading / in front of path names
      --csv                Output in CSV format
  -d, --dir-slash          Append a slash to directory names (default true)
      --dirs-only          Only list directories
      --files-only         Only list files
  -F, --format string      Output format - see  help for details (default "p")
      --hash h             Use this hash when h is used in the format MD5|SHA-1|DropboxHash (default "md5")
  -h, --help               help for lsf
  -R, --recursive          Recurse into the listing
  -s, --separator string   Separator for the items in the format (default ";")

Use "rclone [command] --help" for more information about a command.
Use "rclone help flags" for to see the global flags.
Use "rclone help backends" for a list of supported services.
Command lsf needs 1 arguments minimum: you provided 0 non flag arguments: []

This however works...

./rclone lsf /Users/marcelloh --include='/stack/**'

just output: stacks

5 
./rclone sync –dry-run /Users/marcelloh /Volumes/Kingston-2TB --include='/{stack,data}/**'

also not working:

Usage:
  rclone sync source:path dest:path [flags]

Flags:
      --create-empty-src-dirs   Create empty source dirs on destination after sync
  -h, --help                    help for sync

Use "rclone [command] --help" for more information about a command.
Use "rclone help flags" for to see the global flags.
Use "rclone help backends" for a list of supported services.
Command sync needs 2 arguments maximum: you provided 3 non flag arguments: ["–dry-run" "/Users/marcelloh" "/Volumes/Kingston-2TB"]
6 
./rclone sync –dry-run /Users/marcelloh /Volumes/Kingston-2TB --include='/{stack,data}/**' --delete-during -v -P --modify-window 15s --transfers=5 --checkers=50 --copy-links --ignore-size

Also the same:

Usage:
  rclone sync source:path dest:path [flags]

Flags:
      --create-empty-src-dirs   Create empty source dirs on destination after sync
  -h, --help                    help for sync

Use "rclone [command] --help" for more information about a command.
Use "rclone help flags" for to see the global flags.
Use "rclone help backends" for a list of supported services.
Command sync needs 2 arguments maximum: you provided 3 non flag arguments: ["–dry-run" "/Users/marcelloh" "/Volumes/Kingston-2TB"]
7

Ahh sorry, a typo. I meant --dry-run and have updated my post above.

8

with or without dry-run, only the data is visited, and I think the root of stack.
Also swapped the two, but has the same result.
Only the data dir is "walked". It seems like some bug.

rclone sync /Users/marcelloh /Volumes/Kingston-2TB --include='/{stack,data}/**' --modify-window 15s --transfers=5 --checkers=50 --copy-links --ignore-size
9

How did you determine this?

Please share the commands, output, deductions and thoughts leading to the above statements.

10

Because I can see all the output and I removed some files on the destination stack folder, so they should be picked up.
hmmm. strange: they are picked up, but... there is no output anymore of the files that are synced
Perhaps I miss one of the original params. let me check this...
That was the case :slight_smile: pfff. Thanks (again) for your patience:
I was missing "-v -P"

Case closed :slight_smile:

11

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.