Confirming a command before I use it, and an additional question

Help and Support
1

What is the problem you are having with rclone?

I want to double check that my sync path is correct before I run it, and also had a question about changing it when I need to.

I'm worried about accidentally deleting or overwriting files on the source server. I originally was having trouble getting rclone to work (I'm extremely new at servers!) and managed to use "copy" to get some of the folders on Backblaze initially. However, I ended up deleting a few of them, realizing I didn't need them, and wanting to set up something that would automatically sync on it's own. Admittedly, my command is kind of frankenstein'd from a few others I've seen, so I'm very open to changing it. I have never used cron before, so I was hoping this would work.

If I needed to change the command at any time (because I'm hoping to add a blog/journal's database eventually, when I figure out how to install it) I was just wondering how I would do that.

I did redact the server name, wasn't sure if I was supposed to. It does work, for what that's worth, with the server name there. And I had to strip out some of the log at the bottom due to character count not allowing me to post this.

Thank you so much!

Run the command 'rclone version' and share the full output of the command.

/usr/bin/rclone sync server:ServerName /var/lib/docker/volumes/ --include "/gluetun-config" --include "mealie_mealie-data"  --include "photoprism-data" --include "shared-media" --bwlimit="08:00,512k 23:00,off" --progress --transfers=16

The rclone config contents with secrets removed.

Name                 Type
====                 ====
ServerName            b2

A log from the command with the -vv flag

 directory as --dry-run is set
2023-07-26 00:06:46 NOTICE: navidrome-data/_data/cache/images/a8/e5: Skipped rem                                          ove directory as --dry-run is set
2023-07-26 00:06:46 NOTICE: navidrome-data/_data/cache/images/a8: Skipped remove                                           directory as --dry-run is set
2023-07-26 00:06:46 NOTICE: navidrome-data/_data/cache/images/a7/ea: Skipped rem                                          ove directory as --dry-run is set
2023-07-26 00:06:46 NOTICE: navidrome-data/_data/cache/images/a7/68: Skipped rem                                          ove directory as --dry-run is set
2023-07-26 00:06:46 NOTICE: navidrome-data/_data/cache/images/a7: Skipped remove                                           directory as --dry-run is set
2023-07-26 00:06:46 NOTICE: navidrome-data/_data/cache/images/a6/a5: Skipped rem                                          ove directory as --dry-run is set
2023-07-26 00:06:46 NOTICE: navidrome-data/_data/cache/images/a6/65: Skipped rem                                          ove directory as --dry-run is set
2023-07-26 00:06:46 NOTICE: navidrome-data/_data/cache/images/a6/54: Skipped rem                                          ove directory as --dry-run is set
2023-07-26 00:06:46 NOTICE: navidrome-data/_data/cache/images/a6: Skipped remove                                           directory as --dry-run is set
2023-07-26 00:06:46 NOTICE: navidrome-data/_data/cache/images/a5/c0: Skipped rem                                          ove directory as --dry-run is set
2023-07-26 00:06:46 NOTICE: navidrome-data/_data/cache/images/92/f9: Skipped rem                                          ove directory as --dry-run is set
2023-07-26 00:06:46 NOTICE: navidrome-data/_data/cache/images/92/70: Skipped rem                                          ove directory as --dry-run is set
2023-07-26 00:06:46 NOTICE: navidrome-data/_data/cache/images/92: Skipped remove                                           directory as --dry-run is set
2023-07-26 00:06:46 NOTICE: navidrome-data/_data: Skipped remove directory as --                                          dry-run is set
2023-07-26 00:06:46 NOTICE: navidrome-data: Skipped remove directory as --dry-ru                                          n is set
2023-07-26 00:06:46 NOTICE: mealie_mealie-data/_data/users/e4ea2af8-cd4d-4a29-89                                          ed-02acf3c8b778: Skipped remove directory as --dry-run is set
2023-07-26 00:06:46 NOTICE: mealie_mealie-data/_data/users/9eaf1424-87bd-46e1-9d                                          ea-6ace2e1a9a80: Skipped remove directory as --dry-run is set
2023-07-26 00:06:46 NOTICE: mealie_mealie-data/_data/users: Skipped remove direc                                          tory as --dry-run is set
2023-07-26 00:06:46 NOTICE: mealie_mealie-data/_data/templates: Skipped remove d                                          irectory as --dry-run is set
2023-07-26 00:06:46 NOTICE: mealie_mealie-data/_data/recipes/aa349e21-4141-495e-                                          93f2-8b4479f6b322/images: Skipped remove directory as --dry-run is set
2023-07-26 00:06:46 NOTICE: mealie_mealie-data/_data/recipes/aa349e21-4141-495e-                                          93f2-8b4479f6b322/assets: Skipped remove directory as --dry-run is set
2023-07-26 00:06:46 NOTICE: mealie_mealie-data/_data/recipes/aa349e21-4141-495e-                                          93f2-8b4479f6b322: Skipped remove directory as --dry-run is set
2023-07-26 00:06:46 NOTICE: mealie_mealie-data/_data/recipes: Skipped remove dir                                          ectory as --dry-run is set
2023-07-26 00:06:46 NOTICE: mealie_mealie-data/_data/groups: Skipped remove dire                                          ctory as --dry-run is set
2023-07-26 00:06:46 NOTICE: mealie_mealie-data/_data/backups: Skipped remove dir                                          ectory as --dry-run is set
2023-07-26 00:06:46 NOTICE: mealie_mealie-data/_data/.temp: Skipped remove direc                                          tory as --dry-run is set
2023-07-26 00:06:46 NOTICE: kitchenowl-data/_data/upload: Skipped remove directo                                          ry as --dry-run is set
2023-07-26 00:06:46 NOTICE: kitchenowl-data/_data: Skipped remove directory as -                                          -dry-run is set
2023-07-26 00:06:46 NOTICE: kitchenowl-data: Skipped remove directory as --dry-r                                          un is set
2023-07-26 00:06:46 NOTICE: jellyfin-config/_data/transcodes: Skipped remove dir                                          ectory as --dry-run is set
2023-07-26 00:06:46 NOTICE: jellyfin-config/_data/root/default/Media Library: Sk                                          ipped remove directory as --dry-run is set
2023-07-26 00:06:46 NOTICE: jellyfin-config/_data/root/default: Skipped remove d                                          irectory as --dry-run is set
2023-07-26 00:06:46 NOTICE: jellyfin-config/_data/root: Skipped remove directory                                           as --dry-run is set
2023-07-26 00:06:46 NOTICE: jellyfin-config/_data/plugins/configurations: Skippe                                          d remove directory as --dry-run is set
2023-07-26 00:06:46 NOTICE: jellyfin-config/_data/plugins: Skipped remove direct                                          ory as --dry-run is set
2023-07-26 00:06:46 NOTICE: jellyfin-config/_data/metadata/views/livetv: Skipped                                           remove directory as --dry-run is set
2023-07-26 00:06:46 NOTICE: jellyfin-config/_data/metadata/views/ddf50f8871ecd43                                          192fc411cc55cfd48: Skipped remove directory as --dry-run is set
2023-07-26 00:06:46 NOTICE: jellyfin-config/_data/metadata/views/0fd1b9148d2d701                                          6bc82f295bf41fe59: Skipped remove directory as --dry-run is set
2023-07-26 00:06:46 NOTICE: jellyfin-config/_data/metadata/views: Skipped remove                                           directory as --dry-run is set
2023-07-26 00:06:46 NOTICE: jellyfin-config/_data/metadata/library/ff/ffe0fc5787                                          33fbd660218c3d90dd5ea6: Skipped remove directory as --dry-run is set
2023-07-26 00:06:46 NOTICE: jellyfin-config/_data/metadata/library/ff/ff868d17d8                                          fe24d6a4fe933346dd8625: Skipped remove directory as --dry-run is set
2023-07-26 00:06:46 NOTICE: jellyfin-config/_data/metadata/library/ff: Skipped r                                          emove directory as --dry-run is set
2023-07-26 00:06:46 NOTICE: jellyfin-config/_data/metadata/People/D/Donnie Yen:                                           Skipped remove directory as --dry-run is set
2023-07-26 00:06:46 NOTICE: jellyfin-config/_data/metadata/People/D/Dave Bautist                                          a: Skipped remove directory as --dry-run is set
2023-07-26 00:06:46 NOTICE: jellyfin-config/_data/metadata/People/D: Skipped rem                                          ove directory as --dry-run is set
2023-07-26 00:06:46 NOTICE: jellyfin-config/_data/metadata/People/C/Chrissie Cha                                          u: Skipped remove directory as --dry-run is set
2023-07-26 00:06:46 NOTICE: jellyfin-config/_data/metadata/People/C/Chan Tai-Li:                                           Skipped remove directory as --dry-run is set
2023-07-26 00:06:46 NOTICE: jellyfin-config/_data/metadata/People/C: Skipped rem                                          ove directory as --dry-run is set
2023-07-26 00:06:46 NOTICE: jellyfin-config/_data/metadata/People/A/Akira Ishida                                          : Skipped remove directory as --dry-run is set
2023-07-26 00:06:46 NOTICE: jellyfin-config/_data/metadata/People/A/Akiko Kimura                                          : Skipped remove directory as --dry-run is set
2023-07-26 00:06:46 NOTICE: jellyfin-config/_data/metadata/People/A/Ai Shimizu:                                           Skipped remove directory as --dry-run is set
2023-07-26 00:06:46 NOTICE: jellyfin-config/_data/metadata/People/A/Adam Pak: Sk                                          ipped remove directory as --dry-run is set
2023-07-26 00:06:46 NOTICE: jellyfin-config/_data/metadata/People/A: Skipped rem                                          ove directory as --dry-run is set
2023-07-26 00:06:46 NOTICE: jellyfin-config/_data/metadata/People: Skipped remov                                          e directory as --dry-run is set
2023-07-26 00:06:46 NOTICE: jellyfin-config/_data/metadata: Skipped remove direc                                          tory as --dry-run is set
2023-07-26 00:06:46 NOTICE: jellyfin-config/_data/log: Skipped remove directory                                           as --dry-run is set
2023-07-26 00:06:46 NOTICE: jellyfin-config/_data/data/playlists: Skipped remove                                           directory as --dry-run is set
2023-07-26 00:06:46 NOTICE: jellyfin-config/_data/data/ScheduledTasks: Skipped r                                          emove directory as --dry-run is set
2023-07-26 00:06:46 NOTICE: jellyfin-config/_data/data: Skipped remove directory                                           as --dry-run is set
2023-07-26 00:06:46 NOTICE: jellyfin-config/_data/config/users/Kerri: Skipped re                                          move directory as --dry-run is set
2023-07-26 00:06:46 NOTICE: jellyfin-config/_data/config/users: Skipped remove d                                          irectory as --dry-run is set
2023-07-26 00:06:46 NOTICE: jellyfin-config/_data/config/dlna/user: Skipped remo                                          ve directory as --dry-run is set
2023-07-26 00:06:46 NOTICE: jellyfin-config/_data/config/dlna/system: Skipped re                                          move directory as --dry-run is set
2023-07-26 00:06:46 NOTICE: jellyfin-config/_data/config/dlna: Skipped remove di                                          rectory as --dry-run is set
2023-07-26 00:06:46 NOTICE: jellyfin-config/_data/config: Skipped remove directo                                          ry as --dry-run is set
2023-07-26 00:06:46 NOTICE: jellyfin-config/_data: Skipped remove directory as -                                          -dry-run is set
2023-07-26 00:06:46 NOTICE: jellyfin-config: Skipped remove directory as --dry-r                                          un is set
2023-07-26 00:06:46 NOTICE: jellyfin-cache/_data/temp: Skipped remove directory                                           as --dry-run is set
2023-07-26 00:06:46 NOTICE: jellyfin-cache/_data/omdb: Skipped remove directory                                           as --dry-run is set
2023-07-26 00:06:46 NOTICE: jellyfin-cache/_data/extracted-audio-images: Skipped                                           remove directory as --dry-run is set
2023-07-26 00:06:46 NOTICE: jellyfin-cache/_data/audiodb-artist/f4abc0b5-3f7a-4e                                          ff-8f78-ac078dbce533: Skipped remove directory as --dry-run is set
2023-07-26 00:06:46 NOTICE: jellyfin-cache/_data: Skipped remove directory as --                                          dry-run is set
2023-07-26 00:06:46 NOTICE: jellyfin-cache: Skipped remove directory as --dry-ru                                          n is set
2023-07-26 00:06:46 DEBUG : Local file system at /var/lib/docker/volumes: delete                                          d 1301 directories
2023-07-26 00:06:46 INFO  : There was nothing to transfer
Transferred:              0 B / 0 B, -, 0 B/s, ETA -
Deleted:                0 (files), 1301 (dirs)
Elapsed time:         2.0s
2023/07/26 00:06:46 NOTICE:
Transferred:              0 B / 0 B, -, 0 B/s, ETA -
Deleted:                0 (files), 1301 (dirs)
Elapsed time:         2.0s

2023/07/26 00:06:46 DEBUG : 19 go routines active
2

you can always test your command by running it with -vv --dry-run options and see if it does what you intended.

3

I apologize, I should have mentioned that the log was from me doing -vv and --dry-run. I don't quite understand if it's deleting things on both sides, instead of just transferring them to the destination or deleting files I no longer include or have from the destination. And I haven't deleted what files I have transferred to Backblaze, should I before trying to copy/sync files again?

And a last question, will this do this automatically like I'm hoping? Without me having to run it each time?

I ran it with copy instead of sync and got a completely different log.

2023/07/26 08:38:05 DEBUG : Creating backend with remote "server:ServerName"
2023/07/26 08:38:05 DEBUG : Using config file from "/root/.config/rclone/rclone.conf"
2023/07/26 08:38:05 DEBUG : Creating backend with remote "/var/lib/docker/volumes"
2023-07-26 08:38:06 DEBUG : mealie_mealie-data/_data/.secret: Excluded
2023-07-26 08:38:06 DEBUG : mealie_mealie-data/_data/mealie.db: Excluded
2023-07-26 08:38:06 DEBUG : mealie_mealie-data/_data/mealie.log: Excluded
2023-07-26 08:38:06 DEBUG : gluetun-config/_data/servers.json: Excluded
2023-07-26 08:38:06 DEBUG : photoprism-data/_data/storage/index.db: Excluded
2023-07-26 08:38:06 DEBUG : photoprism-data/_data/storage/serial: Excluded
2023-07-26 08:38:06 DEBUG : photoprism-data/_data/storage/config/hub.yml: Excluded
2023-07-26 08:38:06 DEBUG : photoprism-data/_data/storage/config/settings.yml: Excluded
2023-07-26 08:38:06 DEBUG : photoprism-data/_data/storage/backups/serial: Excluded
2023-07-26 08:38:06 DEBUG : photoprism-data/_data/originals/2023/07/20230708_150038_2E01980F.jpg: Excluded
2023-07-26 08:38:06 DEBUG : photoprism-data/_data/originals/2023/07/20230708_150038_82AB19D2.png: Excluded
2023-07-26 08:38:06 DEBUG : photoprism-data/_data/originals/2023/07/20230708_150038_9208B61C.jpg: Excluded
2023-07-26 08:38:06 DEBUG : photoprism-data/_data/originals/2023/07/20230708_150038_AFDA7DEA.jpg: Excluded
2023-07-26 08:38:06 DEBUG : photoprism-data/_data/storage/albums/album/arxhgc03ra2e0azs.yml: Excluded
2023-07-26 08:38:06 DEBUG : Local file system at /var/lib/docker/volumes: Waiting for checks to finish
2023-07-26 08:38:06 DEBUG : Local file system at /var/lib/docker/volumes: Waiting for transfers to finish
2023-07-26 08:38:06 INFO  : There was nothing to transfer

I also realized that I put the command in the wrong place, I think. Here's what I get after looking for the rclone version:

- os/version: debian 12.1 (64 bit)
- os/kernel: 6.1.0-10-amd64 (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.20.6
- go/linking: static
- go/tags: none

Thank you for any help!

4

You have to explain what you want to do. Otherwise how anybody can tell you if it is OK or not?

So you sync things from /var/lib/docker/volumes/. Any reasons you use --include flag? e.g.:

--include "/gluetun-config" --include "mealie_mealie-data"

you want to include directories? files?

run ls -l /var/lib/docker/volumes/ and explain what you want to include or exclude.

No. There is no magic:) rclone is command line tool you have to run. You can automate it using your OS tools like crontab or systemd.

5

Sorry about that! I ran the command, here's the result:

total 76
brw------- 1 root root 254, 0 Jul 25 10:31 backingFsBlockDev
drwx-----x 3 root root   4096 Jul 19 18:40 gluetun-config
drwx-----x 3 root root   4096 Jul  5 10:16 jellyfin-cache
drwx-----x 3 root root   4096 Jul  5 10:16 jellyfin-config
drwx-----x 3 root root   4096 Jul  5 11:28 kitchenowl-data
drwx-----x 3 root root   4096 Jul 23 10:39 mealie_mealie-data
-rw------- 1 root root  65536 Jul 25 10:31 metadata.db
drwx-----x 3 root root   4096 Jul 14 19:01 navidrome-data
drwx-----x 3 root root   4096 Jul  6 12:00 photoprism-data
drwx-----x 3 root root   4096 Jul  5 09:55 portainer_data
drwx-----x 3 root root   4096 Jul  5 10:36 qbittorrent-nox-config
drwx-----x 4 root root   4096 Jul 14 17:59 shared-media
drwx-----x 3 root root   4096 Jul  5 10:48 shiori-data

I want to include gluetun-config , mealie_mealie-data , photoprism-data , and shared-media so I get the directory with all it's files, but exclude everything else.

The intended goal was to get this to run itself, but I have no idea how crontab works.

Thank you for the help!

6

Ok this is my suggestion. Create text file called e.g. filter.txt with following content:

+ /gluetun-config/**
+ /mealie_mealie-data/**
+ /photoprism-data/**
+ /shared-media/**
- *

These are filtering rules including specified dirs and excluding everything else

then to sync from local to the cloud (I assume that it is what you want) you run:

rclone sync /var/lib/docker/volumes/ server:ServerName --filter-from /path/to/filter.txt --delete-excluded

--delete-excluded flag will delete everything from remote what is not included. E.g. when one day you decide to remove gluetun-config from your filter file then it will be removed from the cloud too.

Always test before with -vv --dry-run

Google is your friend:) you will find plenty of explanations and examples

7

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.