Rclone move specific folder

klapvogn · October 24, 2022, 5:17pm

What is the problem you are having with rclone?

Run the command 'rclone version' and share the full output of the command.

rclone --version
rclone v1.58.1
- os/version: ubuntu 21.10 (64 bit)
- os/kernel: 5.13.0-52-generic (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.17.9
- go/linking: static
- go/tags: none

Which cloud storage system are you using? (eg Google Drive)

Google Drive

The command you were trying to run (eg `rclone copy /tmp remote:tmp`)

Paste command here

The rclone config contents with secrets removed.

[tdrive]
client_id = xxxx
client_secret = xxxx
type = drive
token = xxxx
team_drive = xxxx


[tcrypt]
type = crypt
remote = tdrive:/encrypt
filename_encryption = standard
directory_name_encryption = true
password = xxxx
password2 = xxxx

A log from the command with the `-vv` flag

Paste log here

Hi,

I have the following local on my drive :

/home/xxxx/data/local/downloads/UPLOAD/TL/FOLDER1/CONTENT/

CONTENT = ready to be moved by rclone

inside TL there is FOLDER1 indside the CONTENT are .mkv and .srt files.

The issue I'm facing is: I would like for rclone NOT to copy/remove FOLDER1 but the rest after FOLDER1 being CONTENT

Hope i make sense

asdffdsa · October 24, 2022, 5:32pm

hi,

so you want to exclude files from FOLDER1 but include files from subdirs such as CONTENT

might try a filter, something like this, using rclone ls to view the output, without the risk of rclone move
rclone ls /home/xxxx/data/local/downloads/UPLOAD/TL --exclude="/FOLDER1/*"

to better understand the dir structure, might post the output of rclone tree

klapvogn · October 24, 2022, 5:46pm

Hi,

Yes kinda, except that FOLDER1 has to remain after the rclone move command. and everything inside FOLDER1 should be moved

My gdrive tree is: (where relevant)
tree

rclone tree tcrypt:downloads/UPLOAD/TL
/
├── FOREIGN.MOVIE.BLURAY
├── MOVIE.4K
├── MOVIE.BLURAY
├── MOVIE.BOXSET
├── MOVIE.WEBRIP
├── TV.BOXSETS
└── TV.EPISODES

EDIT:

I have learned this this command you postet, didn't exclude folders when i testet.

rclone ls /home/xxxx/data/local/downloads/UPLOAD/TL --exclude="/FOLDER1/*"

But this one excluded the folder i typed

rclone ls /home/xxxx/data/local/downloads/UPLOAD/TL --exclude="/FOLDER1/" -vv

asdffdsa · October 24, 2022, 6:18pm

good, that filter is working for you.

klapvogn · October 24, 2022, 6:21pm

Hi,

One more thing, I have in my script this : --exclude-from="$HOME"/.config/rclone/excludes

Should i then just add the dirs like:

DIR1
DIR2
and so on

klapvogn · October 24, 2022, 6:23pm

And from a second thought, when I use : --exclude="/FOLDER1/" will it then still move the content in FOLDER1

asdffdsa · October 24, 2022, 6:34pm

not sure, might try something like
"/FOLDER1/*.mkv"

klapvogn · October 24, 2022, 6:41pm

Hi,

Think you have missread somewhere the CONTENT is a folder to

Let me clear up

local path

/home/xxxx/data/local/downloads/UPLOAD/TL/FOLDER1/CONTENT/

CONTENT is a folder to

So rclone should move :
/home/xxxx/data/local/downloads/UPLOAD/TL/FOLDER1/CONTENT/

| CONTENT
| .mkv
| .srt
| .nfo

and leave FOLDER1 behind, so I don't need to create it everytime

Hope it make more sense now

Ole · October 24, 2022, 7:25pm

Hej klapvogn,

Perhaps you are just looking for something like this:

rclone move /home/xxxx/data/local/downloads/UPLOAD/TL/FOLDER1/ tcrypt:downloads/UPLOAD/TL/MOVIE.4K/

klapvogn · October 24, 2022, 7:31pm

Hi,

Just universal so i don't have to change the line after TL/

Ole · October 24, 2022, 7:35pm

What is the real name of FOLDER1? Would it be MOVIE.4K in my example?

klapvogn · October 24, 2022, 7:59pm

Hi,

Yes FOLDER1 can be substituted by this:

rclone tree tcrypt:downloads/UPLOAD/TL
/
├── FOREIGN.MOVIE.BLURAY
├── MOVIE.4K
├── MOVIE.BLURAY
├── MOVIE.BOXSET
├── MOVIE.WEBRIP
├── TV.BOXSETS
└── TV.EPISODES

Ole · October 24, 2022, 8:35pm

OK, I think I got it

Here is a hack that seems to work.

First you need to place a file in the each of the 7 folders like this

rclone touch /home/xxxx/data/local/downloads/UPLOAD/TL/FOREIGN.MOVIE.BLURAY/DO_NOT_DELETE
rclone touch /home/xxxx/data/local/downloads/UPLOAD/TL/MOVIE.4K/DO_NOT_DELETE
....

This file ensures that the --delete-empty-src-dirs doesn't delete those folders in this command:

rclone move --filter="+ /*/*/**" --filter="- /**" --delete-empty-src-dirs /home/xxxx/data/local/downloads/UPLOAD/TL/ tcrypt:downloads/UPLOAD/TL/

Short explanation: Only move files at level 3 or higher (that is inside the CONTENT folders) and then delete all empty source folders after that (this will delete CONTENT, but not MOVIE.4K because it still has the DO_NOT_DELETE file).

klapvogn · October 25, 2022, 7:08am

Morning,

Thanks for this Ole

Just so I don't fuck something up

How would I do it in this :

rclone move "$FROM"/$rPath "dst"$COUNTER"-crypt:/"/$rPath --min-age 7d --delete-after --log-file=$LOGFILE --max-transfer 500G --exclude-from="$HOME"/.config/rclone/excludes --config="$HOME"/scripts/sa-conf.conf --drive-chunk-size 64M --tpslimit 5 --drive-acknowledge-abuse=true -v --delete-empty-src-dirs --fast-list --use-mmap --transfers=2 --checkers=4 --drive-pacer-min-sleep=100ms

Ole · October 25, 2022, 7:49am

Morning, thanks

This is quite a lot of flags and some seems redundant, counter productive or not needed, so let me first understand your command.

I consider this default behavior and only applicable to syncs. Why did you add it?

Why did you decide to increase chunk size and then limit the TPS?

They seem to be pulling in opposite directions and I would expect better performance and less issues by removing both.

Why did you add fast-list?

I often see better performance on Google Drive without it (assuming you remove the above TPS limit too).

This doesn't work well on all platforms. Did you perform a test and see a significant positive effect?

This has no effect when combined with --fast-list (which effective limits you to --checkers=1).

I suggest you remove this initially (to use the default of 8 checkers) and then also tries with 16 checkers (assuming you remove --fast-list above)

This has no effect when combined with TPS=5, which will in effect increase sleep time to 200ms.

I suggest you remove it or reduce to the forum consensus of 10ms.

klapvogn · October 25, 2022, 7:53am

Hi,

Im not at all good at this, so therefore my friend added the stuff. And I would like it to be better with the remarks you made

Ole · October 25, 2022, 8:05am

OK, then I understand and suggest you check my comments with your friend before changing anything, which also goes for the following:

If $rPath is equal to "downloads/UPLOAD/TL/" and you have added the files mentioned above then all you need is to add these flags:

--filter="+ /*/*/**" --filter="- /**" --delete-empty-src-dirs

and please test with --dry-run first.

You can read more about filters here: Rclone Filtering

klapvogn · October 25, 2022, 8:46am

Hi,

thanks so fare I did some testing and here they are:

rclone move --filter="+ /*/*/**" --filter="- /**" --delete-empty-src-dirs /home/xxxx/data/local/downloads/UPLOAD/TL/TV.BOXSETS tcrypt:downloads/UPLOAD/TL/ --max-transfer 500G --transfers=2 --drive-pacer-min-sleep=10ms --dry-run

2022/10/25 10:38:13 NOTICE: xxxxx-DEFLATE: Skipped remove directory as --dry-run is set
2022/10/25 10:38:13 NOTICE: xxxxx-TWASERiES: Skipped remove directory as --dry-run is set
2022/10/25 10:38:13 NOTICE: xxxxx-TWASERiES: Skipped remove directory as --dry-run is set
2022/10/25 10:38:13 NOTICE: xxxxx-TWASERiES: Skipped remove directory as --dry-run is set
2022/10/25 10:38:13 NOTICE:
Transferred:              0 B / 0 B, -, 0 B/s, ETA -
Deleted:                0 (files), 4 (dirs)
Elapsed time:         2.1s

rclone move --filter="+ /*/*/**" --filter="- /**" --delete-empty-src-dirs /home/xxxx/data/local/downloads/UPLOAD/TL/TV.BOXSETS tcrypt:downloads/UPLOAD/TL/ --max-transfer 500G --transfers=2 --drive-pacer-min-sleep=10ms --dry-run -vv

2022/10/25 10:40:45 DEBUG : rclone: Version "v1.58.1" starting with parameters ["rclone" "move" "--filter=+ /*/*/**" "--filter=- /**" "--delete-empty-src-dirs" "/home/klapvogn/data/local/downloads/UPLOAD/TL/TV.BOXSETS" "tcrypt:downloads/UPLOAD/TL/" "--max-transfer" "500G" "--transfers=2" "--drive-pacer-min-sleep=10ms" "--dry-run" "-vv"]
2022/10/25 10:40:45 DEBUG : Creating backend with remote "/home/xxxx/data/local/downloads/UPLOAD/TL/TV.BOXSETS"
2022/10/25 10:40:45 DEBUG : Using config file from "/home/xxxx/.config/rclone/rclone.conf"
2022/10/25 10:40:45 DEBUG : Creating backend with remote "tcrypt:downloads/UPLOAD/TL/"
2022/10/25 10:40:45 DEBUG : Creating backend with remote "tdrive:/encrypt"
2022/10/25 10:40:45 DEBUG : tdrive: detected overridden config - adding "xxxx" suffix to name
2022/10/25 10:40:48 DEBUG : fs cache: renaming cache item "tdrive:/encrypt/" to be canonical "tdrive:encrypt/"
2022/10/25 10:40:48 DEBUG : fs cache: switching user supplied name "tdrive:/encrypt/" for canonical name "tdrive{xxxx}:encrypt/"
2022/10/25 10:40:48 DEBUG : DO_NOT_DELETE: Excluded

2022/10/25 10:40:48 DEBUG : xxxx-TWASERiES/xxxx-TWASERiES.da.srt: Excluded
2022/10/25 10:40:48 DEBUG : xxxx-TWASERiES/xxxx-TWASERiES.no.srt: Excluded
2022/10/25 10:40:48 DEBUG : xxxx-TWASERiES/xxxx-TWASERiES.sv.srt: Excluded
2022/10/25 10:40:48 DEBUG : xxxx-TWASERiES/xxxx-TWASERiES.fi.srt: Excluded

2022/10/25 10:40:48 DEBUG : Encrypted drive 'tcrypt:downloads/UPLOAD/TL/': Waiting for checks to finish
2022/10/25 10:40:48 DEBUG : Encrypted drive 'tcrypt:downloads/UPLOAD/TL/': Waiting for transfers to finish
2022/10/25 10:40:48 NOTICE: Nightflyers.S01.2160p.WEBRip.X264-DEFLATE: Skipped remove directory as --dry-run is set
2022/10/25 10:40:48 NOTICE: xxxx-TWASERiES: Skipped remove directory as --dry-run is set
2022/10/25 10:40:48 NOTICE: xxxx-TWASERiES: Skipped remove directory as --dry-run is set
2022/10/25 10:40:48 NOTICE: xxxx-TWASERiES: Skipped remove directory as --dry-run is set
2022/10/25 10:40:48 DEBUG : Local file system at /home/xxxx/data/local/downloads/UPLOAD/TL/TV.BOXSETS: deleted 4 directories
2022/10/25 10:40:48 INFO  : There was nothing to transfer
2022/10/25 10:40:48 NOTICE:
Transferred:              0 B / 0 B, -, 0 B/s, ETA -
Deleted:                0 (files), 4 (dirs)
Elapsed time:         3.5s

Ole · October 25, 2022, 8:55am

Hi,

Looks like you are one level too deep in your source folders, that is

/home/xxxx/data/local/downloads/UPLOAD/TL/TV.BOXSETS

should be

/home/xxxx/data/local/downloads/UPLOAD/TL

klapvogn · October 25, 2022, 9:00am

Hi,

Ahh yes right, looks like it works now :

rclone move --filter="+ /*/*/**" --filter="- /**" --delete-empty-src-dirs /home/xxxx/data/local/downloads/UPLOAD/TL/ tcrypt:downloads/UPLOAD/TL/ --max-transfer 500G --transfers=2 --drive-pacer-min-sleep=10ms --dry-run -vv

2022/10/25 10:57:36 DEBUG : TV.BOXSETS/DO_NOT_DELETE: Excluded
2022/10/25 10:57:36 DEBUG : MOVIE.BLURAY/DO_NOT_DELETE: Excluded
2022/10/25 10:57:36 DEBUG : MOVIE.BOXSET/DO_NOT_DELETE: Excluded
2022/10/25 10:57:36 DEBUG : MUSIC/DO_NOT_DELETE: Excluded
2022/10/25 10:57:36 DEBUG : WINAPP/DO_NOT_DELETE: Excluded
2022/10/25 10:57:36 DEBUG : TV.EPISODES/DO_NOT_DELETE: Excluded
2022/10/25 10:57:36 NOTICE: MUSIC/XXXX-ENTiTLED/xxxx.m3u: Skipped move as --dry-run is set (size 517)

2022/10/25 10:57:36 DEBUG : MOVIE.4K/DO_NOT_DELETE: Excluded
2022/10/25 10:57:36 DEBUG : FOREIGN.MOVIE.BLURAY/DO_NOT_DELETE: Excluded
2022/10/25 10:57:36 DEBUG : MOVIE.WEBRIP/DO_NOT_DELETE: Excluded