RCLONE equivalent of cp -a?

I mean. You could store a list of files and do a

Echo 0 | rclone rcat remote:$filename

How would you work that into this?
/media/sdah1/duvadmin/bin/rclone move --transfers 10 --min-size 1k --size-only --fast-list --tpslimit 5 /media/sdah1/duvadmin/private/post-processing/ GCrypt: &

pipe it into something else?

Wait. I think I misunderstood.. Is this what you want to do?

Rclone move $file remote:
touch $file

So you have a full size remote copy and a zero byte local copy?

You are 100% correct. If I can accomplish that as a one-liner then I don't need a 2nd box for this hooplah.

I want a 0-size full-replica of my remote.

I suppose it doesn't even have to be a one-liner, as I'm still checking for the existence of the process with [[ $(pgrep -fu $(whoami) 'rclone move') ]] ||

This way it continually recurses the local directory and uploads files with content, and leaves my 0-size markers behind. I do this to trick Sonarr and Radarr into believing those files are still present, ergo no longer searched for (unless a higher quality is found, which is part of the filename).

If I wanted to accomplish this... I'd do this:

Rclone move $file remote: --min-size smallsize
Rclone check local: remote:

  • I'd parse that check and generate a bunch of touch commands and rm commands. Should be straightforward I think as the check command will depict missing things locally or remotely.

The check can be part of the move script or some other periodic one.

If this box is so CPU/mem starved, you should consider reducing the checkers/transfer/bw-limit. I'd also guess with 500 Meg's of ram, the current settings might crash rclone with out of memory issues. When I run rclone on a gcs micro vps, I have to turn down those to keep things stable on memory.

I like this and I'm going to try this first. I'd love to reduce everything to one box. I mean monolithic single-point-of-failure goes against everything I do in my day job but it would reduce the maintenance for this project.

You're right. The odd thing is I was just doing that with manual commands (as screenshot above) and while it did slow down, it didn't seem to crash at any point although I was unable to run it to completion - so the crash may still be in the future of it. These are great suggestions. I noticed rclone check but did not think it could be used like this so thanks again for the good pointers. At least it's a new avenue that I haven't tested yet. That's something.

Okay so I tried this .... I deleted a banner.jpg from the local but here is the output:
[prometheus ~/private/post-processing/Cartoons/The Tick] rclone check . GCrypt:Cartoons/The\ Tick/ --one-way

2019/05/29 03:56:11 ERROR : The Tick.jpg: Sizes differ
2019/05/29 03:56:11 ERROR : fanart.jpg: Sizes differ
2019/05/29 03:56:11 ERROR : folder.jpg: Sizes differ
2019/05/29 03:56:11 ERROR : poster.jpg: Sizes differ
2019/05/29 03:56:11 ERROR : season01-poster.jpg: Sizes differ
2019/05/29 03:56:11 ERROR : season02-poster.jpg: Sizes differ
2019/05/29 03:56:11 ERROR : season03-poster.jpg: Sizes differ
2019/05/29 03:56:11 ERROR : series.xml: Sizes differ
2019/05/29 03:56:11 ERROR : tvshow.nfo: Sizes differ
2019/05/29 03:56:11 ERROR : Season 1/The Tick - S01E01 - The Tick vs. the Idea Men WEBDL-1080p-thumb.jpg: Sizes differ
2019/05/29 03:56:11 ERROR : Season 1/The Tick - S01E01 - The Tick vs. the Idea Men WEBDL-1080p.nfo: Sizes differ
2019/05/29 03:56:11 ERROR : Season 1/The Tick - S01E01 - The Tick vs. the Idea Men WEBDL-480p-thumb.jpg: Sizes differ
2019/05/29 03:56:11 ERROR : Season 1/The Tick - S01E01 - The Tick vs. the Idea Men WEBDL-480p.avi: Sizes differ
2019/05/29 03:56:11 ERROR : Season 1/The Tick - S01E01 - The Tick vs. the Idea Men WEBDL-480p.nfo: Sizes differ
2019/05/29 03:56:11 ERROR : Season 1/The Tick - S01E02 - The Tick vs. Chairface Chippendale WEBDL-1080p-thumb.jpg: Sizes differ
2019/05/29 03:56:11 ERROR : Season 1/The Tick - S01E02 - The Tick vs. Chairface Chippendale WEBDL-1080p.nfo: Sizes differ
2019/05/29 03:56:11 ERROR : Season 1/The Tick - S01E02 - The Tick vs. Chairface Chippendale WEBDL-480p-thumb.jpg: Sizes differ
2019/05/29 03:56:11 ERROR : Season 1/The Tick - S01E02 - The Tick vs. Chairface Chippendale WEBDL-480p.avi: Sizes differ
2019/05/29 03:56:11 ERROR : Season 1/The Tick - S01E02 - The Tick vs. Chairface Chippendale WEBDL-480p.nfo: Sizes differ
2019/05/29 03:56:11 ERROR : Season 1/The Tick - S01E03 - The Tick vs. Dinosaur Neil WEBDL-1080p-thumb.jpg: Sizes differ
2019/05/29 03:56:11 ERROR : Season 1/The Tick - S01E03 - The Tick vs. Dinosaur Neil WEBDL-1080p.nfo: Sizes differ
2019/05/29 03:56:11 ERROR : Season 1/The Tick - S01E03 - The Tick vs. Dinosaur Neil WEBDL-480p-thumb.jpg: Sizes differ
2019/05/29 03:56:11 ERROR : Season 1/The Tick - S01E03 - The Tick vs. Dinosaur Neil WEBDL-480p.avi: Sizes differ
2019/05/29 03:56:11 ERROR : Season 1/The Tick - S01E03 - The Tick vs. Dinosaur Neil WEBDL-480p.nfo: Sizes differ
2019/05/29 03:56:11 ERROR : Season 1/The Tick - S01E04 - The Tick vs. Mr. Mental WEBDL-480p-thumb.jpg: Sizes differ
2019/05/29 03:56:11 ERROR : Season 1/The Tick - S01E04 - The Tick vs. Mr. Mental WEBDL-480p.avi: Sizes differ
2019/05/29 03:56:11 ERROR : Season 1/The Tick - S01E04 - The Tick vs. Mr. Mental WEBDL-480p.nfo: Sizes differ
2019/05/29 03:56:11 ERROR : Season 1/The Tick - S01E04 - The Tick vs. Mr. Mental WEBDL-720p-thumb.jpg: Sizes differ
2019/05/29 03:56:11 ERROR : Season 1/The Tick - S01E04 - The Tick vs. Mr. Mental WEBDL-720p.nfo: Sizes differ
2019/05/29 03:56:11 ERROR : Season 1/The Tick - S01E05 - The Tick vs. the Breadmaster WEBDL-1080p-thumb.jpg: Sizes differ
2019/05/29 03:56:11 ERROR : Season 1/The Tick - S01E05 - The Tick vs. the Breadmaster WEBDL-1080p.nfo: Sizes differ
2019/05/29 03:56:11 ERROR : Season 1/The Tick - S01E05 - The Tick vs. the Breadmaster WEBDL-480p-thumb.jpg: Sizes differ
2019/05/29 03:56:11 ERROR : Season 1/The Tick - S01E05 - The Tick vs. the Breadmaster WEBDL-480p.avi: Sizes differ
2019/05/29 03:56:11 ERROR : Season 1/The Tick - S01E05 - The Tick vs. the Breadmaster WEBDL-480p.nfo: Sizes differ
2019/05/29 03:56:11 ERROR : Season 1/The Tick - S01E06 - The Tick vs. El Seed WEBDL-1080p-thumb.jpg: Sizes differ
2019/05/29 03:56:11 ERROR : Season 1/The Tick - S01E06 - The Tick vs. El Seed WEBDL-1080p.nfo: Sizes differ
2019/05/29 03:56:11 ERROR : Season 1/The Tick - S01E06 - The Tick vs. El Seed WEBDL-480p-thumb.jpg: Sizes differ
2019/05/29 03:56:11 ERROR : Season 1/The Tick - S01E06 - The Tick vs. El Seed WEBDL-480p.avi: Sizes differ
2019/05/29 03:56:11 ERROR : Season 1/The Tick - S01E06 - The Tick vs. El Seed WEBDL-480p.nfo: Sizes differ
2019/05/29 03:56:11 ERROR : Season 1/The Tick - S01E07 - The Tick vs. the Tick WEBDL-480p-thumb.jpg: Sizes differ
2019/05/29 03:56:11 ERROR : Season 1/The Tick - S01E07 - The Tick vs. the Tick WEBDL-480p.avi: Sizes differ
2019/05/29 03:56:11 ERROR : Season 1/The Tick - S01E07 - The Tick vs. the Tick WEBDL-480p.nfo: Sizes differ
2019/05/29 03:56:11 ERROR : Season 1/The Tick - S01E08 - The Tick vs. the Uncommon Cold WEBDL-480p-thumb.jpg: Sizes differ
2019/05/29 03:56:11 ERROR : Season 1/The Tick - S01E08 - The Tick vs. the Uncommon Cold WEBDL-480p.avi: Sizes differ
2019/05/29 03:56:11 ERROR : Season 1/The Tick - S01E08 - The Tick vs. the Uncommon Cold WEBDL-480p.nfo: Sizes differ
2019/05/29 03:56:11 ERROR : Season 1/The Tick - S01E09 - The Tick vs. Brainchild WEBDL-480p-thumb.jpg: Sizes differ
2019/05/29 03:56:11 ERROR : Season 1/The Tick - S01E09 - The Tick vs. Brainchild WEBDL-480p.avi: Sizes differ
2019/05/29 03:56:11 ERROR : Season 1/The Tick - S01E09 - The Tick vs. Brainchild WEBDL-480p.nfo: Sizes differ
2019/05/29 03:56:11 ERROR : Season 1/The Tick - S01E10 - The Tick vs. Pineapple Pokopo WEBDL-480p-thumb.jpg: Sizes differ
2019/05/29 03:56:11 ERROR : Season 1/The Tick - S01E10 - The Tick vs. Pineapple Pokopo WEBDL-480p.avi: Sizes differ
2019/05/29 03:56:11 ERROR : Season 1/The Tick - S01E10 - The Tick vs. Pineapple Pokopo WEBDL-480p.nfo: Sizes differ
2019/05/29 03:56:11 ERROR : Season 1/The Tick - S01E11 - The Tick vs. the Mole-Men WEBDL-480p-thumb.jpg: Sizes differ
2019/05/29 03:56:11 ERROR : Season 1/The Tick - S01E11 - The Tick vs. the Mole-Men WEBDL-480p.avi: Sizes differ
2019/05/29 03:56:11 ERROR : Season 1/The Tick - S01E11 - The Tick vs. the Mole-Men WEBDL-480p.nfo: Sizes differ
2019/05/29 03:56:11 ERROR : Season 1/The Tick - S01E12 - The Tick vs. the Proto-Clown WEBDL-480p-thumb.jpg: Sizes differ
2019/05/29 03:56:11 ERROR : Season 1/The Tick - S01E12 - The Tick vs. the Proto-Clown WEBDL-480p.avi: Sizes differ
2019/05/29 03:56:11 ERROR : Season 1/The Tick - S01E12 - The Tick vs. the Proto-Clown WEBDL-480p.nfo: Sizes differ
2019/05/29 03:56:11 ERROR : Season 2/The Tick - S02E02 - Alone Together WEBDL-1080p-thumb.jpg: Sizes differ
2019/05/29 03:56:11 ERROR : Season 2/The Tick - S02E02 - Alone Together WEBDL-1080p.nfo: Sizes differ
2019/05/29 03:56:11 ERROR : Season 2/The Tick - S02E03 - Armless But Not Harmless WEBDL-480p-thumb.jpg: Sizes differ
2019/05/29 03:56:11 ERROR : Season 2/The Tick - S02E03 - Armless But Not Harmless WEBDL-480p.nfo: Sizes differ
2019/05/29 03:56:11 ERROR : Season 2/The Tick - S02E06 - Bloomsday WEBDL-480p-thumb.jpg: Sizes differ
2019/05/29 03:56:11 ERROR : Season 2/The Tick - S02E06 - Bloomsday WEBDL-480p.nfo: Sizes differ
2019/05/29 03:56:11 ERROR : Season 2/The Tick - S02E09 - Ants in Pants! WEBDL-1080p-thumb.jpg: Sizes differ
2019/05/29 03:56:11 ERROR : Season 2/The Tick - S02E09 - Ants in Pants! WEBDL-1080p.nfo: Sizes differ
2019/05/29 03:56:11 ERROR : Season 2/The Tick - S02E10 - The Tick Loves Santa! WEBDL-480p-thumb.jpg: Sizes differ
2019/05/29 03:56:11 ERROR : Season 2/The Tick - S02E10 - The Tick Loves Santa! WEBDL-480p.nfo: Sizes differ
2019/05/29 03:56:11 NOTICE: Encrypted drive 'GCrypt:Cartoons/The Tick/': 67 differences found
2019/05/29 03:56:11 Failed to check: 67 differences found

The thing is...

[prometheus ~/private/post-processing/Cartoons/The Tick] rclone ls GCrypt:Cartoons/The\ Tick/
   363878 The Tick.jpg
   85069 banner.jpg

So if it's not local, rclone check doesn't notify me that it exists on the remote. Also not a fan it only works on folders and not files. I tried this with -vv and it was the same result.

Rubber duck debugging here: I suppose I could drop the move command into a script, and have the script recurse and store the files found that meet the size requirement in a variable and initiate a single upload per file found and then touch the content of that variable.

Sucks. Just seems like a lot of work. I was really hoping I could get this to be a one-liner based purely on rclone.

I think I'll try a find . -type f -size +0c -exec rclone_uploadcommand with {} to pass in the file.... with a touch afterwords.

If you remove the "one_way" it'll check both ways. You can also modify it to ignore the sizes for less output I believe

Rclone is a sync tool. But here you're actually looking to enforce differences which is the opposite of what rclone is trying to accomplish....

Indeed I am, but I figured out how to do so I think. I don't have the battery strength to test this out... but.... it's going to be something like

find ~/post-processing/ -size +2c -exec rclone_move_cmd {} Crypt:/ ; touch {}
Which will catch a lot of small .nfo files and things that the min-size 1k parameter missed.

Obviously that's a broken command of poor syntax but you get the idea. Rclone may be built around syncing but it's so damn powerful if you put your mind to it. It's one of my favorite packages.

I'm going to post the results here for documentation-sake because the rest of it will be living in a private github repo with credentials stored, so, not sharing that directly. Perhaps someday I'll make a public repo but everyone's configurations are so varied I don't know if I can build a 1-size-fits-all for GDrive.

1 Like

And I DID IT!
find ~/rclone-test/ -type f -size +0c -exec rclone move {} rclone-test: \; -exec touch {} \;

I really hope this helps someone else out too. What an effing great tool rclone is.

1 Like

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.