jamenson
(Jamenson Ferreira Espindula de Almeida Melo)
1
Hi.
Congratulations for the excellent software. rclone é really handful.
I have to download a big amount of files (around 660GB). I own two hard disks: a 500GB one; and another of 320GB. Neither of these disks can alone hold all the files I mentioned I have to download. I tried to merge both of them too (via "mergerfs" FUSE union), but it does not work (the 500GB hard disk is now all dedicated to storage of the files, but the 320GB one not. So, "df -h" says it has only around 25GB available, but I still have to download around 139GB of 660GB). I own an 1TB external hard disk and still have around 180GB to hold the files.
So, my problems are:
1 - the directories where "rclone" check for the files (on the two hard disks; the destination directories) are almost full;
2 - I do not like to plug and work directly on my external hard disk (is it a misconception? Is it have theoretical fundamentals?).
My doubt: can "rclone" compare source and destination directories, but save on a third directory?
There is no problem with external harddrives in essence.
The only problem may be that a lot of "typical" external harddrives are kind of cheap and low quality mass-storage, so they may be less suitable as high-load work-disks that have to do a lot of work.
But this is completely up to the quality of the disk - not a problem with it being external. There absolutely exist external disks that are both high quality and high performance too.
As to your question, there are many ways to do this.
Here is one way using union:
(just add this to your rclone.conf file)
[Union1]
type = union
remotes = C:\folder D:\folder E:\folder
If you download to Union: it will then
compare from all 3 places - and do all additional writing for files that don't already exist to E:\folder (always the last one in the definition).
You may find --max-transfer 100GB (or whatever number) useful to automatically end the transfer before you run out of space, and then do the rest of the transfer to another location (just change the order of the drives in the union).
By the way - maybe you should tell us your "high level goal" here... we may be able to suggest a better aproach.
Are you trying to move the data somewhere else? Is it to another cloud or some other server? If so then a direct transfer may be possible without the need to locally store all the data locally as the middle-man
jamenson
(Jamenson Ferreira Espindula de Almeida Melo)
3
Thank you, thestigma, for your answer.
My "high level goal" is hold the files for my personal use only (several videos teaching e related documents). From time to time, access to the Internet may be an issue. So, I do not want to spend time waiting for the connection work again. Indeed, my television has an USB port. So, if I have the files on my flash mass-storage, then I can watch them in the television (I gain in quality).
Until now, I have downloaded around 500GB.
My approach: I have mounted a directory on an 320GB hard disk; a second one on an 500GB; and a third one via NFS to another machine. So, I have merged them all on a forth directory no the 320GB hard disk. I always invoke rclone in that forth merged directory, so rclone "thinks" it is in one single directory with all the files residing in that directory. But, in fact, the files are on three directories.
Did you get confused?
I appreciate more suggestions. I am almost buying a 2TB pen-drive flash mass-storage. Until that, I am improvising as better as I can.
Merged them via OS links? I guess this could work, though the union approach
I suggested would be easier I think - but whatever works for you
It sounds like you have solved your initial problem - right? If so you kinda have to ask about something else if you want more advice
PS: Be very careful about "2TB" pendrives.
There does exist genuine ones, but they are very expensive. We are talking around 800-900USD ... a prohibitive cost for most people.
Most (or I should realistically say all) "2TB" drives and similar "too good to be true" sizes at a low cost are scams. It is easy to hack the filesystem of a 4GB USB-stick to make it look like it has 2TB - but any data over 4GB is going to just be corrupted and lost of course ...
Don't let scammers exploit you.
if you want a super fast pendrive with large storage, i use this but it is expensive.
on windows, there are issues with usb drives being treated as removable,
this pendrive is treated as a hard drive, which is rare to find, so you can install and boot windows operating systems on it.
i have the 1TB version,
Corsair Flash Voyager GTX 256GB USB 3.1 Premium Flash Drive
don't be sad.
for streaming, plex and so on, any hard drive is good enough.
as part of my consulting business, i use those small fast drives to demonstrate different ways for disaster recovery of window servers.
and i travel a lot.
if you want a small, fast usb key that you can boot a portable windows 10 or portable windows server, plug it into any computer and boot it. i keep it in my wallet.
With USBs with internal mini-SSDs a "system on stick" is finally quite viable yea.
Not that you can't do it with a regular USB, but even light Linuxes get a bit sluggish on those.
if your computer has usb 3.x, you would be suprised how fast an os like windows 10 can be on that small samsung.
you can get a 64GB for $15.90.
if you want, get one and i can guide you on how to do it, super easy to do these days.
