Something does not add up here.... But I am not sure what.
Can you run attempt to copy to yandexunion with DEBUG level output? -vv flag.
I'm not sure what's confusing.
You have one remote.
You have a bunch of shared remotes.
You can only see space on one remote.
Any option you select other than random will always put it on the first remote as they all report the same free space and cannot see into the shared area.
Everything is working as designed and very much expected.
You can't flag around this unless you want random luck to pick a remote and you can use random but that will fail at times.
So for pure testing I tried it on another remote. Union of few folders on one remote. It works perfectly fine with files being written to all folders.
Then if you consider these facts:
It is clearly something else happening here.
All facts I have point otherwise.
What does another remote have to do with this? Are you using a shared yandex remote? What's your config? What's your logs?
Facts? Please share them then as I am missing that part of it.
With no config, you get:
Enter a string value. Press Enter for the default ("epmfs").
That requires free space:
It then picks a single upstream based on the most free space:
If you are writing to all folders, you aren't working with the OP's configuration. You picked a different create policy.
Then every union member returns the same rclone about which does not lead to everything to be attempted to be written to one folder. But when I use rand it ends in all folders roughly equally.
You've been around IT long enough to know random does not mean equally amongst remotes as you know you get lucky in your rolls 
No reall need to debate that is there?
And random does not solve the OPs problem as it'll pick a full upstream eventually and break.
For union free policies to effectively work, you need remotes support free space as anything else, you can rolling the dice.
@kapitainsky, @Animosity022, here's another interesting thing. I tried running the same command but on the normal union and not the encrypted one and it seems to work fine. i also tried running the command on the encrypted remote but changing the destination folder. Below is me running the command and copying to the encrypted one:
rclone copy HAE: yandexunionc:Google-temp-del --filter-from text.txt --transfers 45 -P
Transferred: 323.287 MiB / 283.643 GiB, 0%, 9.760 MiB/s, ETA 8h15m26s
Transferred: 12 / 9737, 0%
Elapsed time: 54.7s
Transferring:
* A/recording_04032022_234200.mp4:100% /2.069Mi, 59.284Ki/s, 0s
* A/recording_05032022_000506.mp4:100% /2.052Mi, 46.773Ki/s, 0s
* A/recording_17112021_014820.mp4:100% /21.151Mi, 581.630Ki/s, 0s
* A/recording_17112021_015448.mp4: 69% /26.824Mi, 450.657Ki/s, 18s
* A/recording_17112021_020119.mp4: 64% /35.660Mi, 731.934Ki/s, 17s
* A/recording_16062021_112319.mp4:100% /3.384Mi, 77.163Ki/s, 0s
* A/recording_16062021_112513.mp4:100% /10.024Mi, 243.562Ki/s, 0s
* A/recording_16062021_113134.mp4:100% /11.546Mi, 299.286Ki/s, 0s
* A/recording_16062021_113410.mp4:100% /5.866Mi, 142.580Ki/s, 0s
* A/recording_16062021_113846.mp4:100% /20.050Mi, 504.397Ki/s, 0s
* A/recording_19112021_153924.mp4:100% /2.002Mi, 45.611Ki/s, 0s
* A/recording_19112021_160602.mp4:100% /2.405Mi, 54.831Ki/s, 0s
* A/recording_19112021_162031.mp4:100% /3.518Mi, 82.371Ki/s, 0s
* A/recording_19112021_163602.mp4:100% /2.185Mi, 49.782Ki/s, 0s
* A/recording_19112021_212311.mp4:100% /2.950Mi, 76.504Ki/s, 0s
* A/recording_19112021_212637.mp4:100% /8.646Mi, 224.144Ki/s, 0s
* A/recording_19112021_213304.mp4:100% /11.816Mi, 306.319Ki/s, 0s
* A/recording_19112021_215657.mp4:100% /5.334Mi, 129.579Ki/s, 0s
* A/recording_20112021_031201.mp4:100% /3.102Mi, 73.663Ki/s, 0s
* A/recording_17032022_131252.mp4:100% /3.382Mi, 87.679Ki/s, 0s
* A/recording_17032022_132503.mp4:100% /5.157Mi, 158.562Ki/s, 0s
* A/recording_17032022_132909.mp4:100% /5.542Mi, 126.346Ki/s, 0s
* A/recording_17032022_133233.mp4:100% /3.359Mi, 76.528Ki/s, 0s
* A/recording_17032022_133823.mp4:100% /3.682Mi, 83.915Ki/s, 0s
* A/recording_17032022_134132.mp4:100% /7.471Mi, 181.581Ki/s, 0s
* A/recording_17032022_134605.mp4:100% /6.019Mi, 171.298Ki/s, 0s
* A/recording_17032022_134726.mp4:100% /7.020Mi, 177.324Ki/s, 0s
* A/recording_17032022_134933.mp4:100% /16.952Mi, 500.024Ki/s, 0s
* A/recording_17032022_135309.mp4:100% /6.137Mi, 150.221Ki/s, 0s
* A/recording_05122021_103820.mp4:100% /2.314Mi, 55.502Ki/s, 0s
* A/recording_05122021_103852.mp4:100% /2.468Mi, 56.241Ki/s, 0s
* A/recording_05122021_104935.mp4:100% /7.920Mi, 205.321Ki/s, 0s
* A/recording_05122021_105303.mp4:100% /3.557Mi, 81.049Ki/s, 0s
* A/recording_05122021_105257.mp4:100% /2.046Mi, 50.153Ki/s, 0s
* A/recording_05122021_105550.mp4:100% /5.601Mi, 441.149Ki/s, 0s
* A/recording_05122021_105840.mp4:100% /29.168Mi, 2.243Mi/s, 0s
* A/recording_05122021_110616.mp4:100% /6.125Mi, 482.495Ki/s, 0s
* A/recording_05122021_110739.mp4:100% /11.516Mi, 1.047Mi/s, 0s
* A/recording_12072021_182509.mp4:100% /4.508Mi, 355.157Ki/s, 0s
* A/recording_12072021_183910.mp4:100% /8.568Mi, 877.314Ki/s, 0s
* A/recording_12072021_184217.mp4:100% /5.074Mi, 472.361Ki/s, 0s
* A/recording_12072021_190245.mp4:100% /2.393Mi, 222.759Ki/s, 0s
* A/recording_12072021_190718.mp4: 0% /5.569Mi, 0/s, -
* A/.outside: 0% /0, 0/s, -
* A/recording_08012022_191103.mp4: transferring
This means that it is in fact possible to use different folders in a union or at least try to find a workaround. The command I was running previously fails:
rclone copy HAE: yandexunionc:Google --filter-from text.txt --transfers 45 -P
But using a new folder as the destination works. @kapitainsky, any ideas on why this might be happening?
P.S. @Animosity022, I read through your replies and comments and I'm not sure I understand them. But that's probably because I'm not a pro with rclone.
Can you run above with -vv flag and post output here?
Yeah sure, doing that now.
With your setup, encrypted or not encrypted doesn't matter.
With the limitations, you are playing an RNG game. Results will vary depending on your RNG luck and how RNG works for you. That's why you keep getting different results as it's the luck of the draw.
If this is the case than it would be all good in this situation. With "RNG game" indeed being a drawback of such solution as expected.
It might be good enough though to backup files.
@kapitainsky, okay so here's the one for the command that's working:
And here's the one that's not:
It was too big to paste here but I hope it makes sense and is readable.
I'd tackle it a bit differently. Random would potentially work well randomly
If you make a read only union remote for reading your files, that gives you read access.
For uploading, make it upload to a remote of specifically your shared drive with the proper space. I'm assuming you are going from one shared, fill it, go to another shared, fill it, rinse and repeat.
So SharedA->B->C->D->E etc.
They are both working but sometimes hit unwritable union member.
also contains
2024/02/29 13:52:08 ERROR : A's Resume.pdf: Failed to copy: [507 - DiskOwnerStorageQuotaExhaustedError] Owner of shared folder has exhausted storage quota. (У владельца общей папки недостаточно свободного места.)
What is happening is that one "share" is full and due to rclone not able to determine it union create rand rule sometime chose it for writing. By default union picks up remote to write to every 120 sec I think.
You have two options here.
- mark full share as "no create", e.g:
upstreams = yandex:Shared2:nc yandex1:Shared3 ...
- make sure all shares are empty when you start using this setup. This way it should statistically work very long without any issues.
Otherwise as mentioned earlier you might have to restart your copy multiple times to finish all upload.
better make it "no create" so all other operations will work (delete for example)
Yeah, I think that could work. Let me give that a try.
One small comment - you are using --no-check-dest. Check docs about side effects of this flag.
This flag is only useful in very specific scenarios - I would not use it when running rclone copy you might have to restart as every time you will be transferring all files again. Or use together with --retries 1
And one BIG comment.
actionPolicy = *policy.Rand, createPolicy = *policy.Rand, searchPolicy = *policy.Rand
I bet this is not what you want. To use rand policy for everything. As with this all your rclone operations results will be truly random:)
What you want is most likely:
action_policy = epall
create_policy = rand
search_policy = ff
Also it means that I would delete what has been transferred before because you might have very random situations like the same file in multiple shares etc. Clear shares directly - without using union. Effectively when happy with testing of your new setup start from zero.
This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.